ashleydrees
Enthusiast
Enthusiast

ESXi 5.5 hacked, next moves

My ESXi server has been hacked - i foolishly left the SSH server on (i forgot) - that is if that is how they got in.

At the root of every disk there is a folder called -HACKED- which i am unable to enter and it seems like something has been done with all the disk files, i say that because the disks seem to have the same amount of data used as if the disk files were there, but as they are either deleted or no longer in the location the vm needs to run, all the vm are currently offline.

i cannot enter the -HACKED- folder (i did not try too hard as i wanted to bring the machine offline. as "cd -HACKED-" tells me that -H is not a flag.. "'cd -HACKED-'" does not work either.

I have a call from vmware scheduled for monday AM, but if anyone else has seen this kind of vandalism - or perhaps it is ransom type activity - i do not know, i guess i am feeling my way round this as i have never had to clean up an ESXi machine before, only linux and windows.

So any thoughts - ideas - anything - i am busy moving backups to other machines - but i have not got 100% backup coverage so am really interested to see if there is anything i can do.

19 Replies
cykVM
Expert
Expert

Hi,

you may try

cd -- -HACKED-

to entrer the directory (-- is end of options) or even

cd ./-HACKED-

should work from the top-level directory.

You should definitely disconnect the machine from the internet and probably put it in a kind of closed (temp) network for investigation.

Check if further access routes/services were opened from the public.

If your VMs still exist somewhere do a brief check if they were altered in any way. I won't bring them up in production straight away.

Also check your backups as the hacker might not be just one person messing around and leaving obvious traces (like the -HACKED- folder). This might have been going on for a longer time with several people accessing your box.

cykVM

cykVM
Expert
Expert

Additionally if you have some kind of monitoring in place you should check if the hacker(s) probably even downloaded your VMs (vmdk and othe files) via SCP by checking the logs and/or high data transfers while those guys were logged in.

CoolRam
Expert
Expert

I would suggest you should always use the default SSH timeout.

This is common mistake people used to do they didn't configure SSH timeout.

Please do follow this to recover Recover Data from VMFS, ESX, ESXi, vSphere Disks

If you find any answer useful. please mark the answer as correct or helpful.
JarryG
Expert
Expert

If you really suppose your ESXi has been hacked, you should take it down asap. You can not be sure what is running there, and even if you got into those -HACKED- folders, you might see nothing (despite of something being there).

So I recommend to take it down, take system-disk of this ESXi out and mount it on some secure workstation. It should contain a few vfat-partitions. Anyway, I think you can not do more than re-install ESXi and restore VMs from backup.

For the future: never (NEVER!) expose management-port to the wild! ESXi is not designed to be able to protect itself alone. Its "firewall" is just very basic filtering script, lacking a lot of functionalities modern firewall has (i.e. connection rate limiting). Honestly, I think it was bad move from VMware to include it in ESXi because now some users have false impression of security. ESXi should either have serious firewall, or none at all...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! 😉
virtualkitten
Enthusiast
Enthusiast

That just happened to a server of mine. ESXi 5.5 a year uptime because it was a standalone server. Password complicated to bruteforce. NTPd and SNMP disabled.

Any ideas about the entry point ? remote exploit ? It is kind of humiliating as well as annoying as I do have several important VMs there, not as worried if they stole the data but...

Here is the message on the server inside the HACKED folder

Hello everyone,

I just want to tell you that your server was hacked. Your protection was completely awful...

While the entrance, we downloaded the virtual machines and deleted them.

(We had to download for many hours -.- Compressed files but huge ones...)

Do not try to search in your logfiles. We deleted the important parts.

If you want to get the backup of your VMs, you should send us an amount of 2.5 BTC (Bitcoin)

for each VM to the address "1BsSKUEn2ktMPr85UZDyXDBPHMhK5gR3M8". After the payment, we will contact you via mail.

Then, we will send you a HDD where the VMs are stored. If you want, we can give you access to our FTP where you can

download them. (Because FTP is faster, remember the shipping time of the HDD after payment)

Please notice, that we will sell the VMs to others if we will not receive these Bitcoins from you. Do not worry,

you have 2 weeks for these payment. After 2 weeks without payment, we will break the VM and sell

all data to our customers (other hackers, spammers, scammers, ...)

(FYI: Some of them may use the data of your customers/employees/... to blackmail them for money. No nice guys, but they pay for that data)

Do not worry: If we receive the BTC, we will send you the backup (or give you full access to FTP) and delete all data here.

(If you want FTP, you can do it for your own) We are hackers, but we want to play fair. If you pay, your data will be secure.

There a short overview about where to buy BTC:

  - www.litebit.eu

  - www.anycoindirect.eu

  - www.happycoins.com

  - www.bitcoin.de

  - www.btcdirect.eu

  - www.clevercoin.com

  - www.bitstamp.net

We wish you a nice week

Kind regards

- Russian guardians

Please think about our offer, your data and your computers...

0 Kudos
cykVM
Expert
Expert

The entry point would be any management service (ssh, webservices etc.) being exposed to the internet.

I gues they had all the time do run scripts on the ssh login.

You should have anything in place to check if they really downloaded the VMs...

0 Kudos
virtualkitten
Enthusiast
Enthusiast

Yeah, I should have many things. But servers are standalone, depends on the company to provide resources. Or use oVirt, as I often recommend but we will not discuss this at the moment. I cannot know when this was actually hacked so they had plenty of time.

SSH was enabled, vSphere GUI was enabled. But the password is strong enough to be bruteforced. Thus, my concern.

NTPD or SNMP exploits... could be, but on that server those were disabled because I knew the server was not patched (as I couldn´t, not without downtime risk,etc.). The attackers deleted all VMs from disk, potentially downloaded (only way would be check network usage but I have backups as well so not easy).

Having SSH enabled is not a security issue, per-se. It is better if you filter or you change the port but anyway...

I have other hosts (even older) I wonder if those are at risk... and why the affected is the most recent one, maybe there was a datacenter leak with passwords or something.

0 Kudos
JarryG
Expert
Expert

Having any management-service exposed to internet unprotected *is* serious issue! With enough of distributed resources (botnets with thousands of computers) no password is strong enough. If you do not have connection-rate limit and/or auto-banning of offending hosts implemented on perimeter firewall, question is not if your server gets hacked, but when this happens...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! 😉
0 Kudos
virtualkitten
Enthusiast
Enthusiast

Yeah, well, those quotes are fine from a marketing perspective. But no, I can have an OpenBSD with SSH open and be perfectly fine unless an exploit for SSH appears. And OpenBSD has PIE, so even with a vulnerability you are protected somehow.

But I understand what you say, however, you need SSH enabled often. It is like saying do not expose Apache when you have a website running. I agree it is better if SSH is only accessible from certain IP addresses but... I believe the service is robust.

Again, I do not think they bruteforced the password, I am not sure about exploits that gives you a shell or creates additional users.

Any idea of what services could be dangerous left with default settings ?

How to protect the vSphere login (you definetively need that run on a standalone host)

thanks

0 Kudos
cykVM
Expert
Expert

Yeah, well, those quotes are fine from a marketing perspective. But no, I can have an OpenBSD with SSH open and be perfectly fine unless an exploit for SSH appears. And OpenBSD has PIE, so even with a vulnerability you are protected somehow.

"protected somehow" sounds very marketingish Smiley Wink and trustworthy, too. Seriously, you compare a full featured OS with a hypervisor OS in terms of security. VMWare never had the intention to replace an OS and therefor at some point they have to make the underlying OS as compact as possible for the hypervisor use.

You also can't directly compare a KVM based hypervisor with VMWare, completely different approaches in terms of a hypervisor.

But I understand what you say, however, you need SSH enabled often. It is like saying do not expose Apache when you have a website running. I agree it is better if SSH is only accessible from certain IP addresses but... I believe the service is robust.

And again comparing apples (ssh) to oranges (Apache). You can perfectly leave SSH enabled with VMWare on your management (internal) network but not exposed to the internet without additional external protection.

How to protect the vSphere login (you definetively need that run on a standalone host)

Yes, but only on the management network.

You can still use a VPN connection to the internal network to manage the host from remote.

I don't think that your box was hacked through an exploit, was just the distributed bruteforce way JarryG mentioned above.

And for your other/older hosts I would take them off the internet ASAP, those guys may come back and hitting the IP of that other host(s).

0 Kudos
virtualkitten
Enthusiast
Enthusiast

So, are they exploits for the web interface for the vSphere or the SSH ? if not, the only thing someone can do is bruteforce the password.

Can be the password bruteforced considering 10 chars with sym, cap, lower, nums ? as long as it has a time out that sounds complicated to bruteforce but can definetively be possible.

Still, even exposing those I do not think someone could simply hack the server. But you may be right.


Leaving that aside:

I have disabled SSH in all the other hosts.

What is the best way to prevent random IPs to connect via vSphere client and/or SSH ? is there a Firewall or you suggest any other way ?

I connect to those servers from remote myself (I do not have physical access) so I wonder about the VPN, can I do that with just the Host itself ? or do I need an auxiliary host on the network ? I do not want to rely in one VM with OpenVPN to manage the server because.

0 Kudos
cykVM
Expert
Expert

What is the best way to prevent random IPs to connect via vSphere client and/or SSH ? is there a Firewall or you suggest any other way ?

Put a (real) firewall in between the host and the internet where you configure connection limits portwise. The ESXi firewall is very basic and üproviding only basic protection.

I connect to those servers from remote myself (I do not have physical access) so I wonder about the VPN, can I do that with just the Host itself ? or do I need an auxiliary host on the network ?

No, ESXi is not capable of doing (real) firewalling, VPN server etc.. So you need a separate box for that. You may even try to implement some kind of port knocking in conjunction with a ssh tunnel for example. But this also need a separate box/firewall.

I do not want to rely in one VM with OpenVPN to manage the server because.

Instead you rely on the host's basic security, not a good idea in my eyes. Maybe a decent router with firewall and VPN is a decent solution for that. If it needs to be not too expensive, I have good experiences with Draytek routers for SMB.

0 Kudos
wootn0w
Contributor
Contributor

Have you had a chance to ID the attackers IP?

It seems this was a broad range attack on multiple esx servers which had (I assume) ssh open.

Seeing log files of this it clearly shows a brute force attack behavior. I'm curious though if atm there are other open vulnerabilities which may allow such access.

0 Kudos
ashleydrees
Enthusiast
Enthusiast

No, we have not tried to find a culprit, though we have asked the D/C to outline the usage over the past while as we are not even sure they have moved the data out, if there is no significant data movement then we are certain the VMs are just deleted.

Machine is now offline and the disks going to ontrask for imaging and investigation.

The police have been informed and have been to visit the sick item, waiting on the cybercrime team to show interest.

Even if we paid up and got the data sets back, to use it would be asking for trouble, it has been seen that in similar cases to this, even though the ransom was paid, the data was still sold on - we will see what Kroll get back, we are recreating what we do not have as backups.  I will spend a lot more time on securing ESXi, possibly to the point of adding a security appliance between it and the wild world as this must not happen again, and as pointed out, the firewall is minimal in ESXi.

This pastebin was from the day before, perhaps it could be traced...

http://pastebin.com/00agVLTT

Seems it has happened before elsewhere in the world.

http://www.webhostingtalk.ir/showthread.php?t=144258

We have some attentive people looking into the B.C. id's, i hope something comes of that.

And, if you are reading this, and it is your work, thanks, you messed up a couple of my days and some very worthwhile projects that were hosted the server got a terrifying shock, they try to help sick and worried, whilst you hold us to ransom on pain of destruction or selling the data.

0 Kudos
cykVM
Expert
Expert

My thoughts on this: They did not download anything and there is no way to get the files back by paying anything. But that's just my opinion from the above message.

There is even no contact details in there so how would they know what data goes to which payer.

I also guess that IPs of the attacker(s) won't help in any way as this looks at least a little more professional than some script kiddies messsing around from their home internet connection.

virtualkitten
Enthusiast
Enthusiast

@wootn0wwootn0w

It could be the vSphere Client as well, were you patched against Heartbleed? NTPD patched ?

Whatever had SSL if your openssl was vulnerable could be affected. Not necessarily SSH.

0 Kudos
Bleeder
Hot Shot
Hot Shot

Even if they were fully patched, ESXi 5.5 still uses SSLv3.0 which is known to be weak.  One really good reason to upgrade to 6.0..

0 Kudos
virtualkitten
Enthusiast
Enthusiast

@Bleeder

Totally agree. I wonder, is it possible to upgrade a 5.5 host to 6.0 without problems ?

0 Kudos
JarryG
Expert
Expert

A lot of hardware has been dropped from supported list in 6.0 (or is not certified yet). For me the biggest problem preventing me from updating to 6.0 is missing smis-provider for my raid-controller. Those folks in LSI are probably still very busy with creating new powerpoint-presentations about bright future lying ahead of them after they have been acquired by Avago...

_____________________________________________ If you found my answer useful please do *not* mark it as "correct" or "helpful". It is hard to pretend being noob with all those points! 😉
0 Kudos