ESX 3.5 : deny of service or system issue ? see screenshot attached

[Dandan712]

Hello everyone,

I have 6 esx 3.5 servers in production, eaxh of them running at minimum 5 virtual machines

this mornin, I found out that all my VMs are down , on all of my 6 ESX servers !!!!

I had a look at the event interface, for each ESX, and I can see, for each of them, the attached screen shot !!!

I do not have any idea of what happened !

[AntonVZhbankov]

Looks like you've got a virus on some machine. Or maybe a hacker.

There definitely were unsuccesful login attempts + attempts to perform SQL injection.

---

VMware vExpert '2009

http://blog.vadmin.ru

[Dandan712]

A virus ? An attack ?

Are sure of that ? Because I have exactly the same on each of my 6 esx servers !!!

Is an attack can directly attempt to an ESX server ?

I had thought that the latest ESX patches would secure them , no ?

[AntonVZhbankov]

Only god can be sure 100%. If he exists. But it looks like SQL injection for me.

I think that is was just a standard trojan trying to infect all machines it see and control them as spam bots but accidentally it made something that powered off your ESXes. Check your network, firewalls and workstations.

---

VMware vExpert '2009

http://blog.vadmin.ru

[tresf]

First, of course God exists !!

Secondly, I don't think of a virus or attack: Why it would affect all of his 6 esx serv at the same time ?

I would rather think of a network issue: service console may tried ( unssuccessfully ) to get network access, and this why he got the Failed login attemps !!

[AntonVZhbankov]

Hackers are overestimated, they cause only 5-10% problems.

And all 6 ESXes gone down the smae time, so thats why I think it's something like trojan. It just scans network, see any web server and tries to infect it.

Failed login attempts are trashy, there is SQL code as logins, so it's unprobable to be done by a human.

---

VMware vExpert '2009

http://blog.vadmin.ru

[uslacker99]

Is this in an HA cluster? If so, check to see if a network switch went down. You can find this in the /var/log/vmkernel file. My guess is that maybe you had isolation mode set.

The event logs are interesting though. Since the logins came from localhost, I would check the /var/log/messages and maybe even use iptables to block unauthorized hosts from accessing ports 443, 902,903 on the ESX host.

[AntonVZhbankov]

Just misread original post. Yes, if VMs are down only and ESXes are still up, isolation most probable reason.

Check if HA policy is set to "power off vms when isolated"

But still pay attention to this garbage on login attempts.

---

VMware vExpert '2009

http://blog.vadmin.ru

[Dandan712]

Hi,

Actually, I do not have any HA Cluster. So Isolation mode may not be the reason.

But, since I've been searching a little since then, I found out that all my 6 ESX servers restarted ( I can see this in the /var/log/messages ).

they all retarted at approximatively the same time.

So I first thought about a power electrical failure... But I don't think so because all my other machines ( not ESX ) didn't reboot at all !!!

so what ? I do not have any idea about that issue !!! I just need to be sure it won't happen again without control...

Any idea ?

[AntonVZhbankov]

In VI Client: File / Export / Export diagnostic data.

Download logs from ESXes and see for something strange just before reboot.

---

VMware vExpert '2009

http://blog.vadmin.ru

[IB_IT]

not sure why all your hosts rebooted...maybe check if there was a power failure in your building? If you want your VM's to power on after your hosts restart automatically, then in the VIC, higlight each host and go to "configuration" tab, "Virtual Machine Startup/Shutdown", and click the "properties" link to change the default settings. You may want to do a little research here before you make any major changes like that...I remember seeing some threads where folks had issues with their VM's restarting when this feature was enabled....but then again I believe that was due to HA issues which it sounds like you do not have.

[Texiwill]

Hello,

Looks like an SQL INjection attack to me as well. It is horribly easy to determine a server is running ESX. Is your Service Console for the 6 nodes on a public facing network? Or behind an administrative firewall.

Those are common SQL 'phrases' used to perform an SQL injection. It also looks like someone may have been successful in doing something bad. Check your logs and secure your Service Consoles using a virtualization administrative network that is NOT internet or anything facing but clearly behind a firewall. If you use vCenter then that should also be behind the firewall.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[Dandan712]

Hello,

All my ESxs and virtual Center are inside the Enterprise network, and so protected behind the company's firewall.

In case you're right and my esx servers would have been infected, what I should I now do to check wether there is realy something bad, and how I could remove the infection is any ?

The only thing I know is that my company have recently been affected by a virus : "Conflicker.A Worm", but I think it's only infecting windows platform...

[Texiwill · Accepted solution]

Hello,

For better security you should place VC, and Service Consoles behind a subsequent firewall. Your normal users, etc. should not be able to reach these machines. Treat the Service Console as you would your datacenter, generaly it is behind lock and key so you use a firewall to do the same thing within your enterprise network.

ESX itself may not be affected, but VMs could be. If you want to be absolutely sure they are not. reinstall the ESX hosts and place them behind an internal firewall.... Place VC behind that same firewall in addition do not allow anyone to use VI Admin tools from anywhere but behind that firewall. GIve each admin a virtual desktop within this new security zone to which they can RDP.

If your VMs are infected you know the rules for those as well.....

I seriously doubt ESX is infected with a virus but soemone could have infected yoru VC server and done damage that way as it is windows after all.

Best regards,
Edward L. Haletky
VMware Communities User Moderator, VMware vExpert 2009
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs -- Top Virtualization Security Links -- Virtualization Security Round Table Podcast

[Dandan712]

Many thanks for all those good advices and best practices.