Crazy Problem - All VMs are Off All of a Sudden

[pcmd]

Hi.

First off, due to budget issues (as in, no longer having one) we let our support contract expire a long time ago (not my doing) and are currently in discussion about how to get back on track with that, financially. So, that's why I'm coming here.

I left work early today (sick). After driving most of the way home, when I got to my street, I got a call from my boss that the entire office is down cause they can't get to the server (I would normally be the first to know from our monitoring system, but that's down right now as well due to other issues).

Anyway....once I got in my house and connected tot he campus network remotely, I discovered that ESX is up and running, but all of the VMs are powered off.

Furthermore, there's nothing on the 'Events' page. So, I have no idea what happened.

It is possible that we lost power to our server room (an old closet with adequate, direct ventilation), but the UPS should have kept it up long enough to survive something like a temporary outage.

Is there anything I can do, anywhere I can go in ESX (or the underlying Linux system) to even get a clue about what may have happened?

Thanks.

steven

:,_`:`.:,_`:`.:,_`:`.:,_`:`.:,_`:`.:,_`:`.:,~`:

Thanks for your kindness and patience as I continue this adventure in the world of virtualization.

Bullies need not reply

[JDLangdon]

One thing you can try is to connect to the ESX console as ROOT and type "uptime" without the quotes. This will give you an idea of how long your ESX server has been up. From there you should be able to determine whether or not the host rebooted while you were driving home.

I don't think you'll see anything in the /var/logs folder to indicating whether or not a power outage caused the reboot but there might be something in the messages file.

If this was caused by a power outage, you can set the VM's to power on automatically when the host server finished booting. It won't solve anything but it will prevent you from having to manually powering on the VMN's.

________________________________

Jason D. Langdon

[pcmd]

Thanks, Jason.

Ya, I'm pretty sure it was a power outage.

The problem is, our UPS is only good for about an hour, and it has no way of notifying me that it is in distress 😛

If it had that capability, I would have known before I left, and had it all taken care of before anyone else knew about it.

As it is, they flick circuits off and on at will around there whenever they FacMan guys (or whomever) are working on something.

Not the best arrangement, but we somehow make it though.

I will definitely check the uptime.

Thanks for that.

By the way, all VMs are back up and the office is back to normal working conditions now.....just a PITA to not be able to get everything in place that I'd like to.

[PaulSvirin]

Did all of them perform shutdown?

Try examining the logs for the virtual machines they could possible get into BSOD and shutdown too.

---

iSCSI SAN software

[pcmd]

This definitely wasn't a BSOD episode; all 12 VMs (Windows, Linux, BSD, etc) were off when I checked.

Oh....Jason,

for some reason, one time a long time ago, one of the VMware guys told me not to have the VMs set to turn on automatically upon ESX boot. I don't recall why, but it made sense at the time for what we were working on.

Usually it's not an issue. I'd rather have to deal with the VMs being off for a while then whatever that other issue was (I'd have to check my notes to be refreshed).

[JDLangdon]

Oh....Jason,

for some reason, one time a long time ago, one of the VMware guys told me not to have the VMs set to turn on automatically upon ESX boot. I don't recall why, but it made sense at the time for what we were working on.

</div>

My guess would be that you were working on something that required rebooting the ESX host multiple times. Each time the ESX server rebooted you would have had to wait for the VM's to power on so that you could shut them down again. That's probably why the VM guy told you not to set anything to automatically power on.

________________________________

Jason D. Langdon

[JDLangdon]

Thanks, Jason.

Ya, I'm pretty sure it was a power outage.

The problem is, our UPS is only good for about an hour, and it has no way of notifying me that it is in distress 😛

If it had that capability, I would have known before I left, and had it all taken care of before anyone else knew about it.

</div>

What type (make/model) of UPS are you using?

________________________________

Jason D. Langdon

[pcmd]

On the reboot issue.....that would normally be possible and seem like what happened, but I would remember that, and I don't think that's it....oh well.

The UPS is a CyberPower PR1500.