VMware Communities: Message List - Big Problems...Whole system down

Re: Big Problems...Whole system down

TylerApache — Mon, 14 Apr 2008 16:47:28 GMT

Looks like we may have configuration update issues. When we went in to CHAP access to change the config, we got an "unable to update settings" message. The engineer is puzzled.

Re: Big Problems...Whole system down

IB_IT — Mon, 14 Apr 2008 15:08:58 GMT

Having delt with VMWare so frequently on different issues, often times they suggest to me blowing the machines away from Virtual Center and reregistering. Having reread your original problem, though, I would tend to agree with wondab that you may need to unpresent/re-present LUNS.

Re: Big Problems...Whole system down

TylerApache — Mon, 14 Apr 2008 15:05:08 GMT

In the queue at the moment...will ask when I get them online.

Re: Big Problems...Whole system down

TylerApache — Mon, 14 Apr 2008 15:00:49 GMT

I did not unpresent/represent and rescan.

Re: Big Problems...Whole system down

IB_IT — Mon, 14 Apr 2008 14:59:41 GMT

We had a similar issue with a power outage. Our hosts were brought up and online before the SAN had a chance to power on, although we did not experience the same thing you are...seems your issue is a bit more drastic. In our case all the VM's were inaccessible or orphaned. It sounds like your hosts are still a bit confused as to who is accessing what VM...I wonder if you could try to "remove from inventory" everything from your cluster and then add back into inventory...your hosts included. If you have an open case with VMWare, see what they think about that.

Re: Big Problems...Whole system down

wondab — Mon, 14 Apr 2008 14:50:44 GMT

What was the fist thing you did after the EMC rebooted? Did you unpresent and represent the LUNs and rescan?

There is a good thread explaining reservation conflicts due to the simultaneous altering of metadata.

http://communities.vmware.com/thread/80561

Big Problems...Whole system down

TylerApache — Mon, 14 Apr 2008 14:03:50 GMT

Two Dell PE2950, 32GB RAM, ESX 3.0.2

EMC NS40 SAN presenting iSCSI

Last Friday we experienced a power failure that caused our SAN to reboot. When I began checking machines that morning things seemed fine but shortly after I began to notice some oddities. Suddenly one of my VMs wouldn't reboot. Another went BSOD. Before I knew it, all of my VMs were corrupt. Most would not make it to a lgoin propmt. Others would BSOD at login. Still others wouldn't boot at all.

I spent the next 26 hrs on the phone with EMC and VMware trying to figure out what was wrong. VMware says everything is OK. EMC says the storage is fine. Ugh!! As a last ditch effort to save the VMs, we cloned one of the BSOD VMs to local storage and it booted perfectly. Ah HA!! I blew away one of the LUNS on the NS40, recreated it and re-presented it to the hosts. When I cloned machines over to it they booted fine. We only lost two machines due to drastic Windows repair efforts. Happy ending right...not so fast.

The next morning after letting the machines settle over night, I came in to begin the process of restoring order. It didn't take long to realize that the machines were beginning to exhibit the same corruption as before. The only machines that have come back uncorrupted and have stayed that way are the ones that I moved to the local storage. Now EMC is saying that they are seeing reservation errors on the SAN. Apparently this happens when two hosts attempt to access the LUN at the same time. I'm waiting on a return call from VMware at the moment. Help!!