Problem With Datastore - All VMs Gone (Urgent)

[Kasraeian]

Hi,

Sorry for my writing but I'm in hurry

Wh have data center which we running ESX 3.5 on it as Infrastructure. after 9 months of working, today suddenly our datastores goes and disappearded.

all our VMs are still working and answering to all commands, but Hosts start generating alarm for datastore and asking for for adding new datastore.

I have searched and find this KB (9453805)language=en_US&cmd=displayKC&externalId=9453805] and did it without shuting down VMs (big mistake as I'm see it until now).

Now, all VMs gone from datastore but the are still working fine, these are screenshots from the state

I'm really appreciate all your helps and support.

we need solution soon because we have to deliver these services soon.

Again thanks for all your helps

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[Troy_Clavell]

from the ESX console it appears atleast one of the VM's is intact, and running with active snapshots. The first place I would start is to restart the vpx agent on the ESX host(s) that these VM's are running on.

service vmware-vpxa restart

Once complete, I would also restart the vCenter Server Service

...and finally maybe hostd

service mgmt-vmware restart

[athlon_crazy]

First let all VMs running for a while until you get permission to schedule downtime and do resignature again since that's what VMware recommended (volume resignature KB). If I were you, I will start calling 911 (VMware Tech Support) right now.

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

[cody_bunch]

I'd actually take both approaches, restart management services, both on the host(s) and vCenter. I'd then also open a case (if you haven't done so already).

Re-signature: http://professionalvmware.com/2009/02/bad-things-happen-at-2am-how-to-resign-a-vmfs-partition/

Support Bundles: http://professionalvmware.com/2009/01/log-bundles-of-the-virtual-center-variety/

Hope this was helpful. Let us know how it turns out.

-Cody Bunch

vExpert, VCP

[Kasraeian]

Thanks for your helps, we still have the problem even when we restarting esx-prv4 (one our host which have problem).

But esx-prv1 didn't have any warning and when we connect to it directly using VI Client it's showing all VM safe and sound.

Its amazing because just one host with directly connection showing the good thing but the reset didn't (all showing the screenshot I had sent - bad one).

Any idea? why only one can shows the good state in one condition (connection via VI Client) and it's showing bad state in another state (connection via vCenter)?

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[athlon_crazy]

Could be the esx-prv1 not affected at all. However, directly connected to esx-prv4 giving you same result? (No VM in datastore)

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

[Kasraeian]

Ok, let me describe the current state in this way:

esx-prv1 -

---

> via vCenter -

---

> no VM (just bunch of vpxa???.log and vpxa???.gz)

esx-prv1 -

---

> via VI Client -

---

> good state, all Folders and VMs are ok

esx-prv4 -

---

> both way (vpxa.log and vpxa????.gz)

All located on the same cluster and managed by same vCenter.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[Troy_Clavell]

I think part of the problem is vCenter. You have restarted the vCenter Server Service? Also, I would still, even though you said you rebooted, restart the vpx agent on both hosts as well as hostd

[athlon_crazy]

Have you try restart related services as told by other poster? BTW, don't tell me this ESX 3.5 is lower than Update 3. If you are aware, on some cases by restarting mgmt-vmware services can result your VM to be restarted.

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

http://www.no-x.org

[Kasraeian]

We already restarted vCenter VM (its run as VM too) and rebooted some of esx-prv (like esx-prv4 and esx-prv5) but no luck.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[Kasraeian]

We are using VMware ESX 3.5 Update 4 and I didn't aware of that problem, thanks for telling me, I'll remember it for sure.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[AndreTheGiant]

From the hosts with the vdf command you can see the datastore?

Have you tried to connect directly with the VIC?

Andre

[Troy_Clavell]

I understand you rebooted, but you really should restart the vpx agents while the ESX Host is properly connected the the cluster. To me it seems if you can see everything normally from connecting directly to the ESX host, it is an vCenter agent problem or a vCenter Server Service.

Please issue these commands on the COS

 service vmware-vpxa restart

wait till ESX host shows back up in inventory, then

service mgmt-vmware restart

[Kasraeian]

Hi,

Yes as I mention before just in some cases (just from esx-prv1 and esx-prv3) the connection from VI Client can see the folders and VMs but in any other cases no luck.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[Kasraeian]

It seem normaly just from two host but other 3 hosts can't see data from direct connection.

All VMs has no chance in seeing data when checking them from vCenter.

And by the way, I checked these commands and theres nothing.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[Troy_Clavell]

>And by the way, I checked these commands and theres nothing

the commands need to be issued with root privileges

[Kasraeian]

It seem normaly just from two host but other 3 hosts can't see data from direct connection.

All VMs has no chance in seeing data when checking them from vCenter.

And by the way, I checked these commands and nothing happend.

It's looked weird to me. why should 2 hosts works fine while others can't in the same condition of connecting via VI Client.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[Kasraeian]

I had test it with root user itself, but no luck.

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard

[cody_bunch]

Have you restarted the vCenter service(s) yet? Also, are you using SQL Express or a full version of SQL? You may have blown the 2GB limit, in turn causing this 'oddness'. Also, have you opened your case with the mothership?

-Cody Bunch

vExpert, VCP

[Kasraeian]

I have restarted the vCenter service as well as vCenter VM (it's running as Vm on esx-prv3 which is ok - at least now )

And about Database, we are using MS SQL Express and as my friend checked, it's far from 2GB.

And by the way, let me point something here which I forgot to mention.

Other problem we have is, after I resignatured the datastores, those hosts seems to pointing to "/var/log/" (or at least we see something like this) and we can't mount or add our main storage (the LUN which contain all VMs).

------

-= If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" =-

-= If there's any mistake in my notes, please correct me! =-

-= Thanks =-

Sohrab Kasraeian Fard