mrlizard08
Enthusiast
Enthusiast

ESXI Hangs When Creating New VM

Hello.  I have ESXI 7.0 Update 2, the DellEMC Customized version, installed on a Dell PowerEdge R540.  The datastore is 17.4 TB on a RAID 6 on 2.4 TB 10K SAS drives.  OS is installed on two 64 GB SD cards in a RAID 1.  For some reason after a certain amount of time after a reboot, the shortest amount of time so far being about 24 hours, I cannot create a new virtual machine.   When I try to create the VM it just hangs at creating VM. 

When this is happening if I try to browse the datastore I get just a red x and "error" on certain folders after a waiting a little while for the little loading icon to go away. 

If I try access any of the logs, or try to open them in a new window it does nothing, the new window just has the blue background, the one that fades up to a lighter blue at the top.  I also noticed that after I click on the actions button the little gear doesn't stop spinning and I can no longer click on it.  If I refresh the page using the refresh page in the management console above the logs then the little blue and darker blue loading bar comes up, it takes a really long time but usually they'll come back up, although I still can't do anything with them.  If they don't come up it gives me a blank error window saying unhandled exception with no information and only a button to show the error details (again shows nothing), or to reload.  

I tried going into settings to turn on SSH to see if I can view the logs manually, but that doesn't work either, the settings pages just sit there with the blue and light blue loading bar forever.

I also can't access the VM's via the console, they are still running normally though as I can remote into them.

It also seems like the longer this goes on the more messed up it gets, for example, when it first starts I can at least view the folders at the root of the datastore in the datastore browser, and it errors if I try to go to subfolders, but as time goes on I can't even view the root of the datastore until its rebooted.

I also can't shutdown the running VM's from the web management console, vmware tools is installed though.  The only way to reboot it is I have to power off the VM's manually, or shut them down from inside the OS by remoting into them, and then reboot the host.  After it comes back up, it works like nothing ever happened.

Essentially all the management functions become completely unusable.

The host isn't overloaded, it only has two VM's on it right now.  I need to schedule a migration to move stuff out of the old server onto this one, but I don't feel comfortable doing that until I figure out why it keeps doing this.  I can't say I've seen an ESXI host do something like this before. 

 

EDIT: Couple minutes after posting this I was able to get SSH to turn on.  But after I SSH'd into it I tried using the command "cat /var/log/vmkernel.log" to view the contents of the vmkernel log but it just hangs forever (see picture below).

mrlizard08_0-1670515323508.png

 

0 Kudos
3 Replies
mrlizard08
Enthusiast
Enthusiast

UPDATE:  Connected with Filezilla via SFTP to try and download the log files, it just hangs there to.  There were other directories that also hang, I've listed them below.  I didn't try all directories, but those ones for sure did hang when I tried to access them in filezilla

/var/logs - all subdirectories hang when trying to retrieve contents

/vmfs/volumes - hangs when trying to retrieve contents of volumes folder

/vmimages - hangs when trying to retrieve contents of subdirectories

 

0 Kudos
mrlizard08
Enthusiast
Enthusiast

UPDATE: It took about 25 or 30 minutes but it finally showed at least a portion of vmkernel.log, hopefully this is helpful in diagnosing the issue, I'm noticing alot of "Status=Transient storage condition, suggest retry" messages.

0 Kudos
mrlizard08
Enthusiast
Enthusiast

UPDATE:  Well it seems this can be disregarded.  This is a known issue with SD cards and ESXI 7.0.  As shown in the below KB and forum post on Dell's website.

 

https://www.dell.com/community/PowerEdge-Hardware-General/VMware-7-0-U2-losing-contact-with-SD-card/...

https://kb.vmware.com/s/article/83376

 

0 Kudos