VMware Cloud Community
AndrewAdvnetsol
Contributor
Contributor

ESXi 6.5.0 Update 2 (Build 8294253) Randomly Freezing or Locking Up

I have a standalone ESXi host, not connected to vCenter, that randomly locks up or freezes.  It can go a few days or a few weeks between lock ups.  When it locks up I am unable to ping the host, the VM running on the host, and when I plug a keyboard into the physical machine I get no response from keystrokes.  Even the num lock key won't light up when pressed.  The only way to get it back online is to do a hard reboot, push the power button on the physical machine.

I have attached the VMKernel log file.

Reply
0 Kudos
9 Replies
diegodco31
Leadership
Leadership

Hi

What's hardware version?

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego
Reply
0 Kudos
AndrewAdvnetsol
Contributor
Contributor

Do you mean what hardware is it running on?

Reply
0 Kudos
AndrewAdvnetsol
Contributor
Contributor

It is running on a server with an Intel Sandy Bridge Platform Xeon CPU E-3-1220 V2 @ 3.10GHz.

Reply
0 Kudos
Ganeshpal
Enthusiast
Enthusiast

Hi Andrew,

Diego Ask Guest VM hardware version not CPU details.

Regards,

Ganeshprasad Pal

Reply
0 Kudos
Dave_the_Wave
Hot Shot
Hot Shot

Curious, can a bad VM really turn a host into a show stopper?

I always expected that it would be isolated, and hijack a few cores at best.

Wouldn't this be a serious security issue for a hosting company that would have many clients each with their own VM on a same host?

Reply
0 Kudos
AndrewAdvnetsol
Contributor
Contributor

Version 8

Reply
0 Kudos
AndrewAdvnetsol
Contributor
Contributor

Here is an update VMKernal log file.  Are there any other log files I should be attaching? I would appreciate any help on what is causing my host to lock up and causing my VM to be offline.

Reply
0 Kudos
continuum
Immortal
Immortal

Are you using a Veeam VM with a 6 TB vmdk on a standalone ESXi to store serious backups ???
Your 20006829 MiB datastore is not alligned correctly - maybe created with very old or buggy ESXi.

Some plugins do not load at boot-time - if they are related to special storage features that you need for Veeam-functions this needs to be troubleshooted.

2018-12-21T16:28:06.341Z cpu1:68361)FSS: 6229: Conflict between buffered and unbuffered open (file 'Veeam Cloud Connect-flat.vmdk'):flags 0x4008, requested flags 0x1

2018-12-21T16:28:06.604Z cpu0:68361)FSS: 6229: Conflict between buffered and unbuffered open (file 'Veeam Cloud Connect_1-flat.vmdk'):flags 0x4008, requested flags 0x1

2018-12-21T16:28:06.796Z cpu0:68361)FSS: 6229: Conflict between buffered and unbuffered open (file 'Veeam Cloud Connect_2-flat.vmdk'):flags 0x4008, requested flags 0x1

2018-12-21T16:28:07.499Z cpu0:68361)FSS: 6229: Conflict between buffered and unbuffered open (file 'Veeam Office 365 Backup-flat.vmdk'):flags 0x4008, requested flags 0x1

2018-12-21T16:28:07.776Z cpu1:69519)WARNING: Fil3: 7869: Found invalid object on 5a795b9f-0c147cb9-c84b-001e67ce70cd <FD c0 r0> expected <FD c0 r40>

2018-12-21T16:28:07.776Z cpu1:69519)Vol3: 3098: Failed to get object 28 type 3 uuid 5a795b9f-0c147cb9-c84b-001e67ce70cd FD a000004 gen 31 :Not found

2018-12-21T16:28:07.791Z cpu2:69519)WARNING: Fil3: 7869: Found invalid object on 5a795b9f-0c147cb9-c84b-001e67ce70cd <FD c0 r0> expected <FD c0 r40>

2018-12-21T16:28:07.791Z cpu2:69519)Vol3: 3098: Failed to get object 28 type 3 uuid 5a795b9f-0c147cb9-c84b-001e67ce70cd FD a000004 gen 31 :Not found

Are those Veeam vmdks thin provisioned ?
I am not at all surprised that this host freezes occasionally and I expect that it will get worse.
I highly recommend to rebuild your datastore - a VMFS-volume with an offset other than 1Mb is unexpected and has a big impact on performance.
Please check:
- are the veeam vmdks thin provisioned ?

- are there NTFS-errors in the systemlogs of your Veeam  VMs ?
- vmware.logs from the Veeam VMs
- which vmfs version do you use ?
- check if the VMFS-features your host offers match with the Veeam demands
Hope you use those vmdks only for temporary, discardable data.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos
continuum
Immortal
Immortal

First of all check your BIOS / EFI settings and enable the keyboard again. The mode you use is not supported.
Every time you have to do a hard reset you damage the filesystem.


________________________________________________
Do you need support with a VMFS recovery problem ? - send a message via skype "sanbarrow"
I do not support Workstation 16 at this time ...

Reply
0 Kudos