Virtual Machines disapear from inventory

[RogerPBS]

Hi everyone,

I'm new here and this is my first post, so please, be kind on me. 

I had an issue with VMware ESXi recently, that it is the second time it happens. All the Virtual Machines in the inventory disapear, they keep working and so, but there are no clue about them in the ESXi. The way to restore it is entering in the Storage inventory and click to register a VM, where you can register again one by one all the machines. 

This is not a big problem by itself, but, I have a backup system that, when this occurs, fails to make the backup of the VM disapeared and the task of the backups have to be reconfigured. 

On the other hand, I'm afraid if this problem could be a syntom of a worse issue or can cause that someday I can't recover the VM by registering them again.

Does anyone knows why this is happening and how to solve it?

Thanks in advance and best regards.

[depping]

Probably would be useful to provide some more info:

1) which version are you using?

2) which interface are you looking at? (vCenter UI or ESXi Host Client?)

3) is this host part of a cluster?

[RogerPBS]

Hi Depping, thanks for your response.

1. The host version is 6.0.0

2. The interface is ESXi Host Client with version 1.33.1

3. Teorically no, its a host hired to SoYouStart alone.

If you need some more info, let me know it.

Best regards

[continuum]

That sounds like the ESXi host temporarily lost connection to one of the datastores.
Can you please download  /var/log/vmkernel.log and attach it to your next reply.

Ulli

[RogerPBS]

Hi Ulli,

Here you have it.

The day I think (based on the error of the backup) that the vmware stopped showing the virtual machines, was May the 20th.

Best regards.

[continuum]

Wow - this does not look good.
Have a look at the problem listed for several vmdks:

Spoiler

2022-05-09T10:20:54.388Z cpu7:101919970 opID=e47af75e)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 07 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-09T10:20:54.399Z cpu7:101919970 opID=e47af75e)FSS: 6245: Failed to open file 'MULTI 07 --flat.vmdk'; Requested flags 0x40001, world: 101919970 [hostd-worker], (Existing flags 0x4008, world: 134670250 [vmx]): Busy
2022-05-10T10:01:17.662Z cpu3:134260116 opID=36461d1c)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:01:17.662Z cpu3:134260116 opID=36461d1c)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 134260116 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - F$
2022-05-10T10:09:26.123Z cpu3:134719123 opID=8b9462a3)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:09:26.123Z cpu3:134719123 opID=8b9462a3)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 134719123 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - F$
2022-05-10T10:09:28.072Z cpu1:134786185 opID=6212e068)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:09:28.072Z cpu1:134786185 opID=6212e068)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 134786185 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - F$
2022-05-10T10:10:50.222Z cpu11:134530424 opID=fe03ae2)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 06 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:10:50.222Z cpu11:134530424 opID=fe03ae2)FSS: 6245: Failed to open file 'MULTI 06 --flat.vmdk'; Requested flags 0x40001, world: 134530424 [hostd-worker], (Existing flags 0x4008, world: 133377108 [vmx]): Busy
2022-05-10T10:10:58.774Z cpu0:134786185 opID=cd604871)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 07 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:10:58.774Z cpu0:134786185 opID=cd604871)FSS: 6245: Failed to open file 'MULTI 07 --flat.vmdk'; Requested flags 0x40001, world: 134786185 [hostd-worker], (Existing flags 0x4008, world: 134670250 [vmx]): Busy
2022-05-10T10:11:14.763Z cpu8:134260116 opID=b045f905)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 09 U18 --000001-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:11:14.763Z cpu8:134260116 opID=b045f905)FSS: 6245: Failed to open file 'MULTI 09 U18 --000001-delta.vmdk'; Requested flags 0x40001, world: 134260116 [hostd-worker], (Existing flags 0x4008, world: 131843398 [vmx]): Busy
2022-05-10T10:11:31.138Z cpu0:134260115 opID=a92710a7)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 14 --000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:11:31.138Z cpu0:134260115 opID=a92710a7)FSS: 6245: Failed to open file 'MULTI 14 --000003-delta.vmdk'; Requested flags 0x40001, world: 134260115 [hostd-worker], (Existing flags 0x4008, world: 125099781 [vmx-vcpu-0:MULTI 14 U12 - Partes tra$
2022-05-10T10:11:42.641Z cpu5:134719119 opID=cd9d9b2d)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 16-flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:11:42.641Z cpu5:134719119 opID=cd9d9b2d)FSS: 6245: Failed to open file 'MULTI 16-flat.vmdk'; Requested flags 0x40001, world: 134719119 [hostd-worker], (Existing flags 0x4008, world: 133361541 [vmx]): Busy
2022-05-10T10:11:59.018Z cpu11:134719123 opID=e1f19946)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI19 BIS --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:11:59.018Z cpu11:134719123 opID=e1f19946)FSS: 6245: Failed to open file 'MULTI19 BIS --flat.vmdk'; Requested flags 0x40001, world: 134719123 [hostd-worker], (Existing flags 0x4008, world: 130344199 [vmx]): Busy
2022-05-10T10:12:14.287Z cpu0:134794384 opID=9d296e25)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI4-flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:12:14.287Z cpu0:134794384 opID=9d296e25)FSS: 6245: Failed to open file 'MULTI4-flat.vmdk'; Requested flags 0x40001, world: 134794384 [hostd-worker], (Existing flags 0x4008, world: 130823339 [vmx]): Busy
2022-05-10T10:12:41.064Z cpu5:134260115 opID=eaa298b9)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:12:41.064Z cpu5:134260115 opID=eaa298b9)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 134260115 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - F$
2022-05-10T10:12:45.699Z cpu11:134786185 opID=74856b5)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 06 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-10T10:12:45.699Z cpu11:134786185 opID=74856b5)FSS: 6245: Failed to open file 'MULTI 06 --flat.vmdk'; Requested flags 0x40001, world: 134786185 [hostd-worker], (Existing flags 0x4008, world: 133377108 [vmx]): Busy
2022-05-17T16:26:46.097Z cpu8:134794386 opID=3fdbfe18)FSS: 5760: Conflict between buffered and unbuffered open (file 'Plesk Paris-flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-17T16:26:46.097Z cpu8:134794386 opID=3fdbfe18)FSS: 6245: Failed to open file 'Plesk Paris-flat.vmdk'; Requested flags 0x40001, world: 134794386 [hostd-worker], (Existing flags 0x4008, world: 135775538 [vmx]): Busy
2022-05-17T16:27:04.400Z cpu0:134719119)FSS: 6245: Failed to open file 'Plesk Paris-000001.vmdk'; Requested flags 0x40001, world: 134719119 [hostd-worker], (Existing flags 0x8, world: 134890856 [vmx-vcpu-3:Plesk Paris]): Busy
2022-05-17T16:38:51.910Z cpu0:134719120)FSS: 6245: Failed to open file 'Plesk Paris-000002.vmdk'; Requested flags 0x40001, world: 134719120 [hostd-worker], (Existing flags 0x8, world: 134890859 [vmx-vcpu-6:Plesk Paris]): Busy
2022-05-23T07:13:48.942Z cpu8:134182636 opID=92f67f4f)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 06 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:13:48.942Z cpu8:134182636 opID=92f67f4f)FSS: 6245: Failed to open file 'MULTI 06 --flat.vmdk'; Requested flags 0x40001, world: 134182636 [hostd-worker], (Existing flags 0x4008, world: 133377108 [vmx]): Busy
2022-05-23T07:14:06.436Z cpu2:136148731 opID=f4196056)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 09 U18 --000001-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:14:06.436Z cpu2:136148731 opID=f4196056)FSS: 6245: Failed to open file 'MULTI 09 U18 --000001-delta.vmdk'; Requested flags 0x40001, world: 136148731 [hostd-worker], (Existing flags 0x4008, world: 131843398 [vmx]): Busy
2022-05-23T07:14:13.154Z cpu5:136148731 opID=f825422f)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 10 --000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:14:13.154Z cpu5:136148731 opID=f825422f)FSS: 6245: Failed to open file 'MULTI 10 --000003-delta.vmdk'; Requested flags 0x40001, world: 136148731 [hostd-worker], (Existing flags 0x4008, world: 135931117 [vmx]): Busy
2022-05-23T07:14:18.712Z cpu0:136254675 opID=fbd7948c)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 14 --000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:14:18.712Z cpu0:136254675 opID=fbd7948c)FSS: 6245: Failed to open file 'MULTI 14 --000003-delta.vmdk'; Requested flags 0x40001, world: 136254675 [hostd-worker], (Existing flags 0x4008, world: 125099781 [vmx-vcpu-0:MULTI 14 U12 - Partes tra$
2022-05-23T07:14:23.377Z cpu8:136254675 opID=590d672)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 16-flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:14:23.377Z cpu8:136254675 opID=590d672)FSS: 6245: Failed to open file 'MULTI 16-flat.vmdk'; Requested flags 0x40001, world: 136254675 [hostd-worker], (Existing flags 0x4008, world: 133361541 [vmx]): Busy
2022-05-23T07:14:27.959Z cpu8:136512734 opID=46cd8958)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 16-flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:14:27.959Z cpu8:136512734 opID=46cd8958)FSS: 6245: Failed to open file 'MULTI 16-flat.vmdk'; Requested flags 0x40001, world: 136512734 [hostd-worker], (Existing flags 0x4008, world: 133361541 [vmx]): Busy
2022-05-23T07:32:07.776Z cpu10:136512734 opID=721724d1)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:32:07.776Z cpu10:136512734 opID=721724d1)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 136512734 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - $
2022-05-23T07:32:47.650Z cpu10:136513312 opID=518d1056)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:32:47.650Z cpu10:136513312 opID=518d1056)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 136513312 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - $
2022-05-23T07:32:49.761Z cpu0:136513312 opID=9966ae6c)FSS: 5760: Conflict between buffered and unbuffered open (file 'new-flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:32:49.761Z cpu0:136513312 opID=9966ae6c)FSS: 6245: Failed to open file 'new-flat.vmdk'; Requested flags 0x40001, world: 136513312 [hostd-worker], (Existing flags 0x4008, world: 135951607 [vmx]): Busy
2022-05-23T07:32:51.119Z cpu2:136512734 opID=5b964d79)FSS: 5760: Conflict between buffered and unbuffered open (file 'Centralita - Free PBX-000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:32:51.119Z cpu2:136512734 opID=5b964d79)FSS: 6245: Failed to open file 'Centralita - Free PBX-000003-delta.vmdk'; Requested flags 0x40001, world: 136512734 [hostd-worker], (Existing flags 0x4008, world: 119772401 [vmx-vcpu-0:Centralita - F$
2022-05-23T07:33:03.111Z cpu6:136513311 opID=fbeb18d6)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 10 --000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:33:03.111Z cpu6:136513311 opID=fbeb18d6)FSS: 6245: Failed to open file 'MULTI 10 --000003-delta.vmdk'; Requested flags 0x40001, world: 136513311 [hostd-worker], (Existing flags 0x4008, world: 135931117 [vmx]): Busy
2022-05-23T07:33:04.056Z cpu0:136513311 opID=e8d7266c)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 09 U18 --000001-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:33:04.056Z cpu0:136513311 opID=e8d7266c)FSS: 6245: Failed to open file 'MULTI 09 U18 --000001-delta.vmdk'; Requested flags 0x40001, world: 136513311 [hostd-worker], (Existing flags 0x4008, world: 131843398 [vmx]): Busy
2022-05-23T07:33:05.142Z cpu8:136148722 opID=da647ca3)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 10 --000003-delta.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T07:33:05.142Z cpu8:136148722 opID=da647ca3)FSS: 6245: Failed to open file 'MULTI 10 --000003-delta.vmdk'; Requested flags 0x40001, world: 136148722 [hostd-worker], (Existing flags 0x4008, world: 135931117 [vmx]): Busy
2022-05-23T14:33:17.102Z cpu7:136148722 opID=a74edac7)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 05 U14 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T14:33:17.102Z cpu7:136148722 opID=a74edac7)FSS: 6245: Failed to open file 'MULTI 05 U14 --flat.vmdk'; Requested flags 0x40001, world: 136148722 [hostd-worker], (Existing flags 0x4008, world: 136551974 [vmx]): Busy
2022-05-23T14:33:36.342Z cpu8:136513312 opID=499f0645)FSS: 5760: Conflict between buffered and unbuffered open (file 'MULTI 05 U14 --flat.vmdk'):flags 0x4008, requested flags 0x40001
2022-05-23T14:33:36.342Z cpu8:136513312 opID=499f0645)FSS: 6245: Failed to open file 'MULTI 05 U14 --flat.vmdk'; Requested flags 0x40001, world: 136513312 [hostd-worker], (Existing flags 0x4008, world: 136551974 [vmx]): Busy

To me this looks like a datastore with some serious corruption issues.
If this VMs are important contact me via skype so I can have a closer look via Teamviewer.

Ulli

[RogerPBS]

Hi Ulli,

Thanks for your answer and your time.

The important ones are backed up and secured. Except for the problem mentioned, I did not see any other issue or malfunction of the host or the virtual machines. Sorry for my ignorance but, why does that messages means that there are corruption issues? Seems like some incoherences, but not corruption. Which options do I have to solve this problems? Backup all the VM and migrate it to another host?

Best regards,

[continuum · Accepted solution]

Those messages mean that there is a mismatch between the data actually written to disk and the data that was supposed to be written.
WARNING: HBX: 1734: 'datastore1': HB at offset 3911680 - HB IO was aborted; disk contents unknown. In-mem:
Anyway - if possible evacuate this datastore as soon as possible. If you run into corrupted vmdks or Input/Output errors let me know.

Ulli

[RogerPBS]

Okey!

Thanks you, I will start looking for a new host to migrate all this machines.

Best regards