VM Got unregistered and vCenter Failed

[farisn]

HI,

Today I notice that my vCenter and other VMs are down, and strange behavior is happening, there is always a trigger for a snapshot. The vCenter VM have a question mark icon on the top of it and I there was a question I need to answer, whether or not should I continue or terminate the session (Continue I need to free some space)

2019-05-15T05:41:26.672Z| vmx| I125: [msg.hbacommon.outofspace] There is no more space for virtual disk 'MyvCenter1_3.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking _Retry. Click Cancel to terminate this session.

I click on Terminate, OK... Then the VM got unregistered. and whenever I try to add it again, I cannot.

Using the web interface for the ESXI Node, I try to "Register a VM", but whenever I click on the folder where the vCenter VM located. it keep loading and it through an error " An Error Occurred, Please try again"

I tried to use PowerCLI to register the VM New-VM -VMFilePath "[Datastore-VM] MyvCenter/MyvCenter.vmx"

The output is

Name                 PowerState Num CPUs MemoryGB

----                 ---------- -------- --------

/vmfs/volumes/5aa... PoweredOff 0        0.000

but also cannot start it up, and its got unregistered automatically.

If I click on other folders, I can access them, but not the vCenter folder, I used WinSCP to access the folder and I was able to open the folder, but when I try to click on any file, like the vmware.log, I got the following error

General failure (server should provide error description).

Error code: 4

Error message from server: Failure

I dont know if the file is being used, but I am able to access vmware0.log ...

and here are some of the errors

2019-05-15T05:53:28.686Z| vmx| I125: Timing out dialog 425937406

2019-05-15T05:53:28.686Z| vmx| I125: MsgQuestion: msg.hbacommon.outofspace reply=0

2019-05-15T05:53:28.686Z| vmx| I125: Msg_Question:

2019-05-15T05:53:28.686Z| vmx| I125: [msg.hbacommon.outofspace] There is no more space for virtual disk 'MyvCenter1_3.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking _Retry. Click Cancel to terminate this session.

2019-05-15T05:53:28.686Z| vmx| I125: ----------------------------------------

2019-05-15T05:53:38.798Z| vmx| I125: VigorTransportProcessClientPayload: opID=ca858ddb seq=22712332: Receiving Bootstrap.MessageReply request.

2019-05-15T05:53:38.799Z| vmx| I125: VigorTransport_ServerSendResponse opID=ca858ddb seq=22712332: Completed Bootstrap request.

2019-05-15T05:53:38.799Z| vmx| I125: MsgQuestion: msg.hbacommon.outofspace reply=1

2019-05-15T05:53:38.799Z| vmx| E105: PANIC: Exiting because of failed disk operation.

2019-05-15T05:53:38.799Z| vmx| I125: Printing loaded objects

2019-05-15T05:53:38.799Z| vmx| I125: [0x5D8DD7000-0x5D9E8ACFC): /bin/vmx

2019-05-15T05:53:38.799Z| vmx| I125: [0x61A484000-0x61A49B448): /lib64/libpthread.so.0

2019-05-15T05:53:38.799Z| vmx| I125: [0x61A6A1000-0x61A6A2F00): /lib64/libdl.so.2

2019-05-15T05:53:38.799Z| vmx| I125: [0x61A8A5000-0x61A8ADF9C): /lib64/librt.so.1

2019-05-15T05:53:38.799Z| vmx| I125: [0x61AAC1000-0x61AD561C4): /lib64/libcrypto.so.1.0.2

2019-05-15T05:53:38.799Z| vmx| I125: [0x61AF87000-0x61AFEFB1C): /lib64/libssl.so.1.0.2

2019-05-15T05:53:38.799Z| vmx| I125: [0x61B1FB000-0x61B30F4DC): /lib64/libX11.so.6

2019-05-15T05:53:38.799Z| vmx| I125: [0x61B516000-0x61B52501C): /lib64/libXext.so.6

2019-05-15T05:53:38.799Z| vmx| I125: [0x61B726000-0x61B80A341): /lib64/libstdc++.so.6

2019-05-15T05:53:38.799Z| vmx| I125: [0x61BA29000-0x61BAA8DA4): /lib64/libm.so.6

2019-05-15T05:53:38.799Z| vmx| I125: [0x61BCAB000-0x61BCBFBC4): /lib64/libgcc_s.so.1

2019-05-15T05:53:38.799Z| vmx| I125: [0x61BEC1000-0x61C021DD4): /lib64/libc.so.6

2019-05-15T05:53:38.799Z| vmx| I125: [0x5DA263000-0x5DA2807D8): /lib64/ld-linux-x86-64.so.2

2019-05-15T05:53:38.799Z| vmx| I125: [0x61C22C000-0x61C246634): /lib64/libxcb.so.1

2019-05-15T05:53:38.799Z| vmx| I125: [0x61C448000-0x61C44995C): /lib64/libXau.so.6

2019-05-15T05:53:38.799Z| vmx| I125: [0x61CA18000-0x61CAAD4B4): /usr/lib64/vmware/plugin/objLib/upitObjBE.so

2019-05-15T05:53:38.799Z| vmx| I125: [0x61CCC6000-0x61CE1C1A4): /usr/lib64/vmware/plugin/objLib/vsanObjBE.so

2019-05-15T05:53:38.799Z| vmx| I125: [0x61D0B3000-0x61D0C7F94): /lib64/libz.so.1

2019-05-15T05:53:38.799Z| vmx| I125: [0x61D512000-0x61D51D1D0): /lib64/libnss_files.so.2

2019-05-15T05:53:38.799Z| vmx| I125: End printing loaded objects

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace:

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[0] 000003d049158420 rip=00000005d9450f27 rbx=00000005d9450a20 rbp=000003d049158440 r12=0000000000000000 r13=0000000000000001 r14=00000005d9b00e60 r15=00000005d9b00e7c

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[1] 000003d049158450 rip=00000005d8f95cec rbx=000003d049158470 rbp=000003d049158950 r12=00000005da0e2d50 r13=0000000000000001 r14=00000005d9b00e60 r15=00000005d9b00e7c

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[2] 000003d049158960 rip=00000005d9105fff rbx=00000005da827330 rbp=000003d0491589a0 r12=00000005d9ebdd00 r13=00000005da827335 r14=00000005d9b00e60 r15=00000005d9b00e7c

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[3] 000003d0491589b0 rip=00000005d90435b1 rbx=00000005da827330 rbp=000003d0491589c0 r12=00000005da8f6190 r13=000003d0491589fc r14=0000000000000000 r15=000003d049158a00

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[4] 000003d0491589d0 rip=00000005d8f9059c rbx=000000061c64d010 rbp=000003d049158a60 r12=00000005da8f6190 r13=000003d0491589fc r14=0000000000000000 r15=000003d049158a00

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[5] 000003d049158a70 rip=00000005d8f911ee rbx=000037281ed6ddf9 rbp=000003d049158b30 r12=000000061c64d010 r13=00000005da8f6190 r14=00000005da52f2e0 r15=0000000000000000

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[6] 000003d049158b40 rip=00000005d8f967f0 rbx=00000005da0e2d80 rbp=000003d049158c60 r12=00000005da7fe8b0 r13=00000005da657f10 r14=0000000000000000 r15=00000005da657f10

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[7] 000003d049158c70 rip=00000005d8f89f06 rbx=0000000000000003 rbp=000003d049158cf0 r12=0000000000000000 r13=00000005d9a14851 r14=00000005d9e8cc20 r15=0000000000000000

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[8] 000003d049158d00 rip=000000061bee18cd rbx=0000000000000000 rbp=0000000000000000 r12=00000005d8f8a878 r13=000003d049158dd0 r14=0000000000000000 r15=0000000000000000

2019-05-15T05:53:38.799Z| vmx| I125: Backtrace[9] 000003d049158dc0 rip=00000005d8f8a8a1 rbx=0000000000000000 rbp=0000000000000000 r12=00000005d8f8a878 r13=000003d049158dd0 r14=0000000000000000 r15=0000000000000000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[0] 000003d049158420 rip=00000005d9450f27 in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[1] 000003d049158450 rip=00000005d8f95cec in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[2] 000003d049158960 rip=00000005d9105fff in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[3] 000003d0491589b0 rip=00000005d90435b1 in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[4] 000003d0491589d0 rip=00000005d8f9059c in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[5] 000003d049158a70 rip=00000005d8f911ee in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[6] 000003d049158b40 rip=00000005d8f967f0 in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[7] 000003d049158c70 rip=00000005d8f89f06 in function main in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[8] 000003d049158d00 rip=000000061bee18cd in function __libc_start_main in object /lib64/libc.so.6 loaded at 000000061bec1000

2019-05-15T05:53:38.799Z| vmx| I125: SymBacktrace[9] 000003d049158dc0 rip=00000005d8f8a8a1 in function (null) in object /bin/vmx loaded at 00000005d8dd7000

2019-05-15T05:53:38.799Z| vmx| I125: Exiting

I have several VM having the same issue and it's getting more critical. Any possible help

I am not sure if I restore the vCenter from my Commvault backup will it help or will it destroy whats running.

[daphnissov]

If this is a critical issue you need to open a support request with VMware.

[farisn]

It is critical, and my support contract has expired, and there is some internal procedure for it to be completed, which may take time.

Adding to the issue, I rebooted the server, hopfully it will register the VM again, but eventually, all the VMs on that server got unregistered too and I cannot add them again.

the only luck I had is these VMs are not critical.

[continuum]

<rant-start>

Thin provisioning comes with a price: you pay with obligatory-monitoring.

If you fill up the datastore once - the deal breaks.
Please forgive me that unfriendly sounding intro.
I have the impressions that too many users are not really aware of that deal.
<rant-end>

> but eventually, all the VMs on that server got unregistered too.
You may not want to hear it - but a reasonable decision needs to be made now.
You can try to fix this datastore - spend a few days on it and probably make it worse.

You can also directly switch to plan B and start evacuation to a new datastore.
You have none ? - that answer is not an option.

Any next start-attempt, any next expand of a running thin provisioned vmdk or snapshot will further increase the damage.

The plan is to clone all VMs and VMDKs to another datastore as quietly as possible..

Do not attempt to do any snapshot operations. Power down all eventually still running VMs to avoid further large vmem-files.

Only use vmkfstools or ddrescue to copy the vmdks and dont run too many jobs side by side.
You may run into vmdks  or deltas that cant be copied.

Do not try other procedures - log those problem-files and I will help you them when necessary.

To sum it up: avoid all operations that require expands of existing files.

Do not try to reboot again - do not try to rename anything.

Copy out in order of priority.
Ok - thats all I can suggest at the moment.
I dont think that you will safe time when you try a fix by deleting unnecessary files.

It is highly recommended to rebuild the datastore anyway ....

If any urgent issues occur - send message via skype.

Ulli