Several VMs are running but are not in Inventory

[MrBromden]

Hello everyone,

I hope someone has experienced this condition and can help.

Here is what I have.

I have 13 VMs (Windows based) that I have been doing DR tests with using EMC RP. All servers are in one consistency group with two LUNs. I have been able to failover the environment to the copy site and then failback to the original location.

However, today when I failed back the instances I have lost 4 VMs.

- I have 9 VMs that I can see in vCenter Server

- The VMs that I have "lost" are running, and I can connect to the using RDP but they do not appear in vCenter Server

- The VMs are in their folders in their datastores. When I browse to the relevant folder and try to Add to Inventory the option is greyed out. Which suggest that the VM is being registered  but when  I search I cannot see the instance.

PowerCLI command revealed that on both DSs I have 9 powered on VMs. As I said all 13 VMs are running.

I have queried  each host in the cluster its vmInventory.xml file and the servers registered on all hosts are 9 instead of 13.

I have not unmounted the DS during the RecoverPoint failover operations, just powered down the VMs.

Luckily, this environment is not in full production yet which is the only good thing.

Has anybody come across this rather weird situation.

If this is relevant, storage array is EMC VNX and the access is all block.

Kind Regards,

[CoolRam]

it sounds like the vcenter server has lost communication to the ESXi host but the ESXi host is still up and running.

You can also try restarting the management agent on the host where those VMs are located: http://kb.vmware.com/kb/1003490http://kb.vmware.com/kb/1003490.

[MrBromden]

Hi CoolRam,

In vCenter there were two hosts with VMs showing as (Inaccessible), however these VMs were not in the VMInventory.xml on the hosts.

I have restarted the hostd on the hosts with no avail. And the weird bit is that all 13 VMs are running but only 9 are listed in vCenter.

Regards,

[brunofernandez1]

what do you see when you connecti directly with the VIClient to the esx server?

are they there visible? if yes, i would recommend to restart the vCenter services and then restart the esxi mgmt agents...

as you said that the vms are running it can only be a displaying problem...

[MrBromden]

Hi Bruno,

Thank you for your reply.

I have forgotten to say that I have logged into each host directly in the cluster and I can see VMs that I can see in vCenter Server, i.e. only 9 instances out of 13. The other 4 are not there.

Looking at their directories on the volumes everything looks OK.

From the beginning those VM were inaccessible, and I removed them. Usually, this is when there is an issues with the array, but I have another few dozen volumes presented to the cluster and they are all fine. When I go the directory of the missing but running VM, and try to add it to the Inventory the option is greyed out.

Any idea what else it could be?

Kind Regards,

[brunofernandez1]

did you allready booted the esx server?

i would recommend to shutdown the VMs with RDP and then reboot the esx server. after this, the VM files, like vmx, vmdk and so on, are no more locked and it should be possible to add them again to the inventory...

[MrBromden]

Bruno,

I was thinking about this but If I were to shutdown the VMs that are not visible in vCenter but still running, how am I going to power them on?

You are right that I have access to the .vmx files through the datastores, but again I fear that I my not be able to power them on.

Also, the visible VMs are spread across a few hosts, so I will have to put them into maintenance mode one by one. I might have to do that for it sounds quite reasonable.

So far, I have restarted two of the hosts that those inaccessible VMs were running on but it did not solve the issue.

That was a helpful suggestion Bruno.

Thank you

[vfk]

Hi, you add the VM back into vCenter using cli with the following

vim-cmd solo/registervm /<path-to-vm>/vm-name.vmx

You don't need to power off the VMs or reboot the hosts.

[MrBromden]

Hi vfk,

I have forgotten to say but I have tried to register the missing VMs from the command line. That was one of my first actions. With no avail. I watched vCenter tasks when running the commands but did not see any tasks running against the commands.

It looks like I have run out of options.

Kind Regards,

[brunofernandez1]

as I know, you can't add a VM to the inventory if its running. for this you first have first to shut down the VM.

the recommendation to reboot also the esxi server is because of possible locked files....but it could also match without rebooting the esx server.

don't worry about the VM. as long as the files are in the datastore, there will be every time a way to import the VM again 😉

have a trie with a VM that is less important and if it works you can do it with the other ones...

[vfk]

MrBromden the command does not produce any output if successful on esxi cli, also once it has been added, I have found sometimes it is does not appear to searchable for a little while, but will eventually be searchable;  so you have to manually look for the VM.

[MrBromden]

Good morning Bruno,

Apologies for the late reply.

So stupid. I have been trying to add VMs that are running even though for some reason not in the Inventory.

So, I have powered down one by one the instances that were invisible and ran vim-cmd /solo/register etc. and the VMs appeared in vCenter Server.

What it made me uncomfortable was the fact that I could not see them in vCenter.

My first troubleshooting steps before posting the question on the Forum were to check the folder/file structure of the missing VMs, then ran vim-cmd vmsvc/getallvms, which did not show the missing VMs, and then vim-cmd /solo/registervm etc command.

Bruno, thank you very much indeed.

Regards,

[MrBromden]

Bruno,

I have tried to mark your answer as a Correct but it does not respond to the click.

Thank you

[brunofernandez1 · Accepted solution]

no Problem you're welcome!