Adding a virtual machine to inventory results in a Queued Status and eventually times out...Orphaned VMs....day from hell here...

[tupelo_operatio]

This has turned into VMware Day from Hell here. It actually started last night when we tried adding a LAMP virtual appliance our VI3 infrastructure. I downloaded the virtual appliance from the Virtual Appliances section of the VMware website and then unzipped it and uploaded it to one of the shared LUNs on our SAN that VI3 is using. I then logged in the VI Client and browsed to the datastore where the virtual appliance had been uploaded. I browsed into the folder and right-clicked on the vmx file and told it to "Add to inventory". The event shows up in the Recent Tasks as Register Virtual Machine with a Target of Discovered Virtual Machine and a Status of Queued. It will sit there for some length of time and will eventually time out. A look in C:\Windows\Temp\VPX at the latest vpxd-*.log reveals this in relation to the Register Process....

============BEGIN FAILED METHOD CALL DUMP============

Invoking on

Arg path:

"sanfs://vmfs_uuid:46549d82-014dd2d1-264b-00145ebd0f9e/Test-SWAPP/Test-SWAPP.vmx"

Arg name:

"Test-SWAPP"

Arg pool:

'vim.ResourcePool:resgroup-75'

Fault has an empty message

============END FAILED METHOD CALL DUMP============

Unable to register Vm sanfs://vmfs_uuid:46549d82-014dd2d1-264b-00145ebd0f9e/Test-SWAPP/Test-SWAPP.vmx

(Test-SWAPP) Got fault vim.fault.Timedout in creating VM

Starting journal rollback

Undo RegisterVm...Host: host-3057 ,VM: vm-3421 ,cfgPath: sanfs://vmfs_uuid:46549d82-014dd2d1-264b-00145ebd0f9e/Test-SWAPP/Test-SWAPP.vmx

I came in this morning to find 3 orphaned virtual machines in one of my resource groups. I called VMware support and was given commands to run on the vms when I was able to power them down. I was able to power down one of the orphaned VMs by VNC'ing into it and shutting it down manually that way. I then unregistered it from one host and registered it another and restarted the VMware Virtual Center Service as instructed....still no go.

Now I have just discovered that it will not allow me to create a new VM. Looks like I get to call tech support again.

:smileyangry:

[oreeh]

Have you already tried and restarted the VC server?

[tupelo_operatio]

Yeah...I did that...I got on another call with VMware support at 10:40 A.M. CST and was on that call for 2 hours while they tried to figure out what had happened to our VMware Infrastructure. I never got a straight answer from the tech but then again I had trouble understanding him at times. I still wish I knew why I cannot get this virtual appliance added to the inventory...has anyone else run into this?

[Sorwyn]

You could try using VMware Converter as a piece inbetween, I've had issues adding appliances directly in the past, at least if the VMware Converter fails it won't hurt your VC stuff (well not that I've found).

[tupelo_operatio]

I am beginning to think there is something seriously wrong with our Virtual Center Server. It is getting to where every task that is submitted to it goes into a queued state and eventually times outs. I also have 3 more orphaned VMs this morning. I am going to reboot the Virtual Center server VM itself to see if that might help things. I just realized something in dealing with these virtual appliances...and it may be only part of the problem...but when they are copied to the datastore...the *.vmx is not set as executable...I will set it as executable if I ever get it added to the mix.

[tupelo_operatio]

Going back to the issue of the orphans. In Virtual Center, one of the orphans thinks it is on the vm3 host. When I putty to the 3 hosts in that cluster and issue a vmware-cmd -l on each host, it shows that orphan is actually registered on the vm1 host. On my many calls to VMware Tech Support yesterday, the tech(s) said to go into the VI Client and right-click on the orphaned VM and remove it from the inventory and then add it back by browsing to the datastore and right-clicking on the vmx file and choosing to add it to inventory. The status of the task shows as "In Progress" and it started at 9:08:03 AM....it is now 9:18 AM and it is still showing as "In Progress"....I am pretty sure it shouldn't be taking this long to register the VM.

[Liam_Clancy]

I've got the same thing happening... "It is getting to where every task that is submitted to it goes into a queued state and eventually times outs"... did you get a fix? Ta.

[mike_laspina]

Connect directly the ESX server where the VM is using the VI3 Client and check if it manageable there. This will rule out VI3 very quickly.

[Liam_Clancy]

Everything works much nicer connected directly to an ESX host (e.g. Starting a VM or creating anew VM)... bit hard to test a VMotion though. Thanks for the quick reply.

[mike_laspina]

This is a VC issue. How many hosts are managed by the VC? Is the database MSDE? What do the Win logs on the VC server look like? What is the current database size?

[Liam_Clancy]

11 hosts.

Database is SQL Server 2000 SP4.

Database size is 2263 MB.

Log is similar to first post in this thread.

Thanks, Liam.

[mike_laspina]

If the database is MSDE you have a problem. It has a maximum size of 2GB.

[mike_laspina]

You should make sure your database recovery mode is set to bulk logged(preffered) or simple and run a backup.

[Daniel_Herrero]

Something similar happened to us.

Try to remove the licensing data, then add it again.

[dwalker-isi]

We are experiencing the same issue when trying to clone an unpowered VM. The task is submitted and sits there for several minutes until it times out with message:

Failed to communicate with the remote host.

Also, other issues with VC are the VMs not reporting performance data and show 0% on CPU, RAM, etc.

I will try connecting directly to ESX server to see what I can do from there. I will also reboot the VC-server to see if that helps.

We use SQL2000 database so it shouldn't be an issue with size of database with only about 15 VMs we have had in place for about 6 months.

Dan