VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
12 Replies Last post: Apr 22, 2009 5:05 AM by penghaug  

error deploying vm posted: Apr 20, 2009 3:52 AM

Click to view Jovi87's profile Novice 10 posts since
Nov 29, 2007

We run esx 3.5 and labmanager 2.5 and this has been working fine until now. On two of our vms in labmanager we recieve this error, see attached file, when we try to deploy the vms.

I checked the raid and everything looks good there. Also the 20 or so other vms still run as normal. Any help is apreciated!

Attachments:

Re: error deploying vm

1. Apr 20, 2009 10:27 AM in response to: Jovi87
Click to view jshao's profile Novice VMware Employees 16 posts since
Jul 21, 2008

Hi Jovi

Have you tried to access the vmx file directly on the ESX server to see if there is any locking issue on the file?

John Shao
VMware Inc.

Re: error deploying vm

2. Apr 21, 2009 12:04 AM in response to: jshao
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007

hi John,

First thanks for your reply.

This seems to be a file system issue, ESX console reports the following message: ls: reading directory .: Device or resource busy
The Virtual Infrastructure Client is reporting 260GB used out of 1TB space, but we can not browse the VMFS datastore on the ESX server.

The management card in the server reports nothing about any disk issue, which we now suspect.

-Pål-André

Re: error deploying vm

3. Apr 21, 2009 12:13 AM in response to: penghaug
Click to view jshao's profile Novice VMware Employees 16 posts since
Jul 21, 2008

This could happen due to a ESX having a rogue lock on the VM you are using.

run esxtop on console of all ESX that have access to the datastore and see if you see the VM running some where. If it does than I can provide a command line to fix that.

John

Re: error deploying vm

4. Apr 21, 2009 12:29 AM in response to: jshao
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007

hi again,

The easy part is that it's only one ESX server connected with local storage to all the VM's. esxtop says that 3 VM's are running, but all guests are undeployed in Labmanager.

-Pål-André

Re: error deploying vm

5. Apr 21, 2009 12:34 AM in response to: penghaug
Click to view jshao's profile Novice VMware Employees 16 posts since
Jul 21, 2008
are the VM showing deployed in VI client? If they are just go ahead and undeploy them from the VI Client by doing a shutdown and unregister VM. that should fix your problem.

John Shao
VMware Inc.

Re: error deploying vm

6. Apr 21, 2009 12:46 AM in response to: jshao
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007

hi,

In the VI client the status of the VM's are:

Unknown (Inaccessible)
Unknown 1(Inaccessible)

and so on, and the datastore is showing space used but we can not browse or see any folders.

-Pål-André

Re: error deploying vm

7. Apr 21, 2009 12:50 AM in response to: penghaug
Click to view jshao's profile Novice VMware Employees 16 posts since
Jul 21, 2008

Hmm that is quite weird, all the other VM is running fine on the datastore but only these 2 is misbehaving. This is the command you can run on the ESX to kill the process for these VM.

#ps auxfww|grep $vmname

#Kill $pid

Remove the VM after that from your VI client with unregister VM from inventory. try to redeploy after you have done the following steps.

John

Re: error deploying vm

8. Apr 21, 2009 1:38 AM in response to: jshao
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007
hi,

No VM is running at all, the guest status is unknown in Virtual Infrastructure Client, and in the labmanager console all guests are undeployed (some with errors, had to force the undeploy on some of the guests)

The confusing part is that if you take a look at the attached image from esxtop which still tells us that 3 VM's are running.

ls -la in the datastore/VM folder gives us the error message:

root@FS-ESXLAB-001 VM# ls -al
ls: reading directory .: Device or resource busy

And ps auxfww|grep $vmname (from any of the listed names in esxtop gives us no PID to kill).

-Pål-André

Attachments:

Re: error deploying vm

9. Apr 21, 2009 1:54 AM in response to: jshao
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007

hi again,

We managed to kill the guests that esxtop reported as running, by killing the pid that was refering to the .vmx files. The names reported by esxtop was not refering to the process name pid, but a template name.

The lock on the filesystem is still there. We're not able to browse or list any folders. Any other tips for unlocking the filesystem? vmkfstools -L did not work see message below.

root@FS-ESXLAB-001 VM# vmkfstools -L release /vmfs/volumes/48ac7113-28843dfc-8333-001a64c56ede/VM
Command release failed
Error: Inappropriate ioctl for device
root@FS-ESXLAB-001 VM# vmkfstools -L release /vmfs/volumes/48ac7113-28843dfc-8333-001a64c56ede
Command release failed
Error: Inappropriate ioctl for device

-Pål-André

Re: error deploying vm

10. Apr 21, 2009 2:24 AM in response to: Jovi87
Click to view grahamwm's profile Enthusiast 48 posts since
Jul 17, 2008

Hi, check to see if your volume ID on your local storage matches the one in the text file you attached : 48ac7113-28843dfc-8333-001a64c56ede

Open a SSH session to your esx host and got to the volumes directory and make sure the volume number is the same. if you moved the VM from one local storage system to another or recreated any local storage volumes, the volume ID number in the VMX file will not be that same as the volume trying to be referenced and therefore will not allow you to access or star the VM.

Re: error deploying vm

11. Apr 21, 2009 12:45 PM in response to: grahamwm
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007

hi guys,

First; thanks for helping out here!

We have not moved any VM's, or recreated any local storage.

Here is what I posted earlier in this thread:

ls -la within the datastore path (via ssh/shell) in the /VM folder gives us the following error message:

--root@FS-ESXLAB-001 VM--# ls -al
ls: reading directory .: Device or resource busy

A normal ls -la output in this folder would list all folders "guests" in the labmanager style. So I still believe that a "rough lock" is on the file system, but need help in solving the lock issue.

Since we can't list any folders in the datastore, labmanager can't access the volume, and no guests is available.

-Pål-André

Re: error deploying vm

12. Apr 22, 2009 5:05 AM in response to: jshao
Click to view penghaug's profile Hot Shot 115 posts since
Feb 21, 2007

Hi John,

We're now able to browse the datastore (VM) folder, what released the lock I really can't tell. Anyways thank you for helping out.

Labmanager now runs the guests as normal.


-Pål-André

VMware Beta Programs

Want to be Considered for Future Beta Programs?

Learn More

VMware Developer

Download SDKs, APIs, videos,
training, and more in the Developer community.

Learn More

Developer
Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld
Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

Only VMware ... Delivers Nexus 1000V

Ensure consistent, policy-based network capabilities to virtual machines across your data center.

Learn More

Communities