Unable to delete (vmdk) files from vmfs

[marc_testing]

Hi,

we're currently running ESX 3.5 (Update04) (STANDALONE installation) on Dell Power Edge 2950 connected over a LSI22320-R SCSI-controler to a Q12+ easyraid.

This hardware architecture is fully supported by Dell, VMware and easyraid.

I want to delete some virtual machines, but it's not possible.

I removed the machines from inventory and:

- try delete all vm files over datastore browser - error: "General fault caused by file".

- try delete via console (rm) - error: "invalid arguments"

- try delete via console (vmkfstools -U) - error: "Failed to delete virtual disk: Invalid argument (1441801)"

The files are not in use! I rebooted the ESX Server and the underlying storage - problem exists.

Is it possible that the vmfs is corrupt and how can I force file deletion?

Does anyone have knowledge/experience/solutions/hints for the problem?

Any input in this subject would be appreciated!

[AndreTheGiant]

- try delete via console (rm) - error: "invalid arguments"

Strange error. What is the complete command line that you execute?

Does the vmdk file has some strange characters?

Andrea

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[ShanVMLand]

Hi,

Can you please try to unregister the VM before attempt to delete?

vmware-cmd -s unregister vmfs/volume/<Storage Name>/<VMName>/<VMName.vmx>

Shan

[marc_testing]

Hi Andrea,

I executed rm -f <full path to vmdk file>

[marc_testing]

Hi Shan,

I unregistered the VM via VI-Client and this works, problems began on try deleting files.

[AndreTheGiant]

If you try do remove directy the entire VM from VIC what happen?

Registered again and then use the option to delete from disk.

Andrea

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[marc_testing]

nothing happens, option to delete from disk show now errors but files remain on storage

[AndreTheGiant]

Very strange. Sure that the file name does not contain strange chars?

Have you tried to do first a cd /path-of-you-vm, and then do a rm * ?

Andrea

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[marc_testing]

Hi Andrea,

yes, there are no "strange chars" in the filenames and I always exexuted rm -f *

[AndreTheGiant]

No idea. ?:|

The logical disk are all with size < 2TB?

Andrea

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[wila]

- try delete via console (rm) - error: "invalid arguments"

Eww.. i do assume that you had enough rights while executing this command (not that i would expect this error then, but...)

Is it possible that the vmfs is corrupt and how can I force file deletion?

Yes, it is possible, I've only seen your type of error on executing rm when there was a storage problem.

In that case there should be some errors saved in the logs somewhere, unless you rebooted after which ESX3i might not have stored those log entries (If this is full blown ESX, search /var/log/messages for a hint.

Does anyone have knowledge/experience/solutions/hints for the problem?

Create a new LUN and start migrating all your VMs over to the new LUN, destroy the old LUN after you are done.

As always... make sure your backups are current before doing something like this.

--

Wil

_____________________________________________________

Visit the VMware developers wiki at http://www.vi-toolkit.com

[kjb007]

Can you rename the folder? If you have another datastore, can you move the files?

-KjB

VMware vExpert

[marc_testing]

Hi KjB,

renaming the folder is possible but moving or copying it to another datastore raises the same errors

[kjb007]

Then, either your vmfs is corrupted somehow, or there is some process, live or defunct at this point, that has those files open, for one reason or another. Are you able to reboot the server? This should at least clear up any active/defunct processes that may be affecting those files.

-KjB

VMware vExpert

[marc_testing]

there are no processes on these files (tested with lsof) and I always rebooted the server (see my first post) - it doesn't help :_|

[kjb007]

lsof won't tell you everytime. Is this server part of a cluster? Was this vm a template?

-KjB

VMware vExpert

[marc_testing]

oh, that's new for me, does any tools exist for esx to retrieve lock informations?

It's a standalone server and the vm was no template

[kjb007]

lsof won't always give you open files if the process is dead. Since you've restarted, this shouldn't apply. But, there used to be an issue when you had templates, deployed from them,and tried to edit the hardware settings at the same time. Sometimes, the template disk is taken over by the vm you just deployed, or at least referenced by it. But if you restarted, then there shouldn't be any vm's running, so that shouldn't apply. You did have all vm's down, right? Can you try to move the files to a new location when all vm's are shutdown, or have you already tried that?

-KjB

VMware vExpert

[marc_testing]

yes I already tried that directly after reboot (no vms online)

[kjb007]

Then I would check the underlying storage as was mentioned earlier by another poster. Move files off, if you can, and recreate the vmfs.

Opening an SR with vmware would be a good alternative as well, just to validate there isn't some known issue we haven't covered.

-KjB

VMware vExpert