VMware

This Question is Answered

1 "correct" answer available (10 pts)
4 Replies Last post: Jan 12, 2009 7:48 AM by alanredo  

VMFS is full, cannot remove vmdk files. ESX 3.5 posted: Jan 7, 2009 6:01 AM

Click to view alanredo's profile Novice 3 posts since
Jan 7, 2009

Hi,

Like the topic says my vmfs ran full a couple of weeks ago and i've been trying some stuff on and off again to remove some guests so that i can get some free space again.

When i access the machine from VI the "Free space" is empty and if i try to boot a guest i get an error stating "Could not power on VM: No swap file" but there is infact a swapfile present, just no space left on the vmfs probably making it impossible to do any writing to it. So i right clicked on a guest and choose "delete from disk" and the machine goes away but still no available diskspace. I then proceed to browsing the storage using VI to try and manually remove the files with just a plain simple error stating that it cannot delete the file specified. So lastly i ssh to the server and go into /vmfs/volumes/491ae4a0-a3e876b6-0cdf-001aa028e698/VeeamBackup and i try to delete the guest using rm and the output i get can be seen below.

root@vmware2 VeeamBackup# rm -rf linux-www\(vm-47\)/
rm: cannot remove `linux-www(vm-47)//replica.vbk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmx': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmsd': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-7.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-8.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-3.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-4.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-5.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-6.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware.log': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www-flat.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www_1-flat.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www_1.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmxf': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.nvram': Input/output error
rm: cannot remove `linux-www(vm-47)//2008-11-22T172955.vrb': Input/output error

What could cause this behavior? is it really so that when the vmfs gets tanked up and there's no space left it just goes bananas or is this something more serious? The machine is a DELL PowerEdge 2950 with 5x136gb SAS drives in RAID5 and it is reported to be healthy.

Click to view Texiwill's profile Guru 10,213 posts since
Jan 13, 2004
Hello,

A few things, you really do not want to overfill the VMFS. 80% is the max I fill a VMFS. THat way I have room to work.

Go into /var/log/vmkernel and report on the SCSI errors you see. Is this a SAN, iSCSI Server, or NFS data store that is full?

Start to shutdown VMs.

I would try to delete other files than full VMs. Files like vmware.log.* would work. That may free up quite a bit of space. Enough so you can safely power down VMs.

In this state, the last time I got into it I had to shutdown my hosts connected to the VMFS, then boot only one host and clean things up from there. There is not much you can do here at the moment.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll
Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links

Click to view Texiwill's profile Guru 10,213 posts since
Jan 13, 2004
Hello,

Since you can reinstall, that would be my suggestion as well. If you did want to save data you may be able to copy some data but that is about all you can do.

I would reinstall and resize the VMFS so that it gets no more than 80-90% full.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll
Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities