VMware Communities > VMTN > VMware Infrastructure™ > VI: ESX 3.5 > Discussions

This Question is Answered

1 "correct" answer available (10 pts)
4 Replies Last post: Jan 12, 2009 7:48 AM by alanredo
Reply

VMFS is full, cannot remove vmdk files. ESX 3.5

Jan 7, 2009 6:01 AM

Click to view alanredo's profile Novice alanredo 3 posts since
Jan 7, 2009

Hi,

Like the topic says my vmfs ran full a couple of weeks ago and i've been trying some stuff on and off again to remove some guests so that i can get some free space again.

When i access the machine from VI the "Free space" is empty and if i try to boot a guest i get an error stating "Could not power on VM: No swap file" but there is infact a swapfile present, just no space left on the vmfs probably making it impossible to do any writing to it. So i right clicked on a guest and choose "delete from disk" and the machine goes away but still no available diskspace. I then proceed to browsing the storage using VI to try and manually remove the files with just a plain simple error stating that it cannot delete the file specified. So lastly i ssh to the server and go into /vmfs/volumes/491ae4a0-a3e876b6-0cdf-001aa028e698/VeeamBackup and i try to delete the guest using rm and the output i get can be seen below.

root@vmware2 VeeamBackup# rm -rf linux-www\(vm-47\)/
rm: cannot remove `linux-www(vm-47)//replica.vbk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmx': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmsd': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-7.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-8.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-3.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-4.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-5.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware-6.log': Input/output error
rm: cannot remove `linux-www(vm-47)//vmware.log': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www-flat.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www_1-flat.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www_1.vmdk': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.vmxf': Input/output error
rm: cannot remove `linux-www(vm-47)//linux-www.nvram': Input/output error
rm: cannot remove `linux-www(vm-47)//2008-11-22T172955.vrb': Input/output error

What could cause this behavior? is it really so that when the vmfs gets tanked up and there's no space left it just goes bananas or is this something more serious? The machine is a DELL PowerEdge 2950 with 5x136gb SAS drives in RAID5 and it is reported to be healthy.

Reply Re: VMFS is full, cannot remove vmdk files. ESX 3.5 Jan 7, 2009 7:49 AM
Click to view Texiwill's profile Guru Texiwill 10,056 posts since
Jan 13, 2004
Moderator
Hello,

A few things, you really do not want to overfill the VMFS. 80% is the max I fill a VMFS. THat way I have room to work.

Go into /var/log/vmkernel and report on the SCSI errors you see. Is this a SAN, iSCSI Server, or NFS data store that is full?

Start to shutdown VMs.

I would try to delete other files than full VMs. Files like vmware.log.* would work. That may free up quite a bit of space. Enough so you can safely power down VMs.

In this state, the last time I got into it I had to shutdown my hosts connected to the VMFS, then boot only one host and clean things up from there. There is not much you can do here at the moment.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll
Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links

Reply Re: VMFS is full, cannot remove vmdk files. ESX 3.5 Jan 7, 2009 11:20 PM
in response to: Texiwill
Click to view alanredo's profile Novice alanredo 3 posts since
Jan 7, 2009

Hi thanks for the quick reply.

This machine is basicly a replica-server, we have two identical servers and this server holds replicas made with Veeam from the main server. So removing them is not really that big of a deal, even a reinstall would not be to horrible but it would be nice if i didnt have to go through all that trouble just to clean up the storage. There is no SAN/NFS/iSCSI solution but just a RAID5 containing said 5x136gb SAS drives, so the system and VMFS/vmstorage is on the same set of disks.

As for the errors reported this is being repeated over and over again.

Jan 8 08:11:22 vmware2 vmkernel: 0:23:38:18.724 cpu5:1039)WARNING: J3: 1378: No space for journal on volume 491ae4a0-a3e876b6-0cdf-001aa028e698. Opening in read-only mode with limited write support.
Jan 8 08:11:22 vmware2 vmkernel: 0:23:38:18.724 cpu5:1039)WARNING: J3: 1381: Set 'OpenWithoutJournal' advanced option to 0 to attempt freeing up space for journal. If this fails, volume is not opened.

And i can't remove just the *.log files either because i get the same input/output error as shown in my first post.

root@vmware2 linux-www(vm-47)# rm -f *.log
rm: cannot remove `vmware-3.log': Input/output error
rm: cannot remove `vmware-4.log': Input/output error
rm: cannot remove `vmware-5.log': Input/output error
rm: cannot remove `vmware-6.log': Input/output error
rm: cannot remove `vmware-7.log': Input/output error
rm: cannot remove `vmware-8.log': Input/output error
rm: cannot remove `vmware.log': Input/output error

So what would be the best thing to do here? Being a long time linux user (+10 years) and having seen similar errors before my guess at the moment is that the only sure thing is to do a reinstall or at best some kind of repair where i could possibly format the vmfs using the installation media.

Reply Re: VMFS is full, cannot remove vmdk files. ESX 3.5 Jan 8, 2009 7:23 AM
in response to: alanredo
Click to view Texiwill's profile Guru Texiwill 10,056 posts since
Jan 13, 2004
Moderator
Hello,

Since you can reinstall, that would be my suggestion as well. If you did want to save data you may be able to copy some data but that is about all you can do.

I would reinstall and resize the VMFS so that it gets no more than 80-90% full.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
Blue Gears and SearchVMware Pro Blogs: http://www.astroarch.com/wiki/index.php/Blog_Roll
Top Virtualization Security Links: http://www.astroarch.com/wiki/index.php/Top_Virtualization_Security_Links
Reply Re: VMFS is full, cannot remove vmdk files. ESX 3.5 Jan 12, 2009 7:48 AM
in response to: Texiwill
Click to view alanredo's profile Novice alanredo 3 posts since
Jan 7, 2009

Hello,

Sorry for the late reply. I ended up reinstalling the machine, it was basicly the only way to get it useable again.

thanks for your help, i'll keep it in mind not to use all the space on the storage.

Actions