VM Keeps eating up space on LUN. HBA out of space errors.

[Phil_White]

For a month in a row now, each Monday I come in to work with one of our VM's throwing an error about running out of space on the LUN where the files are kept. Each week my resolution as been to migrate another VM off of that same LUN to another one to free up space.

I finally took a better look and noticed that ESX's properties for this VM we're having issues shows it as having a 400GB secondary drive on it. When I go to the data store browser, the VMDK file is only 220GB. I then realize that last Monday, that file was only 190GB. I logged on to the VM and noticed that the free space on that hard drive is 220GB.

400GB - 190GB = 220GB

What's happening is that drive is filling up and adding to the 220GB VMDK I see on the LUN. I didn't know ESX 3.5 allows you to dynamically grow a disk?

Here is what else is weird to me. I swear this 400GB drive was a RDM using virtual compatibility mode a while back, is it possible something happened with that RDM and the drive reverted to the redo logs and is now running from the redo VMDK?

[AntonVZhbankov]

Check if your VM has snapshots. Snapshot can grow up to the original VMDK size.

---

VMware vExpert '2009

http://blog.vadmin.ru

[Phil_White]

There are no snapshots, I should have clarified this in my initial post. The VMDK file is dynamically growing as more space fills the virtual drive. Right now the VMDK is 220GB, if I had 5GB of data, it will grow to 225GB of data.

[AntonVZhbankov]

I suppose you migrated VM and RDM became VMDK. Thin disk provisioning was available in 3.5 already, but you had to create thin disks manually with vmkfstools command.

---

VMware vExpert '2009

http://blog.vadmin.ru

[Phil_White]

Yes I am sure we have migrated the VM. I am having a hard time finding documentation that illustrates what happens when the RDM fails or a migration occurs. Both of our ESX servers can see the RDM so what would cause the RDM to turn in to a VMDK? Thanks for your help Anton.

[Texiwill]

Hello,

Virtual Disks are not FIFOs they have well defined sizes. Is this perchance a VM that was thin provisioned? When you look at the Settings for the VM and the virtual disk, what does it say?

There should be a maximum disk size specified for the virtual disk.

Also check the settings is it really a VMDK or an RDM,. it will also say this. Perhaps provide a screen shot.

If you restored the VM from backup, your restored 'RDM' would actually be a VMDK and not a RDM.

Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

[Phil_White]

The VM settings say it is a 400GB VMDK, not an RDM. I spoke with my supervisor and he agreed that that drive WAS an RDM. Something has happened to cause that RDM to convert to a VMDK assumingly via the redo logs used when virtualization compatibility is used. I am going to call VMWare when I have some time to find out what the deal is with this.

You are correct, ESX VMDK files are not FIFO, but this VMDK file is GROWING every time more data is added to the virtual drive. This morning the VMDK was 220gb, now it's gained an additional 500mb... This is truly bizarre.

Here are screen shots. Please note that COLPRINT, WINPAK, and DC01 are empty folders

Here you can see the VMDK size is 220GB...

Now you see where the VM settings say it's 390GB....odd huh ?

[Texiwill]

Hello,

Looks like you have a Thin Provisioned virtual disk file. The maximum size is 410GBs. So it can grow to that amount. A thin provisioned disk can only be created with vmkfstools, a script using vmkfstools, or using VMware View management tools.

Nothing bizarre here, this is the nature of Thin Provisioned disks.

How to fix: use vmkfstools to convert from thin provisioned disk to a zeroed thick disk. But only when the VM is off and you have made a backup.

Best regards,

Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, Virtualization Practice Analyst[/url]
Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

[Phil_White]

Thanks for the reply Texiwill, however this disk was not thin provisioned. I still need to call VMWare support to see how RDM's and their redo logs function if the RDM has problems. There is no room on the LUN that this VMDK file is on becuase it was actually an RDM and the RDM pointer was on this LUN....gonna have to migrate the files to a different LUN.

[Phil_White]

I hvae figured out what happened.

On June 13th we did a major system change and snapshotted VM's before making the change. My co-worker failed to remove the snapshot prior to doing a vmotion which apparently caused this bizarre issue. We just found the old VMDK(guess this wasn't an RDM? We're still confused on why not).

[anoni22]

This much I know... If you migrate to a different host all is well, if you Migrate to a different host AND a different LUN before you remove the raw lun, the raw lun will be cloned into a true vmdk file and no longer be a pointer to the raw lun. ( SAN Space permitting)

Best Practice.. If you want to keep your raw lun and move your vm (os) to another lun, remove your raw lun before you migrate, then add the raw lun to the vm when the migration is over.

[DonalB]

Probably a bit late but from the screenshot you provided above it looks like you had snapshots applied to all disks. The COLSQL.vmdk disk and the COLSQL-000001.vmdk are parent and snapshot I'd guess. If you look at the timestamp for the modification of those disks then you'll that the COLSQL-00001.vmdk is up to date whereas the COLSQL.vmdk hasn't been modified in over a month. So I'd hazard a guess that you've had those snapshots applied for nearly a month.

When a snapshot is applied to a VM the VM config is updated to 'point' the vm at the snapshot disk which is why you see the -00001.vmdk disks as the currently configured disks.Once you commit the snapshots the vm config is updated to point it back at the parent vmdk.

If you checked snapshot manager and it doesn't report any snapshots then this may be caused by the snapshot commit process being run but timing out in VC and the snapshot manager file being updated to show no snapshots when in fact the snapshots are still there. Have been bitten by this a few times and it's a bit annoying. If this has happened you can add another snapshot and see if it restores the view of the existing snapshot in the Snapshot Manager view, if it does I'd recommend committing the snapshots individually at the command-line using vmware-cmd , if it doesn't you can still try the vmware-cmd option and hope that it commits the snapshot anyway, you'll need to check if the parent disk modified timestamp is updating while the vmware-cmd is running to tell if the snapshot is being committed.

HTH

DB

[Chuck8773]

That is a huge snapshot. With a snapshot that large, I would shut the vm down, make a copy of the files, in case something goes wrong, and then remove the snapshot. I would be nervous in your shoes.

Charles Killmer, VCP

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[Phil_White]

I would like to update my post for others who have viewed this thread.

Problem #1 - Employee created snapshot of a VM that is an SQL server. Employee then VMotioned the VM to another ESX server and during this the snapshop assumingly failed. The original VMDK was left abandonded and the VM began to use the snapshot file it had created as the new "D:" drive on the affected VM. This file eventually grew and consumed all of the disk space on the LUN. So what we have is a snapshop file and then the original VMDK of which the snapshot was not appending to the VMDK even though the snapshot WAS deleted.

Problem #2 - VMWare has a KB out there that states to fix an issue like this, that you create a new snapshot of that VM and then delete the snapshot immediately which should force the snapshot to append to the original VMDK that is "abandonded". THIS DOES NOT WORK DO NOT DO IT. I followed the KB and created a snapshot, and then deleted it. Next, my VM went in to a panic and shut itself down. Then, my VM "lost" the snapshot file it was now using as it's "D:" drive on the VM. I figured, maybe ESX is just appending the changes to the VMDK and I should be patient. I waited two hours and nothing had happened, our production SQL server has just been down for two hours. Additionally, I could NOT re-attach the old VMDK file or the snapshot file to the VM. I now have total data loss of that "D:" drive for my VM which held all of my SQL databases.

Resolution to problem #2 - I created a new VMDK file and restored my files from backup. Luckily before making that snapshot to "fix" this issue, I made a complete backup of all of our SQL databases.

Moral of the story, do NOT VMotion a VM if it has a snapshot, heck you even get a warning about it prior to doing so