Datastore Capacity When Thin Provisioned

[keegster]

Hi all,

I have VSphere 4.1 running in a cluster of two host, this cluster uses a Dell Equalogic PS4000 SAN for storage.

The confusion I have is with thin provisioned data stores and my Exchange (2010) server - running on MS Server 2008 R2 Ent.

The exchange server uses three datastores on the SAN, one for the OS, one for the databases and one for the Logs.

The databases datastore was the one giving me problems, it was set as thin provisioned  750GB, when I look on the SAN & through VCenter  it was saying 90% full until it rose to 100% after the SAN emailed me several alerts - the VM then crashed with out of space error.

To cut a long story short I have twol questions

1) In this scenario why does the SAN & VMWare say the datastore is full BUT when I log into the Windows VM it was approx 50% in use?

2) When the datastore containing the databases filled up I can understand why Exchange fell over but why did the VM crash?  The OS datastore still had plenty of space free.

I recovered from this error by increasing the space of the datastore by 100GB to 850GB then the VM came up fine (although Windows was complaining about an unexpected sudden shutdown).

Thanks in advance for your help.

K

[TomHowarth]

Has anybody run a Defrag against the Exchange server?  as Defrag touches and moves every block in a file system, the SAN would see this as a new block and therefore assign space.

[keegster]

Hi Tom, thanks for your reply,

No defrag or similar has been peformed on the Exchange server.

[keegster]

The sudden increase of space (overnight) would have been the backups caching locally before being sent offsite (I think) but I still don't know why it would bring the VM down...

[vGuy]

in vSphere 4.x, if you have deleted/moved large amount of data from the VM running on TP Datastore then it will not be automatically reclaimed by VMware. The fix to the reclamation process is a two phase approach

First is to zero out the unused sectors/blocks using tool such as sdelete (with -c option).

Followed by SVMotion’ng the VM onto another Datastore.

Also, please take note that running sdelete will cause the disk to be ballooned as sdelete will try to allocate large files before the zeroing out process, however this is a temporary situation, once SVMotion is completed all the unused space would have reclaimed.

[a_p_]

The sudden increase of space (overnight) ...

What type of backup do you use. An in-guest agent based backup or an image based backup for ESXi? In case of an image based backup, take a look at the VM's folder on the datastore to see whether there are active snaphots (<vmname>-00000x.vmdk files)! Maybe there were issues deleting these snapshots after the backup, which will cause an increase of the snapshot file's sizes and therefore filling up the datastore.

André

[keegster]

Thanks guys,

A large amount of data being deleted/moved could well be the issue, the backup software I use is Attix 5 Pro so it caches data locally and then streams it offsite.  As this datastore is thin provisioned this seems to fit the symptoms.

Can I change the data store to by thick provisioned?  On the Equalogic it is just a tick box but I don't want to do any damage or risk corruption on a live system.  Looking at VM Ware best practice for Exchange the DS should be thick anyway for better peformance.

I have checked the DS and can see no snapshots - just a Virtual disk of around 786GB.

I'll have to plan a failover over to a secondary server before running any tools such as sdelete - but changing the DS to thick provisioned may fix this anyway?....

Cheers

K

[vGuy]

converting a thin to thick virtual disk is quite straight forward...this blog article seems to have nice step-by step process with screenshots:

http://www.robvanhamersveld.nl/2010/09/15/vmware-convert-vmdk-from-thin-to-thick/

..hth!

[keegster]

Thanks, that's very helpful.  I will work through that blog.

From the SANs perspective as it was thin provisioned and hit 750GB, then 300GB was deleted it would stay at 750GB as it cannot reduce in size - is this correct?

(This would answer why the SAN is saying it is using 750GB of 750GB but Windows thinks there is 300GB ish free).

Cheers

K

[vGuy · Accepted solution]

From the SANs perspective as it was thin provisioned and hit 750GB, then 300GB was deleted it would stay at 750GB as it cannot reduce in size - is this correct?

(This would answer why the SAN is saying it is using 750GB of 750GB but Windows thinks there is 300GB ish free).

yes, that's correct....the unused/dead/deleted space reclamation is not done automatically.

[markpiontek]

Also, just a note, I ws recently reading about virtualizing Exchange 2010 and caveats, etc.  One of the suggestiosn was not to use thin privisioned disks for it.

Might want to just Google Exchange 2010 caveats for kicks.  You may not feel it applies to your environment.  I'm undecided for mine, but it's small enough where I'm leaning towards it wont matter much if it's thin provisioned.