VMware Cloud Community
s1xth
VMware Employee
VMware Employee

Equallogic - Reclaim Unused Disk Space on a Volume with vSphere??

I was hoping maybe someone here the forums with Eql experience explain to me why I am having this issue. Let's say for example, I create a 200GB vol and move a VM that is 10GB to the volume. If I move or delete that VM from the volume the volume STILL reports used space of 10GB. If I move that VM back or create a new VM that is 10GB the Eql shows 20gb used. What's the deal? Anyway to get around this? What do others do?

I did search and found this was an issues under 3.5 jw if it's still a problem 4.0.

Thanks!!

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
12 Replies
binoche
VMware Employee
VMware Employee

could you please check such as any .vmdk left on EQL datastore after vm moved? thanks

binoche, VMware VCP, Cisco CCNA

s1xth
VMware Employee
VMware Employee

I double checked...the volume is empty in vsphere client. Nothing there.

Sent from my iPhone

On Feb 10, 2010, at 9:56 PM, binoche <communities-emailer@vmware.com

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
binoche
VMware Employee
VMware Employee

vmkfstools -P -h -V 10 /vmfs/volumes/ will show how much disk space missing

binoche, VMware VCP, Cisco CCNA

Reply
0 Kudos
IRIX201110141
Champion
Champion

Using Equallogic now for about 2 years and never think about that. It looks for me that you only see the "highest watermark" for disk usage as information. I'am not sure if the SAN can detect you much of used data was in the VMFS.

Regards

Joerg

'Remember if you found this or others answers helpful do not forget to award points by marking an answer as helpful or correct'

Reply
0 Kudos
mellerbeck
Enthusiast
Enthusiast

I am having this EXACT same problem! Just deleted a virtual machine from a volume and then went to clone another virtual machine to the volume and it still thinks the space is being used. I even removed the volume and formatted it again with vsphere and yet equallogic thinks the space is being used. hmmm I have a case open with equallogic. #103174

At least I'm not the only crazy one out there. So what firmware are you running?

-Michael

Reply
0 Kudos
pfuller
Contributor
Contributor

I have been use EqualLogic for awhile now; it can only tell that a block have been used and once it is used EqualLogic can not reclaim it. Another down side to thin-provisioning is that if you over provision your EqualLogic you could knock your thin-provisioned volumes offline.

Source: http://www.equallogic.com/resourcecenter/assetview.aspx?id=5245

A thin-provisioned volume grows automatically due to application data writes. If later the application frees up space, the space is free in the file system but is not returned to the free space in the PS Series pool. The only way to reduce the physical allocation is to create a new volume, copy the application data from the old volume to the new, and then delete the old volume.

o Example: A file share is thin-provisioned with 1 TB logical size. Data is placed into the volume so that the physical allocation grows to 500 GB. Files are deleted from the file system, reducing the reported file system in use to 100 GB. The remaining 400 GB of physical storage remains allocated to this volume in the SAN.

o This issue can also occur with maintenance operations including defragmentation, database re-organization, and other application operations.

Reply
0 Kudos
mellerbeck
Enthusiast
Enthusiast

Yup, exactly the response I got from EqualLogic. Very counter intuitive though!

Unfortunately, although this seems like some sort of problem, it is not. You would expect that after a delete of data from the SAN, that the space would then show as free. And you are rightfully surprised when it doesn’t.

The answer lies in the SCSI protocol. As you know, our SAN is just a really big SCSI drive, for all that the server knows. It doesn’t know that the SAN is off on the network, and that the iSCSI initiator intercepts commands to/from the OS and redirects them over to us.

Unfortunately, the is no ‘delete’ command in the SCSI protocol. The server changes the File Table on one of the blocks of storage we’ve set aside for it the blocks it no longer needs, effectively deleting the data. But it never actually sends us a command ‘delete block 346135.’

Since we’re block storage, we are agnostic to the data stored on our blocks of disk space. The volume could be NTFS, or VMDK, we don’t know. So we dutifully mark any used blocks as dirty, and that’s how they stay, until the volume is deleted, or the server again writes over that block.

The server has the file table. So it knows where the blocks are, and which are used or not. When it reports that only 25GB of the 100GB volume are in use, you can believe it. When we say that 75GB of the 100GB are dirty, then you can believe us that at some point in time, data was written to all 75GB of space.

If you need to recover some of that space, then you will need to move the data off, delete the volume, and create a new, smaller volume. It’s the only way to recover dirty, yet un-used, space.

s1xth
VMware Employee
VMware Employee

Just want to clarify...this issue would still exist even if you arent using Thin Provisioned Volumes??

http://www.virtualizationimpact.com http://www.handsonvirtualization.com Twitter: @jfranconi
Reply
0 Kudos
poweredge2010
Contributor
Contributor

Update:

I did another test that proved I was wrong.

The GB are reported in EQL Group Manager under Volume Used Size

Thick (20GB) Thin (20GB)

-


1. +5GB 5GB 5GB

2. +5GB 10GB 10GB

3. -5GB 10GB 10GB

4. +5GB 10GB 10GB

5. -10GB 10GB 10GB

6. +15GB 15GB 15GB (Warning as over the default 60%)

7. -5GB 15GB 15GB

8. +5GB 15GB 15GB

and the good news is from EQL's reply I got just 5 mins ago.

In response to “reclaiming unallocated array disk space” on the PS Series arrays:

An enhancement request for this feature (reclaim space that was previously used) has already been submitted. Firmware version 5.0.2 does not introduce this feature. Engineering has not updated support as to when such a feature will be available in future firmware releases.

So I think we are safe to use Thin Prov. (TP) in VMFS as well as the problem shouldn't happen. (or will you? Can you double check by running your test again pls?)

Jack

Hi Jonathan,

I don't think this problem exist in normal provisioning (ie, Thick) on EQL, I did a simple test already

Create a 10GB volume, attach to Windows, write 5GB, then remove 4GB, leave with 1GB, to EQL it's 5GB used.

Then I write another 4GB, EQL still reports 5GB, then I write 1GB more, now EQL reports 6GB.

However in my Thin provisioning test for the above 10GB case.

Create a 10GB volume, attach to Windows, write 5GB, then remove 4GB, leave with 1GB, to EQL it's 5GB used.

Then I write another 4GB, EQL somehow reports continously growing to 5GB, then 6GB, then finally bang 9GB. (actually inside, it's still 5GB, you will see later)

HOWEVER, Please note THIS, as I continue to add another 4GB to the volume (now EQL reports 9GB, windows reports 5GB), then EQL reached 10GB max (somehow the volume didn't go offline? why? I don't know), but I can still add this 4GB to the volume, and windows reports 9GB/10GB used.

So in a strange way, even EQL reports the volume has been fully used, we can still add data to it (the thin prov. volume), but it's just TOTALLY CONFUSING.

That's why WAIT UNTIL FW5.0.x or FW5.x coming out with THIN RECLAMATION feature like what HDS's or 3PAR's did. We are probably better to NOT USE Thin Prov. in ESX, what I mean is

Use Thick Prov. in EQL, but Thin in ESX VMFS would be the best way, do you agree?

Jack

PS. Btw, I really enjoy reading your blog regarding EQL User Conf everyday, it's so detailed and very informative, thanks! I wish they have one in Hong Kong or near by city in Asia. Smiley Happy

Reply
0 Kudos
poweredge2010
Contributor
Contributor

So the reply to you question is if you create a 200GB volume (doesn't matter Thick or Thin), then move over 10GB, then remove 10GB, then add back 10GB, now what does EQL group manager shows? It should be 10GB.

Reply
0 Kudos
poweredge2010
Contributor
Contributor

Btw, what's your EQL FW? Is it FW5.0.2?

I tested again both NTFS (Thick only) and VMFS (Thick and Thin)) with FW5.0.2 and ESX4.1, using both Thick and Thin on EQL volumes.

The results are consistant, the new data inside file system will re-used whatever block is avaiable.

So your result should be 10GB instead of 20GB in all my tests.

Reply
0 Kudos
poweredge2010
Contributor
Contributor

Some update from Dell Pro-Support regarding NTFS/VMFS can REUSE the touched blocks somehow.

=======================

A similar problem is when the initiator OS reports significantly more space in use than the array does. This can be pronounced in systems like VMWare that create large, sparse files. In VMWare, if you create yourself a 10GB disk for a VM as a VMDK file, VMWare does not write 10GB of zeros to the file. It creates an empty (sparse) 10GB file, and subtracts 10GB from free space. The act of creating the empty file only touches a few MB of actual sectors on the disk. So VMWare says 10GB missing, but the array says, perhaps, only 2MB written to.

Since the minimum volume reserve for any volume is 10%, the filesystem has a long way to go before the MB-scale writes catch up with the minimum reservation of a volume. For instance, a customer with a 100GB volume might create 5 VMs with 10GB disks. That's 50GB used according to VMWare, but only perhaps 5 x 2MB (10MB) written to the array. Until the customer starts filling the VMDK files with actual data, the array won't know anything is there. If has no idea what VMFS is; it only knows what's been written to the volume.

• Example: A file share is thin-provisioned with 1 TB logical size. Data is placed into the volume so that the physical allocation grows to 500 GB. Files are deleted from the file system, reducing the reported file system in use to 100 GB. The remaining 400 GB of physical storage remains allocated to this volume in the SAN.

� This issue can also occur with maintenance operatiions including defragmentation, database re-organization, and other application operations.

In most environments, file systems do not dramatically reduce in size, so this issue occurs infrequently. Also some file systems will not make efficient re-use of previously allocated space, and may not reuse deleted space until it runs out of unused space (this is not an issue for NTFS, VMFS).

=======================

Reply
0 Kudos