VMware

This Question is Answered

1 "helpful" answer available (6 pts)
1 2 Previous Next 20 Replies Last post: Jul 30, 2009 5:35 AM by bobross  

FC LUNs and data dedupe (NetApp) posted: Jul 1, 2009 8:06 AM

Click to view shane.presley's profile Enthusiast 73 posts since
Oct 16, 2007

I'm curious about how we could take advantage of NetApp data dedupe, if we're using FC LUNs. I realize with NFS the issue is much simpler. But let's assume we don't have NFS for the moment...

If we present a 500GB LUN to ESX, and make that into a VMFS volume..the OS owns the volume. Then we put 400GB of data on that volume. On the back end, that should dedupe down to say 300GB of data. But wouldn't VMware still see 400GB of a 500GB volume used? Where does the space savings happen?

The LUN is still 500GB within the NetApp, so we're not seeing the savings there either (we can't hand out more LUNs thanks to dedupe can we?)

Basically how do we make use of the space savings of data dedupe, if using FC LUNs.

Re: FC LUNs and data dedupe (NetApp)

1. Jul 1, 2009 8:14 AM in response to: shane.presley
Click to view jayctd's profile Hot Shot 152 posts since
Jun 3, 2009

From my understanding as we research netapp the savings comes when you over provision the volume (not LUN) when you are using block level storage (ISCSI and FC) as you are right ESX does not see the space savings the only way to gain it back is to do it on the SAN side by creating more LUN's and "over provisioning"

We are looking at netapp right now and thats how I understood it in the non NFS world

Jered Rassier

##If you have found my post has answered your question or helpful please mark it as such##

Re: FC LUNs and data dedupe (NetApp)

2. Jul 1, 2009 8:53 AM in response to: shane.presley
Click to view bobross's profile Hot Shot 132 posts since
Nov 1, 2007

AFAIK, the NetApp filers can perform ASIS on LUNs (which are just files in WAFL, really). But - the space saved is given to the reserve pool, not the free pool. This means you can't take advantage of the dedup for normal operations. Much different that running ASIS on NFS shares, where the saved space is immediately available.

Reserve space is just that, reserve, and can't be used for new LUNs straightaway. When we tested NetApp at our site, they couldn't dedupe across aggregates either; this may have been fixed, or they may be waiting until ONTAP 8 is released, I don't know. It is rumored that ONTAP 8 will also do 'normal FC LUNs, and not LUN-in-file spoofing as they do in ONTAP 7. Maybe someone from NetApp can speak to that.

Re: FC LUNs and data dedupe (NetApp)

3. Jul 1, 2009 9:53 AM in response to: bobross
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
Dedupe is at the volume level and it's ONLY for NFS volumes, not LUNS. Since dedupe is a file system property it requires access to the block level, which Netapp cannot 'SEE' for a LUN.

Re: FC LUNs and data dedupe (NetApp)

4. Jul 1, 2009 9:57 AM in response to: shane.presley
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
I'm curious about how we could take advantage of NetApp data dedupe

Well Netapp should have explained this to you. You cannot use FC or iSCSI LUNs with Dedupe. DeDupe is for NFS / File shares. Basically you present a file system to ESX and it uses that, but you cannot use LUNS and utilize dedupe.

I realize with NFS the issue is much simpler. But let's assume we don't have NFS for the moment...

Not just simpler but the ONLY way you can implement Dedupe technology. Netapp cannot see inside a LUN, since its a very large chunk of space, what you do inside that space is assigned to a host, which is why you use zones for Fibre. So if you don't have NFS for the moment, you can't use dedupe until you use NFS.

Basically how do we make use of the space savings of data dedupe, if using FC LUNs.

You don't. Basic answer.

Re: FC LUNs and data dedupe (NetApp)

5. Jul 1, 2009 10:01 AM in response to: shane.presley
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
If we present a 500GB LUN to ESX, and make that into a VMFS volume..the OS owns the volume

However if you start using ESX 4.0 / vSphere you CAN use the built in dedupe from VM Ware on a FC LUN :)

You can also use thin provisioning...

Re: FC LUNs and data dedupe (NetApp)

6. Jul 1, 2009 10:03 AM in response to: RParker
Click to view jayctd's profile Hot Shot 152 posts since
Jun 3, 2009

Huh?

It will dedup iscsi and FC luns ... it is just block level deduplication in either case on its WAFL file system it does not have to see inside the lun to dedup

Jered Rassier

##If you have found my post has answered your question or helpful please mark it as such##

Re: FC LUNs and data dedupe (NetApp)

7. Jul 1, 2009 10:17 AM in response to: jayctd
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
t will dedup iscsi and FC luns ... it is just block level deduplication in either case on its WAFL file system

Sorry but you are wrong. The netapp documentation AND VM Ware will tell you the same thing. You can't dedupe a LARGE file that Netapp can't access, how are you going to dedupe what's inside that LUN? You can't so it won't.

read the documentation it CLEARLY states dedupe can ONLY be utilized for NFS, period. Call Netapp to confirm this if you want, but that's how it works.

Re: FC LUNs and data dedupe (NetApp)

9. Jul 1, 2009 10:26 AM in response to: shane.presley
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
Will VMware see shrinking data usage within a datastore?

Well the problem is it does per VMFS datastore. So you can't globally dedupe. So if you have similar VM's on datastore 1 and datastore 2, for instance, it will only dedupe the VM's on 1 and 2 would be done separately.

You will get WAY more space savings hosting ALL the VM's on the netapp using NFS, because it can take 1 VM essentially and dedupe across ALL clones of that same VM. That's why NFS is better, but if you want to continue using FC LUN's, you have some alternative, albeit not as manageable as NFS.

We have the same problem. We had Netapp come out 3 times this year already, and we have been over this for hours.. so that's how I know this. It's been discussed over and over. NFS is the only way to go for Netapp dedupe. We have FC for the being just like you and we have don't have infrastructure (gig switch,NIC's, ports...etc..) to support switching to NFS yet...

Re: FC LUNs and data dedupe (NetApp)

10. Jul 1, 2009 11:32 AM in response to: RParker
Click to view jayctd's profile Hot Shot 152 posts since
Jun 3, 2009


I talked to my engeneer to verify as we were researching the product and we are currently using Iscsi so this is of great intrest to us (Though NFS is also under heavy review)

Netapps answer was fairly simple

1) Dedupe does work on block level storage (FC and iscsi) with block level deduplication content or netapp "Knowing" the content is irellivent as it is just block on disk and any 2 blocks reguardless of content that are identical get de-dupped (Not supporising as other backup technologies do the same thing with encrypted files)

2) To gain back the space from dedup you must uncheck the "space reserved" otherwise the lun reserves the space reguardless of it is consumed (and it wont be because of dedup), you then gain back the space by making more LUN's on that flexvol

3) The dedup happens on the flexvol level reguardless of if the LUN's are iscsi, NFS, FC or any mix of the 3

4) The hosts will never see the savings of block level storage so reclaiming the space needs to be done via #3

I have asked them to send the white paper my way but acording to their engeneers there is no major hurdle.

The ones that I have found out there

http://blog.scottlowe.org/2008/04/24/using-netapp-deduplication-with-block-storage/

http://blog.scottlowe.org/2008/05/20/provisioning-luns-for-use-with-deduplication/

http://communities.netapp.com/docs/DOC-1192;jsessionid=7177122E64AA32F30DD92CEE995AC70E

It appears to work just fine assuming you do the correct configuration ... all the documentation points that way.

Either way we are still looking twards NFS as the host is able to SEE that space reclimation (That and as we have an ISCSI network using software initiators it is not going to cost us more to do)

Jered Rassier

##If you have found my post has answered your question or helpful please mark it as such##

Re: FC LUNs and data dedupe (NetApp)

11. Jul 1, 2009 12:14 PM in response to: jayctd
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
Netapp Sales will tell a different story, they keep pushing NFS route and FC/iSCSI LUN's isn't even an option, for whatever reason.

Maybe because the ESX hosts cannot see the space savings, but in order to use Dedupe requires NFS.

The problem is that Netapp Engineers didn't tell you, or maybe you didn't tell them what was ON that LUN is VMFS. That is a proprietary OS File System. Netapp CAN NOT see into the VMFS structure, so whatever is inside that VMFS file system is NOT visible to Netapp.

Once more, I will state, that you CAN'T dedupe what you CAN'T see. So yes a LUN with files in it CAN be deduplicated, but VMFS CAN NOT!

NFS is the ONLY way to take advantage of DeDupe and Netapp for VM Ware ESX. That's the whole story.

I assumed (apparently incorrectly) that since this is a VM Ware forum, that in the context of DeDuplication and Netapp meant as it operates WITH VM Ware / ESX servers. You can't use DeDupe.

A LUN CAN be deduplicated, if it's NOT containing a VMFS datastore.

Re: FC LUNs and data dedupe (NetApp)

12. Jul 1, 2009 12:10 PM in response to: jayctd
Click to view jayctd's profile Hot Shot 152 posts since
Jun 3, 2009

I found the white paper

http://media.netapp.com/documents/tr-3505.pdf

Its section 4.15 "Deduplication and LUNS"

Jered Rassier

##If you have found my post has answered your question or helpful please mark it as such##

Re: FC LUNs and data dedupe (NetApp)

13. Jul 1, 2009 12:12 PM in response to: RParker
Click to view jayctd's profile Hot Shot 152 posts since
Jun 3, 2009

It can see the blocks ... thats why its block level deduplicationrather than file level deduplication


Their white papers and engeneers and my test envyronment all agree

Jered Rassier

##If you have found my post has answered your question or helpful please mark it as such##

Re: FC LUNs and data dedupe (NetApp)

14. Jul 1, 2009 12:15 PM in response to: jayctd
Click to view RParker's profile Champion 5,282 posts since
Dec 6, 2006
Its section 4.15 "Deduplication and LUNS"

Yes, native data, like RAW storage not for VMFS / ESX.

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities