VMware Cloud Community
georgetog
Contributor
Contributor

Vsphere 5 File Server Design vmdk. 2TB limit

Hi everyone,

We are about to deploy a new file server that needs about 8TB one volume. There`s no more problems with datastore on ESX5 but there`s still an issue with vmdk size. I was thinking to have multiple 2 TB vmdks put together as a single volume under Windows dynamic-spanned.

Would there be any issues with VMware when doing storage vmotion later on ? or Performance issue ?

Thanks.

Reply
0 Kudos
20 Replies
weinstein5
Immortal
Immortal

Welcome to the Cmomunity --  There shouldn't be any issue with storage vMotion perfomrance is all going to depend on the how busy the server is -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
Reply
0 Kudos
a_p_
Leadership
Leadership

It may work spanning multiple virtual disks in the guest OS. However, if you have shared storage and the storage is able to present 8 TB LUNs, then you may consider using an 8 TB pass-through (physical mode) RDM.

André

Reply
0 Kudos
RParker
Immortal
Immortal

georgetog wrote:

Hi everyone,

We are about to deploy a new file server that needs about 8TB one volume. There`s no more problems with datastore on ESX5 but there`s still an issue with vmdk size. I was thinking to have multiple 2 TB vmdks put together as a single volume under Windows dynamic-spanned.

Would there be any issues with VMware when doing storage vmotion later on ? or Performance issue ?

Thanks.

What's wrong with RDM?  I wouldn't use soft RAID, a direct LUN or NFS would be much better, than trying to use multiple VMDK.

Reply
0 Kudos
malaysiavm
Expert
Expert

Is officially supported in esxi 5 with rdm on physical mode  Regards,  Craig

Craig vExpert 2009 & 2010 Netapp NCIE, NCDA 8.0.1 Malaysia VMware Communities - http://www.malaysiavm.com
Reply
0 Kudos
georgetog
Contributor
Contributor

This is a pysical server with fast seq read/write local storage just to avoid  ISCSI traffic. I will still make it a VM as I want to be able to back it up with Veeam V5 , V6 ...  so that`s why I think I need vmdk instead of RDM.  

I am still very scared to use 2 or 3 vmdks put together under Windows to spann them in one large volume. The reason is about snapshots, anything happends to one vmdk my whole volume is gone... In the end I might have to have 3 volumes each 2TB instead of one large one... which I hate it so bad...

VMware took care of datastore but no vmdk WHYYY ?

Reply
0 Kudos
a_p_
Leadership
Leadership

VMware took care of datastore but no vmdk WHYYY ?

I'm pretty sure VMware is working on this. However since it is not available at the moment and you need/want image backups (Veeam) your options are limited. Depending on the data structures you may consider to implement DFS to provides a single share to the users but have independent virtual disks on your server. This would at least avoid possible issues with spanned virtual disks.

André

Reply
0 Kudos
georgetog
Contributor
Contributor

thanks a lot guys. We have DFS as well but that limit is killing us. I wish I would know when they will fix that as I will probably wait it`s not an emegency to deploy this second local FS.... Don`t know what to do... I might just go with 2 2TB vmdks in one spanned volume and that`s that until VMware fixes that...

Reply
0 Kudos
RParker
Immortal
Immortal

georgetog wrote:

This is a pysical server with fast seq read/write local storage just to avoid  ISCSI traffic. I will still make it a VM as I want to be able to back it up with Veeam V5 , V6 ...  so that`s why I think I need vmdk instead of RDM.  

I am still very scared to use 2 or 3 vmdks put together under Windows to spann them in one large volume. The reason is about snapshots, anything happends to one vmdk my whole volume is gone... In the end I might have to have 3 volumes each 2TB instead of one large one... which I hate it so bad...

VMware took care of datastore but no vmdk WHYYY ?

Umm.. Veeam back's up the VM, it doesn't care if it's using VMDK or RDM.  I think the issue is with snapshots.. but call Veeam, I am pretty sure their product will backup RDM's with no problem

Reply
0 Kudos
RParker
Immortal
Immortal

georgetog wrote:

thanks a lot guys. We have DFS as well but that limit is killing us. I wish I would know when they will fix that as I will probably wait it`s not an emegency to deploy this second local FS.... Don`t know what to do... I might just go with 2 2TB vmdks in one spanned volume and that`s that until VMware fixes that...

There is no limit killing you.. YOUR implementation and HOW you want to make File Servers work is killing you.  So in essence you are shooting yourself in the foot.

Use RDM, the fact that VMDK's are limited to 2TB is irrelevant.  That's why there is more than one way to accomplish something.  Not to mention you are using iSCSI, that means you have the LAN infrastructure so make an NFS store there is no file limits to using that.  NFS is better for file servers anyway, because a file server is a file level server, not block level.....

Reply
0 Kudos
georgetog
Contributor
Contributor

RParker buddy we are not using ISCSI, nor NFS.  We have folders that might exceed 2TB limit so that`s why we need larger volume. RDM will not be backuped by Veeam cuz veeam works with snapshots and VMDKs.

Reply
0 Kudos
Baddos
Enthusiast
Enthusiast

I would recommend against using local storage for your file server and move to a SAN or NAS environment. You'll be missing out in host redundancy, centralized storage w/ backup controllers, load balancing and possible DR replication.

If you must stay with local storage, then go ahead and use multiple vmdks with a software raid. I don't think the feer of losing 1 vmdk from a set would be any different from losing 1 giant vmdk. In fact, if you did lose 1 vmdk the recovery window would be shorter than restoring that 1 giant vmdk.

Reply
0 Kudos
georgetog
Contributor
Contributor

Oh guys... u don`t get it. We have 2 ISCSI SANs that`s not the issue. We have a file server on a SAN.... but nothing can beat DAS performance on the second file server... in our enviroment..

Reply
0 Kudos
RParker
Immortal
Immortal

Baddos wrote:

I would recommend against using local storage for your file server and move to a SAN or NAS environment. You'll be missing out in host redundancy, centralized storage w/ backup controllers, load balancing and possible DR replication.

If you must stay with local storage, then go ahead and use multiple vmdks with a software raid. I don't think the feer of losing 1 vmdk from a set would be any different from losing 1 giant vmdk. In fact, if you did lose 1 vmdk the recovery window would be shorter than restoring that 1 giant vmdk.

The loss of a VMDK has nothing to do with fear of losing 1 VMDK, it has to do with performance and stability.  A Software RAID is just not a good idea on any OS.  I guess if you MUST have a RAID then that's one thing, but a File Server should need a RAID first of all.  A typical file server has multiple shares, so split the shares / files across the different VMDK.

If you lose a VMDK you can't restore / use a software RAID until you get back ALL the VMDK's either.. so your theory is flawed.  1 BIG VMDK vs multiples running across smaller VMDK (RAID) amounts to the same thing.  So individual VMDK is better than trying to RAID them.

Reply
0 Kudos
RParker
Immortal
Immortal

georgetog wrote:

Oh guys... u don`t get it. We have 2 ISCSI SANs that`s not the issue. We have a file server on a SAN.... but nothing can beat DAS performance on the second file server... in our enviroment..

Something is wrong with your SAN then, because iSCSI SAN is every bit as capable as any DAS.  The CONTROLLER / STORAGE ARRAY determines IOPS / SPEED of disk *NOT* the topology used to connect anyway... A File Server isn't about speed in any case.

And that's why I said you obviously have a LAN capable of high speed if you are using iSCSI.  My point was just setup your SAN to use NFS for your File Server, 1 BIG VMDK, no software RAID, and no limitation on file size......  AND you can backup with Veeam

Reply
0 Kudos
georgetog
Contributor
Contributor

Oh I see what you mean. NFS vmdk is not limited to 2TB... yeah that makes sense but no we don`t want our second file server to be located on any SANs as we don`t have much performance out of them..they are used for other things... I prefer to see a server there in case my SAN blows out I still have all the data on a local server. We achive very high seq read /write perf over 700MB/s with cache. From our SAN currently under load we don`t get more than 50MB/s read and 130MB/write. But with this new server will be able to push with quad nics about 400MB/s both read and write seqencial.

That`s the reason of our second file server as our first server is slow cuz of the slow SAN ..and other loads...

But yeah in the end I`ll just have to decide between

1. 2 volumes each 2TB....

2. Windows dynamic disk spann 2 vmdks of 2TB which will give me 4TB good enought for couple of years from now.

The only reason I would go with 2 is much faster to fix my file-1 which almost is running out of space on 2TB volume so would be easy to just add another vmdk and expand it...

Reply
0 Kudos
RParker
Immortal
Immortal

georgetog wrote:

Oh I see what you mean. NFS vmdk is not limited to 2TB... yeah that makes sense but no we don`t want our second file server to be located on any SANs as we don`t have much performance out of them..they are used for other things... I prefer to see a server there in case my SAN blows out I still have all the data on a local server. We achive very high seq read /write perf over 700MB/s with cache. From our SAN currently under load we don`t get more than 50MB/s read and 130MB/write. But with this new server will be able to push with quad nics about 400MB/s both read and write seqencial.

That`s the reason of our second file server as our first server is slow cuz of the slow SAN ..and other loads...

But yeah in the end I`ll just have to decide between

1. 2 volumes each 2TB....

2. Windows dynamic disk spann 2 vmdks of 2TB which will give me 4TB good enought for couple of years from now.

The only reason I would go with 2 is much faster to fix my file-1 which almost is running out of space on 2TB volume so would be easy to just add another vmdk and expand it...

OK, so we are agree with the size limitation.  We are agreed on your configuration.

What I don't agree with is your assessment of the performance of the SAN.  A SAN is a conglomerate of disks.  right?  So you can configure disks to be separate from OTHER disks.  If nothing else is touching those disks nothing else can affect performance of those disks, right?

So.. A File Server is not performane driven.  Performance issues asside, your SAN should be fine for file server.  I agree that you want to use local disks and not waste "high performance" SAN drives for File Server, I have the same reservations..

All I am saying is that for your purposes, you shouldn't be concerned with how bad or what kind of configuration you have on the SAN, it may not compare with OTHER high performance solutions (like local disk) on the SAN for database use.  A file serer is just a bunch of files that people use to install, make copies, backup, or share between departments.. that's fine.

Users will not be affected by the performance or lack thereof, if you have the space, use it.  In this case, don't let the SAN affect your judgement because you think it won't be adequate for responsiveness, it will.. for a File Server. That's all I am saying.

Reply
0 Kudos
sergeadam
Enthusiast
Enthusiast

I'm in the same boat. I have 2 servers that I want to virtualize. One has a 4TB datastore and the other a 8TB datastore. I have folders that are larger than 2TB and they can't be broken up. I also want to use VEEAM to backup. Is there a time frame for VMWare to support VMDKs larger than 2TB?

Seems to me that supporting large LUNs is almost worthless if I can't use them.

Reply
0 Kudos
RParker
Immortal
Immortal

sergeadam wrote:

I'm in the same boat. I have 2 servers that I want to virtualize. One has a 4TB datastore and the other a 8TB datastore. I have folders that are larger than 2TB and they can't be broken up. I also want to use VEEAM to backup. Is there a time frame for VMWare to support VMDKs larger than 2TB?

Seems to me that supporting large LUNs is almost worthless if I can't use them.

so what were you doing for the last 5 years?!?  This isn't new, and the limitation is for files.  I am sure there is some technical problem with files that big, just like ZIP files can't be over a certain size either.  It's probably an addressable space problem in memory or the file system.  Other than virtualization, where else do you see 2TB files?  Databases MAYBE... but those are no on block level storage either, so I think you are forgetting this isn't limited to VM Ware.

They give you RDM and NFS, why the insistance to use VMDK on BLOCK datastores, when there ARE alternatives.  The pressure should be on Veeam, Quest to allow backups for RDM.. Since there ARE people like you that seem to want this functionality.

Reply
0 Kudos
sergeadam
Enthusiast
Enthusiast

RParker wrote:

sergeadam wrote:

I'm in the same boat. I have 2 servers that I want to virtualize. One has a 4TB datastore and the other a 8TB datastore. I have folders that are larger than 2TB and they can't be broken up. I also want to use VEEAM to backup. Is there a time frame for VMWare to support VMDKs larger than 2TB?

Seems to me that supporting large LUNs is almost worthless if I can't use them.

so what were you doing for the last 5 years?!?  This isn't new, and the limitation is for files.  I am sure there is some technical problem with files that big, just like ZIP files can't be over a certain size either.  It's probably an addressable space problem in memory or the file system.  Other than virtualization, where else do you see 2TB files?  Databases MAYBE... but those are no on block level storage either, so I think you are forgetting this isn't limited to VM Ware.

They give you RDM and NFS, why the insistance to use VMDK on BLOCK datastores, when there ARE alternatives.  The pressure should be on Veeam, Quest to allow backups for RDM.. Since there ARE people like you that seem to want this functionality.

The past 5 years, I've been using physical servers with large RAID arrays. It's not individual files, it's entire folders. I have a business application where it is not unusual to have a single folder over 2TB with 10s of thousand of files.

The pressure also needs to be on VMWare to remove limitations that seem to be there for no reason.

Reply
0 Kudos