Max no of vm per lun

[tom_brechet]

Hi guys

I seem to remember that VMware stated with ESX 2.x that the maximum Numbers of VMs per Lun is above 100 so I never cared about it. Now there is this "SAN System Design and Deployment Guide" from VMware, released March 2007 which states on page 166 that no more than 16 VM should be placed on a lun? I don't seem to be the only one remembering the old limits as in the HP "VMware Infrastructure 3, HP StorageWorks best practices" there is this statement: While VMware Infrastructure 3 supports up to 80 VMs per LUN for a "light" I/O load... 40 VMs per LUN is a conservative number.

So, what is correct? Am I really that mistaken? 16 VM per LUN, is that possible?

Would be glad to have an answer on that.

Cheers

Tom

[habibalby]

Hi,

Well, you can have up to 100 VMs per LUN, 32 VMs per LUN may often be acceptable. But that all depends on your needs / VMs I/Os.

It's recommended to keep 8 ~ 14 machines per LUN.

[habibalby]

Read here as well

http://www.experts-exchange.com/Storage/Storage_Technology/Q_22410495.html

[jhanekom]

You must have a different version of the Storageworks best practices document. Mine specifically state on page 7:

"Carefully determine the number of VMs you can place on a single LUN in your particular environment. Note that, with new VMware Infrastructure 3 capabilities such as iSCSI and NAS, this determination has become even more complex. While VMware Infrastructure 3 supports up to 100 VMs per LUN, 32 VMs per LUN may often be acceptable. However, HP recommends being more conservative and using 8 – 10 VMs per LUN. In a NAS or iSCSI environment, HP recommends using 8 – 16 VMs per LUN. If your VMs are not heavily impacting I/O, you may be able to exceed the conservative recommendations above."

The document I have is dated "6/2006 -2" on the very last page.

[tom_brechet]

Yes, my documentation is definitvely different then. Please see in the attached document page 8 second bullet in Chapter "General SAN connectivity". This seems to be a newer version dated february 2007.

[tom_brechet]

>Hi,

>Well, you can have up to 100 VMs per LUN, 32 VMs per LUN may often be acceptable. But that all depends on your needs / VMs I/Os.

>It's recommended to keep 8 ~ 14 machines per LUN.

Hi,

Do you have an source for that? Just wondering as I seem to remember the same, never implemented it this way my LUNs where always sized for 10-20 VMs but now I read that i should place 16 VMs on a LUN

Cheers

Tom

[habibalby]

From HP StorageWorks Best Practice

http://h20219.www2.hp.com/ActiveAnswers/downloads/VMware3_StorageWorks_BestPractice.pdf

Page 7

General SAN connectivity

VMware Infrastructure 3 and HP StorageWorks storage combine to provide a robust solution for

consolidation, reliability, and disaster recovery. This section provides general guidelines for

connecting VMware Infrastructure 3 to an HP StorageWorks SAN.

• Ensure each LUN has the appropriate RAID level and storage characteristics for the VMs that will

use it.

• Carefully determine the number of VMs you can place on a single LUN in your particular

environment. Note that, with new VMware Infrastructure 3 capabilities such as iSCSI and NAS, this

determination has become even more complex.

While VMware Infrastructure 3 supports up to 100 VMs per LUN, 32 VMs per LUN may often be

acceptable. However, HP recommends being more conservative and using 8 - 10 VMs per LUN.

In a NAS or iSCSI environment, HP recommends using 8 - 16 VMs per LUN.

If your VMs are not heavily impacting I/O, you may be able to exceed the conservative

recommendations above.

• To optimize array performance, spread I/O loads over the available paths to the storage.

• HP suggests configuring one Virtual Mirror File System (VMFS) volume per LUN. With VMFS-3 you

no longer have to set accessibility.

• In an environment with HP BladeSystem servers, a shared diagnostic partition is appropriate.

• Local SCSI controllers and Fibre Channel HBAs can be allocated to the VMkernel. However, it may

be preferable to use local RAID solutions for the Service Console and Fibre Channel HBAs for

external storage such as SANs.

[Texiwill]

Hello,

How many VMs per LUN depends on the storage device involved and the size of the VMs. A good algorithm for sizing a VM is:

Size of VMDK * 2 (possible snapshots) + Size of Allocated Memory + 15MBs (sundry files) + 4GBs (logfile explosion) = Size of VM

Assume a 24GB VMDK and 2 GB of Memory for the VM.....

24 * 2 + 2 + .15 + 4 = 54.15 GBs per VM

I you only have a 512GB LUN... The math states:

Size of LUN * 0.80 = Useable Space on LUN / Size of VM = Total # of VMs (rounded down).

512 * .8 / 54.15 = 7 VMs

Some may argue that 24GBs for a snapshot in this case is unreasonable but I have seen 100GB snapshots because the backup software failed to properly delete the snapshot. Plus, logfiles fill up the disk quite a bit as well. In an error situation you will burn through disk space with each logfile quite easily. If the VMFS fills up, things stop working.....

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[habibalby]

Yes, I do agree entierly with Taxiwill. SAN Calculation for VMDK files is the best strategy and planning for your VMs. It's not only depends on the LUN Size and How many VMs per LUN? It depends on the Backup Strategy, Snapshot, logs. if they are stored to the same SAN Storage for Backing up to the Proxy Server and Tape Library. Blah Blah Blah.

[tom_brechet]

Hi Edward

Thanks for that formula. But what I need to know is probably exactly the opposite: Not how many VMs per given LUN size but how big a LUN size for the VMs. I've had LUN sizes varying from 256 to 1GB in my past designs and all worked quite well. Now I have a customer with a 1GB LUN which has some trouble. This is when I try to find the info that existed back then (at least in ESX 2.x) that VMware supports over 100 VMs per LUN. While searching I only find this new to me info that one should never place more that 16 VMs on a LUN.

Am I the only one catched cold by this?

Cheers

Tom

P.S.: the consultant with 18 happy and one unhappy VI-Design customer

[Texiwill]

Hello,

The math really does not change that much when you start to decide on how big a LUN you want..... The last calculation changes.

BTW, it was not 100 VMs per LUN in ESX 2 days it was 192 files per LUN.... + 32 Files for every Extent added. And the more Files on the LUN the slower the LUN went. If for example there were 6 files per VM (2 VMDK, 1 vMotion log file, 1 suspend file, 2 REDO files), you could not put more than 32 VMs per non 'extent'ed VMFS. And most people cut that in half even so. So 100s of VMs, never, but 100s of files, perhaps. You could perhaps have 192 VMDKs, but that does not necessarily translate to 100s of VMs.... At least I never once suggested more than 16 VMs per LUN even in v2 days.

For ESX v3, to calculate the size of a LUN based on a count of VMs, reverse the algorithm mentioned previously some:

Size of VM * # of VMs / 0.8 = Size of LUN required

Assume a 54.15GB VM, and 16 VMs.... = 1083 GBs or 1.083TBs of LUN space required.

Remember you want to fill a LUN no more than 80% just in case.

Also, it is often much better to have several smaller LUNs than one massive LUN. This will help with alleviate SCSI Reservation Conflicts. These conflicts are often what dictates the LUN count more than anything else. It depends on the storage used as well.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[williambishop]

I have luns that vary from 1 vm to 150 vm's. They are all scaled according to IO and the array they are connecting to.

So the answer is ALWAYS...._.IT VARIES_.

[tom_brechet]

Hi Edward

>BTW, it was not 100 VMs per LUN in ESX 2 days it was 192 files per LUN.... + 32 Files for every Extent added.

Am still battling my war and found this info here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1269

>Up to 100 virtual machines may share a VMFS-2 volume if they have less disk-intensive workloads, subject to other system constraints such as CPU and memory resources.

Good to know I'm not completely off my rocker and do at least remember some things correctly

BTW: Got a statement from a Field-SE hat all these numbers are no limits but mere recommendations. The only limiting factor for VMware seems to be performance and thats not my question at the moment.

Cheers

Tom

[Texiwill]

Hello,

The max # of files available for VMFS-3 is ~31000, I actually think it is 31715 or something close to that.. Consider that there are at least 1 directory and 9 files per VM, but most likely 15 considering all the logfiles created, there can ~3100 VMs per VMFS-3. I would say this is the maximal count allowed, but you are ideally looking at ~1900 VMs. It is all based on file count and the average # of files/directory for a VM with a single VMDK is ~16.However with a 2TB limit on the size of a VMFS-3, VMDK;s may need to be .5GBs for this to work.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[tom_brechet]

Hi Edward

Now we are talking. That statement or any other is exactly what I need from VMware. How many VMs they support on a single LUN. Be it 10 or 100 or 1000 or whatever.

I've given you 2 helpful answers now as your post have indeed been the most infomative. I'd like to reserve the "correct answer" for whoever can point me to an official statement of VMware what the exact number is.

Many thanks

Tom

[Texiwill]

Hello,

Unforunately there is no exact number. It is a combination of the maximum vCPUs per host, and the size of the VMDKs in use, plus the limit of 2TB VMFS-3.

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education. CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354, As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[eagleh]

"Carefully determine the number of VMs you can place on a single LUN in your particular environment. Note that, with new VMware Infrastructure 3 capabilities such as iSCSI and NAS, this determination has become even more complex. While VMware Infrastructure 3 supports up to 100 VMs per LUN, 32 VMs per LUN may often be acceptable. However, HP recommends being more conservative and using 8 - 10 VMs per LUN. In a NAS or iSCSI environment, HP recommends using 8 - 16 VMs per LUN. If your VMs are not heavily impacting I/O, you may be able to exceed the conservative recommendations above." -

---

<VMWare3_StorageWorks_BestPractice.pdf>

[williambishop]

Again, those numbers are not fixed by vmware itself. You'll note that most of the limit numbers were from the storageworks best practice guide....Which is only one array of many. Vmware prefers you not to overload the luns....but it varies according to how strong the array you are using. A symmetrix is easily capable of 150+ vm's per lun.

[rmgbasker]

Hi Dued's

Can you please explain this calculation need 4GB for log file explosion because the maximum log file size 1020kb and if the log file reach 10 then automatically old log files are deleted.Could you please explain about clearly on this below vm sizing calculation and let me know the sundry file details as well

Size of VMDK * 2 (possible snapshots) + Size of Allocated Memory + 15MBs (sundry files) + 4GBs (logfile explosion) = Size of VM

[FredPeterson]

Discussions of LUNs per VM can be annoying because ya you want 20 VM's per LUN, figure something like 25GB per VM, you need a 500GB LUN. 500GB / 146 = 4 disk RAID 5.

But will a 4 disk RAID 5 work for the required IO? Thats something like 300 IOPS, or 15 IOs per VM. Is that enough? Do you need 5 or 6 disks? That directly increases the size of the LUN leaving "empty" space which will get annoying to look at over time. That is unless you carve the size of the LUN and let that space go unassigned because you don't want it assigned to anything else. It would suck to have LUN performance suffer for your VMs because that "extra" 100GB was assigned to something else and its pounding the hell out of the spindles while your VMs are sitting quietly.

Ahhh... the joy of why a VMware administrator needs understanding of storage and how things are setup.