thoughts on "growable" storage (local, non-SAN) in ESXi 5

[ljwobker]

I'm looking for advice/thoughts on what sort of storage setup I should consider for a small deployment of VMs.  I'll try to provide some general guidelines and what sorts of tradeoffs/limitations I'm working with.

• no SAN: all storage is local.  As of now I'm working with a couple of Cisco UCS servers, with anywhere from ~300G of SSD storage to ~4TB of SATA drives.  Again: all drives are local, a SAN isn't in scope/budget for this.  Willing to accept that this is suboptimal but also need to do the best I can here.
• VM operating systems are mostly win2008, win7, linux.  Possibly others but these are dominant.
• Key requirement is to be able to "grow" the filesystems over time.
  • We know that *some* of these servers will grow significantly in their storage requirements.  Unfortuneately, we do not know which ones, so it's not possible to pre-provision large drives on all of them up-front.

So i have quite a few high level questions... some of these are probably basic/stupid, and for that I apologize.  My background is heavy network, reasonably heavy PC/server systems, but very light on storage.

• How effective is thin provisioning?  If my physical hard drive is just a 2TB RAID5 array shared over 20 VM's... is it possible/reasonable to just configure every VM with a thin-provisioned 2TB disk, and then monitor the actual storage as it grows over time?
• Assuming my RAID controller supports on-line addition of disks to the array, is this a reasonable thing to consider?  I *am* in a position where taking down the physical hosts for maintenance isn't that big a deal, so if I can add more disks to the array later it's OK if this takes a couple hours of off-line time.  I'm trying to understand if the following workflow is possible, and if possible is it considered reasoanbly safe?
• assume a host with 4x500G SATA drives in a RAID5 array, this is a single large VMFS datastore where everything lives...  On this host, we have a number of linux VMs each with one small OS drive, and a larger data drive (call it 600GB).  At some point, the vmfs datastore gets close to full, and I want to expand it... what are my options here?
  • can I do something like this:

1. power down VMs, power down server.  (this is OK...)
2. add more physical drives to the machine, and grow the RAID array at the controller level (i realize this could take a long time)
3. boot, extend the vmfs datastore within ESXI (is this possible?  if so, how?)
4. increase the size of the disk assigned to the storage-hungry VM within ESXi
5. boot the VM, expand the partition, then grow the filesystem (I know how to do this in linux, is it doable in windows??)

so anyway... that's probably more than enough questions to get started.  I appreciate any advice/thoughts that folks might have.

[mcowger]

• How effective is thin provisioning?  If my physical hard drive is just a 2TB RAID5 array shared over 20 VM's... is it possible/reasonable to just configure every VM with a thin-provisioned 2TB disk, and then monitor the actual storage as it grows over time? - Growth increment is 16MB at a time, so its VERY effective .  As long as you can effecitvly monitor your commitment rate and react to problems quickly, its reasonable.

• Assuming my RAID controller supports on-line addition of disks to the array, is this a reasonable thing to consider?  I *am* in a position where taking down the physical hosts for maintenance isn't that big a deal, so if I can add more disks to the array later it's OK if this takes a couple hours of off-line time.  I'm trying to understand if the following workflow is possible, and if possible is it considered reasoanbly safe?

     Its reasonably safe & proven, yes.  Backups are always a good idea

• assume a host with 4x500G SATA drives in a RAID5 array, this is a single large VMFS datastore where everything lives...  On this host, we have a number of linux VMs each with one small OS drive, and a larger data drive (call it 600GB).  At some point, the vmfs datastore gets close to full, and I want to expand it... what are my options here?
  • can I do something like this:

1. power down VMs, power down server.  (this is OK...)  Maybe not - if you have open drive slots, most decent servers let you add them online
2. add more physical drives to the machine, and grow the RAID array at the controller level (i realize this could take a long time) - Some RAID controllers can do this, most cant. 
3. boot, extend the vmfs datastore within ESXI (is this possible?  if so, how?) Once you have done step 2, right click the datastore and choose 'Extend'.
4. increase the size of the disk assigned to the storage-hungry VM within ESXi - Right click the VM, choose edit, expand the virtual disk size.
5. boot the VM, expand the partition, then grow the filesystem (I know how to do this in linux, is it doable in windows??)  Its doable in windows with dynamic disks.

so anyway... that's probably more than enough questions to get started.  I appreciate any advice/thoughts that folks might have.

[ljwobker]

• How effective is thin provisioning?  If my physical hard drive  is just a 2TB RAID5 array shared over 20 VM's... is it  possible/reasonable to just configure every VM with a thin-provisioned  2TB disk, and then monitor the actual storage as it grows over time? - Growth increment is 16MB at a time, so its VERY effective .  As long as you can effecitvly monitor your commitment rate and react to problems quickly, its reasonable.
• Cool - that sounds good.
  

• Assuming  my RAID controller supports on-line addition of disks to the array, is  this a reasonable thing to consider?  I *am* in a position where taking  down the physical hosts for maintenance isn't that big a deal, so if I  can add more disks to the array later it's OK if this takes a couple  hours of off-line time.  I'm trying to understand if the following  workflow is possible, and if possible is it considered reasoanbly safe?

     Its reasonably safe & proven, yes.  Backups are always a good idea

• assume  a host with 4x500G SATA drives in a RAID5 array, this is a single large  VMFS datastore where everything lives...  On this host, we have a  number of linux VMs each with one small OS drive, and a larger data  drive (call it 600GB).  At some point, the vmfs datastore gets close to  full, and I want to expand it... what are my options here?
  • can I do something like this:

1. power down VMs, power down server.  (this is OK...)  Maybe not - if you have open drive slots, most decent servers let you add them online

Right.. but even if I can add them online, I don't have any way to provision them on the RAID controller without dropping back into the controller BIOS  ("web bios" in LSI Megaraid speak, which is what I have)... right?  Or is there some way to talk to the RAID controller while the host is up and running that I'm not aware of?

1. add  more physical drives to the machine, and grow the RAID array at the  controller level (i realize this could take a long time) - Some RAID controllers can do this, most cant. 

If my server is on the can't list... does this limit me to adding a second RAID array, which gets presented to ESXi as a second datastore/device?

1. boot, extend the vmfs datastore within ESXI (is this possible?  if so, how?) Once you have done step 2, right click the datastore and choose 'Extend'.
2. increase the size of the disk assigned to the storage-hungry VM within ESXi - Right click the VM, choose edit, expand the virtual disk size.
3. boot the VM, expand the partition, then grow the filesystem (I know how to do this in linux, is it doable in windows??)  Its doable in windows with dynamic disks.

so anyway... that's probably more than enough questions to get started.  I appreciate any advice/thoughts that folks might have.

[mcowger]

Right.. but even if I can add them online, I don't have any way to provision them on the RAID controller without dropping back into the controller BIOS  ("web bios" in LSI Megaraid speak, which is what I have)... right?  Or is there some way to talk to the RAID controller while the host is up and running that I'm not aware of?

That depends on the controller.   Some of them can do it online, including some of the LSI WebBIOS ones by using a remote API tool.    The vst majority done, so you'd need the downtime.

1. add  more physical drives to the machine, and grow the RAID array at the  controller level (i realize this could take a long time) - Some RAID controllers can do this, most cant. 

If my server is on the can't list... does this limit me to adding a second RAID array, which gets presented to ESXi as a second datastore/device?

No, you can still create a second LUN and export to VMware, then use it as an extent or new VMFS.

[ljwobker]

Ok, so the next stupid question would be "how is a new LUN different from a second RAID 5 array" as they're presented upstream from the LSI controller?

[scottyyyc]

Each separate volume of disks will be presented as it's own voume/LUN. So if you have two RAID 5 volumes on a single controller, it will show as two volumes/datastores in VMware.

Generally speaking, I think you're on the right track. It will be up to the RAID controller as to how easily you can grow the volume. Every RAID controller is slightly different, so I'd highly recommend TESTING this as much as you can. It's not always as easy as adding another disk and clicking 'expand'. On the VMware level, it's pretty easy though -- once your storage is expanded, rescan the storage controllers, and then it should see the new size. From there, expanding the size of the datastore is just a couple clicks away. Obviously local will never be as flexible as a SAN, but with some decent RAID controllers, it's about as good as you can do.

Also keep in mind that you can easily move VMs from datastore to datastore (live with storage vmotion, otherwise the VM simply has to be powered off). This will mean that in the event you need to move a VM from one datastore to the other, it's not the big of a deal - one of the core benefits of virtualization to begin with.

As previously mentioned, backup is key. If you want a nice, simple, but powerful backup solution, look into Veeam. It also has a variety of replication tools, making moving VMs even that little bit easier...

Thin provisioning works well. The only real danger is over-allocating and getting yourself in trouble in the future, but if you keep an eye on things (or don't have crazy growth rates), it's generally not an issue.

[ljwobker]

OK, cool... that's very helpful.  Regarding the monitoring/management aspect of the arrays, is there some way within vCenter to have it notify me as certain storage thresholds are reached/crossed?  Ideally I'd just like the system to send me an email -- as this environment is definitely not "my day job" there's no guarantee that I'll be smart enough to remember to check it regularly, and at this point I really have no idea how fast the utilized storage is going to actually grow.

[psplnalin123 · Accepted solution]

To monitor the Growth of the VMFS Datastore and what i understand you want a automatic mail drop incase a threshold defined by you exceed you first need to configure the smtp server IP address from vCenter Settings. After that go to the Alarm in datacenter level and go to the Datastore usage alarm properties go the to reporting tab select add and define the email address of yours over there.

For your reference

http://www.vmadmin.co.uk/vmware/36-virtualcenter/117-alarmsemailalerts