VM deployment brainstorming

[icontrolyourpow]

Years ago I had to write some logic to handle deployment of batches of VMs. The most crucial part of the deployment was assuring that the entire batch could be deployed before deployment started. I tried for a few days to figure out the combinatorial calculations to get the most disks for every VM in the batch on the same datastore, but that proved too difficult so I resorted to basically the following:

1. Create a shadow list of all valid datastores for the target location of the VM and their free space.
2. For each VM in the queue:
    a. For each of its disks:
        1) For the first disk, loop through the shadow list and subtract the disk size, memory size, and some additional space for logs and other metadata from the free space of first datastore in the shadow list which could accommodate that quantity and still retain at least X% free afterwards.
        2) For all subsequent disks, try the previous disk's datastore subtracting only the size of the disk and checking the desired datastore %free target threshold.
        3) Try every other datastore in the shadow list if the datastore of the previous disk didn't meet the requirements.
3. Fail the batch if anything can't be placed or deploy the machines according to the disk/ds assignments determined above if everything was successful.

With datastore clusters I can get rid of all that code, but I still need a way to validate before deploying a batch of VMs. Easy to create a shadow list and subtract total VM sizes from free space, but that won't provide 100% assurance of a successful deployment. For example, if I had 2x 2 TB datastores and each housed a single 1.5 TB disk, there would be 1 TB of free space in the datastore cluster, but there would be no way to place any single disk greater than 500 GB. If I were only deploying one machine at a time, I suppose I could use the method for getting recommendations as a validation step, but that would not work for validating multiple VMs at once. Perhaps I could total the disks for all queued VMs into a single fake configuration just to get recommendations on that, but that would only work until the quantity of disks in the batch exceeded the maximum number of disks for a single VM.

Any genius suggestions on how to validate with 100% accuracy that a batch of any number of VMs, each with any unique number and size of disks could be validated together for successful deployment to a datastore cluster? I'm not looking for code, just ideas.

[LucD]

I would try using the RecommendDatastores method.

If there are no recommendations, check the drsFault property in the StoragePlacementResult to see why

[icontrolyourpow]

LucD wrote:

I would try using the RecommendDatastores method.

If there are no recommendations, check the drsFault property in the StoragePlacementResult to see why

Considered that, but I'm almost certain it only takes the configuration of a single VM as input. Haven't tried passing it an array, but nothing in the docs or my experience with similar methods leads me to believe it would work with an array. I need to validate that 10, 100, or 1000 unique VM configurations can be deployed before deploying the first one.

[LucD]

Yes, you're right, that method only takes 1 VM.

That would mean you need to go for an object placement algorithm.

There are many available.

How does your input look ?

How do you specify the batch of VMs to be created ?

[icontrolyourpow]

LucD wrote:

Yes, you're right, that method only takes 1 VM.

That would mean you need to go for an object placement algorithm.

There are many available.

How does your input look ?

How do you specify the batch of VMs to be created ?

I think creating even a rudimentary object placement algorithm would be a daunting task for someone of my ability level. I understand the basic problem; given a quantity of containers of a fixed size, determine if a quantity of items of variable size can be arranged to fit within the containers. I just don't know where to start with the calculations unless I were to find every possible permutation of the total quantity of items and test each one individually. With just 10 disks I would have to test over 3 million possible layouts, so that's not going to work. Even if I were able to create something, it would have to function just like whatever SDRS does or it would yield an inaccurate validation assurance.

I think for now I may have to create a temporary VM configuration containing all disks from the batch and try to get a single recommendation. This limits me to batches of VMs with less than 60 disks, but that should be ok most of the time. Are you aware of any code samples for RecommendDatastores for the context of creating a VM? Documentation indicates Folder, ConfigSpec, ResourcePool, PodSelectionSpec, and HostSystem are required. Despite those being provided, I am met with with a non-specific error of "optional value not set." I've tried populating additional properties with objects as described in the API documentation at various levels of the hierarchy without success.

Message was edited by: icontrolyourp…  Nevermind on the example code request. Onyx to the rescue.

[Chrisxxxx]

Maybe it would be simpler just to simulate an actual deployment as a sequence of steps instead of a single formula?

Real life

1. For VMs in CSV
   1. Choose datastore with most free space (used or provisioned, depending on your environment).
   2. Deploy VM to that datastore
   3. Go to next VM

Simulation

1. Export datastore info to a CSV
2. Import that CSV to a variable, such as $DSTs ($DSTs is now an array variable)
3. For VMs in CSV
   1. Sort $DSTs so that datastore with most free space is first, at position $DSTs[0]  ($DSTs = $DSTs | sort ProvisionedSpaceGB -descending)
   2. Substract the VM's disk space from that datastore.  ($DSTs[0].ProvisionedSpaceGB =  $DSTs[0].ProvisionedSpaceGB - VM's disk size)
      1. Error chech that there is enough free space, break out of script if there isn't
   3. I would log to a file or the console what was done (VM "X" deployed to datastore "X", old datastore sizing, new datastore sizing).  Probably to the screen so I could watch what was happening until the script was stable, then log to a file instead.
   4. Go to next VM

Each pass through the loop sorts the $DSTs variable so that the datastore with the most free space is the first item in the array.  It should run pretty quick and avoids any hint of math. 

I think this logic works because it's the process I use to create VMs now.  If I'm cloning I have a clone function that clones one VM and calls another script (Get-DatastoreMostFree.ps1) that returns the datastore with the most free space in that cluster.  (If I have lots of cloning I just feed a 2 column CSV with sourcevm and targetvm into a loop.)  If it's deploying from from template I fill out a CSV with all the details and pass that into a script that creates each VM one at at time, and that script will call Get-DatastoreMostFree.ps1 for datastore placement.

hth

-Chris

p.s.  Thin provisioning is your BEST friend, after vmotion, svmotion, dvswitches, etc.