Script to attach VMware tools and HW version update baselines to a set of VMs (from csv file)?

[golddiggie]

This environment is large, alright it's downright HUGE (about 15k VMs on ~1200 hosts in four physical datacenters). We need to kick off a VMware tools and HW version update task on about half of the VMs. Looking to use csv files for the targets on this (since we have a short window to perform these updates).

So, I'm hoping that someone has a solid method to flip the switch in the settings to update tools on next power on, as well as the schedule HW update on reboot. Or, a way to pull said list into VUM to then push the updates to the targets (including taking snapshots of these VMs before executing the tasks).

If it wasn't for the high VM count we need to target, I'm sure we could do it a less automated way. But when you have this many and such a short amount of time, scripting/automation is required (for sanity sake).

I'm hoping to have something we can test out in the next few days in our sandbox (or on our canary set) to confirm it's going to work. This also needs to have many running concurrently. Since if they go in a serial fashion, we'll run out of time LONG before they are done (or even a set of them get completed). One of the people in the company had created a script, but it was set to run in a serial fashion, with it taking about one minute per VM. We have a few two hour windows to do half of the VM count. :smileyshocked:

[golddiggie]

OK...

Is there an easy way to get the cloning script to do more than one execution at a time?? Also, when I ran it to do 100 it bombed. I split it into four 25 clone task chunks and they didn't bomb on me. If I could even have 5-10 clone tasks running at the same time to get the 100 count, that should be enough to make it go fast enough. I've removed the line to make the VMs thin provisioned since the source already is. So it can clone at the original provisioning type.

At least the VMs didn't all get dumped into a single datastore this time.

We also did a test run of the VMware Tools and HW version update script against the 100 clones. We had a 10% failure rate on the targets. It also took about 30 minutes to get them completed. We're looking into how we're going to do this at scale, inside the time frame we have. We need to get up to about 350 VMs processed in less than a two hour window (at a time). Some of the vCenter's will have upwards of 130 being processed at a single time (we have 15 vCenter Servers in total).

I'm wondering if having the NIC disconnected (on boot too) on the test targets is part of the issue. We don't need them to have any network access, since they're only to test the update tasks/scripts.

Side question since I can't recall exactly... Do you need to update the tools or hardware version first? Does it matter with 6.5u1? I seem to recall reading to do the HW version first, then tools. Right now, I think the tools are getting updated first, then the HW version.

[LucD]

The script I provided runs the New-VM with the RunAsync switch, meaning the cloning task runs in the background.

So there should be, provided the vSphere environment has the capacity, be running more than 1 of those.

Remember that vCenter can limit the number of parallel tasks as well.

I would need more info to determine why the run with 100 clones gave errors (the error messages).

Also for the 10% failure on HW and Tools upgrade, I would need to see what errors those 10% gave.

Isn't the Update Manager determining the order of the HW and VMware Tools upgrade?

[golddiggie]

The lab vcenter, and setup, isn't being hit with anything else since it is out sandbox lab to use/abuse. Three hosts with 7 datastores (in a cluster, 2TB per DS). No other processes are running, or people using this setup while I am.

If there are specific logs you want pulled, let me know and I'll see what I can do. I was just hoping to get this more time efficient, but also have it balance across the datastores in the cluster. When I kicked it off yesterday, it would do each clone task in turn and not progress to the next until that was finished. Not the behavior I saw in the other script. The issue with the other script was it would blast all the clones to a single datastore and make it go to a yellow alert state. With SDRS enabled (storage cluster), it's a pain in the ballsack to try and manually balance it after the fact. I would have hoped that it would do it on generation (accept recommendations automagically).

[LucD]

You could provide the datastorecluster instead of a single datastore on the Datastore parameter of the New-VM cmdlet.

Then SDRS would decide on the placement of the clone.

Do those 10% failures always show the same error?

Can you include the error in here?

Not sure if you now want multiple clones running in parallel or not.

And if in parallel would you want to impose a maximum of these tasks?

All options are possible.

Perhaps attach the current script you are using again, and explicitly state which of the above options you want to take.

[golddiggie]

I have been using the datastore cluster name in that field. In the first script (not yours) that I used, it would only go to one DS, even when I used the cluster name.

I can't post the error right now since we're in the process of testing out a few methods to get these updates applied. The problem, IIRC, was that tools crashed when we went to execute the update on those VMs. Some would come back after a reset, but not all. We ran a different command set against a smaller set of VMs (spread out over several vCenters/host clusters) this morning and out of 32 targets, we had one that we had to uninstall and reinstall the tools to get that one resolved. Tools were running, and not reporting any issues, before we kicked off the update task. It was driven through VUM. Now, we're looking to see about flipping the switch for 'update on boot/reboot' for the tools and HW version. That way, since OS patches also are going to be applied, the VM will reboot and apply the tools update during those reboots. Same thing for the HW version update.

I did export the events from the past 36 hours and it didn't show any errors. Even though we saw them in the tasks. Looking in the tasks listing, I can see errors, but they're generic. The target VMs have been deleted since the tasks were done (to get into a clean state with the VMs).

Of course, ONE of the vCenters is still on 5.5 and the thick client was being a total ballbag to us today. Lagging like a you would not believe. The web client was fine, except it doesn't have VUM available. But, we were able to get the tools updated (struggled to get the 5 VMs that were in that set to remediate) and then we ran the script again to also get the HW version updated. That configuration is slated to be decommissioned sometime in the coming months, so no version updates are going to happen there.

I have 53 VMs to update on my window tomorrow at 10am (EST). That includes the 5.5 vCenter/environment. Plus another 6 vCenter Servers/environments.

[LucD]

Ok, thanks for the feedback, and good luck with the upgrades tomorrow.