I've been doing some stress testing of our cluster by creating a large number of VMs (40+) from a single template (about 12GB of disk space each) running Ubuntu Linux. To see how well VC and ESX handle this, all VMs are created using a for loop in PowerShell with indexed names (vm_001, vm_002, etc.).
Afterwards, all VMs are powered up using the same for loop, so queued relatively fast, and pushing the cluster a bit in terms of CPU, Memory, and Disk utilization.
We've seen numerous times that many VMs hang while this is happening, most of the time at the BIOS screen when the virtual screen resolution changes.
We've also seen the VirtualCenter service crash and restart while queueing a large number of Tasks like this. Never a pleasant thing to have happen.
Has anyone else done these types of tests and been successful? Unfortunately, VMware's support is pointing me in the direction of SDK support, and I can already see the answer coming "VI Toolkit is in beta". Alas, I'll attempt to submit as much as possible about the problems.
We have also written a script to reboot all 40 VMs using a similar for loop (gracefully restarting the Guest OS). This actually works quite well, and the cluster handles it pretty elegantly... but... after doing this every 5 minutes (enough time for all VMs to finish booting and remain idle for a minute or so) for about 12 hours, many of the VMs have hung. We've even had some VMs claim corrupted storage with an fsck error during boot. Not a good sign.
The hardware is all HP Proliant equipment, including the fiber-channel SAN equipment (MSA series), which has been rock-solid for 2+ years. We just haven't pushed it this hard since it wasn't very easy to do so without the toolkit. We do, however, push the storage pretty hard on a daily basis with esXpress with "no" problems whatsoever. So, I'm not convinced that we're having a hardware issue.
Any comments? Or anyone else with similar/different results?
This is with VC 2.0.2 Update 1 (not the latest) and ESX 3.0.2 with some hotfixes, but not all.
Eric K. Miller, Genesis Hosting Solutions, LLC
- Lease part of our ESX cluster!
Eric K. Miller, Genesis Hosting Solutions, LLC
http://www.genesishosting.com/ - Lease part of our ESX cluster!
