Configuration
Have two ESXI-6 servers. Across these two servers, have 6 resource groups containing 36 virtual machines. Each server has 24 cores. vCPU assignments on each machine don't exceed 21 vCPUS. The Guesthost types are RHEL 6 and Solaris 10. On server 1, I have 1 additional RHEL 6 Guesthost that is the 'controller' application that uses VIM25 to start, stop, and otherwise manage the other instances..
Issue
Using VIM25 webservices, I issue startup of the first resource group of 7 servers. Followed within 30 seconds the second resource group of 7; these 14 vms are located on the first physical server. I then within 30 seconds start the 3rd group of 7 on the second second physical server. At this point, two issues occur:
1) On the server using the VIM25 client to do these startups, the load average of the machine grows to over 9, even hits 11 however, the CPU usage on the java pid containing the vim25 client calls, is well under 100 (bounces 55-60). My research thinks this behavior is due to vim25 client calls not completing thus causing contention??
2) The VIM25 client calls to poweron guesthosts will show up on the event log on server 2 and the status of that event goes into PROCESSING and stays there
At this point, server 2 becomes 'frozen', as well as the "controller" application Guest host. In some cases the only way to recover is to go to the text console of the server and issue a shutdown with force VM shutdown.
Need help with how to troubleshoot this issue. We have further complications, in that these servers exist in a secure lab and I cannot provide any type of logs from those servers.
