VMware Cloud Community
cbass64
Contributor
Contributor

VMware tools drastically decreases performance with T2000 TSS

I have several hundred testers running as VMs. When they use T2000 Test System Software and have VMware tools installed the system becomes completely unresponsive for the duration of the t2k command. The commands also take 4-5 times longer to complete with VMware tools enabled.

Has anyone else seen this? Anyone found a work around (other than disabling the VMware tools service, because we require this service to run in our production environment).

If I run 't2kctrl restart' with no VMware tools installed, you see the normal spike in CPU usage but you can continue to use the VM to do other tasks and it's still quite snappy. This process takes ~68 seconds.

If I install VMware tools and run the same command the system becomes completely unresponsive, but the CPU usage sits around 6-8% with vmtoolsd.exe using the most CPU. The command takes ~300 seconds to complete. Some Googling on this subject leads me to believe there is a link between the VM Processor counter in perfmon disapearing and VMware tools is trying to rebuild it and is robbing CPU cycles to do so, but this is leading me nowhere. Essentially, the counter times out while waiting for the sample to be collected and the counter disappears and when VMtoolsd sees that the counter is missing it spends CPU cycles trying to replace the counter, only it's an endless loop wasting CPU cycles until the t2k command is complete.

If I disable vmware tools, the problem goes away, back to ~68 seconds and snappy VM.

I know it's caused by VMware Tools but I don't know how much of it is related to the perfmon issue and I have been working with VMware Support for a couple weeks now and they basically have no clue what is going on I have tried 3 versions of VMware tools, including a version the support team sent me.

Has anyone seen this and found a fix that doesn't include disabling VMware tools?

FYI, I have tried disabling performance counters and VMware Tools still sees this as the counter being gone and still wastes CPU cycles trying to replace it...although not as drastic as when the counters are enabled (t2k command takes ~250 seconds, rather than ~300).

Reply
0 Kudos
1 Reply
cbass64
Contributor
Contributor

Update - This is on XP SP3 32bit. It seems to happen when any t2kctrl commands are ran, but is most noticable when the start and restart commands are ran. I have more and more users seeing this issue. I had a support ticket open with vmware support for 6 months and no solution was found (multiple versions of vmware tools tried, some dll's disabled, the only solution is to disable the tools vmware service, and we require vmware tools on these VMs for some API calls). I'll try running on ESXi 5 next but it will be quite some time before we are ready to make the upgrade.

Reply
0 Kudos