epa80
Hot Shot
Hot Shot

Composer Bottlenecking

Jump to solution

We have come across an issue where one of our VDI environments experiences a pretty severe case of bottlenecking when running  a large recompose task. We have pretty much isolated it down to a single composer server/database instance, but, we're unsure of where to look now.

In a nutshell what happens is, say we kick off a recompose of 560 VMs. For the first 20 minutes it seems to run fine. I'd say it does about 200 VMs in those 20 minutes. After that though, we see maybe 40 done over the next 10, then perhaps 5, then maybe just 1 or 2. In a test we ran, it took 5 hours for 560 VMs to recompose, but in another block where we have a separate composer server/DB instance, the same task (same type of VM pool) will finish in ~ 45 minutes.

Block 1 goes to an isolated Composer server, ViewCPSR1. Block 2 goes to its own as well, ViewCPSR2. Each of those servers connects to a SQL server, ViewSQL1, and on that, there is a DB for Block1 Composer, and a DB for Block2 Composer.

Both Composer servers are identical in resources, they both live on the same vCenter/distributed switch, go to the same back end storage, etc. They each provide Composer services for a unique vCenter. In the vCenter where the problem exists, we thought we had spotted the issue, as Block1 had a high usage of CPU for the vmware-invsvc constantly, even when not running a task like a recompose. However, we threw a few more CPUs at the vCenter, the spike went away, yet the recompose bottleneck continued.

One oddity we found: deleting the 560 VMs outright and letting them re-create, seems to run much smoother. The recompose though, it's brutal.

We opened a ticket with support, but, wanted to see if anyone else has seen anything similar.

Thanks in advance.

0 Kudos
1 Solution

Accepted Solutions
epa80
Hot Shot
Hot Shot

We were experiencing high CPU in the vCenter where the bottlenecking was occurring. The vmware-invsvc was through the roof, and it turned out the cause was the root file system was running out of free space due to excessive logging in audit.log. This KB was related to it some.

vCenter Appliance root Partition 100% full due to Audit.log files not being rotated (2149278) | VMwa...

View solution in original post

0 Kudos
2 Replies
parmarr
VMware Employee
VMware Employee

Hello, do you have any updates from the Support Case you had open?

Sincerely, Rahul Parmar VMware Support Moderator
0 Kudos
epa80
Hot Shot
Hot Shot

We were experiencing high CPU in the vCenter where the bottlenecking was occurring. The vmware-invsvc was through the roof, and it turned out the cause was the root file system was running out of free space due to excessive logging in audit.log. This KB was related to it some.

vCenter Appliance root Partition 100% full due to Audit.log files not being rotated (2149278) | VMwa...

0 Kudos