ESXi 4.1 and SQL Performance

[lbenton]

Hello,

We just finished migrating our vm's from old IBM hardware with Xeon x5460 processors to a new Cisco UCS solution running with Xeon x7560 processors and have noticed the performance of that versus our old hardware is a lot worse, particularly for our SQL 2008 R2 server.  SQL jobs and queries run on average 3 times longer.  I've messed with a lot of settings trying to find the culprit but nothing has had an impact.  I can get closer to same performance as the old servers by upping the vCPU count to 4, but to be honest we were hoping for a much larger improvement.

The only thing that I can think of at this point is perhaps the processor frequency difference may be the cause.

I do have cases open with Cisco and VMWare on this, but I wanted to see if anyone else has experienced a difference in performance of their vm's under similar circumstances and if they ever got a resolution.

Thanks in advance,

Lewis Benton

[idle-jam]

have you try to look into the disk configuration? it could be the culprit.

[lbenton]

Yep, we've looked at the disk (along with VMWare) and didn't see any bottleneck there.  Also, if we power off the vm and vmotion it back to the older hardware (same storage) it performs faster.

[idle-jam]

i would advise turning on esxtop and look into the values and see which is causing the delay. http://www.yellow-bricks.com/esxtop/

[lbenton]

So far nothing jumps out inside ESXTOP. The vm is getting the resources it needs, when requested.  Also tried reservations with no effect.  VMWare is analyzing some performance snapshots I took as well so maybe they'll see something.

Microsoft came back and said the SQL queries are using a lot more CPU though on the new servers than the old servers.

[alefestaedist]

Hi there, I was digging around the net and found this post on the technet forums that could help you.

http://social.msdn.microsoft.com/Forums/en/sqldatabaseengine/thread/e94641b4-2c7a-4ee5-84b6-d1007c76...

Alex

[lbenton]

Interesting.  I'll have to point that out to Cisco.  I have tried vmotioning the SQL vm to other blades with no luck, but that doesn't mean they couldn't be a bad batch.  We did buy them at the same time.

[lbenton]

I apologize to anyone who has had similar issues but the issue took several months to resolve.  The issue boiled down to the processor C1E state not being properly controlled by ESXi. 

Cisco, at the time, didn't allow the C1E state to be disabled in BIOS and ESXi 4.1 was not correctly controlling it so the CPU speed was being throttled.

Cisco released a BIOS update for the B230 blades in the middle of last year that allowed the C1E state to be disabled.  Once we did that, performance was as expected.

I think Cisco and VMWare came to the joint conclusion that the ESXi was at fault for not correctly controlling that state but the reason why ESXi didn't control it correctly (and why only on Cisco servers) I don't think has been determined.