Since converting an MSSQL server to a virtual machine we have seen severe performance drops and an array of unusual errors. Here's the scenario:
3 Dell PowerEdge 2950 servers each configured as follows:
2 PROC, Quad core Intel E5430 2.66Ghz
8GB RAM
2 onboard Broadcom GB NICs (used for service console and admin network only)
1 Intel GB 4 port NIC (2 for VMotion, 2 for Production VMs)
2 QLogic QLA2432 4GB HBAs
EMC CX300 SAN
Admin, VMotion and Production networks are physically separate, each with it's own GB switch.
We are running ESX 3.5 on all hosts with VC 2.5 setup as a VM. HA, DRS, VMotion all enabled. There are six VMs running currently arranged like so:
Host1
WebServer VM (4 vCPU, 3GB RAM, Win2K)
Host2
WebServer2 VM (1 vCPU, 1GB RAM, Win2K3), secondary sites, minimal traffic
WebServer3 VM (1 vCPU, 1GB RAM, Win2K3), not in production yet, no traffic
VirtualCenter VM (1 vCPU, 2GB RAM, Win2K3)
DataImport VM (1 vCPU, 1GB RAM, Win2K), processes data files to load into SQLServer
Host3
SQLServer VM (4 VCPU, 3GB RAM, Win2K, MSSQL2000 sp4) primary backend database
The WebServer, DataImport and SQLServer VMs were all converted using Converter Enterprise.
All VMs are stored on the SAN in LUN A, except SQLServer in LUN B.
All SQLServer databases (.mdf files) are in two RDMs to LUNs C & D.
All SQLServer log files are in the VM.
We operate approximately 200 SQL databases ranging in size from 20MB to 20GB. Most are around 750MB. There is a lot of SQL activity. We receive daily update files from clients to load into their databases. These files are stored in a separate physical server and processed by the DataImport VM which does most of the insert/update/delete functions. The web portals themselves are mostly just queries. Very few inserts or updates. The past couple weeks since converting everything to VMware we have seen a lot of performance issues. Web Portal requests routinely return ASP timeouts while waiting for responses from SQL. We have also seen a lot of other intermittent errors since the conversion like these:
"The RPC Server is unavailable" when accessing some pages.
and
"Microsoft OLE DB Provider for SQL Server error '80004005'
DBNETLIB ConnectionOpen (PreLoginHandshake()). General network error. Check your network documentation."
Both of these are intermittent and will often go away after refreshing a page. We've also seen these creep up on our web server: "Out of process application '/LM/W3SVC/1/ROOT/xxxxx' terminated unexpectedly."
After some monitoring with esxtop, VI client Performance, perfmon, and our switches the network, SAN fabric and memory do not appear to be dragging. The only thing that is jumping out at me is the SQLServer CPU usage. Since converting the VI client performance tab and esxtop is showing consistently above 90% CPU usage when our DataImport programs are running. If I shut them down the importers, usage drops to 30% or below. Also perfmon and taskman in the guest are showing barely 25% usage at the same time esxtop shows 100%.
I'm at a loss of where to look next. SQL Server was able to handle all site and importer requests perfectly prior to the conversion and that was with lesser hardware. Any help would be greatly appreciated.
Thanks.
Tom
