Exchange server slows down dramatically after migration to virtual machine on ESX 3.5

[gnetaneli]

Hello,

In the last month or so we installed 2 brand new ESX servers and used Converter to to migrate a number of Windows 2003 servers from physical to virtual machines.

All of the virtual machines are working well, in fact, we see performance imporovements on all servers other than Exchange . The virtual machines all reside on a single VMFS3 file system, that is configured on an EMC Cellera NS350 storage array using an iSCSI LUN. The LUN is built on a RAID 5 (4+1) RAID group.

The Exchange server is Exchange 2003 SP2 standard edition, with about 48GB database size, providing services to approximately 200 users.

The Exchange server latency is not apparent from the server it self, but rather from Outlook. When you go to the Calendar option of Outlook and try to create a new appointment with many people, outlook loses connectivity to the Exchange server and it takes a long time before you are able to continue. While this is happening, the Exchange server Disk I/O increases considerably and the number of RPC operations also increase, but CPU and memory are not affected.

When you look into the storage array performance counters, everything looks like its under 10% utilization.

We made tweaks to the Exchange virtual machine (added more memory, made some registry changes, boot.ini changes) and it seemed to help resolve some of the latency issue, but not all of it.

Can anyone suggest a course of action to remedy this situation?

[khughes]

How many vCPU's are you running for your exchange server? I had a discussion thread open on exchange a few days ago, a helpful whitepaper which was posted had some good information about Exchange 2003 and VMware.

[Milton21]

Since you are running all these systems on a Raid 5 with only 4 disks I would bet that this is your problem.

[gnetaneli]

We have 2 VCPUs allocated to the Exchange server.

Keep in mind that before we migrated the Exchange server, it was running on a physical server with with local drives, so the migration to a VM running on a NAS should imporove performance considerably. I realize that if we had other RAID configuration we could improve performance, but this cannot explain performance degradation.

[Milton21]

The storage array counters that you are using. Do they look at one system, or all of them? Alos look at the NIC traffic on the ISCSI.

[khughes]

The performance loss could definately be due to poor performance on your iSCSI RAID, but you may consider dropping it down to 1 vCPU. If you look on your perforance charts for cpu usage you should see that the CPU's don't match eachother.

[gnetaneli]

All of the virtual machines are running on the same VMFS3 file systems, which means that the storage array performance couters relate to all virtual machines. The NIC (Dual NICs) configuration is dedicated to iSCSI (all IP networking is configured on different dual NICs), which means that no other machine is fighting for this resource.

All in all, the networking aspect looks very good in terms of performance.

[Milton21]

I still think it is your disk IO. The other systems would seem fine unless they were doing alot of paging. try running this tool.

http://www.softpedia.com/get/Internet/Servers/Server-Tools/Microsoft-Exchange-Server-Jetstress-Tool....

Also look at.

http://technet.microsoft.com/en-us/library/aa997052.aspx

[A13x]

did you ever solve this issue?

[davver]

Just use perfmon to check this.

Take a look at the average disk queue lenght.

Regards,

Davy

[azn2kew]

Have you check out your NAS I/O throughput performance using IOmeter or using Jetstress as mentioned above to test your Exchange server? Did you follow those tips/tricks for virtualizing Exchange 2007 on VMworld.com guide? Try to reduce to 1vCPU to see how CPU are scheduling and monitor performance both RAM/CPU?

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[azn2kew]

It seems like your storage is not optimize to run well with Exchange or SQL databases and 90% of the performance problems is related to storage architecture, so double check it again. You can do some analysis using the guide here for details http://communities.vmware.com/docs/DOC-10095

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

VMware vExpert 2009

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant