ESXI5 Vmotion performance issue

[wvanderspek]

Hi,

I have a strange performance issue on our vmware production environment. We are running 5.1 on our 12 Hosts (DL380G6 And DL380G8) machines. Now we just upgraded our entire Citrix farm to xenapp 6.5 and Windows 2008 R2.

With this upgrade we gave the machines more virtual memory (12GB). And now when de vm's are being migrated to a different host we see a period of about 5 minutes with extreme lag and performance issue on the vm.

This can show in when typing text the letters are stopping to appear for a brief moment or the citrix session hangs for about 2-10 seconds.

A little background information.

We use a separate iSCSI network (2GB (2x 1gb trunked) this on our Sans (Netapp fas2020 6x and 2220 2x) (the 2220 are also connected by a 10GB connection).

The DL380G8 cluster is completely connected with the 10GB Network. The iSCSI network is connected to 4 HP 3500 Switches.

On both the clusters (DL360G6 and DL380) we see the same performance issue's on the different types of Netapps (so 10GB or 2GB)

We also tried with empty sans but that didn't make any difference. Our Vmotion LAN is also a standalone network with 1GB Connection.

Any Idea's?

[tomtom901]

Is this only with XenApp? Does a standard Windows 2008 R2 virtual also display this issue?

[jrmunday]

It makes sense that VM's with more virtual memory will take longer to vMotion but I've not seen it degrade as you describe.

How is your virtual networking configured for vMotion - are you using VSS or VDS, and how many physical NIC's are you using?

Do you make use of multi-nic vMotion? This is a relativeley easy and cheap to implement and offers significant performance benefits.

Do you see the same issue if you migrate a single VM, or does it only happen when you run concurrent vMotion operations?

[wvanderspek]

We have it with all 2008 servers, with the 2003 machines this is also happening but much shorter (max 1 minute)

[wvanderspek]

How is your virtual networking configured for vMotion - are you using VSS or VDS, and how many physical NIC's are you using?

We use a standard switch configuration for V-motion, actually for all vmware traffic. We are planning to switch to VDS, but this is taking some time to plan and configure.

Do you make use of multi-nic vMotion? This is a relativeley easy and cheap to implement and offers significant performance benefits.

No not at this moment, but as i mentioned above we are changing to a VDS configuration and at that moment we can use 2 nics for v-motion

Do you see the same issue if you migrate a single VM, or does it only happen when you run concurrent vMotion operations?

When we test this or troubleshoot this problem we are migrating a single machine at a time

[wvanderspek]

Still troubleshooting this issue, sometimes we see a smaller lag on some of our Xenapp servers.

Will test soon with our 10GB network if this is performing better without lag.