Vmotion killing Prod Network

[fastpud]

Hi,

I have 2 x ESXI 4.1 host clustered and this morning when migrating over a number of test server from one host to the other, my production network was swamped affecting end users access to critical systems.

The hosts are configured with 3 NICs on the Production Virtual Switch and 2 NICs on the vMotion Virtual Switch. Vmotion is disabled on the Production Virtual Switch but management traffic is enabled. The virtual switches are in different address spaces.

I expected all vMotion traffic to go over the vMotion NICs / switch and not to swamp the production, but clearly this did not happen. There was little or no traffic on the vMotion NICs for the duration of the migration.

Have I missed some basic configuration or am I misunderstanding how vMotion works?

Any help is appreciated as this can't happen again.

Thanks in advance.

[schepp]

Hi,

had the same problem some weeks ago when patching my ESX and ESXi servers.

The network wasn't usable for 10 minutes, wireshark showed insane network-activities between the two vMotion IPs and only 1% of other IP pakets were able to go over the network.

fastpud wrote:

I expected all vMotion traffic to go over the vMotion NICs / switch and not to swamp the production, but clearly this did not happen.

Well I guess all those NICs are connected to the same phyiscal switch? Do you use VLANs?

I took this bug as an opportunity to get some money and buy a new switch to seperate the vMotion Network physically from the productive network. No problems till then.

Regards

[weinstein5]

Please post a screen shot of your ESXi Hosts network configuration - 

[fastpud]

Hi,

I have attached a screen shot of the hosts network configuration.

Now that I re-look at it I noticed that the vMotion & Production virtual switches are in the same address space. Could that be the cause? However no traffic went over the vmotion nics at all which seems strange.

Regards.

[logiboy123]

You seem to be using the same subnet for Management and vMotion. This is not a recommended (maybe even not supported) configuration from VMware.

I would seperate out your vMotion network to it's own seperate VLAN and go from there. It sounds like there are other issues with your configuration, but this is a good starting point.

Check out my ESXi 5 host network diagrams for more info;

http://vrif.blogspot.com/2011/10/vmware-vsphere-5-host-network-designs.html

Regards,

Paul

[rickardnobel]

fastpud wrote:

Now that I re-look at it I noticed that the vMotion & Production virtual switches are in the same address space. Could that be the cause?

Yes, that is the reason. When the IP packet shall leave the host to the vMotion IP on the other host - it will go by the first available interface on that subnet, which is not your wished interfaces. Just put another IP subnet range on the vMotion network. Best security practise is to also separate into an isolated VLAN too.

[fastpud]

Hi,

OK thanks to all who took the time to help.

I am going to make those changes now , separate IP address space and isolated VLAN

Cheers