Can vMotion take a network down?

[kopper27]

hi guys

ESXi 4.1 Update 2 (2 hosts)

vCenter 4.1 Updat 2

I've setup vmware multiples times....

But this time I used 2 vmnics for Management (vmnic0 active - vmnic0 passive) - vMotion (vmnic0 passive - vmnic0 active)

and vmnic 2 and 3 for server traffic....Cool I've done this multiple time....most of the time vmotion and management in the same network as servers.... I know it should be separated for vmotion.....

Ok but this time this is happening....

One day while deploying this enviroment a host had an issue then I putit in maintenance mode....vMotion kicked in.....so this 4 vmotion at the same time took my customer network down.... IP Phones, printers, internet all that had issues...they disconnected and no one could use Phones, internet, and so on and after the vmotion finnished all was OK again.

Any idea what going on? or what should I check?

when I vMotioned a single VM this does not happen only when 4 simultaneously

thanks a lot guy

[weinstein5]

Beside security that is the second reason why you want vmotion seprated - when multiple vmotions are attemtped it can saturate the network impacting time sensitive network traffic like VOIP - 

[kopper27]

so looks like it can happen

thanks

[Josh26]

Just because it "can" happen doesn't mean it should.

What kind of switches were involved here? Saturation to a point that other devices become impacted tends to refer to a small switching backbone.

[vGuy]

to avoid this situation you can look at using NetIOC to limit the vMotion network bandwidth utilization.

http://www.vmware.com/resources/techresources/10119

you can also limit the no. of concurrent vMotions per host:

http://virtualizationreview.com/blogs/virtual-insider/2011/04/limit-concurrent-vmotions-in-vsphere-4...

[kopper27]

it's a cisco WS-C6509-E (R7000) processor (revision 1.5) with 458752K/65536K bytes of memory.Processor board ID SMG1431N060.

Cisco IOS Software, s3223_rp Software (s3223_rp-IPSERVICESK9_WAN-M), Version 12.2(33)SXH5, RELEASE SOFTWARE (fc1)

[rickardnobel]

These devices that got disconnected as printers and similar, they are of course physical devices and the clients - are they also physical PCs? That is, did this vMotion event break connections between physical devices?

[kopper27]

yes physical IP Pones losses their connection and for instance when testing (we went to the customer to try to find out the cause) I got no internet connection....and also they have monitoring system that report all server down since there are no pings

[rickardnobel]

kopper27 wrote:

yes physical IP Pones losses their connection and for instance when testing (we went to the customer to try to find out the cause) I got no internet connection....and also they have monitoring system that report all server down since there are no pings

It looks like you either has something incorrectly configured on the physical switches or that the backplane capacity is way too small and the whole switch goes down when the vMotion interfaces gets highly loaded.

Typically you should be able to get decent switches quite cheap today without these problems (if it is a performance problem on your switch.)

[kopper27]

back at this topic....

the networking guys have checked the Switch and they say it does not look saturated.... CPU load is OK so they says it's weird that vMotion does not impact the Switch but it is affecting all the traffic like I said IP phones, internet, printing....

any other idea guys?

[kopper27]

someone knows if this affects ESXi 4.1 Update 2?

http://www.vmadmin.info/2011/04/vmotion-unicast-flood-esxi.html

According to this

Right now my Management and vMotion are like this (2 hosts)

Hosts 1

Management - 192.168.23.240

vMotion - 192.168.23.241

Gateway 192.168.23.1

Host 2

Management - 192.168.23.242

vMotion - 192.168.23.243

Gateway 192.168.23.1

so I should create a vMotion with 10.10.10.x ???? for instance?

vMotion host 1 : 10.10.10.5

vMotion host 2 : 10.10.10.6

and same Gateway 192.168.23.1

or

something like this might be enough?

vMotion host 1 : 192.168.30.5

vMotion host 2 : 192.168.30.6

Let me know guys

thanks a lot

[rickardnobel]

kopper27 wrote:

Hosts 1

Management - 192.168.23.240

vMotion - 192.168.23.241

Gateway 192.168.23.1

Host 2

Management - 192.168.23.242

vMotion - 192.168.23.243

Gateway 192.168.23.1

So the Management and vMotion is on the same IP subnet? This is typically not recommended, you should absolutely separate them into different IP networks and, if possible, different layer two VLANs.

It still should not bring down half your network, but it could be a good way to start and make your hosts more like "best practice" and I think we should be able to find the solution after you have change this.