Takes a LONG time to vmotion a VM.

[prutter]

Hi All,

Just a bit curious about some vmotion issues.  It seems that for some reason in the past few months, when a VM needs to get moved to another host it seems like it takes a long time to move it.  In past versions on VMware it took only a few seconds to move any vm no matter what size it was.  It seems like ever since we went to version 5, and now 5.1, it takes a couple minutes to migrate a VM.  All the networking is setup the same way it's always been, switchports are all the same.  Anyone have an idea why this could be happening?  Any comments or suggestions are welcome.  Thanks!

Perry

[homerzzz]

tail the vmkernel.log while performing a vmotion to verify you are seeing the network bandwidth you expect.

example:

2013-12-06T19:00:47.810Z cpu6:1416255)VMotion: 4502: 1386356437578944 S: Another pre-copy iteration needed with 517806 pages left to send (prev2 2097152, prev 2097152, network bandwidth ~691.005 MB/s, 783% t2d)

2013-12-06T19:00:50.886Z cpu3:1416255)VMotion: 4502: 1386356437578944 S: Another pre-copy iteration needed with 300384 pages left to send (prev2 2097152, prev 9, network bandwidth ~680.553 MB/s, 164% t2d)

2013-12-06T19:00:53.472Z cpu16:1416255)VMotion: 4502: 1386356437578944 S: Another pre-copy iteration needed with 105996 pages left to send (prev2 9, prev 29, network bandwidth ~692.835 MB/s, 174% t2d)

[prutter]

Well there seems to be a problem.  Here is what the vmkernel had

2013-12-06T17:35:01.611Z cpu2:503348)WARNING: VMotion: 6323: 1386351353616894 S: Detected 8Ms round-trip latency between source host <192.168.20.110> and destination host <192.168.20.112>, but vMotion is not configured for latency tolerance.  Per

18:01:23.079Z cpu14:131351)WARNING: VMotionUtil: 5028: 1386352935830521 S: vMotion vmknic is undefined.  vMotion will attempt to proceed, but may not reach optimal performance.

Not sure why it says not defined.  I have one nice assigned on every host that has vmotion enabled.  Any thoughts??

[homerzzz]

The first message shows you have some network latency and you are not licensed at Enterprise Plus. That latency can affect your bandwidth. What kind of bandwidth are you getting?

The second message I ignore. I do not know why it shows up.

[prutter]

Not sure about the bandwidth but the two nics in question (192.168.20.110 and 192.168.20.110) are both setup to use the iSCSI network running Jumbo Frames. The .20 subnet is all iSCSI.  Also a little confused why it says no vmknic is not defined.

[homerzzz]

Your vmotion network and storage network are one in the same?

[prutter]

Yes.  Same subnet

[homerzzz]

Well, if they are both sharing the same NICs, when you upgraded to 5, did you implement network IO control? (to limit the vmotion traffic)

Did you upgrade to new hardware when you upgraded to 5? Is jumbo frames configured on all the hosts?

If your port group, virtual switch,uplink, nic configuration and HW carried over when you upgraded to 5, then I would look at the physical network. Other than whats already been mentioned, I do not know what else in 5 would slow your vmotion performance.

[prutter]

No that's not the case.  I was referring to thempointing to the same subnet.  They are bnot using the same nics.  Here are what 2 hosts look like.

HOST 1

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type

vmk1       IP Storage          IPv4      192.168.20.13                           255.255.255.0   192.168.20.255  00:50:56:6d:82:bb 1500    65535     true    STATIC

vmk1       IP Storage          IPv6      fe80::250:56ff:fe6d:82bb                64                              00:50:56:6d:82:bb 1500    65535     true    STATIC, PREFERRED

vmk2       JumboVMotion        IPv4      192.168.20.113                          255.255.255.0   192.168.20.255  00:50:56:69:86:bf 9000    65535     true    STATIC

vmk2       JumboVMotion        IPv6      fe80::250:56ff:fe69:86bf                64                              00:50:56:69:86:bf 9000    65535     true    STATIC, PREFERRED

vmk0       Management Network  IPv4      10.50.5.106                             255.255.255.0   10.50.5.255     b8:ca:3a:ec:b4:ee 1500    65535     true    STATIC

vmk0       Management Network  IPv6      fe80::baca:3aff:feec:b4ee               64                              b8:ca:3a:ec:b4:ee 1500    65535     true    STATIC, PREFERRED

HOST 2

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type

vmk1       JumboVMotion        IPv4      192.168.20.110                          255.255.255.0   192.168.20.255  00:50:56:79:ac:81 9000    65535     true    STATIC

vmk2       IP Storage          IPv4      192.168.20.10                           255.255.255.0   192.168.20.255  00:50:56:74:b3:17 1500    65535     true    STATIC

vmk0       Management Network  IPv4      10.50.5.105                             255.255.255.0   10.50.5.255     84:8f:69:e1:c8:4a 1500    65535     true    STATIC