Unified vMotion vmk usage?

JustinL3 · ‎12-28-2012

Has anyone else noticed when migrating a vm between 2 ESXi 5.1 hosts which do not have shared storage that the storage copy portion of the vmotion doesn't use the vmotion vmknic portgroups defined?

I have the following vmk port groups on each host:

vmk0 - management traffic only

vmk1 - vMotion traffic only

vmk2 - vMotion traffic only

When I initiate a vmotion between hosts, I notice all the traffic is initially between vmk0 on both hosts until the storage copy is done. vmk1/vmk2 are only active during the last few seconds of the vmotion when the memory is transferred. (see below esxtop results)

Wondering how to tell the the storage copy portion of the vmotion to use the vmotion vmknics I've defined so I can control the VLAN used as well as take advantage of multi-nic for the storage copy portion as well as the memory copy portion?

The below shows typical esxtop information during the majority of the vmotion while the storage is being copied. Notice it is all vmk0 traffic which is not even specified as a vmotion vmknic.

11:52:42pm up 8 days 8:07, 349 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.01

PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PKTRX/s MbRX/s %DRPTX %DRPRX

33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00

33554434 vmnic0 - vSwitch0 29661.75 15.30 82627.22 934.17 0.00 0.00

33554435 Shadow of vmnic0 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00

33554436 vmk0 vmnic0 vSwitch0 29661.94 15.30 16530.30 900.88 0.00 0.00

33554437 vmnic1 - vSwitch0 0.00 0.00 19.61 0.01 0.00 0.00

33554438 Shadow of vmnic1 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00

33554439 vmk1 vmnic0 vSwitch0 0.00 0.00 0.93 0.00 0.00 0.00

33554440 vmk2 vmnic1 vSwitch0 0.00 0.00 0.93 0.00 0.00 0.00

The below shows typical esxtop information during the last few seconds of the vmotion while the memory is transfered. Notice that vmk0 utilization drops and vmk1/vmk2 are active.

11:55:12pm up 8 days 8:10, 371 worlds, 1 VMs, 0 vCPUs; CPU load average: 0.05, 0.03, 0.02

PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PKTRX/s MbRX/s %DRPTX %DRPRX

33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00

33554434 vmnic0 - vSwitch0 7586.46 4.30 32505.95 373.34 0.00 0.00

33554435 Shadow of vmnic0 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00

33554436 vmk0 vmnic0 vSwitch0 28.61 0.49 34.74 0.09 0.00 0.00

33554437 vmnic1 - vSwitch0 7546.34 3.80 32509.67 373.82 0.00 0.00

33554438 Shadow of vmnic1 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00

33554439 vmk1 vmnic0 vSwitch0 7558.23 3.81 6586.19 360.17 0.00 0.00

33554440 vmk2 vmnic1 vSwitch0 7546.71 3.80 6599.38 360.77 0.00 0.00

vmk1/vmk2 on both hosts can reach each other via vmkping etc. Logs from the receiving host show connections made on both vmk1/vmk2, yet the storage migration still only used vmk0 as shown above. Only the memory copy uses vmk1/vmk2 which are the designated vmotion vmknics.

2012-12-28T23:55:08.189Z cpu14:475203)Migrate: vm 475204: 3273: Setting VMOTION info: Dest ts = 1356738745905490, src ip = <172.21.165.196> dest ip = <172.21.165.195> Dest wid = 0 using UNSHARED swap
2012-12-28T23:55:08.199Z cpu14:475203)Tcpip_Vmk: 1213: Affinitizing 172.21.165.195 to world 475212, Success
2012-12-28T23:55:08.199Z cpu14:475203)VMotion: 2688: 1356738745905490 😧 Set ip address '172.21.165.195' worldlet affinity to recv World ID 475212
2012-12-28T23:55:08.199Z cpu14:475203)Hbr: 3308: Migration start received (worldID=475204) (migrateType=1) (event=0) (isSource=0) (sharedConfig=0)
2012-12-28T23:55:08.347Z cpu0:8771)MigrateNet: vm 8771: 2061: Accepted connection from <::ffff:172.21.165.196>
2012-12-28T23:55:08.347Z cpu0:8771)MigrateNet: vm 8771: 2131: dataSocket 0x410024909fc0 receive buffer size is 563272
2012-12-28T23:55:08.347Z cpu2:475209)MigrateNet: 1165: 1356738745905490 😧 Successfully bound connection to vmknic '172.21.165.195'
2012-12-28T23:55:08.348Z cpu0:8771)MigrateNet: vm 8771: 2061: Accepted connection from <::ffff:172.21.165.196>
2012-12-28T23:55:08.348Z cpu0:8771)MigrateNet: vm 8771: 2131: dataSocket 0x41002490ad20 receive buffer size is 563272
2012-12-28T23:55:08.348Z cpu0:8771)VMotionUtil: 3184: 1356738745905490 😧 Stream connection 1 added.
2012-12-28T23:55:08.349Z cpu0:8771)MigrateNet: vm 8771: 2061: Accepted connection from <::ffff:172.22.165.196>
2012-12-28T23:55:08.349Z cpu0:8771)MigrateNet: vm 8771: 2131: dataSocket 0x410024b23b10 receive buffer size is 563272
2012-12-28T23:55:08.349Z cpu0:8771)VMotionUtil: 3184: 1356738745905490 😧 Stream connection 2 added.
2012-12-28T23:55:08.382Z cpu20:475212)XVMotion: 1768: Allocating pool 0.
2012-12-28T23:55:08.383Z cpu20:475212)XVMotion: 584: Bitmap page: len = 16384, pgLen = 1, bitSet = 5199, bitClear = 11185.
2012-12-28T23:55:08.383Z cpu20:475212)XVMotion: 584: Bitmap block: len = 131072, pgLen = 4, bitSet = 95824, bitClear = 35248.
2012-12-28T23:55:10.567Z cpu20:475212)VMotionRecv: 570: 1356738745905490 😧 Estimated network bandwidth 58.517 MB/s during disk copy.
2012-12-28T23:55:11.730Z cpu2:8240)<6>tg3 : vmnic0: RX NetQ allocated on 1
2012-12-28T23:55:11.730Z cpu2:8240)<6>tg3 : vmnic0: NetQ set RX Filter: 1 [d4:ae:52:b5:b3:63 0]
2012-12-28T23:55:14.996Z cpu20:475212)VMotionRecv: 638: 1356738745905490 😧 Estimated network bandwidth 215.124 MB/s during pre-copy
2012-12-28T23:55:15.347Z cpu6:475203)Swap: vm 475204: 1185: Swapfile already initialized. No work to do.
2012-12-28T23:55:15.375Z cpu21:475212)VMotionRecv: 2386: 1356738745905490 😧 DONE paging in
2012-12-28T23:55:15.375Z cpu21:475212)VMotionRecv: 2394: 1356738745905490 😧 Estimated network bandwidth 218.587 MB/s during page-in
2012-12-28T23:55:15.385Z cpu6:475203)VSCSI: 3734: handle 8198(vscsi0:0):Using sync mode due to sparse disks
2012-12-28T23:55:15.385Z cpu6:475203)VSCSI: 3775: handle 8198(vscsi0:0):Creating Virtual Device for world 475204 (FSS handle 9566254)
2012-12-28T23:55:15.577Z cpu9:475204)VMMVMKCall: 208: Received INIT from world 475204
2012-12-28T23:55:15.579Z cpu9:475204)PVSCSI: 3070: scsi0: wdt=1 intrCoalescingMode=2 flags=0xf
2012-12-28T23:55:15.580Z cpu9:475402)Net: 1858: connected .Windows_2008_R2_Standard_SP1 eth0 to OSS_1113, portID 0x200000c
2012-12-28T23:55:15.586Z cpu9:475402)NetPort: 1380: enabled port 0x200000c with mac 00:50:56:a9:e3:c0
2012-12-28T23:55:15.586Z cpu9:475204)VMotion: 5679: 1356738745905490 😧 Received all changed pages.
2012-12-28T23:55:15.588Z cpu9:475204)VmMemMigrate: vm 475204: 5005: Regular swap file bitmap checks out.
2012-12-28T23:55:15.589Z cpu9:475204)VMotion: 5458: 1356738745905490 😧 Resume handshake successful
2012-12-28T23:55:15.593Z cpu9:475402)Hbr: 3405: Migration end received (worldID=475204) (migrateType=1) (event=1) (isSource=0) (sharedConfig=0)
2012-12-28T23:55:15.593Z cpu18:475208)XVMotion: 584: Bitmap page: len = 16384, pgLen = 1, bitSet = 5199, bitClear = 11185.
2012-12-28T23:55:15.593Z cpu18:475208)XVMotion: 584: Bitmap block: len = 131072, pgLen = 4, bitSet = 95824, bitClear = 35248.
2012-12-28T23:55:15.603Z cpu7:475403)Swap: vm 475204: 3254: Starting prefault for the migration swap file
2012-12-28T23:55:15.619Z cpu7:475403)Swap: vm 475204: 3429: Finish swapping in migration swap file. (faulted 0 pages, pshared 0 pages). Success.
2012-12-28T23:55:15.839Z cpu12:9984)Config: 347: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)

Thank you |

a_p_ · ‎12-29-2012

That's interesting. According to http://frankdenneman.nl/vmotion/vsphere-5-1-vmotion-deepdive/ this should not be the case!?

Please provide the details of your network configuration (IP settings, VLANs, ... of Management and vMotion), maybe some screen shots will help.

Which build of vCenter Server and ESXi do you use?

André