Unable to migrate VM's on iSCSI SAN. Failing at 10% mark. Can vmping to all addresses

[KobusBensch]

Hello everyone

Unable to migrate VM's on iSCSI SAN. Failing at 10% mark. Can vmping to all addresses

I recently installed a new VM setup and I am experiencing the above. My setup is as follow:

2 HP DL380 with 1CPU 16GB ram and 500GB shared space on iSCSI SAN.

Networking setup as follow:

each host has 6 network ports that is load balanced and fault tolrated into 3 VMSwitches. VMSwitch0 goes to 2 trunked ports on a cisco 4510e. On this switch I use VLAN tagging for different servers that perform different functions. This works fine.ie. vlan 5 is for management, vlan 31 is for secure servers vlan 37 is forinternal webservers.

VMSwitch1 is a vmkwenelport for the iSCSI SAN connection. This is set to vlan 200 on the switch so no vlan tagging takes place her. We have segregated ourusual networks from our san network.

vmswitch2 is used for backup purposes and is used in a similar way as vmswitch0. trunkied ports and the backup services is vlan tagged to go to the correct servers.

I am able to ping betwneen the allowed vlans. I can also vmkping to all the san ip addresses from both hosts.

What I am unable to do is to migrate live guests between hosts. It always fails at 10% with the following error:

Migrate virtual machine iws3xx-pop1.mns.tlg.private A general system error occurred: The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration.

I have looked in the vmkkernel log and I get the following:

Oct 9 15:05:38 esxn1-pop1 vmkernel: <7>fn_scroll_back not implemented<7>fn_scroll_back not implemented<7>fn_scroll_back not implemented<7>fn_scroll_back not implemented7:05:04:55.236 cpu2:6534)Migrate: vm 6535: 2094: Setting VMOTION info: Source ts = 1255097135856641, src ip = <172.31.207.21>

Oct 9 15:05:38 esxn1-pop1 vmkernel: dest ip = <172.31.207.22> Dest wid = 6447 using SHARED swap

Oct 9 15:05:38 esxn1-pop1 vmkernel: 7:05:04:55.236 cpu2:6534)Tcpip_Vmk: 1107: Affinitizing 172.31.207.21 to world 6899, Success

Oct 9 15:05:38 esxn1-pop1 vmkernel: 7:05:04:55.236 cpu2:6534)VMotion: 1807: 1255097135856641 S: Set ip address '172.31.207.21' worldlet affinity to send World ID 6899

Oct 9 15:06:53 esxn1-pop1 vmkernel: 7:05:06:10.234 cpu3:6899)WARNING: MigrateNet: 668: 1255097135856641 S: Connect to <172.31.207.22>:8000 failed: Timeout

Oct 9 15:06:53 esxn1-pop1 vmkernel: 7:05:06:10.234 cpu3:6899)WARNING: Migrate: 295: 1255097135856641 S: Failed: The ESX hosts failed to connect over the VMotion network (0xbad010d) @0x0

Oct 9 15:06:53 esxn1-pop1 vmkernel: 7:05:06:10.245 cpu1:6538)WARNING: Migrate: 3267: 1255097135856641 S: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.

The vmx log says:

Oct 09 16:07:38.856: vmx| http://msg.checkpoint.migration.nodata The VMotion failed because the destination ESX host did not receive any data from the source ESX host on the VMotion network. Please check your VMotion network settings and physical network configuration and ensure they are correct.

Can anybody help with this please

[sultanahmed]

Your Vmkernal port should be on the same VLAN as your vmConsole port, try that and let me know if it works.

[KobusBensch]

Does that mean that if my console port is on vlan 5 on vswitch0 i need to create another console port on the iSCSI san network with another IP Address which is on vswitch1?

[sultanahmed · Accepted solution]

No, iSCSI traffic will only work if both vmKernal and vmConsole are on the same vswitch. possibly on the same vLAN depending on how your vLANS work.

[KobusBensch]

That did it.

I had a vmkernel for Iscsi which i also set to the vmotion port.

I have now created another vmhernel port on the same vlan as the vmcosnole and it works.

Thanks.

[sultanahmed]

Cool, don't forget my points :smileygrin:

[KobusBensch]

I think I have done your points. Is that auto assigned when I mark it as answered?