VMotion on 3i

[BernieLedwick]

Has anyone got VMotion working on 3i? I have two servers (31, sp1; VI 2.5 build 84767) each with 2 nics on a 172.23.1.0/24 network (vswitch0) for general and management access and a separate VMotion nic on a different network (vswitch1) 192.168.253.0/24. When I try and "Configure a host for HA", I get a "Configuration of host IP address is inconsistent on host xxxx address resolved to 192.168.253.1" (i.e. the address of the VMotion nic). I have checked the hosts files and they contain only the loopback address. All addresses are resolved through DNS. There is also a storage iSCSI network on a differnet subnet. Everything works fine if I take out the VMotion network, except that I don't then have a VMotion network. Anyone any ideas what is going on here please?

[VMKR9]

check your vmkernel gateway IP address

[BernieLedwick]

The defailt gateway is showing as 172.23.1.254 on both vswitchs. If I change it on either switch it changes it on both. So I assume that there is only one gateway allowed. As the VMotion network is a VMotion network I don't see that it should require a gateway, although I would like to throw a second NIC in there for redundancy as well, if I ever get it working with one nic.

[weinstein5]

There is only one gateway allowede per vmkernel port - you can have multiple gateways for service console ports -

[VMKR9]

Use the same gateway as your iSCSI network...

[BernieLedwick]

The iSCSI network doesn't require a gateway either. The gateway 172.23.1.254 is correct. What I don't want HA to do is take any notice of the VMotion vswitch (or the iSCSI storage network) when it is looking at the network identity of the service/management vswitch.

[VMKR9]

I believe HA needs to be able to ping the default gateway of the vmkernel to set up correctly, you need to set the vmkernel gateway to something that is pingable. Try that and see what happens

[BernieLedwick]

Thanks for the suggestions, and, yes it does need to be able to see the gateway, but I can connect to the default gateway. That isn't the problem. The problem is that HA for some reason is thinking that the VMotion network address is something to do with the management/service network. Have you got a working 3i VMotion system? If so does it use iSCSI?

[BernieLedwick]

I have regressed to ESX 3.5 on both servers, applied the same confiuration and it works perfectly. Surely someone out there must have 3i and VMmotion?

[andreal]

vmotion worked just fine with VC 64192 and esx3i 65301.

A couple of questions:

- Did you enable vmotion the the 192.168.x.y vmkernel nics?

- What is the output of your esxcfg-route -l? ( Keep in mind that for vmkernel nics, we have one route to the network kernel stack for each subnet. For example. if you have two vmknics that are on the same subnet, then these two vmknics will share the same route to vmkernel.)

- would you please run "vmkping" for the two vmkernel interfaces?

- you could also try to remove the 192.168 vmkernel nics and enable vmotion on management interfaces, and see if it works(this might result sluggish performance but let us isolate the issue step by step.).

[andreal]

Please attach output for vmkping, esxcg-vmknic -l, esxcfg-route -l, esxcfg-vswitch -l, and /var/log/messages from both hosts after you reproduced the problem.

Thanks.

[andreal]

From the desktop where VC(where HA is running) is installed, can you ping the 192.168 vmkernel IP?

[BernieLedwick]

Hi, Thanks for you thoughts on this. However, having reinstalled ESX, I can no longer run the commands you suggested. However, on 3i I can't run those commands on the servers anyway as there is no service console. I need to get the systems out the door for our customer and will run with esx 3.5 for now. We will then build test systems and work on that when time permits. The VMotion network needs to be on a separate switch, and trying to include the vmtion network in the management switch, either on the same or different subnet as the management network, causes no-end of issues. When I get to a solution on 3i I will post the answer back. For now I'm just going to assume 3i is broken. I'm also a bit skeptical (although, I see the resoning) about the lack of a service console on 3i, as it does make it almost impossible to debug issues like this, even with rcli. Thanks again for your input.

[andreal]

It worked for me just fine.

Manament and vmotion/iscsi could be on the same vSwitch(managemnet interface and vmknic cannot be on same subnet for esx3i release). I do not know what your topology looks like, so cannot be of much help here.

As for RCLI, please do let me know if there are commands that are nice to have but not yet being implemented. As far as I know, I could run lots of esxcfg commands via RCLI, and they are pretty much sufficient for most common debugging.

I just tried vmotion on my boxes, and it worked as expected. Management and vmotion interfaces are on the same vSwitch.

host1:

~ # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.17.21.80 255.255.252.0 10.17.23.255 00:1a:a0:0c:73:a4 1500 65536 true STATIC

vmk1 VMkernel3 IPv4 192.168.16.80 255.255.255.0 192.168.16.255 00:50:56:72:55:bd 1500 65536 true STATIC

vmk2 VMkernel5 IPv4 192.168.55.56 255.255.255.0 192.168.55.255 00:50:56:73:66:50 1500 65536 true STATIC <----

---

vmotion interface

~ # esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 64 7 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks

VMkernel5 0 1 vmnic0

VM Network 0 3 vmnic0

Management Network 0 1 vmnic0

host2:

~ # esxcfg-vmknic -l

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk0 Management Network IPv4 10.17.21.68 255.255.252.0 10.17.23.255 00:1a:4b:af:ea:cc 1500 65536 true STATIC

vmk1 VMkernel3 IPv4 192.168.16.68 255.255.255.0 192.168.16.255 00:50:56:73:27:cb 1500 65536 true STATIC

vmk2 VMkernel5 IPv4 192.168.55.55 255.255.255.0 192.168.55.255 00:50:56:7c:ee:30 1500 65536 true STATIC <----

---

vmotion interface

~ # esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 64 8 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks

VMkernel5 0 1 vmnic0

VM Network 0 4 vmnic0

Management Network 0 1 vmnic0

Please let me know your build numbers(esx3i and VC), and I will try it out with your version.