Upgrade to ESXi5 - NIC strange behaviour

[nhoareau13]

Hello,

Currently running a vSphere 4.1 (ESX) environment on a HP Blade c7000 with Flex Fabric infrastructure.

Blades servers are BL460c g7 with CNA (Emulex OneConnect 10Gb/s) for FC/FCoE and Ethernet.

12 ESX nodes. Everything running fine.

I recently decided to upgrade to ESXi5.

So I did first start by upgrading vCenter and sub-functionnality (VUM...) : Eveything's OK.

Then I give a try with 1 blade server :

I did a fresh install of ESXi5 on the blade server, reconfigured it and inserted it to the vSphere Datacenter.

After practicing some tests, I noticed that everything "sounds" working fine, except that I got a very strange behaviour regarding the vMotion Network : this network did not work properly.

I can not either vmkping other vMotion IP addresses (from other blades) or myself...

While looking around I saw that all nics are "connected"

I don't think it could be a driver network as :

- Nic driver for Emulex OneConnect is embedded inside ESXi-HP Customized image

- Other Nic (using the same driver...) are working fine.

ESXTOP does not show any kind of traffic for vMotion interfaces.

Configuration :

vSwitch0 :

Management (vLAN2)

VLan2, 16, 20

Attached to vmnic4 & 5 (active/active)

vSwitch1 :

vMotion (no vlan)

Attached to vmnic0 & 1 (active/stand by)

vSwitch2 :

DMZ (vLAN101)

VLan2, 16, 20

Attached to vmnic2 & 3 (active/stand by)

Does anyone have an idea ?

Thans in advance,

[rickardnobel]

nhoareau13 wrote:

vSwitch1 :

vMotion (no vlan)

Attached to vmnic0 & 1 (active/stand by)

One thing to try on vSwitch1 is to make the other vmnic active and remove the vmnic used at the moment. This will make it possible to see if there is any problem with the configuration on any of these vmnics.

(Later, since you have only the vMotion VMkernel adapter on this vSwitch there should be no gain in having the interfaces in active/standby. You could make both active on vSwitch level and the Vmkernel will make its own selection and failover. This would make make the setup somewhat simpler.

Also later, in ESXi 5.0 you could attach multiple vmkernel adapters in a certain way to vMotion to increase the vMotion performance.)

[nhoareau13]

Hello Richard,

Thanks for your reply,

I have tested to :

- activate other nic and remove used one,

- activate both nics,

- add another vmkernel for vmotion on vswitch1 (with another ip add),

- remove existing vmkernel for vmotion port,

Each time I completely reboot the Server, but it still does not work 😕

Esxtop does not show any kind of trafic on vswitch1.

Since this is a flex fabric (HP CNA) architecture I can not add vmotion to another vswitch/nic without help of m'y network engineer...

Any other ideas ?

Thanks again,

[KamilAzmer · Accepted solution]

Just my one cent info,

I saw that your management console and vmotion in the same range of the subnet, The reason that your vmotion traffic is not being used because of the route for the vMotion traffic is using your vmk0 (management console), because the management console able to work as vmotion traffic, I suggest that you changed to isolated network ip address if you want to you vMotion traffic, you may tried on it.

you must see the route which interface adapter is being used for the ip that you had set for vmotion network.

hope it help you.

[rickardnobel]

KamilAzmer wrote:

I saw that your management console and vmotion in the same range of the subnet,

Very good spotted, this is indeed very likely the reason.

There is also a VLAN number on one of the vmk but not the other, and still on the same IP network, which breaks connectivity.

Just select any unused IP network, like 192.168.180.0/24 and set on all vmkernel vMotion interfaces.

[nhoareau13]

Hum....

Thank you Richard and Kamil,

Sounds that you are right, I will give a try on next week but have to wait my network engineer to be available.

This is a strange behavior since the configuration you saw is the running config for my other vSphere4 nodes.

Is it possibly due to service console ?  (vSphere4 nodes are ESX not ESXi).

Thanks again for your troubleshoot, will keep u in touch,

Nho

[nhoareau13]

Richard, Kamil,

Does this mean that vMotion should have its own gateway ?

[KamilAzmer]

Vmkernel network doesn't required any gateway, and doesn't need any gateway for vmotion network

Sent by Maxis from my BlackBerry® smartphone

[nhoareau13]

Hello Guys,

You were right, I had to modify vMotion Subnet to an isolated one.

What I cannot explain to myself is that in vSphere 4.1 (ESX) configuration (the currently running one), vMotion is working while being on the same subnet...

Anyway it does not really matter as I now have the solution

Thanks again for your help Kamil and Richard !

Best,

[dplema]

Hi All,

thanks for this post.  It's the exact same behavior I'm seeing in the field.  I was told at one point this would be fixed with U1, but alas, it hasn't been fixed.

It works in 4.0, 4.1, but not 5.0.

But thanks for clarifying!  I'd appreciate if we could get some closure from VMware support regarding this.  Doesn't seem right that the network stack in VMware is "lazy" insofar as it only looks at the subnet instead of the actual IP Address for the VMKs.

Take Care>>>Dustin

[nhoareau13]

Hey Dustin,

Thanks for your comments,

What I would like to know regarding this is : is it a prerequisites (or a bast practices…) to configure vMotion Network on an isolated and different subnet ?!!?

[dplema]

I'm sure it is a best practice although I don't have the specific VMware reference at the moment.  In the field we always try to isolate the networks.  However, in a PoC situation where were not performance testing, this bug is a problem.

So now that I know, it's OK, however, it makes us look like we have egg on our face assuming a feature that once worked no longer does.

C'mon VMWARE, fess up!

Take Care>>>Dustin