ESXi 3.5 update to build 158874 network failed

[hudsonmanx]

We have two HP DL380 G5 servers which came with VMware ESXi 3.5 on USB which is embedded in the server.

Today using VMware infrastructure update it updated to build 158874, it completed successfully and asked for a reboot of the server.

It rebooted fine but since doing so the server now cannot see the network, all configurations are correct I restarted network services and tried the test of management network but nothing worked.

Has anyone else experienced this problem?

Regards,

Mark

[frankdenneman]

Hi,

I think the second onboard nic is the adapter used by the service console.

This happens quite often on HP machines.

Have you tried to place the networkcable in the other onboard nic?

-

---

Blogging: http://frankdenneman.wordpress.com

Twitter: If you find this information useful, please award points for "correct" or "helpful".

[hudsonmanx]

I seem to have found the issue, before updating everything was working great.

Our ESX servers are setup with the following network config;

Vswitch0 = Virtual Machine Port Group (VM Network) & VMKernal Port (Management Network 192.168.35.3) we have 3x Physical Network Adapters assigned to this (vmnic1, vmnic2 and vmnic3)

Vswitch1 = VMKernal (10.0.0.21) this has 2x Physical adapters vmnic0 and vmnic4

Vswitch2 = VMKernal VMotion (10.1.1.1) this has 1x Physical adapter vmnic5

If I remove vmnic3 from Vswitch0 the network comes back to life and everything functions as it should, the moment I introduct vmnic3 back into Vswitch0 we lost contact with the server it cannot be pinged and I cannot access any of its resources.

vmnic 0 and 1 are the embedded network cards on the server

vmnic 2 and 3 is a HP NC360T (82571EB Gigabit Ethernet Controller)

vmnic 4 and 5 is a HP NC380T (NetXtreme II 5706 Gigabit Ethernet)

Seems strange by adding vmnic3 into the adapters on Vswitch0 that everything just falls over.

Has anyone else experienced this kind of behavior?

Thanks,

Mark

[kjb007]

Do you have failback enabled on your vswitch / portgroup settings? I would separate the management away from your vm portgroup, anyway. If you want to pair it, pair it with vmotion, and make both redundant to each other. Check to see if that port is physically in blocking mode. If it is, and you have failback enabled,then the traffic will start going back through it when it comes up.

-KjB

VMware vExpert

[hudsonmanx]

My only concern of moving the management to the Vmotion port group is that its on a different network subnet which our production LAN doesnt have access to.

I guess I just use the same IP on that port group?

[kjb007]

That's how it should be. Are you using VLANs? As long as those two NICs are trunks, you'll be fine, as your portgroup will carry the VLAN tag. If they are access ports, then you have less options. I would still separate it from the others, but you won't be able to create a redundant pair of NICs to the network.

-KjB

VMware vExpert

[hudsonmanx]

So within our HP switch I need to trunk VLAN1 to the VLAN which I am using for Vmotion?

[kjb007]

In HP terms, what you will need to setup are tagged ports. And add both VLANs to those two interfaces. Once that is done, you'll need to edit your portgroups, and add the appropriate VLAN ID to the portgroup settings.

-KjB

VMware vExpert

[hudsonmanx]

So for instance my HP switch has 3x VLANS, VLAN1 is production network (servers & desktops) this is the one which currently has the management IP for VMware.

VLAN100 is for iSCSI storage and VLAN200 is used for VMotion.

So I have added 2 more ports to VLAN200 and tagged both those ports, I plugged in vmnic3 from both VMware servers into those two ports, what else do I need to do?

[hudsonmanx]

I think I have sorted the VLAN, I tagged ports 33-34 in VLAN1 and VLAN200 and that seems to work.

How do I move Management Network from vswitch0 to vswitch2?

[kjb007]

Create a new port on vSwitch2, in the same segment as your current management network, then remove the previous one. Be advised, since the gateway is on the original network, you will need to have access to the server console to update the config. It would be easier to move the vmotion network to vSwitch0, and move the VM network to vSwitch2.

-KjB

VMware vExpert

[hudsonmanx]

Thanks for your reply, obviously if I were to move the VM Network to vswitch2 would that cause downtime?

[kjb007]

I don't see any VLAN ID's in your networking config above, so where are you "tagging" your VLANs?

-KjB

VMware vExpert

[hudsonmanx]

That was before we changed our VLAN configuration on the HP switch, my question was to your comment about changing the VMNetwork to vswitch2 would that cause downtime if we moved it.

I am slightly confused because our management IP is 192.168.35.3 so that we can access ViClient using our production LAN subnet.

If we moved our VMNetwork over to vswitch2 and then made vswitch0 for management and Vmotion I am concerned that Vmotion is on a separate subnet 10.0.0.1 I just wanted to know would moving the VMNetwork cause downtime as we have another identical server which currently has all the VMs running on it so we would have to do the same.

Also would these changes mess up virtual center?

[kjb007]

Changes would not mess up virtual center in any way. Also, you can't really move a portgroup. You'll have to create a new portgroup, and then move the active vm's to this new portgroup, one at a time. This will cause a disconnect from the previous portgroup and a connect to the new portgroup, arp updates, the disconnect time should be similar to vmotion. If you have portfast enabled, you may miss a ping or two, if that.

The difference between moving managment and moving the vm network in this case is if you add a new management portgroup to the vSwitch2, you can't provide more than 1 default gateway. So, you can create the portgroup, and it will work, but the routing occurs from the 1st management network, and not the 2nd. The moment you remove the 1st management network to provide the gateway to the 2nd, your connection to ESX will drop. That is why it's easier to move the vm network, as a connection drop of a couple of seconds won't disconnect your from ESX with no way to get back in without a KVM console.

-KjB

VMware vExpert

[hudsonmanx]

I have now moved the VM Network to vswitch2 as pictured I left the Management Network on vswitch0 as shown.

I have moved VMNIC1 and 2 to vswitch2 so that the production LAN subnet will be available to the virtual machines.

My question is on vswitch0 the management IP of the esx server is for instance in my case 192.168.35.3 which is on the same subnet as the production LAN, however when I go to add the VMkernel for Vmotion and input the Vmotion subnet IP which is 10.0.0.1 it says I cant have to VMkernels on the same vswitch. How do I get around this?

Attached is an image of my network configuration.

[Dave_Mishchenko]

You can't add the 10.0.0.1 IP while you have a vmkernel port with the IP of 10.0.0.21. You would have to delete that vmkernel port or change that IP address to another subnet first before you could use 10.0.0.1 for another vmkernel port.

[hudsonmanx]

my mistake I meant to say 10.1.1.1 all these subnets I got mixed up.

Subnet 10.1.1.1 is on VLAN200 so I guess I just add that to vswitch0 as a new VMKernel for Vmotion and input the VLAN id 200 in the optional VLAN box.

[kjb007]

Correct. You can't have two vmkernel ports on the same subnet, regardless of how many vSwitch you have.

-KjB

VMware vExpert

[hudsonmanx]

Ok, I think I am configured right now attached is my new network setup in VMware.