vSphere 4.1 guest no network connectivity

[JaysonDss]

i've done a fresh install of vsphere 4.1.0, 348481 ESXi on an HP Proliant DL360 G5 server which has dual NICs.

The install appears to run through correctly. Both NIC ports are plugged into a cisco 2960 router, I have vlans setup, 1 for the local traffic and 2 for storage traffic/SAN.

I noticed that if I have both NICs plugged it, on the first boot after installation it will not get an IP address from the DHCP server. If I unplug the SAN nic it will grab an IP and work fine from that point forward with both NIC's plugged in. I am able to connect with the vsphere client, I can add NFS storage and ISCSI storage with no issues, and even add a second vswitch for the SAN traffic.

I added VM guests that I had running on other hosts to this new host, they will boot up with no errors, but will not have a network connection. I can ping the host from other external PCs, and also ping out from the host console and always get a response. It is like there is some type of firewall that is blocking just the guest VMs.

I also created a new VM guest and installed a new OS and the same situation continues.

On the console screen alt-f12 I am seeing messages when I start up the guest vm it says it is connected to vm network eth0 and gives it a portID, the next line is NetPort: 982: enabled port with mac, it then goes though a bunch of VSCSI: 2245, 2519, 2319  Reset request on FSS Handle, and reset, there are no retries or outstanding commands, this appears to run about 7 times, then I see NetPort: 1157: disabled port and the same port ID at the guest VM..

So, I am assuming this is why it doesn't have a network connection, but can't figure out why?

The baffling thing is, I have another HP Proliant DL360 that is running ESX 4.0.0, 261974, and that works fine and has been our production machine for quite a few years now.

[iw123]

How have you configured you vswitches/nics ? If this is going to be a production server I would configure it with a static address rather than let DHCP do it.

[JaysonDss]

Eventually it will be a production Host/server. I left it at all the default settings to try and keep things simple while trying to get it up and running.

I had the vswitch settings at the defaults from when it was first installed. I only had 1 NIC plugged into the switch, so vswitch0 was only showing  vmnic0.  After you asked the question, I went and plugged in the second NIC into the switch. They are both on the same vlan.. I then added the second nic vmnic1 to vswitch0, both are set as active.

vSwitch0 only has  VM Network and

I then powered on the guest machine I installed on the local storage and that now had network connectivity, not sure why it wasn't working when I only had vmnic0 plugged in.

I then added from the NFS storage one of the vm guests. powered it on and I am seeing the same issue with no network connectivity, and the alt-f12 screen is showing that it is disabling the port. 

Right now, I am coping the guest vm from the NFS store to local storage to see if that makes a difference.

I am connecting to the storage on the same vswitch0 network as the guests are connected to, I haven't seperated the NICs or created a vswitch1 yet.. But I'm thinking since it didn't work with a single vmnic0, it won't work when I move vmnic1 to vswitch1 when I create it.

[a_p_]

Now that you described the issue you have, please elaborate on the network configuration.

• How did you configure the vSwitch? Any non-default settings (e.g. Policy)?
• Which VLAN's do you use? Maybe a screen shot of the virtual network would be helpful?
• Is the DHCP server on the same VLAN or did you configure a DHCP helper?
• How are the switch ports configured? (EtherChannel, spanning-tree portfast trunk) Can you provide the configuration?

Although I'm pretty sure you are aware of this, please take a look at http://kb.vmware.com/kb/1004074

André

[Jawdat]

Hello guys,

We're too seeing the entries like the extract below in the vmkernal logs. Our host is not hanging though and there are no performance issues or anything, the reason we referred to the logs and investigating is because of some possible network disconnections in couple of the VMs causing us problems.

So far I could not find much on the nature of these entries as to when and why they are logged. This thread is the nearest so far to what we're seeing and quite recent, but no clear answer to what could be causing them. I hope we'll get some more updates soon.

Thanks.

ug 25 08:44:25 <esxhost-name> vmkernel: 63:20:14:07.755 cpu2:12512)NetPort: 1157: disabled port 0x200001f
Aug 25 08:44:55 <esxhost-name> vmkernel: 63:20:14:37.375 cpu1:12514)NetPort: 982: enabled port 0x200001f with mac 00:50:56:88:00:05
Aug 25 15:22:35 <esxhost-name> vmkernel: 64:02:52:17.032 cpu3:12484)NetPort: 1157: disabled port 0x200001b
Aug 25 15:22:35 <esxhost-name> vmkernel: 64:02:52:17.032 cpu1:12487)NetPort: 1157: disabled port 0x400000d
Aug 25 15:22:35 <esxhost-name> vmkernel: 64:02:52:17.032 cpu3:12485)NetPort: 1157: disabled port 0x200001c
Aug 25 15:23:05 <esxhost-name> vmkernel: 64:02:52:47.213 cpu2:12487)NetPort: 982: enabled port 0x200001b with mac 00:50:56:88:00:26
Aug 25 15:23:05 <esxhost-name> vmkernel: 64:02:52:47.491 cpu1:12487)NetPort: 982: enabled port 0x200001c with mac 00:50:56:88:00:14
Aug 25 15:23:06 <esxhost-name> vmkernel: 64:02:52:48.653 cpu3:12487)NetPort: 982: enabled port 0x400000d with mac 00:50:56:88:00:16
Aug 25 15:29:39 <esxhost-name> vmkernel: 64:02:59:21.516 cpu6:12496)NetPort: 1157: disabled port 0x200001e
Aug 25 15:29:39 <esxhost-name> vmkernel: 64:02:59:21.516 cpu7:12497)NetPort: 1157: disabled port 0x200001d
Aug 25 15:29:39 <esxhost-name> vmkernel: 64:02:59:21.517 cpu5:12495)NetPort: 1157: disabled port 0x400000e
Aug 25 15:30:12 <esxhost-name> vmkernel: 64:02:59:54.627 cpu7:12498)NetPort: 982: enabled port 0x200001d with mac 00:50:56:88:00:2a
Aug 25 15:30:13 <esxhost-name> vmkernel: 64:02:59:54.966 cpu4:12498)NetPort: 982: enabled port 0x200001e with mac 00:50:56:88:00:10
Aug 25 15:30:13 <esxhost-name> vmkernel: 64:02:59:55.191 cpu4:12498)NetPort: 982: enabled port 0x400000e with mac 00:50:56:88:00:1a

[virtualeye]

Did you ever resolve this issue?

Having same problem, periodic guest network disconnects and near identical log entries as yours.

Dell R610

CentOS 5.7 64-bit

VMware Tools latest (8.3.x)

VMXNET3 vNIC X3 (eth0 is only NIC affected, which happens to handle majority of network traffic)

VMware is great but the number of "did you ever resolve this?" threads on the net is more than a little disconcerting, flying without wings it seems....

[JaysonDss]

I had actually given up on installing the second VMhost and moved onto other projects for a while.  I just recently came back to it and decided to give Vsphere 5.0 a try and see if that resolved the problem.  I had the same exact problem.. I was careful to make sure the hosts were set up exactly the same in the networking layouts.

What it turned out to be was the Router settings. In the Port Role Setup/Smartports I had to configure them as "Desktop" ports. Oddly enough on the working Host server it has ports marked at Desktop and blank for the Smartports setup and it was working.  I do not understand the Router settings enough to know exactly what did it.. But the problem was in the router and not on the host.

[virtualeye]

Jayson, thanks for the reply.

I have no "Desktop" or SmartPort option anywhere that I can see in vSphere client on ESXi 4.1.

Perhaps you're referring to your own physical router? (i.e. not Vmware related)

Keep posted...

Thanks (congrats on solving your particular issue, btw)

Noah

[JaysonDss]

Sorry for not being clear, but yes it was on the physical router, ours being a Cisco 2960 that I had to change the settings on to get things to work properly.

Purely a guess, but the enabling and disabling might have been in relation to the router actually disabling and enabling the ports. I don't recall seeing anything within a VM guest itself that would indicate a port going up and down.  But if you confirm your vmhost is setup correctly..Maybe find a "dumb" Hub or router and see if you can get that to work..

Seems the better things get the more difficult they are to use..

[virtualeye]

Ah, good, thought I was missing something in vSphere.

ESXi host sits behind a Cisco ASA 5505, could be related to the router issue you were having.

ASA logs will hopefully have some information to work with...ESXi host and Linux guest are useless for troubleshooting the issue, just vmkernel error and kernel error in respective logs. Great, now what caused the error? 😉

Having run bare metal linux boxes for years, the switch to powerhouse servers and virtualization has been both interesting and disturbing.  On the one hand you have the feeling of being on the bleeding edge, and on the other, wondering whether or not it's a good idea to be on the edge in production :smileyconfused: