ESXi 4.1 lost all network connection

[targus]

I can't connect to my ESXi 4.1 server.

I rebooted into bios to disable the onboard network adapter because of messages in the log file.  When I rebooted, I was unable to connect using vSphere.  I then tried the Test Management Network at the console and it failed to ping the dhcp server.  I tried reassigning network adapters but had same result.

After that I rebooted into bios again and re-enabled the onboard network adapter to see if going back to the original configuration might make the connection work again.  It did not.  Even though the console is getting an IP address from the router the console still can't ping the dhcp server?!?

Wow, I'm really confused and with no connection of any network to the ESXi server I'm dead in the water with my email server offline.

The orignal configuration was 4 network adapters.  1 onboard, a single Intel and a dual Intel.  I can't ping the ESXi server from anywhere and it can't ping the dhcp server that is setting it's default management network IP.  What should I do?

[Hoschi201110141]

First: stay cool, in god and vmware we trust

Second: Don't know what is wrong with your server but when the LAN Interfaces are enabled and should work but they don't.... maybe your Networkinterface's are not anymore asociated with your configured vSwitch(s).

You should first conntact VMware Support... or when you are brave:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100367...

Try unsupported mode and take a look if you can repair the issue with esx*xy commands.

[Dave_Mishchenko]

If you login to the DCUI which NIC is enabled for the management network?  You may need to make a change there to re-enable the original NIC.

[jamesbowling]

I would first attempt to set the Management Network from the DCUI manually with a static IP instead of relying on DHCP to do this.  Once you have done that you can use the Test functionality in the DCUI to have it run some quick ping and resolution tests for you.  That would be the first place to start.

[targus]

I tried several things but nothing is working.

I checked to see if the correct nic was selected in the DCUI.  It wasn't but is now.  It should be vmnic0 which I'm certain is the onboard nic.  It's status in that screen is connected.  

After that I set the IP to static and tried the Test Network.  Failed.

After that I set IP to different static value and did Test Network.  Failed.

After that I set it to DHCP and in my router I changed the IP address for that mac and watched in the DCUI that it also changed.  Test Network Failed.

The DHCP server is clearly setting the IP address in the DCUI.  Yet the Test Network from the DCUI fails.  Why?

Is there something else I can do to verify the configuration?

[jamesbowling]

Verify that your gateway is set correctly for the management network.

James Bowling

[targus]

Yes, the gateway address is correct.

[Dave_Mishchenko]

How complex is your host configuration?  If it is pretty much as is then reset your configuration, reboot and then register any VMs that were running on the host.  I would also check the MAC address for the management port for ESXi.  That's found in /etc/vmware/esx.conf.  If it matches the physical MAC for vmnic0 you might want to change that.  I would also flush the arp cache on your management PC and switch.

[targus]

Dave Mishchenko wrote:

How complex is your host configuration?  If it is pretty much as is then reset your configuration, reboot and then register any VMs that were running on the host.  I would also check the MAC address for the management port for ESXi.  That's found in /etc/vmware/esx.conf.  If it matches the physical MAC for vmnic0 you might want to change that.  I would also flush the arp cache on your management PC and switch.

It's not terribly complex, I'm willing to rebuild it if that is the way to go.

Questions:

How difficult is it to register VM's that are on the local datastore?

Will I have to set up the local datastore drive again?

About flushing the arp cache:  Where is it?  If on external switch then not a problem becasuse I'm not using managed switches.

[Dave_Mishchenko · Accepted solution]

For the system reset of ESXi it'll just reset the host configuration but the datastore will remain (along with it's contents).  After the reset (root password is blank / IP is set by DHCP)  you'll login, browse the datastore and find the VMX file for each VM.  Right click on the VMX file and select Add to Inventory.  If you created any port groups you'll want to recreate those before you add the VMs back.  The VMs will have a reference to the old port group by name (case sensitive). For an unmanaged switch i'd just cycle it as well as clear the arp cache on your PC.

[targus]

Dave Mishchenko wrote:

For the system reset of ESXi it'll just reset the host configuration but the datastore will remain (along with it's contents).  After the reset (root password is blank / IP is set by DHCP)  you'll login, browse the datastore and find the VMX file for each VM.  Right click on the VMX file and select Add to Inventory.  If you created any port groups you'll want to recreate those before you add the VMs back.  The VMs will have a reference to the old port group by name (case sensitive). For an unmanaged switch i'd just cycle it as well as clear the arp cache on your PC.

Ok.  But before I reset I have one more question.  This problem started after I had disabled the onboard nic.  Since I'm resetting, can I just go ahead and disable it now?  And if I do will ESXi renumber the remaining nic's and pick one for console to use?

[Dave_Mishchenko]

Yes,  ESXi will just use the 3.  What sort of hardware are you using?

[targus]

Dave Mishchenko wrote:

Yes,  ESXi will just use the 3.  What sort of hardware are you using?

Thanks.  I got the email flowing again and the onboard nic is disabled.

I'm using a white box server.  The onboard network adapter is a realtek 8111DL that was supposed to work but I was seeing problems in the log and my NFS datastore was disconnecting randomly.  So my first step in troubleshooting it was to eliminate the nic.  I wasn't expecting it to blow up my server in the process.