NIC Port Missing After a ReStart

[BernardJL]

This appeared to happen before but we don't remember what happened to fix this.

We are running ESXi 4.0 on an HP ProLiant ML370 G6 which has a Quad Port NIC on the machine. This machine has been up and running fine for the past three months. Today we needed to restart the machine.

When ESXi came up, it did not detect the fourth port on the NIC. So we had three virtual machines that lost all connectivity. We were able to move them to the three other ports to get them back on the network.

So what happened? And more importantly what do we do to make that NIC port come back?

As noted we did see this happen before when we were testing the machine but for somehow the problem could not be duplicated then.

Please help.

Thanks in advance for your assitance.

[kjb007]

Card going bad, maybe? If you're seeing other ports on the same card, then the card was detected, the driver loaded, and made available to the vSwitch. If one port is missing, maybe that chip on the card is going bad? I've seen it happen on blade servers, which is 1 reason I always team my NICs across cards, and not simply within ports on the same card.

-KjB

[rkobiske]

I have this problem with my DL380G6's also. The fix i had was to move the 4 port NIC to a 8x PCIe slot. Then all four ports would then show up.

[BernardJL]

KjB, thanks for the insight. I seems unlikely the port is bad since it has been running right along since we went into production. We did see this problem when initially installing ESXi on the machine. And it seems to us when this first happened we re-started the machine and did something else (maybe remove the cabling from all the NIC's).

We have not logged any errors on the NIC or that particular port.

Since we do not right now have another Quad Port card to place in the machine we are sort of limited in trying another NIC.

I will keep you posted as we move along with this.

Thanks for the quick help.

[BernardJL]

RKosiske, thanks for the quick response.

Well if you have seen this same problem may be we can give it a try. Did you need to move the NIC on all of your DL380G6's to another PCI slot?

Does HP have any comments on this issue?

Let us know, and thanks for the suggestion.

[rkobiske · Accepted solution]

yes, i had them origionally in the 4x slot, and it would only find 3 ports out of 4. I think one time though it detected all four, but upon reboot it only then detected three. Exactly like what you saw. I never opened a case with HP on it as moving it to a 8x slot fixed the issue. We have three 380G6's and it happend on all three. We are using a HP NC364T PCI Express Quad Port Gigabit Server Adapter.

[BernardJL]

RKobiske, sound like our problem.

Although we are not so sure what to do about it.

In Our ProLiant ML370 G6 all the slots are at lease PCIe2 8x slots. There are two PCIe2 16x slots so maybe we can try one of those.

I suppose if you went from a 4x to a 8x the logic would indicate moving our NC375i from its 8x to a 16x slot might work.

This is just odd. I wonder if there is some ESXi driver update that is needed for the NC375i to work properly?

When we can take the machine out of service we will give it a try.

Thanks for any updates.

[rkobiske]

I may be mistaken and we moved ours to 16x PCIe slots then. Now that you mention it, i cant remember if i moved it from a 4x to an 8x or an 8x to a 16x. I'd try the 16x to be safe and see if that works.

[BernardJL]

RKobiske, no problem---good enough for us to give it a try.

We will let you know the outcome as soon as we make the change.

Thanks again you help and experience are appreciated.

[Josh26]

I have a feeling if you run the HP firmware update CD it will recommend an update for the NIC.

[BernardJL]

Josh, thanks for the quick response.

We have run the latest Firmware Maintenance CD Release 8.60 and updated all the BIOS and Firmware before going into production. After getting your message we checked if there was a new release of the CD, but 8.60 is the latest one there.

If you have a link to a newer release or firmware please let us know. Perhaps we missed it or something.

[DSTAVERT]

There are individual updates to firmware. The 8.60 firmware CD is August. Go to the HP drivers section for your server and look under VMware ESX(i) and you should find several updates to firmware. You can also look under firmware for the 4 port card as well.

[BernardJL]

DSTAVERT, thanks for your quick response. We did check the location. However, the only new firmware shown there was for the disk array controller. And that one we will need to create a USB image to install it.

There was nothing listed there for the NIC unfortunately.

Thanks for the help.

[BernardJL]

RKobiske, well that worked perfectly. A bit odd but it works.

We move the NIC from the slot it was in from the factory. We just moved it over one slot to the faster one and rebooted. ESXi 4.0 saw all four ports.

We did a number of warm and cold starts and it worked each and very time perfectly.

It is odd and it was suggested by someone else to check about upgrading the firmware for the NIC and we are going to look into that one. However, at this point we don't see any firmware updates that we have not applied to the NIC.

So with things running smoothly we can work research the firmware issue.

Thanks for being so kind as to share your experiences and help with this issue. Your efforts are appreciated.