topic Re: Problem with ESXi 5 or Cisco servers (hosts disconnected) in ESXi Discussions

Problem with ESXi 5 or Cisco servers (hosts disconnected)

MartinBergeron — Mon, 06 Feb 2012 13:55:16 GMT

Hi, since we change our servers for our ESXi 5 cluster, we have sometime host deconnection.

Servers specs is : 3x Cisco UCS C200 M2 (2x Xeon X5675 @ 3.06GHz and 96Gb memory)

Some times, without warnings, the host disconnect in vCenter.

We can still ping the host but cannot SSH or log with vSphere Client directly. If I want to log locally, as soon as I type the password and ENTER, it freeze and the only fix is hard restart. (and lost local logs)

While the host is disconnected, guests are disconnected but fully operationnal (remote desktop, ping). We can't vMotion or change settings on the guest since they are (disconnected)

Is anyone already saw that problem, it's really annoying.

Let me know if you need more informations.

Thanks and have a nice day,

Martin Bergeron

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

jimraina — Mon, 06 Feb 2012 21:27:27 GMT

have you did some basic troublshooting like services are port number .and also dont forget that service blocked by any virus or firewall.

____________________
Always desire to learn something useful.

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

marcelo_soares — Tue, 07 Feb 2012 00:59:26 GMT

You may be facing some local disk problems. Add a vMA to your environment and use it to collect the logs from the ESX servers: http://kb.vmware.com/kb/1024122

Also I am quite sure there is a syslog server included on vCenter installation. Use it if vMA does not work.

I had a very simmilar issue where the KB http://kb.vmware.com/kb/1030265 resolved the issue. You can try on a test ESX to check if this makes the problem stop.

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

MartinBergeron — Tue, 07 Feb 2012 15:19:07 GMT

Thanks,

It really seems to be a HDD controller or HBA so I tried that solution http://kb.vmware.com/kb/1030265 but need to wait some days to make sure that servers don't go down again.

I'll let you know in few days.

Thanks again!

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

jsuarep — Thu, 16 Feb 2012 14:50:12 GMT

Hello Martin,

We have the same issue and I cannot resolve it.

We have review http://kb.vmware.com/kb/1030265 but we can´t find the line ALERT: APIC: 1823: APICID 0x00000000 - ESR = 0x40 in the log.

If you have resolved, please help with the solution.

Thanks!

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

MartinBergeron — Thu, 16 Feb 2012 19:52:30 GMT

9 days and still up...

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

Rauldelaflor — Mon, 20 Feb 2012 17:21:44 GMT

Hi Martin,

we have the same issue. After reading your post we applied the solution of disabling the interrupt mapping.

The problem is that we don't see the ALERT message on the logs.

Is your enviroment still OK?

Did you find the ALERT message on logs?

Thanks and Best Regards,

Raul de la Flor

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

MartinBergeron — Mon, 20 Feb 2012 17:26:41 GMT

Hi Raul,

I’m still up since 12 days from now and I didn’t see the ALERT because when the problem occurred and host has been cold reboot, every logs was lost.

But everything seems to be ok now.

I'll update thread in few days to let you know.

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

MartinBergeron — Thu, 01 Mar 2012 15:36:02 GMT

Disable interrupt mapping on the host did the trick for me.

23 days now without disconnecting

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

Rauldelaflor — Thu, 01 Mar 2012 15:42:14 GMT

Thanks a lot!! 15 days with no disconnections!!!

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

marcelo_soares — Thu, 01 Mar 2012 16:05:58 GMT

Good to hear.... bad to Cisco/VMware, they really need to address this thing up...

Re: Problem with ESXi 5 or Cisco servers (hosts disconnected)

CBorg64 — Thu, 01 Mar 2012 21:22:28 GMT

ER 89872 Cisco Bios issue found. From Cisco: "I have an update on this. Today we were able to connect the dots and found that we recently fixed a problem related to interrupt remapping in the VT-d. Apparently, the work around of disabling interrupt remapping may not always solve the problem.

In the past we have seen many different manifestations of this issue, adapter disconnects and sometimes even PSODs.

Contact Cisco to obtain necessary Bios Updates.