Need Help!! Cannot access hosts via vcenter, webclient or directly using vsphere client. ESXi 5.5

[sheel]

Hi everyone,

yesterday morning I had a VM switch itself off after a failed back up using VDP 5.8.

I tried to turn it back on and got an error -14 (broken pipe)

I tried a few things with this VM like changing hosts and changing datastore but I kept getting the same error no matter what I did.

All of a sudden both my hosts (I have 2 in the cluster) became "not responding" and all the VMs were showing as "disconnected"

I am not able to manage any of the VMs.

I tried restarting the Vcenter several times and restarting the managment network on each host.

nothing has changed.

I tried using my local vsphere client to directly access the hosts using the local root admin account but I get an error saying 503: server unavailable

I enabled SSH on both hosts and I seem to be able to log in but if I try to do a simple command such as listing the VMs I get an error saying once again 503 service unavailable.

when running a managment network test on the hosts they ping all services OK

HAS anyone encountered this sort of behaviour before?

[brunofernandez1]

do you have tried to restart the management agents?

VMware KB: Restarting the Management agents on an ESXi or ESX host

are the VMs on the hosts still running?

[sheel]

Yes have tried to restart management agents but made no difference. test on management network resulted ok for dns servers and default gateway. FQDN is correct.

Yes VMs are all still running.

the VM's running would indicate that there is no issues with network connectivity in my mind...yet I can't connect to manage them!

it's really bizarre

[brunofernandez1]

i had a similar problem problem when our storage admins made some maintenance on the SAN.

we had to reboot each esxi....but wait...maybe someone else can help you to find out the problem.

what are the logs on the esxi server saying?

/var/log/hostd.log

/var/log/hostd-probe.log

/var/log/syslog.log

[sheel]

Yeh I'm tempted to reboot the hosts but I am scared if the issue does not ressolve I will have no way of booting VMs. All the VMs are in production so they can't be down!

I can't access the logs it says permission denied, I'm logged in as the local admin account so I'm not sure why!!!

I think maybe SSO might have something to do with it? but I'm not sure how that could have an effect when logged in locally on each esxi host

[brunofernandez1]

what do you mean with the "local admin"? are you using the root user? of not, try it with the root user

SSO is only used for vCenter authentication an not for esxi. Therefore i don't think that SSO causes this issue...

[sheel]

Sorry by Local admin I was referring to the "root" user.

I agree SSO will only effect authentication in the Cluster environment and should have no bearing on logging in direct to the host either by using the VI client or SSH.

I can log in via SSH but when ever I try to run any commands it gives an error : 503 service unavailable