KInvestments
Contributor
Contributor

ESXi web ui error 500 (Internal Server Error)

pastedImage_0.png

When trying to access my WebUI, it runs very, very slow and then completely goes unresponsive. When I run developer tools, I'm met with several internal errors(500) as well as 'main.js:379' tied to the error code 500.

Has anyone experienced something similar to this, and if so any solutions or a good place to start looking? I've done some digging on the web with no luck. I havent tried restarting the Server as I cannot guarantee the Servers will come back online, and without Web access there is no way I know of to turn them on IF they dont come up. SSH is also disabled by default, for security reasons and can only be enabled via the WebUI.

to add some context, this happened after we upgraded to

update-from-esxi6.5-6.5_update02

Product:ESXi (Embedded and Installable) 6.5.0

Download Size:486.6 MB

05/03/2018

8294253
KB53097

It was working fine up until then.

Any feedback would be greatly appreciated.

0 Kudos
12 Replies
msripada
Virtuoso
Virtuoso

Is that ESXi connected to vcenter? ESXi webui  queries hostd management agent.

are you facing this issue with one esxi host or multiple esxi hosts and as soon as you update them?

If it is one esxi host, then possible hostd hung or unresponsive. If ssh is disabled, can you take shell access via dcui and restart only management agents or check the hostd log

Thanks,

MS

0 Kudos
KInvestments
Contributor
Contributor

it isnt connected to vCentre.

Just the 1 ESXi host, it's our only host too, in happened within a day of the update.

Thank you for the advice. I will check the logs to find out some more details.

We have dcui shell disabled as well.. so it wont let me in that way either.

0 Kudos
msripada
Virtuoso
Virtuoso

Is the strict lockdown mode enabled on the ESXi ?

Thanks,

MS

0 Kudos
KInvestments
Contributor
Contributor

Probably, all access outside of WebUI is most likely off, the defaults were set and left alone as far as I know.

No way for me check as far as I know.

0 Kudos
msripada
Virtuoso
Virtuoso

Do you have direct ESXi host connectivity? Can you login to DCUI using root?

thanks,

MS

0 Kudos
KInvestments
Contributor
Contributor

There is no login option when using DCUI, only a method of how I would like to connect. I load up TeraTerm, it recognizes the Serial port. And then after selecting connect, I have a black terminal with no prompts or shell UI. pressing the keyboard doesnt move or show a cursor.

0 Kudos
msripada
Virtuoso
Virtuoso

Then probably we can try to connect to teraterm and copy the hostd log after reproducing the issue and check the logs. Also check in rhttpproxy.log if the request is coming to host

thanks,

MS

0 Kudos
KInvestments
Contributor
Contributor

I'm unable to access the hostd log via TeraTerm; Console or SSH. The only entrance we have into the system is the WebUI, and only from the webUI can I unlock/allow SSH/Console. and the WebUI is not responsive, Im unaware of any other method in order to retrieve the hostd logs or rhttpproxy.log

Thank you for your continued help thus far! Smiley Happy

0 Kudos
Finikiez
Champion
Champion

Just power off VMs correctly and reboot the host Smiley Happy

0 Kudos
KInvestments
Contributor
Contributor

That runs the risk of my VMs not working once I reboot. Not sure if I can afford to do that. IF I cant rely on the WebUI to allow me to manually turn them back on.

after waiting through the constant webUI crashes and refreshes, I was able to verify that DCUI is running, though SSH is disabled. that being said Im not sure why I cant actually see a UI for ESXi when I console into the IBM Server.

0 Kudos
Finikiez
Champion
Champion

That means that something goes wrong with management agents.

If you can login with DCUI then go to troublshooting options and try to restart management agents.

But if you can't do that that means that you don't have any option to check what's going on and you have to restart the host.

0 Kudos
KInvestments
Contributor
Contributor

I'm beginning to think it might be related to the physical server itself. As not even the usb keyboards are working, yet the ethernet and VMs on the server are running completely fine.

I got into the SSH, so I'll continue to troubleshoot from there.

I've identified the hostd.log file but not sure what Im looking for. I've seen the error and googled a bit but to no avail.

----

July 23rd, after swapping out the motherboard, the same issue is happening. So it's probably VMWare related.

I think I have found the causepastedImage_0.png

This keeps ticking every second or so, until the webUI eventually comes to a crawl.

0 Kudos