VMware Cloud Community
rjf7r
Enthusiast
Enthusiast

how to diagnose a failure: hypervisor ran fine for months, then VMs slowed to a crawl, then hypervisor became non-responsive

We have a very small installation running ESXi-5.1.0-799733-standard with a vSphere Essentials license on a single physical machine.  We run 10 very different servers that support our software development needs.

We haven't had any problems with the hypervisor, perhaps for over a year.  However, this morning first we noticed some of the VMs stopped responding, and about a half-hour later the hypervisor itself (accessed through vSphere Client) became unresponsive, including the hardware console itself.

We power-cycled the hypervisor hardware and everything came up fine with no error conditions or messages.  All the VMs are working fine now.

What happened?  How do I find out what happened?  How would I copy the system logs to another system where I can study them?

The "Events" tab in the vSphere Client displays only a limited span of the most recent events.  Is there a way to enlarge this log?  Is there a file behind this display that retains more history?

Thanks for any advice,

Bob

Tags (1)
Reply
0 Kudos
27 Replies
greco827
Expert
Expert

If you are on 5.1 U3, your build version should be 2323236 or higher.  Assuming you are, your driver version should be ahci version 3.0-15vmw, as 3.0-13 stopped being compatible from 5.1 U2.

I would consider updating that driver.

If you find this or any other answer useful please mark the answer as correct or helpful https://communities.vmware.com/people/greco827/blog
rjf7r
Enthusiast
Enthusiast

It looks like I didn't actually apply Update 3 even though I was set to do it, probably because of issues regarding scheduling downtime.

I will do that as soon as I can schedule the downtime.

Thanks.

Reply
0 Kudos
unsichtbare
Expert
Expert

You could run:

#All servers can apply VMware patches by running lines 1-6 and lines 11-12

vim-cmd hostsvc/maintenance_mode_enter

esxcli network firewall ruleset set -e true -r httpClient

esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep ESXi-6

# Determine latest appropriate build of ESXi for your host and insert it in place of ESXi-X.X.X-XXXXXXXXXXX-standard on the next line

esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-X.X.X-XXXXXXXXXXX-standard

#The next two lines are specific to HP servers ONLY

#esxcli software vib install -d http://vibsdepot.hp.com/hpq/latest/index-drv.xml

#esxcli software vib install -d http://vibsdepot.hp.com/hpq/latest/index.xml

#The last two lines need to be run with all servers

esxcli network firewall ruleset set -e false -r httpClient

reboot

I saw this online somewhere and the step by step instructions with screenshots at: Patch your ESXi Hosts from the command line easily and quickly | JohnBorhek.com

+The Invisible Admin+ If you find me useful, follow my blog: http://johnborhek.com/
Reply
0 Kudos
rjf7r
Enthusiast
Enthusiast

So I can update ESXi 5.1 to Update 3 (or whatever is the latest) without bringing it or the VMs down?

Reply
0 Kudos
greco827
Expert
Expert

The host needs to be in maintenance mode, which means no VM's will be hosted on it.  The host should be rebooted, but since it has no VM's, it won't matter.... i.e., no impact.

If you find this or any other answer useful please mark the answer as correct or helpful https://communities.vmware.com/people/greco827/blog
Reply
0 Kudos
rjf7r
Enthusiast
Enthusiast

Makes more sense.

I was thinking of just booting the update DVD.

Reply
0 Kudos
greco827
Expert
Expert

After updating, but before removing it from maintenance mode, check the host out, then apply the new driver.

If you find this or any other answer useful please mark the answer as correct or helpful https://communities.vmware.com/people/greco827/blog
Reply
0 Kudos
unsichtbare
Expert
Expert

This set of commands implies both Maintenance Mode and Reboot

+The Invisible Admin+ If you find me useful, follow my blog: http://johnborhek.com/
Reply
0 Kudos