ESXi

I am running a whitebox and when I first got it up and running, it was running flawless for weeks. It eventually started locking up and becoming absolutely unresponsive. I can't ping the host IP, I can't log in with VClient, and I can't even get the console to respond to keyboard inputs.

I first thought that I had too many VM's running on it, so I looked at the historical performance graphs in VCenter Server (Eval Version) and it was all low usage of everything (RAM, CPU, Disk). I have upto 5 VM's running (4 2008 64bit and 1 XP 32bit). I started moving the VM's to a VMServer and had it down to 1 VM on the ESXi host. I dont know if I was impatiant but it seemed to run fine for 5+ days so I started moving VM's back to ESXi one by one (using the free converter). It seemed to run fine for a bit, and now it's back to locking up about once every day or two. It had at one time gotten where it was locking up multiple times per day.

My next step was I thought maybe having the two network ports plugged in, something was getting confused as to which interface was used for management without actually doing any additional configuration on the Vswitch. Although I setup the one NIC as the management interface, I figured I try unplugging the second NIC. It's still locking up

I found one other thread about the same type of symptoms, but that persons problems seem to be locking up every couple of hours, at a minimum. I'll go anywehre from 12 hours to 2-3/4 days. My VM's are not resource intensive by any means. I have an AD server with one user, two Exchange servers with one mailbox, and an XP machine doing some media sharing to my network

My hardware is as follows:

Supermicro C2SEA

Xeon X3360

8GB RAM (if someone wants to know specifically what, I can find out)

Intel Pro gigabit PCIx NIC

LSI 3Ware 9690 Raid card

***5 1TB HDD's in raid5 with two volumes configured on that raid to split it in half

***I had to use the update host utility to install the 3ware drivers from their website

***I have all VM's installed on one of the two volumes, the second volume is empty

74GB 10kRPM WD HDD - this is where ESXi is installed, nothing else is on this datastore

I have not done anything much different than default when setting up ESXi or the guests or the network. I have also not done anything for resource allocation because I have not taken the time to learn how to set that up correctly yet. I also have the host attached to an eval version of VCenter Server.

I have looked through the logs and nothing shows up. It doesnt even recognize the fact that it's become unresponsive. The only way to pull it out of this "state" is by doing a hard reset. I also lose all connectivity to the VM guests too.

If anyone has any ideas, maybe I missed something or there is a known issue with one of the pieces of hardware I am using, I'd greatly appreciate any guidance.

Unless you have moved the log location to a datastore the logs you see will be after the fact since logs are lost after a reset. Remember time in the logs will be UTC time not local. You can point the logs to a datastore location so that they survive a reboot. Do that from the Client Configuration Tab / Software / Advanced under syslog. That at least will give you a place to look.

Do the VMs become unresponsive?

There are other posts about issues with resets on the 3ware cards. Have a search through the forums on 3ware.

I have not researched 3wares site. Another note to consider, I have the 74GB WD drive where ESX is installed on one of the onboard SATA ports, not through the RAID card. I would guess that if the RAID card is causing problems, wouldnt it only affect the VM's? Not ESX, being it's not going thru the RAID card? My next step in testing this is to actually pull ALL VM's off the RAID card and put them onto a single 1TB drive connected to another onboard SATA port. This will definitely rule out the card from any question.

The VM's do become unreachable when it locks up. The whole system goes down. Once, I even saw the console screen all "blurry" (for lack of a better term) when I went to reset the server. It had horizontal lines across the screen

Don't count out the RAID card just because ESXi isn't installed on it. I would run any hardware diagnostics you can find. I would run Memtest for one. Let it run for an extended period of time overnight at the very least.

Did you do anything special to get ESXi to install?

ESXi installed perfectly. The only thing that didnt go "as planned" or "out of the box" was in-fact the RAID card. But I installed the manufactures drivers from their website via the host update utility.

I have pulled all my VM's off the RAID and removed the card all together. Now all I have is 3 standalone HDD's. One is the original 74GB "system" drive, second is a 500GB, third is one of my 1TB drives I robbed from the RAID which is the new home for my VM files. I'll see how this runs and then think about doing a memtest.

Is there any known issues with running an Intel G45 ICH10 chipset? I saw some posts about this combination, but those posts were a fairly old. I'd think that if ESXi didnt support it or run on it, it just wouldnt install, just like another box I attempted to install on. It just flat out didnt install.

Well, it locked up agin today. Ran memtest as suggested and it checked out clean

I noticed while i was in the host bios that the system clock was off. I know I had issues with time when I first built the server and got everything installed. Back then, I got all the time corrected and havnt checked it, except for periodic checks in the ESXi configuration.

Today, I set the host bios clock, then booted. ESXi clock was then off, so I corrected. Checked the virtual bios on all 4 VM's. 2 of the 4 were off, so I corrected. Then went into the guest OS's, and only 1 OS clock was off, so I corrected.

What would make the time change and become out of sync like this? Does anyone think that this could be the cause of it locking up every day or two?

Have you set up the Time sync in the Client Configuration tab? Use two or more NTP time sources. PC clocks are notoriously bad. On shutdown or reboot ESXi will sync the software clock to the hardware clock. On start up it will reference the hardware clock until it has completed a time sync with an NTP source and sets it's software clock.

In ESXi some things do have issues with incorrect time but I don't think this would cause lock up.

I would run memtest for an extended period of time.

I would move my log files to a datastore so you have something to refer to when a lockup occurs. The logfiles do not survive a reboot since they write to ramdisk.

If you configure syslog logging then you'll likely be able to capture something that might help you diagnose your lockup issue.




Dave

VMware Communities User Moderator

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.

So, I think my problems may have been related to dirty power coming into the server. I thought it was running thru my UPS, but when I was doing some other stuff, noticed I had it plugged directly into the wall socket. Hmm. I know my UPS for my desktop is always going on battery for a second or two through out the night. Probably happens a couple times per night. Never during the day.I have good quality power supplies on all my PC's, so I dont even know if this truely is/was my issue.

Since I plugged it into the UPS, it's been running solid ever since. Going from no UPS and not being able to stay up for more than 2 days to now on the UPS and been up for 13 days straight now.

To add, I tried going into the settings and changing where the syslog writes to. By default, I dont remember anything being there. I filled it in, but when I go to the datastore, I dont see any text files I can pull down and view. Am I missing something or filling in the wrong field? I'm attaching a couple screen shots.

Try to create the folder first and you can specify it as LOGS/host.log . Note that it will be case sensitive and it will start writing to the file right away after you save the setting.




Dave

VMware Communities User Moderator

Now available - vSphere Quick Start Guide

Do you have a system or PCI card working with VMDirectPath? Submit your specs to the Unofficial VMDirectPath HCL.

It locked up again today. Attached is my logfile. My last thing logged is at Dec 22 14:47:38 UTC time (Page 279 pf 374). This is when my performance logs also cut off this morning. The next event at 16:58:10 is rebooting the server. Nothing really stands out.

I am lost now. I thought for sure there would be tons of errors leading up to the "freeze"

Attachment(s)

Dec 22 11.doc   1.14 MB 1 version