The express patches have been posted. This thread is long.
We've just encountered a serious bug with our ESX cluster - serious enough that I thought I should post about it here as a prior warning for others running ESX 3.5 Update 2.
The VMWare tech support person we spoke to wouldn't 100% confirm whether this was / would be affecting all ESX3.5u2 installs, but he strongly alluded that it was widespread. For others sake I hope I'm wrong and it's limited.
Starting this morning, we could not power on nor VMotion any of our Virtual Machines. The VI Client threw the error "A general system error occurred: Internal Error".
Further digging lead us to messages like this one in /var/log/vmware/hostd.log, and the log file for any virtual machine we tried to power on or VMotion:
Aug 12 10:40:10.792: vmx| Be sure that your host machine's date and time are set correctly.
Aug 12 10:40:10.792: vmx| There is a more recent version available at the VMware Web site: "http://www.vmware.com/info?id=4".
A call to tech support confirmed this as a known problem with a temporary workaround.
Turn off NTP (if you're using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s "08/10/2008" at the Service Console command line on the ESX hosts.
As soon as the date was reset to the 10th - problem solved.
Note that running VMs were operating fine, this only seems to affect initial VM power-on (including from suspended state) and VMotion.
So, it sounds like a serious licensing bug has crept into 3.5u2. Further testing shows that the problem begins as soon as the date hits 12th August - 10th is fine, 11th is fine, 12th and the problem appears.
There wasn't any real reference to similar problems in the forums as far as I could see, but it's quite possible we're seeing this before most of the rest of the world as we're in Australia, and therefore the date here ticked over to the 12th "before" those in Europe, America, etc.
Hope this helps others... took us a couple of hours to get this far - at least we can power on VMs again though!
Message was edited by: JohnTroyer to add new thread links.
Also in Australia and experiencing the same issue just as you have described. Looking forward to a proper fix from VMware as we have raised a support case as well.
Same happened here.
Except we're unable to change times due to legal obligations. oh well... I'm sure this will be fixed once the US and Europe start calling in too.
Has anyone experienced this problem with a fixed/expiring licence?
> Except we're unable to change times due to legal obligations. oh well... I'm sure this will be fixed once the US and Europe start calling in too.
Tech we spoke to wouldn't commit to a timeframe for a permanent fix, but I'd imagine they'll have to get something out today.
> Has anyone experienced this problem with a fixed/expiring licence?
Yup - our hosts are fully licensed (Enterprise licenses) using a license server (VM, shared with VirtualCenter).
Thanks mate! I just wasted 4 hours at least trying to solve this issue.
This was time right when my RAID failed too. Rebuilt esx and couldnt figure out why it wouldnt start any vm's. Keep us informed if ESX bring a patch out! This is a major problem!
Thanks. I was just about to open a SR on this exact issue.
I can confirm that I've also seen exactly the same behaviour and that disabling NTP and setting the date backwards a week via the client or so did resolve it for the moment.
Time to set a job to keep resetting the date I guess (from external management host) until VMWare can release a fix.
One thing to note is that it seems to stop collection for all the performance monitoring graphs on the VM servers (but actually being able to turn on VMs is more important to me)
Yep we got it too luckily not all of the servers in our DRS cluster are upgraded yet, so the powered on ones are OK but nothing new on the "broken U2" servers, doh!
My uneducated guess is that the beta for U2 expired on 12th August...
Don't forget if the answers help, award points
Just got off the phone with tech support and they've been inudated by calls about this very problem. It only affects ESX and ESXi 3.5 Update 2. Setting the time back a day in ESXi should be ok because if memory serves, VMware Tools will only reset time forward and not backwards.
Been bashing my head on this last couple of hours, thought licence issue of some kind.
ESX 3.5 update 2 misbehaving here as described.
Everything was working fine on Friday.
NTP/Date work around working (phew).
Dear VMware customers,
We are actively working on rootcausing the problem. Once we know the appropriate action to take here, we’ll provide an update.
Apologies for any inconvenience.
The ESX Product Team
If your VMs are already up and running on the server you are fine. You just cannot power up or VMotion VMs. If you need to do something with the server set the date prior to the 12th August
ie From VI CLient, click on ESX Host > Configuration > Time Configuration > Properties and change the time to 10 aug 2008
Then do what you have to on the box and then flick the date forward if you need it to be correct
Don't forget if the answers help, award points
We have the same issue may have to wait for the fix to come out as not sure what affect the work around will have on our SQL, oracle and maxdb servers.
Note that the time being referred to is the time of the ESX host, not the guest VMs running on the host. So, assuming changing the time on your ESX hosts doesn't change the time in your guest VMs, you should be OK...
If your VMs are already up and running on the server you are fine. You just cannot power up or VMotion VMs.
That's a very good point that I should have mentioned in my original post. Our experience was the same, running VMs were fine, but you can't start (even from a suspended state) or VMotion.