VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
1 ... 36 37 38 39 40 ... 47 Previous Next 704 Replies Last post: Aug 25, 2008 11:18 PM by dipaksharma   Go to original post
Click to view awbc-au's profile Novice 10 posts since
Aug 12, 2008

the difference between the word "licensing" and "time bomb" is semantics in this case... even the vmware logs use the term licensing in the error message they output when the vm can't be started...

I agree that technically the issue is caused by the time bomb, which revokes the "license", which causes the guests not to boot.. the problem is with the licensing code, whether it be a legitimate expired license or a timebomb it's the same thing and the result is the same.... if you forget to renew your license the same problem occurs, if your VC host goes down for a number of days (cant remember exaclty how many) and the host can't contact it this will also occur as well... VMWare wont let you start your guests until you re-license....

Agreed the issue is an inconvience as works arounds can be put in place.. I can set the time back, rebuild the ESX server or even try and uninstall the U2 patch (as someone else mentioned) and all of these things will allow me to get the servers back online.. The issue with all of these things is either technical (ie you can't set time back for other reasons) or resourcing (ie you need to have someone perform those tasks, and I doubt an ESX rebuild can be done in 30 mins as someone commented...) by the time you reattach all the LUN's, setup agents, enable the SSH etc..etc.. this all takes time and needs to be paid for... especially compaines that don't have ESX skills in house and outsource.... Is VMWare going to cover those costs?

Everyone is different,.. for some people it's not a big deal and they have commented as such with all the "dont get to worked up" posts.. for other people it is a major problem and is causing downtime that in some cases they can't control.... either way everyone is awaiting the patch and communication so we can get back to a reliable base again... personally we have very complex servers with scripts and all sorts of stuf that will take a long time to reconfigure... we also have a lot of them and to do it on all would take days.... It's a real shame that benefits for VMWare in having a common code library between the ESX and ESXi products means a time bomb meant for the free version slipped into the production ESX codebase and caused this error. Thats probably the biggest issue for VMWare to deal with, and hence my comment that a change in the licensing restrictions needs to be put in place, especially for the flagship ESX server, which is the product that generates the coin to be able to build the free ones i nthe first place..

Click to view Tibmeister's profile Enthusiast 87 posts since
Jan 9, 2006
Definatly understand that rmumford, please don't take my post that you shouldn't be upset, it just seemed like emotions were getting in the way of some people seeing how to recover from this. Now that I have a confirmed way of getting around this, I got a chuckle out of knowing that the folks at VMware are not super-human, just human. I used to do a ton of programming and had this happened to me once before, so It's kinda funny to see it happen to a major company. Although it very annoying, it is a real possibility when dealing with technology.
Click to view rollin71's profile Novice 11 posts since
Jan 24, 2008

We chose our jobs knowing that times like this always pop up but never at a convenient time.

At least i didn't have any vacation scheduled although i might take some after this.

Click to view sam.god's profile Lurker 3 posts since
May 23, 2007

Helpful Tip: As the load on cluster members change DRS tries to VMotion VMs accross host. With the current problem VMotion is failing with "Operation Timeout" Error messages. To avoid this Select your DRS cluster and set VMware DRS Automation Level to MANUAL. This will stop un-necessary failing VMotions.

After that access the VMs on the Host through console and open VMware Tools, remove sync time with host. For windows machine set time server to external time server in internet time. for Linux use ntpd. Once All VMs time update from ESX stopped then change time of ESX server to 01/AUG/2008. Now we should be evacuate one host (other may get overloaded). Then on evacuted host we will be able to apply the patch and remediate the evacuated host. Set the appropriate time on the remediated host or set it to use Time Server. Now instead of moving orginal VMs back from overloaded host move VMs from another affected host to remediated host. This will ensure that no changes in VMware Tools for those VMs for time sync setting required as detination ESX host time is correct.

We are still awaiting for patch at this moment, so we are unable to test the last part of this activity. But if you can migrate VMs to ESX 3.5 U1 then it should work with U2 also. I do not have any servers of older version than U2. So can any body test a migration of a test VM from affected system to ESX 3.5 U1 host.

If the migrations are successful we will be able to apply express patch on all the hosts one by one this way and remediate the systems quickly.

Click to view APN.NZ's profile Novice 15 posts since
Feb 26, 2007

So Where is the express patch? as promised.

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today.

The target timeframe is 6pm, August 12, 2008 PDT.

Click to view qmcnetwork's profile Enthusiast 37 posts since
Dec 9, 2004

awbc-au,

I regularly build fully configured (firewall, agents, etc.) ESX 3.5 servers in about 20mins. If you have a good ks.cfg script and mount a custom .iso via an iLO, it can be done very quickly. And ESXi is even faster.

Sometimes working hard at being lazy pays off.

Click to view Gonecase's profile Novice 14 posts since
Jul 2, 2008

The patch is out:

kb.vmware.com/kb/1006670

All the best


Click to view joncellini's profile Novice 13 posts since
Jun 17, 2008
Our TAM keeps telling us soon
Click to view missysadmin's profile Lurker 2 posts since
Jul 17, 2006
As pathetic as this entire mess is, and it sure is completely pathetic, what seems worse to me is that VMware did not even take the initiative to send out a support notice to us via email warning our enterprise of this potential issue. Sad.
Click to view FireDog7881's profile Enthusiast 73 posts since
Mar 26, 2007
I bet they didn't plan on the huge amount of traffic to the download of the
patch.
Brandon Meyer, MCP, CCA

Enterprise Windows System Engineer

Quebecor World, Inc

Cell - 630-215-7538

Office - 847-230-1731
Click to view Tibmeister's profile Enthusiast 87 posts since
Jan 9, 2006

awbc-au, I agree with part of you post:

the difference between the word "licensing" and "time bomb" is semantics in this case... even the vmware logs use the term licensing in the error message they output when the vm can't be started...

Not sure if I agree with this, ESX has some of the worst error handling I have ever seen, what the hell is a General System Error tell me? Not much.

I agree that technically the issue is caused by the time bomb, which revokes the "license", which causes the guests not to boot.. the problem is with the licensing code, whether it be a legitimate expired license or a timebomb it's the same thing and the result is the same.... if you forget to renew your license the same problem occurs, if your VC host goes down for a number of days (cant remember exaclty how many) and the host can't contact it this will also occur as well... VMWare wont let you start your guests until you re-license....

Exactly, it's a nasty cause-effect loop. The time-bomb expires the products, preventing the licensing module from running, and with out the licensing, you can't use Vmotion. The time-out is 15 days BTW.

Agreed the issue is an inconvience as works arounds can be put in place.. I can set the time back, rebuild the ESX server or even try and uninstall the U2 patch (as someone else mentioned) and all of these things will allow me to get the servers back online.. The issue with all of these things is either technical (ie you can't set time back for other reasons) or resourcing (ie you need to have someone perform those tasks, and I doubt an ESX rebuild can be done in 30 mins as someone commented...) by the time you reattach all the LUN's, setup agents, enable the SSH etc..etc.. this all takes time and needs to be paid for... especially compaines that don't have ESX skills in house and outsource.... Is VMWare going to cover those costs?

So my ESXi servers take about 30 minutes to install, I have established steps and procedures that are followed. The hardest part is setting up the VSwitches and such, but in comes Powershell to do the leg work for me. I do not believe something as critical as an ESX server should ever have anything other than itself on the box. No extra scripts, agents, etc. That sucker is too critical. Plus, the VMware pipeline only has the ESXi version available, so all these scripts will be a moot point, but that is not a debate I think anyone should have with anyone else, I have no clue what your environment looks like, or what your needs are, or what resources you have available. So with that, I will take my comments and concede the point.

Everyone is different,.. for some people it's not a big deal and they have commented as such with all the "dont get to worked up" posts.. for other people it is a major problem and is causing downtime that in some cases they can't control.... either way everyone is awaiting the patch and communication so we can get back to a reliable base again... personally we have very complex servers with scripts and all sorts of stuf that will take a long time to reconfigure... we also have a lot of them and to do it on all would take days.... It's a real shame that benefits for VMWare in having a common code library between the ESX and ESXi products means a time bomb meant for the free version slipped into the production ESX codebase and caused this error. Thats probably the biggest issue for VMWare to deal with, and hence my comment that a change in the licensing restrictions needs to be put in place, especially for the flagship ESX server, which is the product that generates the coin to be able to build the free ones i nthe first place..

I don't think it has anything to do with ESXi's free release, which was probably part of u2. I think it's just a bad coincidence.

Click to view mjlin's profile Novice VMware Employees 19 posts since
Jun 18, 2007

Dear VMware Customer,


The express patches are now available for download that will resolve the ESX/ESXi 3.5 Update 2 issue which causes the product license to expire as of August 12, 2008. Please go to http://www.vmware.com/go/esxexpresspatches for more information.

Thank You,

The VMware ESX Product Team

Click to view esiebert7625's profile Guru User Moderators vExpert 6,796 posts since
Oct 23, 2006

Not sure if I agree with this, ESX has some of the worst error handling I have ever seen, what the hell is a General System Error tell me? Not much.

I would guess that the VI Client was not equipped to handle that type of situation (time bomb code) and therefore displayed the general system error instead of the exact error message. Normally the VI Client would not have to deal with that type of situation because it should only exist in the beta code.

Eric Siebert
VMware Communities User Moderator
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-
Check out my website: VMware-land
Read my virtualization blog: SSV Blog
-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Click to view akmolloy's profile Enthusiast 45 posts since
Jul 9, 2007
So here's a dumb question... can I suspend the VMs on a Host and put it into Maintenance Mode and apply the patch? I really don't want to change the time on one of my hosts and then Vmotion machines to it. I have on Host that is running some VMs that could tolerate a suspend for the 15 minutes it would take to run this patch. I have no idea if maintenance mode will work with suspended machines... anyone try this?

VMware Beta Programs

Want to be Considered for Future Beta Programs?

Learn More

VMware Developer

Download SDKs, APIs, videos,
training, and more in the Developer community.

Learn More

Developer
Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld
Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

Only VMware ... Delivers Nexus 1000V

Ensure consistent, policy-based network capabilities to virtual machines across your data center.

Learn More

Communities