VMware Cloud Community
admin
Immortal
Immortal

Letter from Paul Maritz & Patches Posted

A letter from VMware CEO Paul Maritz has been posted over on the Console blog.

Please go read the whole thing, but here are his closing paragraphs:

We are doing everything in our power to make sure this doesn't happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

Tags (2)
Reply
0 Kudos
35 Replies
esiebert7625
Immortal
Immortal

It's good to see VMware publicly addressing the issue, I guess only time will tell if their QA process improves as promised.

Eric Siebert

VMware Communities User Moderator

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Check out my website: VMware-land

Read my virtualization blog: SSV Blog

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply
0 Kudos
jasonboche
Immortal
Immortal

It's a good update from the CEO. The first thing I noticed was the stern-looking photo of Paul. It's a good response to the current atmosphere. Very serious looking. Not the smiley CEO photo. Well played.

Thank you,

Jas






[i]Jason Boche[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
maishsk
Expert
Expert

Just a personal thought here...

I wonder whose head will role here ?????

And Don't you love it when Vmware prviodes you with Job security?

Been a busy last 24 hours...

But time to calm things down....



Maish

Systems Administrator & Virtualization Architect

Maish Saidel-Keesing • @maishsk • http://technodrone.blogspot.com • VMTN Moderator • vExpert • Co-author of VMware vSphere Design
Reply
0 Kudos
COS
Expert
Expert

Can someone please explain to me why a "FREE" product has a friggin expiration date? It just does NOT MAKE SENSE!

Even worse, is a Licensed version one has paid for has this expiration "BUG".

YES IT IS A BUG!

QA missed the big fat cock roach in their process.

Here's what I want. I want install Enterprise binaries that has NO expiration. If our agency paid in excess of $15,000.00 to license our servers (Enterprise version) the friggin thing should never ever expire.

I feel like VMware does not trust it's customers eventhough they have already spent their thousands of dollars on their product. MS 2003 Server does not expire does it? NO.

Enough with this garbage about software piracy and protecting your product, the same old boring played out song. If we paid for it it should never expire. VMWare's method is basically saying "you're all pirates and eventhough you paid up the a(insert letters here), I still don't trust you worth doodoo".

That's why this issue is here.

Hmmmm....

This scenario should sound familiar to VMware, remember the "FREE" vmware server release?....tick....tick....tick....

Reply
0 Kudos
COS
Expert
Expert

Personally I consider job security a functionally working envoronment because I set it up correctly using the thousands of dollrs spent from tax payers money with software that works flwlessly.

Chaos is not job security. That's just me because I don't get overtime pay.

I think the new CEO's head should roll and EVERYONE who purchased the ESX 3.5 Enterprise (like we did) should get something back.

Reply
0 Kudos
esiebert7625
Immortal
Immortal

The BETA version had the expiration as is common with Betas because they do not want you using it past a certain date after it becomes a final build. The final build shouldn't of had the expiration in it. And you do own binaries with no expiration, it's called ESX 3.5 Update 1 or any of the previous ESX versions. It was a mistake plain and simple, if you've read the letter mentioned above it did explain this. Go download the new version of ESX 3.5 Update 2 when it is re-released and you will again have the latest ESX binaries with no expiration.

Eric Siebert

VMware Communities User Moderator

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Check out my website: VMware-land

Read my virtualization blog: SSV Blog

-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=-

Reply
0 Kudos
TomHowarth
Leadership
Leadership

All beta's have expiration code in them, that is standard practice. it is too stop nefarious people running Beta code past the sell by date (ie GA) and to make sure that GA code is used, GA code is usually faster as it does not have all the de-bugging code running in it.

I don't understand where you think VMware is accusing you of being a pirate!! (but if the cap fits Smiley Wink ) Yes VMware missed the relevant code, Bad QA and processes need to be looked at and issues addressed. but stop kicking them, I think they get the message. things like this has happened with other vendors and I have not seen their CEO writing a public applogy and statement (think Vista and the "Oh my got my update manager no longer works bug"). and 36 hours for a turn around and fix is impressive.

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
Reply
0 Kudos
COS
Expert
Expert

"And you do own binaries with no expiration, it's called ESX 3.5 Update 1"...

So what your saying is that pre Update 2 VMware trusted us then for for some crazy a(insert letters here) reason in Update 2 the trust was broken?

Are you also saying that someone put this BUG into our Licensed copy of ESX 3.5 because it wasn't in my Licensed U1.

I was fortrunate in that my lazy a(insert letters here) put off patching because I'd rather let the newbies who are interested in getting the latest and greatest get slammed.

My gripe isn't that we lost productivity, it's that the tax payers money spent was on a product that has blatently showed a major flaw. "Flaw" meaning not only how the product works but on how the products development integrity is flawed.

This has put a major wrench in the push for virtualization. Too many eggs (VM Guests) in one basket (ESX Host). VMWare just made it more difficult for me to make the sell to virtualization.

Reply
0 Kudos
TomHowarth
Leadership
Leadership

Right and the next time you make a mistake at work your boss gets fired, ah-emm. to err is human to forgive is divine.

Mistakes happen. The thing to look for is how the lessons have been learnt. and how a company responed to the problems. VMware have reacted very well and on the whole kept us informed of progress. it is sometimes dificult to remember to tell somebody the the fire is nearly under control when you are still pouring water on it, you do not want to leave it to make the call as it may well flare up again.

Yes people are angry about the issue, but it is time to calm down and get the patching done.

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
Reply
0 Kudos
COS
Expert
Expert

Tom, your sooo totally wrong. If I do something wrong NOBODY gets fired and the blame is on the "Process" or lack of training not the person. Government worker, remember, one basically has to sexually offend or kill someone to get fired... 🐵

I totally agree to err is human. In a recent post I mentioned this just shows VMware is "Human". 2% of everything a human does is in error.

Now, since my test environment was just rebuilt today I have the luxury of rebuilding it but I want the .iso file that has the latest and greatest update. Do you have an eta on that.

Oh, did I mention i'm a lazy a(insert letters here) and I don't like patching.

:oP

Reply
0 Kudos
TomHowarth
Leadership
Leadership

"And you do own binaries with no expiration, it's called ESX 3.5 Update 1"...

So what your saying is that pre Update 2 VMware trusted us then for for some crazy a(insert letters here) reason in Update 2 the trust was broken?

Are you also saying that someone put this BUG into our Licensed copy of ESX 3.5 because it wasn't in my Licensed U1.

No that is not what was being said. ESX3.5 Update 2 would be sent to core customers for external beta testing as per all Beta Code it contains time out code. the removal of this code prior to GA was missed. I know you work for the govenment, but beleive me this is no conspirisy, just a mistake. If you have ever been involved in Beta testing and tried to run a Beta after GA you will find it has timed out.

I was fortrunate in that my lazy a(insert letters here) put off patching because I'd rather let the newbies who are interested in getting the latest and greatest get slammed.

in a round about way, I agree but I worry about your tone Smiley Happy

My gripe isn't that we lost productivity, it's that the tax payers money spent was on a product that has blatently showed a major flaw. "Flaw" meaning not only how the product works but on how the products development integrity is flawed.

so you have never had an issue with Windows OS then, or Novell in the day. or any other peice of software. I have been using ESX for nearly 4 years now. I have had 2 major issues. in that same time I have lost count of the issues I have had form other Vendors. I still install and recommend the other vendors to customers, I see no reason why I can not continue recommending VMware.

This has put a major wrench in the push for virtualization. Too many eggs (VM Guests) in one basket (ESX Host). VMWare just made it more difficult for me to make the sell to virtualization.

Not really, it is the same message. VMware may have made it more dificult to sell VMware but not virtualisation.

Tom Howarth

VMware Communities User Moderator

Tom Howarth VCP / VCAP / vExpert
VMware Communities User Moderator
Blog: http://www.planetvm.net
Contributing author on VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment
Contributing author on VCP VMware Certified Professional on VSphere 4 Study Guide: Exam VCP-410
Reply
0 Kudos
finebanana
Contributor
Contributor

I just patched my EXSi host and all is good now. :smileygrin:

Reply
0 Kudos
j-swift
Contributor
Contributor

It seems like there are a few things to learn from this.

License problems are very serious - in all cases the code should warn for 1-2 weeks before expiration without taking any action. This happens all the time with shareware and other mainstream applications. Only an idiot would write sloppy code which provided no warning, but simply stopped the product working.

Seems like some people have been running their Virtual Centre/Infrastructure as a VM on their hosts. I think, to be fair, VMware said this wasn't a good idea. Keep one on there as a backup in case your physical machine goes down, but have it as a normal install on a separate server. So some haven't been following VMware's advice on this.

Nobody should be running all infrastructure services on their ESX hosts - you should always have at least one DC on a real physical machine, and DNS should be there too.

VMware have to take the blame for recommending people use their VMware tools to timesync guests. I clearly remember them stating that this was efficient - but in the end it's caused a problem. The moral is to use virtualisation (applied to all products not just VMware) for what they are good at, and leave DC, timesync as separate infrastructure, and keep your VC on a separate physical machine.

But the main point is to never again have timeout code in your product unless it gives warnings 2 weeks BEFORE it stops working - that time you won't be caught with your trousers down. I dread to think what VMware engineers blood pressure levels were overnight !

Jonathan

Reply
0 Kudos
estadler
Enthusiast
Enthusiast

the following was in the email i got this morning, does this mean there will be a solution that does not require the host being put in maint mode or a reboot? you would just apply patch, problem resolved and life goes back to being good?

"We are currently testing an option to apply the patch without requiring VMotion or VM power-off and re-power-on at the point of patch application."

Reply
0 Kudos
sgorham
Contributor
Contributor

I have to give some kudos to Paul Maritz and VMware for this. Yes, they screwed up and they owned up to it. I was impressed where he pointed out the two areas VMware failed in the process.

It is refreshing that a comapny actually takes responsibility for the bad as well as the good.

Reply
0 Kudos
Dollar
Enthusiast
Enthusiast

Two Things:

  1. VMWare has to improve their communications. Too many of us learned of the issue WAY after it was known at VMWare, and we learned of the issue because when we came in yesterday morning things were broke. As an Enterprise customer, had when been notified earlier, at any point of the previous day or night, we could have taken actions to further minimize the impacts. In addition, we had to "discover" the cause of the issue on our own; with (for most of the day) the best source of information being a Forum thread, not "official" information from VMWare. Sorting through reality in a public forum thread can be problematic.

  2. It's just my opinion, but the occurrence of this issue, and actions subsequent to its discovery, are an indication that Marketing and Sales forces have taken control of the company. I understand that in order for a company to grow, marketing and sales play a major role, but they serve little benefit if the product is lacking. And if these forces cause "Go Fever" in getting new products and/or updates out the door, the product WILL BE lacking. Another symptom of this are indicated in the initial communications from VMWare, where priority appeared to be placed on getting updated ISOs and TARs released.... In other words, get the new product out the door first. This apparently changed later in the day with the "Express Patch" coming first. If this is an incorrect understanding of the sequence of events, then it goes back to "Issue 1", communication.

All of that aside, I remain a VMWare proponent. VMWare Infrastructure 3 is an "Enterprise Ready", stable product and this incident should have no bearing on decisions of its use. This specific incident resulted in more "Risk" than it did in actual problems and qualified individuals quickly discovered workarounds for the problems that did occur, primarily because of the flexibility provided by ESX Server.

Reply
0 Kudos
THP
Contributor
Contributor

I think it is vital that VMWare learn that it is RELIABILITY that puts them head and shoulders above other providers of virtualisation.

I have had production services running for several years without interuption on ESX 2.5.x and even GSX.

Even before this current problem the 3.0/3.5 releases have been a little less stable than the older versions which I can only attribute to a pressure to get new and marketable features out the door as fast as possible.

I would much rather wait an extra few months for extra features so I can be secure in the knowledge that it won't blow up on me.

I appreciate that VM have more competition than before but I would imagine that yesterdays events will not have helped them and will have been a champagne day for their competition's sales teams.

Obviously there is a burden on us end users to test our patching before putting critical services on it - but I can't imagine that many people test the consequences of rolling clocks back and forwards before applying a release?

Reply
0 Kudos
jasonboche
Immortal
Immortal

VMware is clearly the virtualization market leader. I don't know that anyone would argue that. Other vendors in the territory are in "catch-up" mode with the vision that some day they will be able to match product offerings with VMware.

As part of the fallout from recent events, I witnessed a few comments from this and other sites and blogs generally eluding to "Goodbye VMware, I'm switching to Brand X!"

If we role play the theory that some folks talk about, blaming VMware's product rush to market as the root cause to the slip in quality control, think of what those who are abandoning ship are subjecting their environments to: Migrating to a competitor who's current product offerings force them to bring their product to market at a much faster pace than VMware is today just to catch up. Can we imagine what's in store for them?

Something to think about.

Jas






[i]Jason Boche[/i]

[VMware Communities User Moderator|http://communities.vmware.com/docs/DOC-2444][/i]

VCDX3 #34, VCDX4, VCDX5, VCAP4-DCA #14, VCAP4-DCD #35, VCAP5-DCD, VCPx4, vEXPERTx4, MCSEx3, MCSAx2, MCP, CCAx2, A+
Reply
0 Kudos
RParker
Immortal
Immortal

> VMware publicly addressing the issue

Not just publicly, but quickly and efficiently as well. This shouldn't have happened in the first place, but at least they put fourth the effort and made the change and everything seems to be working.

Not like SOME companies which shall rename nameless (Intel -- P60 Pentium) they saw fit to dodge it first, THEN address once someone proved they were at fault.

I saw VM Ware did the right thing and they deserve aknowledgement for a 24 hr turn around on a wide spread problem.

Reply
0 Kudos