VMware Cloud Community
admin
Immortal
Immortal

Letter from Paul Maritz & Patches Posted

A letter from VMware CEO Paul Maritz has been posted over on the Console blog.

Please go read the whole thing, but here are his closing paragraphs:

We are doing everything in our power to make sure this doesn't happen again. VMware prides itself on the quality and reliability of our products, and this incident has prompted a thorough self-examination of how we create and deliver products to our customers. We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes.

I want to apologize for the disruption and difficulty this issue may have caused to our customers and our partners. Your confidence in VMware is extremely important to us, and we are committed to restoring that confidence fully and quickly.

Tags (2)
Reply
0 Kudos
35 Replies
RParker
Immortal
Immortal

I wonder whose head will role here ?????

First of all it's spelled: R-O-L-L haha.. too much coffee this morning, not enough sleep??!? Smiley Happy

OK, secondly someone made a mistake, not like that woman working @ 911 they burned at the stake for sending the ambulance to the wrong address and it caused someone to die.. Whoever is at fault, I am sure it was a group effort. I am also sure that maybe someone will pay for this with their job, but I for one would be the FIRST to stand behind them and say "OK, you really put your foot in it this time, but hold your head up and just use this as a learning experience. Everything is going to be fine".

Whoever has NEVER committed a serious error in judgement or has never made a mistake in their lifetime, say "I". If anyone reponds that they have never made a mistake.. is LYING!

We ALL make mistakes, the degree to which results play out depends on how bad it was, but we are certainly ALL capable of doing wrong, and I am sure MANY of us have committed MANY mistakes. It's all part of learning, and we should NOT be quick to condemn.

"There is so much good in the worst of us, and so much bad in the best of us, that it ill behooves ANY of us to find fault with the rest of us"

Reply
0 Kudos
RParker
Immortal
Immortal

So what your saying is that pre Update 2 VMware trusted us then for for some crazy a(insert letters here) reason in Update 2 the trust wasbroken?

And how much about progamming do you know? Apparently NONE.

> Can someone please explain to me why a "FREE" product has a friggin expiration date? It just does NOT MAKE SENSE!

> Even worse, is a Licensed version one has paid for has this expiration "BUG". YES IT IS A BUG!

In the first place you posted this SAME EXACT message 3 times. WE SAW it the first 2 times, how many more times do you need to post that same message..

We KNOW it was a bug. No one targeted you because you were using "FREE" version. The BUG was affecting ALL the code. Like Tom said EVERY company puts a built in limit on their software at some point. (Set your date ahead on your computer about 5 years and see what happens to Windows....). It's called "life cycle".

VM Ware just missed it in QA. They made a mistake, they fixed it, so how many more times are you goig to bash them over the head over it? You made your point, now can you manage to come up with something original and not copy / paste previous posts?

Reply
0 Kudos
RParker
Immortal
Immortal

If anything, this proves we need to get some of the community involved BEFORE the product goes to press.

There needs to be an OPEN beta BEFORE release so many of us bleeding edge, testers can try it before ANYONE else gets hit. THEN if it passes the test you can move it to the website for release.

There needs to be another level of BETA. Probably why MS hasn't ever suffered a debacle this bad, is because they OFFER BETA and PRE-Release candidates of their software.. to about 10,000 people... someone is bound to see something the developers missed..

"another set of eyes..."

Reply
0 Kudos
RParker
Immortal
Immortal

....other sites and blogs generally eluding to "Goodbye VMware, I'm switching to Brand X!"

If there was a primary example of "hot air" Microsoft has been the subject of ridicule for 2 decades now.. and they are STILL here. People like to complain, it doesn't mean they will ACTUALLY follow through, and go with something else. Once they realize that there isn't a better alternative, they will calm down. M$ $ucks, Windows is junk.. Yeah well why are you still using it?!?!?

BRAND X in this case is FAR behind VM Ware.. I am not worried. For one thing its like that old saying "There is no such thing as bad press...". If anything VM Ware will get MORE recognition.. This will only hurt the sales for August. September will be record sales... you watch, because.. they FIXED the problem, and it's like ValuJet.. they were scrutinized by the FCC for years.. they finally changed their name.. and How many problems have you seen from Airtran. Almost NONE!

So fixing a process is a wake up call, and the industry will realize this. No one is going to forget this for some time, but HOW you handle adversity is the challenge.

"The mettle of a company is tempered in the fires of adversity".

This is VM Ware's uppercut.. they may have been knocked to the canvass, but WHEN they get back up.. that's when you should take notice. VM Ware will be stronger from this, and we will get better process and certifications as a result.

Reply
0 Kudos
COS
Expert
Expert

3 posts? I remember only 2. Musta been too many beers last night....

Programming yeah did a little bit of that, also did QA code checks. Thats where documentation came in handy with check marks to indicate what features, modules and other stuff was completed and by whom. This eliminated any errors and we could finger point. Can we see your dcumentation?

Well, my posts were of obvious anger and frustration. I have since calmed down. Patched and working for this time.

Just need to thank you forum folks fo all the help and "updates" on the patching, how you did it and what you experienced. Helped out.

Reply
0 Kudos
COS
Expert
Expert

This is just going to make it more difficult to sell the virtualization strategy on folks who do not want to put all their eggs (VM Guests) in one basket (ESX Host). We have many folks in this mindset.

Hmmm....I noticed something. Here's your quote..."we will get better process and certifications as a result."

You work for VMWare?

Reply
0 Kudos
wila
Immortal
Immortal

All beta's have expiration code in them, that is standard practice. it is too stop nefarious people running Beta code past the sell by date (ie GA) and to make sure that GA code is used, GA code is usually faster as it does not have all the de-bugging code running in it.

As you say it is standard practice to put a "kill date" in a beta, there's one very important reason you missed.

with such a date in your beta code, you know "for sure" that after a certain date no more people are going to call you with the beta running, this is important for support as they can then focus on the GA release just like it should. As VMware I would deem that even more important as stopping piracy, as support for beta's has a direct cost, while piracy doesn't (not that I approve software piracy at all)

--

Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
Reply
0 Kudos
audienceofone
Contributor
Contributor

Personally I am impressed with the letter. Being fairly new to the whole VMware world, I am glad to hear someone admit mistakes, and take action to fix it. I worked half a dozen years in the support department for one of the primary backup software companies, and too often code was rushed out the door when it wasn't ready, and there was no response from management, when backups were lost, unable to be restored or cause major issues. Developers make mistakes, sometimes it was their fault other time QA people didn't have the time do really do a good job before being told, "well, it's been released now anyway" Smiley Happy I definately saw a major version be released that was nowhere ready (sort of reminds me of vista :smileygrin:).

Reply
0 Kudos
khughes
Virtuoso
Virtuoso

Personally I wasn't affected by all of the madness yesterday due to we don't go along the cutting edge for this very reason. I was extremely surprised to see the amount of comments that people should be fired, that they were going to leave for Microsoft etc... Was it a big mistake, yes. Is this the reason to jump ship and switch to microsoft, or have VMware's stock drop 5$ a share, no.

Like rparker stated, everyone of us has made a serious mistake at some point. They handled it correctly, I'm sure had everyone and their mother in working on the problem to get things going. We'll see how things change, but everyone in the developing and QA departments will be under a microscope for a while so I doubt something like this will happen again any time soon.

  • Kyle

-- Kyle "RParker wrote: I guess I was wrong, everything CAN be virtualized "
Reply
0 Kudos
THP
Contributor
Contributor

Personally I wouldn't rush to move away from VMWare as we have had largely good experience until now - but this one has certainly shaken my confidence a little. It also makes it far harder to push for further virtualisation when talking to non-technical people who have just read about this problem and don't have the hands on positive past experiences to offset it.

I think the biggest casualty will be those companies planning to enter the market rather than those already involved - a lot of directors perhaps suggesting that their staff at least examine other options before adopting VM and so on.

Also you can assume that the segment of the market represented by the angrier posters might well not flee VMWare today but those people aren't going to be acting as in-house salesmen for the product as they have been during it's rise from infancy to today - reading between the lines they feel some personal betrayal which may not be 100% logical but will still have a negative impact on VMWare's bottom line

M$ can pull this type of stunt - they have the market dominance to get away with it, mores the pity. I'm not sure VMWare are in that position.

The response to this issue was initially very slow but accelerated as VM became more aware of just how seriously this was being taken - credit for that and getting the patch out earlier than originally stated. It also installed well for me although I'll be keeping a closer than usual eye on the farm for a few weeks.

We'll just have to let the dust settle a bit and see if the recent niggles with the product go away and we can return to trusting the product to do it's thing while we think of new ways to make 100% uptime sound interesting in the monthly reports!

Reply
0 Kudos
gi-minni
Contributor
Contributor

I fully agree with you, we need an extra time period before delivering golden code to ensure product stability and quality, but honestly time bombs can only be found if

a professional team of software engineers do a lot of regression and NFR tests following strict rollout guidelines with a lot of automation testing.

I know that VMware use this time bomb procedure since the early days of Workstation 2 to ensure beta code is not used after a certain time period.

My question is now: Even they don't implement this the very first time, how could this be undiscovered couple of weeks after release?

I think that complexity, market and competitors pressure does not allow to test every things and side effects in a decent manner

and we all know this needs time, money and excellent skilled staff.

I am confident that VMware learned a lot from yesterday and hope the black Thuesday remains a singular event in VMware history.

Reply
0 Kudos
tvleavitt
Contributor
Contributor

Yo, Paul.

You talk about improving Q.A., but when I go here:

http://www.vmware.com/landing_pages/esxexpresspatches.html

and click on the links to the KB articles (which point to kb.vmware.com), I get redirected here:

http://kb2.vmware.com/kb/1006716.html

and get "Server not found."

If I change "kb2" to "kb" in the URL above, I get to the proper URL. Seems like this is another Q.A. problem...

This is a minor irritation, but folks under stress may not think to edit the URL, and it certainly doesn't make a positive impression. Suggest you fix this, ASAP.

Regards,

Thomas Leavitt

Reply
0 Kudos
admin
Immortal
Immortal

Hi Thomas --

I'm not Paul, but I'll try to help. I can't reproduce the problem now (via Comcast in CA) -- the redirect to kb2 works fine -- but I will report it to the appropriate teams.

In addition, problem reports to customer service, tech support, or webmaster should get routed the same teams, so if it's a widespread problem, we should soon know.

John

VMware Communities

Reply
0 Kudos
admin
Immortal
Immortal

Hi Thomas --

The team suspects you may have stale data in your DNS cache. (The page is being served worldwide through Akamai, which gives you a geographically-specific IP address to pull from.) Can you flush your DNS cache and try again? You can try doing a DNS lookup before and after to see if the IP has changed, as well.

John

Reply
0 Kudos
tvleavitt
Contributor
Contributor

Thanks for the quick response, John

You're probably correct. I will do this when I'm on site at that particular client again tomorrow in the morning. I was most likely incorrect in assuming that kb2 had just been created and that the site's servers wouldn't have anything cached for it as a result.

Regards

Reply
0 Kudos
admin
Immortal
Immortal

Dear VMware Customers,

In addition to the express patch and the re-issued ESX/ESXi 3.5 Update 2 release, we now have an alternative installation process for customers who haven't applied either to hosts that were affected by the product expiration issue.

Below are the details we list at http://www.vmware.com/landing_pages/esxexpresspatches.html:

*Known VMware ESX 3.5 Update 2 Express Patch Installation Challenges

The following message is applicable ONLY for customers who had installed the impacted release of ESX 3.5 Update 2 (build number 103908), but not yet applied the express patch.

We are aware that you may encounter the following challenges installing the express patches needed to correct the problem:

  • Internal change control procedures

  • No available server to VMotion running VM's onto

  • Unable to schedule a maintenance window

If you experience one of the challenges listed above, please contact your support provider and indicate you need assistance with the U2 Alternative Install Process (U2 AIP). The support team can assist customers with this alternative installation procedure.

Thanks,

The VMware ESX Product Team

Reply
0 Kudos