Why does VMware refuse to educate their customers ?

[continuum]

Looks like this new forum is the right place to ask some old questions again.

Why is there no in-depth documentation of vmx-parameter ?

Why is there no in-depth documentation of the Windows-tool vnetlib.exe ?

Why is there not even a rudimentary guide on vmdk/snapshot repair ?

Why is there not even a rudimentary guide on trouble-shooting log-files ?

Me and other long-term members of VMTN have requested all this docs over and over again since years ...

Regards Ulli

description of vmx-parameters:

VMware-liveCD:

[admin]

Hi, Ulli! It is a great pleasure to see you here. I've followed your VMTN posts over the years with great interest. And, like many folks at VMware, I am a big fan of sanbarrow.com.

The topic you raise has been of personal interest to me for all my years at VMware, and even longer: I was a VMware customer before I was an employee. I've participated in many internal discussions on the subject, some sparked by your own posts, and some sparked by our training customers. I will share my thoughts on the subject here.

For the benefit of other readers, let me explain the issue Ulli raises. VMware publishes no official, centralized table of all the parameters one may place in a .vmx file, which is the file that defines a virtual machine. VMware also does not provide some other kinds of interesting technical documentation. Some people outside VMware, such as Ulli, have maintained volunteer websites documenting what they have discovered and learned from others. It is reasonable to ask why VMware does not take on this job itself.

Although you phrased your inquiry in the form of a question, Ulli, I realize that you want not answers but action. You want VMware to change its behavior. My response will probably not satisfy you, because I have come to the conclusion that VMware's stance is, in general, the least bad of all the alternatives. Furthermore, I can speak officially only on behalf of VMware's Customer Education program. Although I hope you'll find it interesting, this note must be considered one man's opinion, not a response on behalf of VMware as a whole.

Like Microsoft and Oracle, VMware is a proprietary, closed-source software company. Every closed-source software product consists of two parts: implementation and interface. The difference between the two parts is ultimately related to the agreement between the software company and its customers. The interface of the software product is all those parts that are documented; on which the company will accept support incidents; and that the company can reasonably be expected not to change in a surprising way. An example of this last criterion: imagine that, in Office 2008 Service Pack 3, Microsoft quietly dropped support for Rich Text Format files. Everyone would be very angry. But if they renamed one of the run-time libraries that make up Office, that would at least be fair, although annoying to people who use those .DLLs in a creative way. What kinds of file Office opens is part of its interface, but what libraries compose it is part of its implementation.

In a closed-source product, everything that isn't interface is implementation. In a pure open-source product like Emacs or the GNU C compiler, there is no distinction between interface and implementation, because all parts of the product can be inspected and modified by users. And, conversely, Richard Stallman has no obligation to help you if you encounter a problem.

Closed-source companies must decide what parts of their products are implementation and which are interface. This is ultimately an economic decision. Marking a feature as interface is tantamount to making promises to customers, morally if not legally. The company must document the feature, accept support calls on it, and preserve its stability across at least point releases, and probably minor releases too. All the costs to fulfill those promises must ultimately be paid for either by customers or by stockholders.

I have watched VMware's management of the contents of the .vmx file over the years, because our training customers ask us the same questions Ulli does. Here is what I have observed: for VMware, the .vmx file is interface, not implementation, except in opportunistically chosen cases. Our customers generally have the right to demand that they be able to use their VMware product to do what it's documented to do without editing any .vmx files. (In the case of VMware Player, that statement is trivially true, because creating or reconfiguring virtual machines is not among that product's documented features.)

I did say "...except in opportunistically chosen cases." An example: when ESX Server 2.0 came out, the company found that the most rapid, cost-effective way to help customers who were running Terminal Services workloads in virtual machines, and still ship that product on time, was to ask them to add workload.TerminalServices = "TRUE" to their .vmx files. Later versions of the user interface made manually editing the .vmx file unnecessary, and ESX Server 2.5 made using any special flag at all for Terminal Services workloads unnecessary. But, for a while, workload.TerminalServices = "TRUE" graduated from implementation to interface.

In general, though, the .vmx file is interface, not implementation. VMware may change the meaning of .vmx file parameters without notice between product version x.y.z and x.y.z+1, although that behavior would certainly be out of character (and it'd take me by surprise!). And VMware does not accept generally support cases concerning manual edits to .vmx files--unless, of course, they're intended to address a situation like that workload.TerminalServices = "TRUE" issue from the old days.

Workstation, Player, Server, ESX Server, VirtualCenter, Fusion... all these products ship with helper binaries like vnetlib.exe. Some are documented, and some are not. The ones that are documented are the ones VMware has chosen to treat as interface. For example, on the ESX Server 3.5 box I have here, I can type man vmkfstools and get a nice Unix-style manual page on everything I can do with this command. But there are plenty of other vm* commands in /usr/bin that have no man pages. The lack of man pages is VMware's way of telling me I should not bet my business on the continued existence of this command from one release to the next.

You might reasonably ask: "Why does there have to be such a sharp line between implementation and interface? Why not a third category? Why can't, for example, VMware publish documentation on, say, .vmx file contents or helper commands, with a disclaimer saying that this could change at any time? That way, people could use it for whatever purposes they liked, but they have been adequately warned that it could change."

That's a good idea, but its costs have to be weighed against its benefits. Many customers would say, "What's the point of documentation I can't rely on for more than a point release at a time? I would rather see VMware spend its money on improving the documentation it already has! Or improving the products themselves! Or hurrying up and releasing a version of the products localized into my language, because I am tired of using my English dictionary with the product! Or...." These customers have a good point, and it applies particularly well to .vmx file contents. Every VMware product supports a somewhat different set of parameters, and each parameter has its own lifecycle of birth, usefulness, and then obsolescence (as with workload.TerminalServices). Accurately tracking them all would be doable, but far from free of cost.

Other customers might say, "Never mind whatever disclaimers you and your lawyers compose! If I download the documentation from *.vmware.com, I expect to be able to rely on it." I'd find it hard to criticize a customer for this position.

Anyway, thank you for indulging this very long post about my own personal opinion. To sum up: the current model can be annoying, not only for Ulli but also for VMware employees. (My own personal "pet peeve": keyboard.typematicMinDelay. My enhancement request to get that blessed as official is in the system.) But as long as VMware remains a closed-source company, it will have to divide its products between interface and implementation. And as long as some parts of VMware products are implementation, not interface, it'll be tough for VMware to spend resources on documenting those parts instead of better documenting the supported parts of the products.

[continuum]

Hi Brian

First of all I have to say I am impressed by your quick and long answer.

I had not expected any answer - or something like "oh no - not again ..."

Although you phrased your inquiry in the form of a question, Ulli, I realize that you want not answers but action. You want VMware to change its behavior.

You are right - this was a rhetorical question - it has been answered before and your long answer doesn't bring any new point.

Yes - I really wanted that VMware changed its behaviour the first one hundred times I asked this questions

Ok - I guess it doesn't make a lot of sense to repeat this discussion for the 101 time if we don't change the pattern.

=========== break ========================================================================

Hi Brian

Since 2003 I do a part-time job in VMware-support. It was a lot of fun the first years but now it becomes increasingly boring.

Customers ask the same question over and over again while I would like to discuss things that are way beyond the manual instead.

Lets think about ways to reduce the repetitive work - before I get tired of it.

In the last years I made an observation ... when ever some user had problems with VM and CDs or Isos and I told them to study

this users were able to fix the next problem with CDs themselves.

When I told those users to click on the "xy" button in the GUI or read the VMware manual they usually came back after some weeks with a slightly different problem ...

As this forum is about user-education ...

In my book education means : first explain how things basically work and then help the novice to develope his own skills so that he can analyze and solve future problems himself.

In VMware-context this means : An educated user that runs into a problem reads the last logs, tries to understand what is going wrong and in best case finds a solution himself.

For this he needs a basic understanding of vmx-parameters, vmdk-parameters and usable hints in log files.

It is not necessary that this user knows about monitor_control.restrict_backdoor = ....

Brian - all VMware documentation that I have seen follows a different path:

To get A click here.

To get B click there.

Regular user is clueless when he wants C

The knowledgebase is a large collection of

If you have problem X click here.

If you have problem Y use this line in vmx.

Regular user is helpless when he runs into problem Z

Brian - I believe I understand the reasons you have not to document all possible vmx-parameters.

But is it really necessary to follow this rule so strictly as you do it ?

I would like to see a more pragmatic attitude towards this issue.

Ulli

[oreeh]

I would like to see a more pragmatic attitude towards this issue.

Me too!

[admin]

Hi, Oliver!

Ulli, I like the way you phrased this...

I would like to see a more pragmatic attitude towards this issue.

...because some parts of VMware (including the part I have direct influence over) do take a more pragmatic attitude towards the issue. In one of the training courses we in VMware Customer Education offer to customers, we discuss some of the contents of the .vmx file and the log files, though not as exhaustively as sanbarrow.com does.

We include this material because one of the course's objectives is to train people on how to troubleshoot: in this case, ESX and VirtualCenter. Typically, the people who need this material are their own companies' experts or internal support people, who must help their co-workers when problems occur. If you are trying to help somebody else with a VMware product, it's often fastest and easiest to look at the contents of the .vmx file, or to interpret the (often revision-specific) messages in a log file.

Why can we in Customer Education get away with different behavior from the rest of the company? Well, because we must. Real support organizations, such as VMware's own support team and those of its big customers, use these resources, just like the increasingly fatigued VMTN experts! Of course, all of our training courses are aimed at a particular revision of the software, and we make clear that these details may change in other revs. We try to focus on (a) useful details that are among the least likely to change and (b) details that are specifically relevant to the current version (such as workload.TerminalServices).

But here's my question for you, Ulli and Oliver. Imagine that VMware produced, for each of its products, a really good document called "Troubleshooting productname". It would contain all the under-the-surface detail that you'd need to troubleshoot this product: vmx file flags, important log file messages, etc. Would that be good enough? I claim no.

Such a document would help with the product's supportability, but it would be a total surrender on the product's self-supportability. Isn't it reasonable for people to expect to be able to troubleshoot most problems using the normal GUI they do all their other work in? Imagine that your Mercedes had no fuel gauge. To measure the amount of fuel left in the tank, you would need to stop the car and insert a dipstick into the fuel tank. Our friends in Stuttgart would never accept this compromise.

In the early days of VMware, we could assume that the users of our products were familiar with the command line and with text configuration files. We can't assume that any longer. Our goal has to be making the product's interface complete. Unless there's a software bug, people should be able to solve their own problems, without recourse either to VMware Support or to the experts on VMTN.

Here's an idea I have been developing as an enhancement proposal for our products. One of the main ways you troubleshoot a virtual-machine problem is to compare your broken VM to a working VM, correct? I am writing up a feature proposal for a new item in the Virtual Machine menu: Compare to... . If you chose this item, you'd be prompted to select another virtual machine. (In Workstation, maybe you'd have the option to compare to a virtual machine on a different host computer, if you supply a username and password.) The result would be a window showing, in human-readable terms, only the differences between the virtual machines: if one had an LSILogic SCSI adapter and the other had a Buslogic adapter; if one had 384 MB of memory and the other had 1024; and so on. (Of course trivial differences, like different UUIDs, would have to be surpressed. And the tool would probably only give reliable results between VMs of the same version. Comparing a Workstation 6 VM to an ESX VM would not be very informative.)

What do you think about that idea? Any suggestions for improvement? Or is it a dumb idea?

Let me summarize my position. Of course it'll never be enough for our customer training to cover the information you need for troubleshooting, because only a percentage of our customers take part in our training. And if the objective is to make people stop posting the same questions over and over again on VMTN, writing documents on troubleshooting will never be enough either. Many customers never read manuals. The ultimate goal has to be building self-supportability into the products.

For the vast majority of our customers -- all except for those in support jobs -- I disagree with the assertion that we should be educating them on the details of our implementation, such as the internals of a .vmx file. More and more VMware customers are like my father. He is not a programmer or even much of a computer guy; he just wants his programs to work. If a VMware product expects him to learn about all kinds of command-line stuff and its competitor does not, he will choose the competitor.

Anyway, what do you think? Am I correct in my belief that, to really make a dent in VMTN gurus' workload (not to mention VMware Support's workload), we have to include more supportability and self-supportability features in the products? Or am I wrong, and documentation really is the right answer?

If I had a budget of $X, how much should I spend on self-supportability features, and how much should I spend on writing new troubleshooting documents?

Can you suggest other self-supportability features?

[oreeh]

Brian, first let me say that I really appreciate this "discussion".

Anyway, what do you think? Am I correct in my belief that, to really make a dent in VMTN gurus' workload

(not to mention VMware Support's workload), we have to include more supportability and self-supportability features in the products?

I agree that self-help features are required but at the same time wonder if this would really help that much.

After all VMware products are far from being "normal" software.

A good start would be some useful defaults during VM creation - but I'm wandering from the subject...

Or am I wrong, and documentation really is the right answer?

Both! Self-help facilities for the beginners / average users / easy to fix problems and some more insights for the pros - to help the pros help the average.

I'm thankful that I use the forum long enough to be aware of Petr's posts - Petr was one of the guys you could learn from by simply reading his posts carefully.

Same goes for a few other developers.

IMHO self-help options are more a requirement of the "consumer" products (WS, Player, Server, Fusion) and not the enterprise products.

Mainly because most of the enterprise users either have educated / trained employees or have easy access to them.

A few self-help topics are already covered in the different FAQs - question is how to turn these FAQs into working software.

[admin]

After all VMware products are far from being "normal" software.

Unfortunately, now many VMware customers expect its products to be normal software. Consider VMware Fusion. It's aimed explicitly at my dad: a consumer user, not an IT professional, who just wants his applications to work. In the evenings and weekends, I spend time answering questions on the Fusion VMTN forum; it is a very compelling intellectual challenge. Explaining VMware concepts to people who are not computer experts is endlessly humbling.

most of the enterprise users either have educated / trained employees or have easy access to them.

Even this is no longer true. I speak from personal experience: In the early days of VMware's customer education program, basically all customers who showed up for training on our enterprise products already had prior experience with virtual machines, and the vast majority had command-line skills. Today, most new attendees of our entry-level ESX/VirtualCenter training class have never used any virtualization product. And, although they are system administrators, few have significant experience at the command line (Linux or Windows). Few have scripting skills. Most expect the GUI to be their sole work environment.

VMware's products are increasingly mainstream. That means people have mainstream expectations for them. This is why I do not think that documenting .vmx files and the like will decrease VMTN gurus' workload. More and more VMware customers, if they found such a document, would think, "Hmmm, this document simply does not apply to me, because it makes assumptions about my prior experience that are not valid."

This is not a criticism of those customers; on the contrary, it's a challenge for VMware. For one thing, our documentation has to evolve to meet the challenge, and even ESX guys like me try to pitch in: But our software has to evolve to meet the challenge too. Hence the Compare to... proposal I'm working on.

There is a third way: supporting our gurus better. I think that the best way to do that would be for us to do a better job with documenting the top N issues for each product. That way, fewer uninteresting problems would make it through to gurus. I am formulating some ideas about how to do this; your suggestions are most encouraged.

[kwajalein]

While we're on the subject of educating customers, I'd like to make two observations.

1. The VMWorld materials often present focused topics, such as performance best practices and the handling of virtual/physical/machine memory, much better than the classroom materials do (at least the two classes I have taken). The class materials, in my experience, are little more than barely-glossed powerpoints, sometimes badly edited, and of little value months after the class. It would be great to see some of this existing expertise recycled to enhance the training materials. And, oddly, the classroom VMware classes I took were each individually more expensive than VMWorld's cost.

2. It's true that VMwre is becoming a mainstream technology, and this means different customers and different expectations will be applied to it. Learn from the best practice of other mainstream technologies, such as Microsoft. They have white papers, excellent student course manuals, walkthroughs, entire books at several levels of depth, webcasts, podcasts, etc. This undisputably accelerates mainstream adoption of their technologies. It's worth emulating. The IT world teems with the bones of companies that thought they could compete with Microsoft because they had a better product. All of us here want VMware to succeed and thrive.

[admin]

Hi, Kwajalein. I'm sorry you found the classroom training you experienced not well organized. I hope you gave us unvarnished feedback at the time. If you didn't have a chance to, or if you'd like to send me specific comments on particular courses, modules, slides, or labs, my email is brice at (obvious domain).

From your use of plurals, I take it that you took part in more than one classroom course. Thanks for the gesture of trust that represents. I greatly appreciate the point you made in your last paragraph.

You drew a contrast between the presentations you saw at VMworld and VMware's classroom training. It might surprise you to learn that, for the last several years, VMworld presentations that were heavily drawn from our course materials were among VMworld's top five best-rated. As you point out, we need to do a better job of harvesting information in the opposite direction: reaping detail from the best VMworld presentations and rolling them into our training offerings. We're doing a lot more of that, notably with the VI design courses we're building.

We're constantly working to improve the classroom materials we use in our instructor-led courses. The materials we deliver today are a lot more than just Powerpoints, and customers do tell us they use them intensively after class. Customers also tell us that there's work we need to do to make the materials easy to use after class use: we need to provide better indexes and references, and bundle them with more job aids. We're working on that.

You mentioned webcasts, podcasts, and walkthroughs. We in Customer Education are building a lot of new things online, and many are available free. An example is our new YouTube channel. Another area we're working on improving is how we integrate online training (whitepapers too, which you also mentioned) into our instructor-led classes.

In another message in a different thread, you asked how our online training works. I hope you don't mind if I answer that in a separate thread, to make it easier for others to find.

Thanks again--

--

Brian Rice

Education Services Product Manager

Professional Services Organization, VMware, Inc.

[kwajalein]

Many thanks. I'm an MCT, and my feedback for both classes was broad and deep. I must emphasize the instructors were excellent; it was the materials which lagged. In fact, I am stunned to hear that so many best-rated presentations at VMworld were based on classroom materials! I can only conclude they have evolved from the time I took classes, in late 2006 and early 2007. At that point (I'm paging through my student manual for VMware Infrastructure 3: Install and Configure as I write), the content left a lot to be desired. The model I had in mind were the student manuals from MS or Citrix classes, which were deep, broad, and has tons of supplementary materials, even on CDs. The MS course manuals in particular are usually enough for you to learn everything on your own. I'd love to see VMware materials on a par with those.

Can't wait to check out the youtube materials. I love the mp3s from VMworlds past; on the treadmill the other day I was listening to Kit Colbert explaining virtual/physical/machine memory from last year; great stuff!

[continuum]

Hi Brian

shame on me - I really missed your answer from 1 april - sorry

But here's my question for you, Ulli and Oliver. Imagine that VMware produced, for each of its products, a really good document called "Troubleshooting productname".

It would contain all the under-the-surface detail that you'd need to troubleshoot this product: vmx file flags, important log file messages, etc.

Would that be good enough? I claim no.

Well - I guess you simply don't have the manpower to write such "a really good document".

Very likely it would be outdated when it appears ..

The really tricky problems occur when unexperienced users work with your products - you can't write those troubleshooting guides you mentioned before the product meets the public I guess.

I think that only a small number of folks is really interested in "hard core" troubleshooting - when I remember the times Petr was visiting VMTN only a few guys were listening to him like to a teacher.

IMHO this was a pretty cost-effective way of support for your company.

Folks like Oliver, Rob , me and many others learned a lot - now we continue the work in VMTN or by running trouble-shooting websites like

Its a pity that this channels are closed nowaday - I also don't think its clever.

I guess I followed Petr through maybe 5 snapshot-repairs - in the following times I solved hundreds of this problems on my own - and have told lots of other how to help themselves.

Brian - every bit of information that you give to use - reduces your support-work ten and hundredfolds.

Why are you so closed ?

Bring back some teachers - or at least some folks who answer our technical questions ...

Such a document would help with the product's supportability, but it would be a total surrender on the product's self-supportability.

Isn't it reasonable for people to expect to be able to troubleshoot most problems using the normal GUI they do all their other work in? Imagine that your Mercedes had no fuel gauge.

To measure the amount of fuel left in the tank, you would need to stop the car and insert a dipstick into the fuel tank. Our friends in Stuttgart would never accept this compromise.

Hmm - that comparison doesn't match I think - I thought we were talking about a level of troubleshooting deeper than the service you get in a gas-station.

I guess users don't expect that they can recover their data after a systemcrash using the GUI.

Like if the gasoline is empty they go to a gas-station (GUI) - if they have a flat tyre they go to a garage in the neighborhood.

Brian - Oliver, Rob, me and folks like us do the garage-work - if you help us - you help yourself

If I had a budget of $X, how much should I spend on self-supportability features, and how much should I spend on writing new troubleshooting documents?

What about a third option ? - spend some money for trouble-avoiding education.Take an example - the tricky part in snapshot-repair on hosted platforms is the fact that you have to hexedit in the default case.

Setting a user-friendly default would be very cheap - compared with writing docs on how to safely hexedit ...

By the way - I like your idea of a VM-comparing tool

Sorry for late reply

Ulli

___________________________________

description of vmx-parameters:

VMware-liveCD: