larstr

When I was first introduced to the Service Console back in the ESX 2.0 days (in 2003) I was delighted with what VMware had done. The Service Console is a linux (sort of) VM which is what you see when you boot up an ESX server. The environment you see is a linux environment, but the hardware seen from this environment is not the full blown one as it is running as a virtual machine with some special privileges that normal VMs don't have. This means that you can also access and configure the VMware (vmkernel) environment. The MUI (management user interface) was also nice, but everything you saw of statistics in the MUI was derived from stats that you could find within the proc nodes of the Console OS, which was the name it was known by back then. It made sense that a GUI wasn't always as correct as the “real deal” where the raw numbers were sitting, so if you were in doubt of anything you could always double check it if you knew where to look. If you knew some basic unix scripting you could also in a short amount of time write a script that monitored what you where looking for.

When ESX 3.0 was introduced things had changed a bit. The Service Console still looked the same at first glance, but when you had a closer look you could tell that things had changed beyond the change of Linux version. These values that you could look at where still there, but they didn't make much sense. It turned out that /proc/vmware wasn't the main source of statistics anymore, as VMware had created their own stats interface called VSI(VMware SysInfo). The proc nodes within 3.0 was either a leftover from 2.0 that wasn't complete for 3.0 or it was a half done conversion interface from VSI. I don't know which one, but surely enough you couldn't rely on the proc nodes for anything anymore even though it was there.

The reasons for abandoning the proc nodes may have been valid. The proc nodes is where all linux distros today and many other unix variants (aix, solaris, etc) present their performance numbers through. The resolution of stats within the proc nodes on linux is always 100Hz even though the system timer could be higher. The stats within the kernel could be of a higher resolution, but the numbers presented to applications are always 100ms accurate as per USER_HZ. The real system timer rate is not accessible to user applications because it should not be necessary for them to know the real timer rate (there exists patches that overrides this). The reason for this is compatibility reasons with normal linux tools such as ps, top, uptime, etc. This means that the accuracy of stats were only as +accurate as 100ms while the accuracy in the stats in the VSI interface was 1000ms. Wouldn't normally mean a lot on most stats, but more accurate graphs are of course for the win.

The main console tool for monitoring ESX performance is esxtop. Esxtop in version 3.x looked quite similar to the one in 2.x, but it had been rewritten from a different project than before. In esx 2.x, esxtop was derived from GNU code, while in 3.x it was now derived from BSD code. VMware had also extended it quite a bit since 2.x with various stats that they retrieved from VSI. The best tools for dumping values from VSI was however esxcfg-info that dumped all available stats in a given category. In 3.5 you could also choose the preferred formatting of the dumped stats if you wanted them in xml or perl table formats.

When troubleshooting performance issues on ESX you would normally start with the stats in viclient to get an overview, then look at esxtop, and for further details you could use esxcfg-info and vscsiStats for more detailed information. esxcfg-info and vscsiStats where new tools introduced to the Service Console in ESX 3.0 and 3.5.



There was never much documentationof the statistics of neither the values in the proc nodes or in esxcfg-info, but it was possible to interpret what they were by comparing the stats you were looking for to the ones shown in esxtop and in the viclient. There was eventually some documentation provided by vmware on the esxtop stats (by performance guru Scott Drummonds).

The announcement of the death of the Service Console @ vmworld 2007 has been received with mixed feelings. In ESX 3 and newer you can do everything you need from the GUI and may never need to know that there's a Service Console available. They also introduced a console less hypervisor (ESXi) we can all see the advantages it brings. As the Service Console is a Redhat based linux install, it needs additional patching which is in reality unrelated to VMware's core business of virtualization. By having a thin hypervisor of the ESX hosts you get a smaller attack surface and it should in theory also give you slightly better performance as you don't have the additional load of an extra linux VM running (even though it's fairly lightly loaded unless you've filled it with agents).

ESXi (or ESX 3i as it was known as upon announcement) was introduced to be the future successor of normal ESX, but without a Service Console. In ESX3 you can do everything that you need to configure and manage your ESX environment from the GUI (viclient), but troubleshooting issues is often easier done from the command line. They couldn't just stop shipping normal ESX with a console right away since many third party vendors have software agents that depend on the Service Console.



To provide command line access similar to the Service Console, VMware initially introduced the Remote CLI appliance. It was based on Debian 3.1 and wasn't that bad of an idea. It provided you a command line interface where you could run resxtop and similar commands to get statistics from an ESX host and you could also to a fair bit of other management related tasks. It did include a version of esxtop, but both esxcfg-info and vscsiStats were missing. It also came with all of the developer tools a console oriented linux programmer needed to interface with a VMware environment. VMware also had their own debian repository available for this appliance so you could add the packages you needed similar to a normal debian distro. In the original Service Console, there was no online software repository, but the ESX install CD provided extra packages that you could install.

The Remote CLI appliance's software repository was suddenly gone and VMware introduced another appliance that had similar capabilities. The VIMA=Virtual Infrastructure Management Assistant. It was an appliance that included the same remote cli command set, but was still different. It did not include any development tools, it included a method of setting up a trust between the ESX servers and this appliance so that you didn't need to authenticate for each command (vi-fastpass). It also didn't have the ability to add any packages, but was based on RHEL 5.2 so you could take packages from RHEL risking breaking any support from VMware if you should need to call them.

In ESX4 this appliance has a new version and also a new name. It's now known as vMA 4.0, vSphere Management Assistant and is basicly the same appliance as VIMA 1.0, but with updated packages and you now also have the ability to setup a trust to the vCenter server in order to run commands on all of your ESX servers instead of setting up trusts to each ESX server in your environment. It still lacksboth esxcfg-info and vscsiStats for deep level performance troubleshooting.

ESX4 still has the vmware proc nodes present, and whether they are a reliable source of information or not I'm not 100% sure. But I doubt they have done much with it since no software relies on it anymore and the core info lives in the VSI.

A new (official) feature that was introduced with ESX4 is thin provisioning. Thin provisioned disks only store the blocks that have data on them and is a good way of saving data while utilizing your storage subsystems better (not always performance wise, but atleast data wise). One should however be aware that when viewing a thin provisioned disk from the local Service Console, it will always show the full size of the disk, even when a fraction of the disk is allocated physically. To get the correct disk usage of this VM you will have to use the viclient (GUI) so things are now beginning to be the other way around from what we were used to in the “old days”. I don't know what other features that comes out wrongly in the console nowadays that are correct in viclient, but wouldn't be surprised if this wasn't the only one.

The death of the Service Console is indeed coming up. VMware have stated that it will die with the next release of ESX. I'm not sure if that means ESX4.5 or 5.0, but it surely will die soon enough and we already see that the focus is shifting in that direction. I hope that when that day comes, all of the functionality we are using today will be available through supported means on the new platform.

In 1996 I attended some basic training and did my first firewall installation. During the following year I did a few more and every time I had a question I called other people in my company or the distributor. In 1998 I discovered that this product had a mailing list. I joined it and it opened a new world for me regarding known problems and solutions. There were a few very knowledgeable people on the list who seemed to know most of the common questions. These users where consultants or end users of the product who out of their own curiosity by the product learned the ins and outs of the product, offering free support for other peer users. There were also members of the list working for the vendor itself, but they would normally post from their personal accounts. Sometimes we would however see official postings or responses from the vendor.

After joining this mailing list and being there for a while I understood the product much better and the quality of my work also became much better. I could also more easily discuss different questions with my customers as the questions they had was often related to something I had discussed before on the list.

This was my first involvement in a mailing list, but I had used news groups (nntp) for years, and before that I had used similar discussion groups on different BBS networks (over modem/pots). Even though comp.sys.firewalls was nice, it’s user mass wasn’t product specific enough to give you the “little extra” of info that the vendors mailing list provided.

Having a mailing list where anyone can post their questions is something I regard as a very good thing, but it has a few limitations:

• Threads are not implemented equally across all email clients
• Searchability – Searching emails will return single postings, not the full thread
• History – Hard to refer to a previous thread with a link
• Formatting – Each email program will display emails differently so text only is often the only common denominator
• No user profiles or stats per user
• Spam
• RSS feed?

I was introduced to VMware ESX Server 1.5 in 2002, but the project that I was part of waited for version 2.0 that was shipped in September 2003 before it was implemented. Installation was done by the HW vendor and I was thrilled by this new technology. First of all, the Console OS was based on RedHat 7.2 which was something I knew from before. Secondly, there were active news groups for this product that were fairly active. I think I posted my first posting on the VMware news groups in mid-September 2003.

A bit later, VMware announced on the news groups that they had started a web based forum. At first I didn’t really see the point why they had done that since the news groups were working so perfectly. I thought it was something that would just go away so I refused to try it. Why use a web browser when you could use your favorite news reader (tin)? A web based forum was likely to be much slower and you couldn’t use your own preferences of threading/searching etc that you were used to in your news reader.

News is quite similar to mailing lists and has the same limitations except that threads are working well across different clients.

After a few months I noticed that the traffic on the news groups was decreasing. In March 2004 I decided to check out what the entire buzz was about so I created a bogus account. I didn’t think I would log in to this system twice so why bother using real info? After seeing the quality of the content of these forums I logged out and created a real account that I could use in the future. I didn’t give up on the news groups right away, but I was more and more often using the web forums after that. Why? First of all, there were a few VMware employees on the forum that would have answers to questions nobody else could know. Things that wasn’t documented. Things that would require digging in the source to figure out. Documentation back then wasn’t nearly as good as today and a knowledgebase didn’t exist. Having access to discuss topics directly with those who can figure out such things, just make you want to hang around more to find out what else you have missed lately. It turned out to be highly addictive. And quite a few got addicted.

Another thing that didn’t hurt the forum activity was the “Forum rewards-program”. They announced that points were now to be awarded by the question posters and that VMware would send gifts to those who got enough points based on that. VMware also sponsored a trip+hotel+access to VMworld 2005 for the top 5 forum users where they would be a panel for an “Ask the community experts”-session. Having good products is one thing, but when you add things like this you risk that people start loving the whole concept. It attracted many smart users who in turn helped keep the quality of the forums at a very high level. When people start loving a company’s concepts they will help the company build a good reputation. We have during the past couple of years seen the blogosphere around VMware grow quite substantially and it wouldn’t do that if people didn’t feel good about VMware and the products. A large part of the bloggers has also a background as active users on the VMware community forums.

If I’m questioned about products/technology I don’t know well I usually check if the product has an active community. If it does, it’s often an indication that the products are good or that the products have great potential. You will also be able to get non-marketing answers to questions about the product line even if you’re not an existing customer. An active community does certainly not replace an official support line, but it can offload it quite a bit for the commonly known issues. An official support line is still needed for resolving the more serious issues.

A community is also normally something that potential customers can join and not only existing customers/partners. I’m still amazed that not all vendors understand the value of such a community and also charge their users extra for being able to join or post to such a community. By limiting their forums like that they are probably loosing potential customers and potential active users. If their product is interesting enough it can attract technical people who are willing to spend their free time on their forums. This is a win-win situation for all parts. Potential customers and existing customers are being helped with their basic questions on the forums and the official support center can concentrate about the more serious issues. Users and consultants are getting to know the most common challenges that people are having and special solutions to things in a way that is often more trustworthy than a marketing brochure.

A while ago I was introduced to a product that looked very interesting. It had a very nice design, some smart solutions and they could refer to a handful of major customers. Googling the product did however not give too many results and a user community didn’t exist. I evaluated the product and had a quite close contact with the distributor during this process. If the vendor had community forums I suspect I may have gotten answers to some of my questions quicker and saved myself some time during my testing.

While this product wasn’t bad, I still prefer using a competing product even though it’s a more expensive one. Not because the competing product is so much better for the end user, but because it has good community forums, a larger user base, and less complex installation procedures.

Personally I find the VMware forums an excellent resource, but I’m still waiting for the next version of the forums to be released as the current version is too slow for my patience and the rich text editor is useless. The current version was introduced in late September 2007 and had major problems shortly after being put into production. The stability problems were resolved, but the performance and formatting issues still isn’t perfect. As I’m not currently as active on the forums as I once was, I’m now blogging using the same system, just to be reminded of how bad or good the situation is. I could have used Wordpress or another known working blog platform, but then I wouldn’t get reminded of the status of the VMware forums.

There are still lots of activity on the VMware community forums, so it seems all are not as impatient or formatting oriented as me. I have tried joining back at some occasions, but found myself switching to another tab while waiting for a response from the community. Having the forums available is still something I value highly even if I’m not active there on a daily basis.

By subscribing to RSS feeds on Planet v12n (“planet virtualization”) I’ve been able to get a great deal of info without using the forums. The weekly VMTN Roundtable podcast on Talkshoe is also good. It also has chat during the show so it’s much more valuable in a live session than a recording.

There is also an irc channel on freenode, #VMware that I’ve been part of for a few years now and that’s also a place where it’s possible to discuss VMware related topics with other peer users.

In addition to these arenas I’m now also following the progress in the virtualization field via Twitter. It’s amazing how Twitter has attracted the right users who will point you to blog posts and good community postings on a regular basis while there’s also some direct discussion going on right there.

To sum things up, the bottom line is: VMware have some very good products that people like. VMware have supported their users by giving them an arena to discuss their product and share their experiences in an unmoderated fashion. This has lead to users spending a lot of their spare time writing about the products on both the community forums and in other arenas. The users now know more about the products than they would by only reading the documentation. This means that the users are now better at what they do, certified or not certified. Several books have been written by community members. The importance of such a community can never be underrated.