This week I attended the “VMware vSphere: Troubleshooting” class. It’s a four day class that gives you the right to certify for VCP similar to a few other classes. This is a fairly new class and this was the first class in the Nordic region. Our instructor was not dutch, even though his last name was Blom. He was from Sweden and was fairly well informed about the topics of this class and VMware in general.

The troubleshooting class sounded like an interesting class. I have never attended a VMware class before, but since I have been using VMware products for quite a while this class sounded like the most reasonable one to take.

This class had the following modules:

-     ESXi Command Line Troubleshooting

-     ESX/ESXi & vCenter Server log files

-     Network troubleshooting

-     Management troubleshooting

-     Storage troubleshooting

-     VMotion troubleshooting

-     VMware Infrastructure troubleshooting

-       vSphere 4 DRS Cluster Troubleshooting

Note that this class is not about performance troubleshooting. It’s only about troubleshooting configuration problems. A performance class is coming up soon from VMware educational services.

This class was around 70% labs and 30% lecture time. Most of the labs were performed as two-student teams. Most  labs involved configuring something, either from the command line or from the vSphere client, and later a configuration issue was being introduced to our environment (by the instructor) that we should try to resolve.

This class was a good look into managing your infrastructure from vMA and we learned how it differed from the ESX console and the unsupported ESXi console. VMware’s educational labs were used and compared to the labs I attended at VMworld 2009, these actually worked.

A few of the labs were a bit too repetitive and the scripts run by the instructor to bring problems into our configuration didn’t always work as intended. Most of the problems introduced did also IMHO not reflect problems you see in most real life environments. Since you were not allowed to watch the “Tasks and events” when troubleshooting a problem it made the troubleshooting a bit strange for some issues (you couldn’t monitor the progress of your VMotion if VMotion was the issue). It was still a very valid reason we couldn’t watch the logs when problems were introduced as it would have made it too easy to solve the issues, but it still made the troubleshooting situation a bit weird compared to real life.

The class was nice because of all the labs and for me it was especially useful because it made me get more familiar with vCLI which is required for supported command line management of ESXi. As we all know : ESXi is the future and the future is now.

I’ve had feedback from each of the vendors that I’ve blogged about so it's probably time to update the status on these topics

NetApp: NFS vs FC

http://communities.vmware.com/blogs/larstr/2009/02/06/nfs-vs-fc

The first vendor I  blogged about was NetApp and some material that was part of their ASAP training. I had done some testing using NetApp storage back at VMworld 2006 that gave results that didn’t match those slides from the training. When these slides came up during training I told the trainer that I thought these slides could not be valid and told him about the testing I had done myself a few years back. He couldn’t comment any further than that these slides where part of the official NetApp course.

A few months later I received an email from Mike Shea (@NetApp) who had read my blog and informed me that “the slides from NetApp is factual, but it is old and very out of date.”.

VMware also released a paper during the summer confirming that FC is still king of the hill on storage performance.

http://www.vmware.com/files/pdf/perf_vsphere_storage_protocols.pdf

Even though FC gives best performance I still prefer to use NFS for my customers with NetApp storage due to the extra benefits you get from using it.

I don’t know if the ASAP course still is teaching about superior NFS performance. Maybe someone who has taken it more recently would care to share their knowledge?

CheckPoint: Riding the virtualization wave

http://communities.vmware.com/blogs/larstr/2009/06/17/riding-the-virtualization-wave

The second blog posting was about CheckPoint’s virtual firewall solution for VMware that had insane pricing and licensing. A few days after writing this post I also discussed this issue with a local CheckPoint SE who wasn’t aware of this issue, but he agreed with me that this didn’t look right. He checked my findings and he could also only find the same information about this as I did. All he could offer me was special pricing if I showed him a customer case, but I told him the point here was not special pricing for a single case.

1.5 half monts later there was a CheckPoint sales presentation at my office. They where talking about upcoming products and here also the virtual firewall appliance (“Virtual Edition”) was mentioned. The upcoming version was supposed to be much better than the previous one as it was using VMsafe and it would be a specialized solution for VMware environments. I raised my hand and asked if they were planning to fix their broken licensing model too.They were not aware of the licensing model, but since this was a non-released product licensing wasn’t necessarily 100% nailed yet

I sent them an email with a link to my blog posting and it reached the people within CheckPoint responsible for this product and they wanted a conference call with us. In the end they came visiting the local CheckPoint office for a meeting about this.

At VMworld I also visited the CheckPoint booth for a demo of the upcoming version and it surely looks nice.

We still haven’t seen the final results of this, but I’d be very surprised if their licensing model for VE remains unchanged.

Trend Micro: My view on the new TrendMicro Smart Scan Server Virtual Appliance

http://communities.vmware.com/blogs/larstr/2009/06/26/my-view-on-the-new-trendmicro-smart-scan-server-virtual-appliance

The third vendor I blogged about was TrendMicro who had just released a series of virtual appliances where I did a peek into one of them during a customer install. TrendMicro was the ones I had the quickest feedback from.  Within an hour  I got feedback on twitter from a Trend Micro employee stating: ”Thank you sir, I have already passed your suggestions on to the dev team.”

http://twitter.com/rik_ferguson/status/2344954320

During VMworld in San Francisco I also talked to a Trend Micro marketing guy (@Solutions Exchange) from whom I also learned that they would now start using .ovf files instead of .iso files for distributing their virtual appliances (great move). If they will also start having VMware Tools preinstalled remains to be seen.

When I was first introduced to the Service Console back in the ESX 2.0 days (in 2003) I was delighted with what VMware had done. The Service Console is a linux (sort of) VM which is what you see when you boot up an ESX server. The environment you see is a linux environment, but the hardware seen from this environment is not the full blown one as it is running as a virtual machine with some special privileges that normal VMs don't have. This means that you can also access and configure the VMware (vmkernel) environment.  The MUI (management user interface) was also nice, but everything you saw of statistics in the MUI was derived from stats that you could find within the proc nodes of the Console OS, which was the name it was known by back then. It made sense that a GUI wasn't always as correct as the “real deal” where the raw numbers were sitting, so if you were in doubt of anything you could always double check it if you knew where to look. If you knew some basic unix scripting you could also in a short amount of time write a script that monitored what you where looking for.

When ESX 3.0 was introduced things had changed a bit. The Service Console still looked the same at first glance, but when you had a closer look you could tell that things had changed beyond the change of Linux version. These values that you could look at where still there, but they didn't make much sense. It turned out that /proc/vmware wasn't the main source of statistics anymore, as VMware had created their own stats interface called VSI(VMware SysInfo). The proc nodes within 3.0 was either a leftover from 2.0 that wasn't complete for 3.0 or it was a half done conversion interface from VSI. I don't know which one, but surely enough you couldn't rely on the proc nodes for anything anymore even though it was there.

The reasons for abandoning the proc nodes may have been valid. The proc nodes is where all linux distros today and many other unix variants (aix, solaris, etc) present their performance numbers through. The resolution of stats within the proc nodes on linux is always 100Hz even though the system timer could be higher. The stats within the kernel could be of a higher resolution, but the numbers presented to applications are always 100ms accurate as per USER_HZ. The real system timer rate is not accessible to user applications because it should not be necessary for them to know the real timer rate (there exists patches that overrides this). The reason for this is compatibility reasons with normal linux tools such as ps, top, uptime, etc. This means that the accuracy of stats were only as +accurate as 100ms while the accuracy in the stats in the VSI interface was 1000ms. Wouldn't normally mean a lot on most stats, but more accurate graphs are of course for the win.

The main console tool for monitoring ESX performance is esxtop. Esxtop in version 3.x looked quite similar to the one in 2.x, but it had been rewritten from a different project than before. In esx 2.x, esxtop was derived from GNU code, while in 3.x it was now derived from BSD code. VMware had also extended it quite a bit since 2.x with various stats that they retrieved from VSI. The best tools for dumping values from VSI was however esxcfg-info that dumped all available stats in a given category. In 3.5 you could also choose the preferred formatting of the dumped stats if you wanted them in xml or perl table formats.

When troubleshooting performance issues on ESX you would normally start with the stats in viclient to get an overview, then look at esxtop, and for further details you could use esxcfg-info and vscsiStats for more detailed information. esxcfg-info and vscsiStats where new tools introduced to the Service Console in ESX 3.0 and 3.5.

There was never much documentationof the statistics of neither the values in the proc nodes or in esxcfg-info, but it was possible to  interpret what they were by comparing the stats you were looking for to the ones shown in esxtop and in the viclient. There was eventually some documentation provided by vmware on the esxtop stats (by performance guru Scott Drummonds).

The announcement of the death of the Service Console @ vmworld 2007 has been received with mixed feelings. In ESX 3 and newer you can do everything you need from the GUI and may never need to know that there's a Service Console available. They also introduced a console less hypervisor (ESXi) we can all see the advantages it brings. As the Service Console is a Redhat based linux install, it needs additional  patching which is in reality unrelated to VMware's core business of virtualization. By having a thin hypervisor of the ESX hosts you get a smaller attack surface and it should in theory also give you slightly better performance as you don't have the additional load of an extra linux VM running (even though it's fairly lightly loaded unless you've filled it with agents).

ESXi (or ESX 3i as it was known as upon announcement) was introduced to be the future successor of normal ESX, but without a Service Console. In ESX3 you can do everything that you need to configure and manage your ESX environment from the GUI (viclient), but troubleshooting issues is often easier done from the command line. They couldn't just stop shipping normal ESX with a console right away since many third party vendors have software agents that depend on the Service Console.

To provide command line access similar to the Service Console, VMware initially introduced the Remote CLI appliance. It was based on Debian 3.1 and wasn't that bad of an idea. It provided you a command line interface where you could run resxtop and similar commands to get statistics from an ESX host and you could also to a fair bit of other management related tasks. It did include a version of esxtop, but both esxcfg-info and vscsiStats were missing.  It also came with all of the developer tools a console oriented linux programmer needed to interface with a VMware environment. VMware also had their own debian repository available for this appliance so you could add the packages you needed similar to a normal debian distro. In the original Service Console, there was no online software repository, but the ESX install CD provided extra packages that you could install.

The Remote CLI appliance's software repository was suddenly gone and VMware introduced another appliance that had similar capabilities. The VIMA (Virtual Infrastructure Management Assistant). It was an appliance that included the same remote cli command set, but was still different. It did not include any development tools, it included a method of setting up a trust between the ESX servers and this appliance so that you didn't need to authenticate for each command (vi-fastpass). It also didn't have the ability to add any packages, but was based on RHEL 5.2 so you could take packages from RHEL risking breaking any support from VMware if you should need to call them.

In ESX4 this appliance has a new version and also a new name. It's now known as vMA 4.0, vSphere Management Assistant and is basicly the same appliance as VIMA 1.0, but with updated packages and you now also have the ability to setup a trust to the vCenter server in order to run commands on all of your ESX servers instead of setting up trusts to each ESX server in your environment. It still lacks both esxcfg-info and vscsiStats for deep level performance troubleshooting.

ESX4 still has the vmware proc nodes present, and whether they are a reliable source of information or not I'm not 100% sure. But I doubt they have done much with it since no software relies on it anymore and the core info lives in the VSI.

A new (official) feature that was introduced with ESX4 is thin provisioning. Thin provisioned disks only store the blocks that have data on them and is a good way of saving data while utilizing your storage subsystems better (not always performance wise, but atleast data wise). One should however be aware that when viewing a thin provisioned disk from the local Service Console, it will always show the full size of the disk, even when a fraction of the disk is allocated physically. To get the correct disk usage of this VM you will have to use the viclient (GUI) so things are now beginning to be the other way around from what we were used to in the “old days”. I don't know what other features that comes out wrongly in the console nowadays that are correct in viclient, but wouldn't be surprised if this wasn't the only one.

The death of the Service Console is indeed coming up. VMware have stated that it will die with the next release of ESX. I'm not sure if that means ESX4.5 or 5.0, but it surely will die soon enough and we already see that the focus is shifting in that direction. I hope that when that day comes, all of the functionality we are using today will be available through supported means on the new platform.

In 1996 I attended some basic training and did my first firewall installation. During the following year I did a few more and every time I had a question I called other people in my company or the distributor. In 1998 I discovered that this product had a mailing list. I joined it and it opened a new world for me regarding known problems and solutions. There were a few very knowledgeable people on the list who seemed to know most of the common questions. These users where consultants or end users of the product who out of their own curiosity by the product learned the ins and outs of the product, offering free support for other peer users. There were also members of the list working for the vendor itself, but they would normally post from their personal accounts. Sometimes we would however see official postings or responses from the vendor.

After joining this mailing list and being there for a while I understood the product much better and the quality of my work also became much better. I could also more easily discuss different questions with my customers as the questions they had was often related to something I had discussed before on the list.

This was my first involvement in a mailing list, but I had used news groups (nntp) for years, and before that I had used similar discussion groups on different BBS networks (over modem/pots). Even though comp.sys.firewalls was nice, it’s user mass wasn’t product specific enough to give you the “little extra” of info that the vendors mailing list provided.

Having a mailing list where anyone can post their questions is something I regard as a very good thing, but it has a few limitations:

• Threads are not implemented equally across all email clients

• Searchability – Searching emails will return single postings, not the full thread

• History – Hard to refer to a previous thread with a link

• Formatting – Each email program will display emails differently so text only is often the only common denominator

• No user profiles or stats per user

• Spam

• RSS feed?

I was introduced to VMware ESX Server 1.5 in 2002, but the project that I was part of waited for version 2.0 that was shipped in September 2003 before it was implemented. Installation was done by the HW vendor and I was thrilled by this new technology. First of all, the Console OS was based on RedHat 7.2 which was something I knew from before. Secondly, there were active news groups for this product that were fairly active. I think I posted my first posting on the VMware news groups in mid-September 2003.

A bit later, VMware announced on the news groups that they had started a web based forum. At first I didn’t really see the point why they had done that since the news groups were working so perfectly. I thought it was something that would just go away so I refused to try it. Why use a web browser when you could use your favorite news reader (tin)? A web based forum was likely to be much slower and you couldn’t use your own preferences of threading/searching etc that you were used to in your news reader.

News is quite similar to mailing lists and has the same limitations except that threads are working well across different clients.

After a few months I noticed that the traffic on the news groups was decreasing. In March 2004 I decided to check out what the entire buzz was about so I created a bogus account. I didn’t think I would log in to this system twice so why bother using real info? After seeing the quality of the content of these forums I logged out and created a real account that I could use in the future. I didn’t give up on the news groups right away, but I was more and more often using the web forums after that. Why? First of all, there were a few VMware employees on the forum that would have answers to questions nobody else could know. Things that wasn’t documented. Things that would require digging in the source to figure out. Documentation back then wasn’t nearly as good as today and a knowledgebase didn’t exist. Having access to discuss topics directly with those who can figure out such things, just make you want to hang around more to find out what else you have missed lately. It turned out to be highly addictive. And quite a few got addicted.

Another thing that didn’t hurt the forum activity was the Forum rewards-program. They announced that points were now to be awarded by the question posters and that VMware would send gifts to those who got enough points based on that. VMware also sponsored trip+hotel+access to VMworld 2005 for the top 5 forum users where they would be a panel for an “Ask the community experts”-session. Having good products is one thing, but when you add things like this you risk that people start loving the whole concept. It attracted many smart users who in turn helped keep the quality of the forums at a very high level. When people start loving a company’s concepts they will help the company build a good reputation. We have during the past couple of years seen the blogosphere around VMware grow quite substantially and it wouldn’t do that if people didn’t feel good about VMware and the products. A large part of the bloggers has also a background as active users on the VMware community forums.

If I’m questioned about products/technology I don’t know well I usually check if the product has an active community. If it does, it’s often an indication that the products are good or that the products have great potential. You will also be able to get non-marketing answers to questions about the product line even if you’re not an existing customer. An active community does certainly not replace an official support line, but it can offload it quite a bit for the commonly known issues. An official support line is still needed for resolving the more serious issues.

A community is also normally something that potential customers can join and not only existing customers/partners. I’m still amazed that not all vendors understand the value of such a community and also charge their users extra for being able to join or post to such a community. By limiting their forums like that they are probably loosing potential customers and potential active users. If their product is interesting enough it can attract technical people who are willing to spend their free time on their forums. This is a win-win situation for all parts. Potential customers and existing customers are being helped with their basic questions on the forums and the official support center can concentrate about the more serious issues. Users and consultants are getting to know the most common challenges that people are having and special solutions to things in a way that is often more trustworthy than a marketing brochure.

A while ago I was introduced to a product that looked very interesting. It had a very nice design, some smart solutions and they could refer to a handful of major customers. Googling the product did however not give too many results and a user community didn’t exist. I evaluated the product and had a quite close contact with the distributor during this process. If the vendor had community forums I suspect I may have gotten answers to some of my questions quicker and saved myself some time during my testing.

While this product wasn’t bad, I still prefer using a competing product even though it’s a more expensive one. Not because the competing product is so much better for the end user, but because it has good community forums, a larger user base, and less complex installation procedures.

Personally I find the VMware forums an excellent resource, but I’m still waiting for the next version of the forums to be released as the current version is too slow for my patience and the rich text editor is useless. The current version was introduced in late September 2007 and had major problems shortly after being put into production. The stability problems were resolved, but the performance and formatting issues still isn’t perfect. As I’m not currently as active on the forums as I once was, I’m now blogging using the same system, just to be reminded of how bad or good the situation is. I could have used Wordpress or another known working blog platform, but then I wouldn’t get reminded of the status of the VMware forums.

There are still lots of activity on the VMware community forums, so it seems all are not as impatient or formatting oriented as me. I have tried joining back at some occasions, but found myself switching to another tab while waiting for a response from the community. Having the forums available is still something I value highly even if I’m not active there on a daily basis.

By subscribing to RSS feeds on Planet v12n (“planet virtualization”) I’ve been able to get a great deal of info without using the forums. The weekly VMTN Roundtable podcast on Talkshoe is also good. It also has chat during the show so it’s much more valuable in a live session than a recording.

There is also an irc channel on freenode, #VMware that I’ve been part of for a few years now and that’s also a place where it’s possible to discuss VMware related topics with other peer users.

In addition to these arenas I’m now also following the progress in the virtualization field via Twitter. It’s amazing how Twitter has attracted the right users who will point you to blog posts and good community postings on a regular basis while there’s also some direct discussion going on right there.

To sum things up, the bottom line is: VMware have some very good products that people like. VMware have supported their users by giving them an arena to discuss their product and share their experiences in an unmoderated fashion. This has lead to users spending a lot of their spare time writing about the products on both the community forums and in other arenas. The users now know more about the products than they would by only reading the documentation. This means that the users are now better at what they do, certified or not certified. Several books have been written by community members. The importance of such a community can never be underrated.

Back in the mid-late-nineties there were one global market leader in the firewall/vpn market: CheckPoint. There were some other players as well (Gauntlet, Raptor, Watchguard), while the big players, Cisco and Microsoft had no solutions at the time.

CheckPoint's solution was a software solution that injected a firewall driver between the ip stack and the nic drivers, and could because of this also protect the ip stack of the OS it was running on. It supported several platforms such as SunOS, Aix, HPUX and Windows NT. CheckPoint's development at the time was mainly targeting SunOS and it was back then the preferred platform.

I did my first CheckPoint training and installs back in '96. These were however on HPUX due to current vendor ties of my employer. In the years coming I was involved in quite a few installs in both small and large businesses. Most of them were in the process of getting an internet connection while some of them needed firewalls internally or for vpn's to business partners. With CheckPoint you had a nice GUI, an inspect language for defining new protocols (based on content of the data stream), application proxies, and a very good log viewer. CheckPoint was king of the hill, functionality wise. But there were annoying issues with licensing and the process for acquiring access to 3des licenses/software was quite awkward. The price model was also of the more expensive type.

Cisco had bought firewalling technology back in '95 that they had been developing further and tried adjusting the command set be similar to IOS. I attended Cisco Pix training in 1999 after Cisco had some campaigns where they had shown off their Pix firewall and declared it ready for business. Cisco's main argument was performance, but they also used other tricks in their marketing. The cisco Pix was also priced cheaper than CheckPoint, and for small customers it didn't really matter so much what product they chose as both would solve their core need of seperating/protecting networks.

We did however get a request from a larger customer in the financial sector who had decided to go for the Cisco Pix. We had installed CheckPoint at several similar environments before, but it's always hard to estimate how much time you're going to use as each environment is unique. In that particular case I think it's safe to say that we did get everything working in the end, but we probably used five times as many hours of setting it up than we would if we had used CheckPoint. One of the comments that came up during troubleshooting these Cisco issues was that you can compare Cisco with a Rhino: "They can have a slow start with products or arrive late to the table, but once they're gaining some speed you can't stop them." And at that point Cisco's product wasn't as mature as we had wanted it to be.

During this same period (late nineties-2001) we also used Axent Raptor Firewall for quite a few customers. Raptor was a product that had a very slick management interface, that would allow relatively complex environments to be configured easily. It was one of the earliest MMC applications that I came in touch with. This technology was later bought and brought to failure by Symantec who replaced the MMC interface with a java based one.

CheckPoint have however done little to their enterprise pricing other than that they have included more and more functionality into their package. For the SOHO market they launched appliance boxes through their daughter company Sofaware that were reasonably priced.

Several other competitors were also joining this field and most of them have taken a slice of the cake. Cisco also launced a new and much improved technology in 2005 with their ASA series to a fraction of the price that CheckPoint demands. Microsoft also launced MS Proxy server (later known as ISA), but it's functionality and performance wasn't covering the same area as CheckPoint and was often used to bring complementary functionality alongside with a "standard" firewall.

There is no doubt that CheckPoint have lost customers during these past 10 years (especially in the non-enterprise market), but they have also got new ones. In the SMB market they have lost a quite huge part of their share. For big environments, CheckPoint still have a large market share due to their still superior management software, even though their pricing haven't changed much (even with the new price model launced 3 weeks ago). According to themself, 100% of fortune 100 and 98% of fortune 500 are using a checkpoint security solution (they now also have many other products than firewalls). Other than having a good product line it has also helped that they have had many partners providing supplementary and/or prepackaged solutions. So it is possible after all to survive a Rhino, but you probably can't do it without some good products, partners, marketing, focus and timing. Having good products still wasn't good enough to keep SMB customers though, where price often is a more important factor than functionality as long as cheaper competing products are "good enough".

So how does VMware fit into this picture? Well, VMware is facing a similar situation now as there are several players that have just entered their core market. Citrix XenServer is the product that currently seems to beat VMware on raw performance, while Microsoft Hyper-V seems to be the most important comptetitor in the long term. In this game, Microsoft has taken the role as the Rhino and they have put some tremendous effort in order to get into the hypervisor game (and they missed their own deadline by a few years). Until now, VMware has shown to have smart roadmap in order to face the competition by being first to the table both with new technology and by bringing free products to the market (first Player, then Server and now last year ESXi was launched). We have seen Microsoft preaching the importance of the functionality that will be available in Hyper-V R2 since 2006, and they are repeating this at every opportunity now.

Even though Citrix have had fairly good technology since 2007, it seems that the mainstream wide adoption of XenServer isn't coming too quickly, even though they also include live migration for free in the version that will be available in two weeks. Citrix have over the years bought many types of products, but there's only one product package (that changes name every leap year) that people associate with Citrix. I still think that we'll see some adoption of XenServer in the product area that Citrix know best: Server based computing for end users (XenApp & XenDesktop).

Citrix and Microsoft have been both close partners and competitors for quite some time, and on the virtualization side, they are cooperating to bring functionality to each others hypervisor while competing with VMware.

In the months to come we will probably see a lot of marketing and technical sessions from MS showcasing Hyper-V R2 and in the meantime it will be exciting to see VMware launch vS^HH the next version of Virtual Infrastructure with lots of new bells and whistles. While no marketing stunts can compete with the stunt of giving something away for free, VMware aren't doing so bad either. Their focus on Fusion for OSX has IMHO been a ground where they have showed excellent marketing skills, making them the current market leader also on OSX. I guess that it has been a good drill for the upcoming era of new products being launched. I'm surely looking forward to an exciting time ahead.