ronzo

(Yippie, first BLOG post of the year 2008)

We've been testing the GA release of ESX 3.5 almost continuously since it came out and except for our GUI problem, the rest is looking good! Actually it seems a little more peppy. The client over VPN is faster and more responsive, at least for me.

Here is the link explaining the GUI problem:
http://communities.vmware.com/thread/117115

Besides the problem with serial ports as a pipe, the rest was good. We made a change to put the serial pipes in another path, that is allowed by the /etc/vmware/configrules, and disabled the GUI for now, and it all works.

Because most of what we do is actually in a VM (the VBA) we had very little to change. Even on the actions we do in the console of the host (creating the VBAs and such) nothing needed to be changed (except for the serial pipe). This makes esXpress pretty flexible, as it will run on any version of ESX patch levels. So far, VMware API changes have not affected us.

In our version 2, we ran in the console space, and when we moved to ESX 3.x we moved into a VBA for the backup itself. The whole VBA idea was a good one, it was the next evolution of esXpress.

Our next version under development right now will take this to a new level.

People always ask us about the VBAs, why are we the only one doing backups like this?

How did we come up with this idea?

Even though the VBA was my idea (most of the other good ideas came from Caleb Shay) for a change, I cannot take total credit for it. If it was not for a conversation here on the VMTN (http://communities.vmware.com/message/422323) we would not of been pushed to come up with a better solution then running in the console like we currently were doing at the time.

I remember the day, it was rainy out, and we (Ken and Caleb too) were at a customer site, and after reading that post we had to go out for a smoke and a meeting. We were sitting in my truck (Ford F250 Harley Edition) and discussing... I was like "Let's take Jason at face value and assume he is correct. What can we do?". After about 10 minutes of talking (quizzing Caleb, our Linux expert) I came up with running it all in a tiny Linux VM, which we coined as a VBA.

Thank you jason for pushing us into this solution using VBAs!
And we just passed the 1,000 customer mark, and still going. If anything, our development is moving faster.

Since testing has been good (for the most part) with ESX 3.5, we finally released our esXpress version 3.1. GA was released today (Jan 1, 2008).

Thank you to everyone who has helped with the testing of esXpress, we really do appreciate all the feedback we get.

thanks
ron

It's been a long 2 weeks, a lot of testing. We've been doing the QA testing on esXpress v3.1 RC-8 and we are getting close with a final release of version 3.1. Version RC-8 just went up.

A little history on our releases:
2006-03, First Beta of esXpress v2
2006-05, GA of esXpress v2
2006-08, Beta of esXpress v3.0
2006-12, GA of esXpress v3.0
2007-06, Beta of esXpress v3.1
2007-12, GA of esXpress v3.1 (predication)

Let’s talk about software development cycles.

Our version 2 of esXpress was originally released in March 2006 for public beta and GA release was May 6, 2006.

We released a few maintenance versions after that, but we have not updated the code since Dec 30. 2006 when we released esXpress 2.3-5 for ESX 2. We only released this version because of some things we learned from creating version 3 of esXpress and back-ported some of the features.

In terms of software reliability, version 2 of esXpress is pretty stable. It has been almost a year since any changes has been made to the code, and we know a lot of people are still using it. You can install version 2 of esXpress onto any version of VMware ESX 2.x and it works. We want that same level of comfort with our version 3 of esXpress. We believe in solid releases that work, not the version of the day. It's pretty impossible to call a 2 week beta cycle a real beta test.

Now back to software testing. I’ve been doing the QA for the 3.1 versions, and I’ve let a few bugs slip by. But in the past few weeks we have found and squashed a few old bugs including a Divide by Zero error in the stats, when you are running a VM backup in parallel mode, and all disks fail.

The best part about VMware, we created a monstrous QA test using literally a 100 VMs that cover just about every scenario that esXpress might encounter. VMware even makes testing VMware third party products easier!

On my dev box I run about 8-12 VMs at any time for my testing before moving to the big box. Just to give you an idea of how much we test, I personally have run 687 backups in the past 9 days on this box. This does not even count the other testing others have done before it was turned over to me. This is just to certify the 3.1RC-8 version is good.

This RC-8 version has some minor enhancements over the RC-7 version. We have streamlined the backup process to make the 'Backup All' better. If you have 8 VBAs enabled, we now try to achieve that and get 8 of them running at a time, and keep 8 of them running for the entire backup run.

Previously, the delay between starting up new backups was not as quick as it needs to be. It worked well here on my dev box with 12 VMs, but at one our test sites it did not. When you have Hosts that are really running 100 VMs, things work a little different then a Host with a couple of VMs.

We also added some fuzzy logic to the backup ordering. When you have to backup 100 VMs (remember, this is on one host) you need to be able to automatically figure out the best order without having to manually manipulate each of the VMs. We give each VM a weight, and for each different status (such as vm_order, power, snapshot, version) we add extra weight. The more weight a VM gets, it sinks to the bottom of the backup order list. (Note: this is another good feature to steal.) This way VMs that will probably be skipped will be at the bottom of the list and attempted last.

Sometimes it seems like most of the work we do is at a level that most people we never even know it exists. But that is a good thing, it means that esXpress will probably mostly work unless you have some type of environmental failure in VMware, Storage, Network or Backup space. We've spent the last week really working the obscure bugs out of the engine that runs the VBAs. Even adding another layer of having the backup try again on failure, if the failure was something immediate.

Besides running hundreds of tests on my dev server, I’ve also been running them at a customer’s location for a reality check. I’m backing up 8 hosts that represent some 400 VMs, and 12 TB of data. All this gets backed up nightly. Two of those hosts have 100 VMs each.

This customer used to be the biggest customer of those other guys (200 CPU), but not anymore. And what makes it more amusing, he gave us a list of feature requests earlier this year, and that is what drove most of the development for version3.1. Gotta luv users who know what they want. This is just one customer among many who let us use their environments for our QA testing (which they get personal support in return for). It is nice when customers trust you enough to give you root access into their systems.

Hopefully we did enough testing on version 3.1 to make it as reliable as possible, but only time will tell.