4 Replies Latest reply on Jun 5, 2008 8:41 PM by wizdude2

    Beta 2 in *Production* server - it works and it Rocks!

    markokobal Enthusiast



      I've been using VMware for some years now. From the release of Server 1.0 also in development and light-weight production servers environments. As 1.0 has some known limits (only beta 2-way SMP, memory limits, no paravirtualization...) I was waiting for 2.0 with great anticipation - 2.0 beta1 was a disappointment, so I decided to wait for 2.0 beta2.



      From what Azmir told as at the VMware World Europe 2008 I give it a test drive the day beta2 was released. And I was more then pleased!



      (For the ones that do not want to read the whole story just skip to the end where I put up the necessarily optimizations.)



      My Linux choice is Debian - it has always been. So I put up a host - installed Debian Etch 4.0p3, recompiled its kernel with bigmem support and was ready to roll-in the VMware Server 2.0 beta2. It couldn't be easier! I've actually got running the 2.0 beta server from scratch in less then one hour.



      The machine isn't some high-end piece of hardware - just an old but good IBM xSeries 225:



      • 2x 2.4 GHz Intel P4 Xeon, hyper threading enabled (so 4 logical processors)

      • 8GB RAM (4x 2GB DDR266 Kingston)

      • 1x 80 GB IDE 7.2k single

      • 2x 36 GB SCSI 10k in RAID1

      • 4x 500 GB SATA 7.2k in RAID5

      • 1x 1 Gbps Broadcom NetExtreme



      My first move was to check out the Web interface. It works, it actually works. Cool. So, let's put up some virtual machine on.



      Virtual machine:

      • 2way SMP

      • 3584 MB RAM

      • virtual disk1 on RAID1 host field

      • virtual disk2 on RAID2 host field



      For web servers (and many other services), my choice is FreeBSD - it has always been. So I've rolled-in the FreeBSD 7.0 for the guest virtual machine. At the first glance it's not looking good. The idle CPU usage of the guest as shown in the host is about 35%. OK, lets go on - install vmware-tools (need to use the FreeBSD 6.0 compatibility libraries - come-on  VMware, make the tools for FreeBSD 7.0!), then I recompiled kernel with standard tweaks (100 Hz kernel timer...) and with enabled the all-new ULE_SCHEDULER. Reboot and woila - the idle CPU usage of the guest as shown in the host is about 3-5% which seems to be reasonable.



      I did some more testing and the virtual machine seemed to be stable and performing good. So I said to myself - let's take a chance and move to this vm a real high-weigh production web server. So I did - and after a week I can say just this - Server 2.0 beta2 Rocks!



      OK, It's true that I have only one vm, but still, this server serves about 200 web pages, all supported by MySQL DBs for content managers. I've got about 5 Mb/s of constant network traffic and around 120 select queries / second in MySQL server. So it is quite a load. And it just works.  






      As much as I love FreeBSD, I hate VMware & FreeBSD Foundation for not supporting it a little bit more in VMware virtualization. Knowing this I'm using Linux as guest where true performance is needed.



      So I went on and installed a Debian Etch 4.0p3 as second guest. I wanted the paravirtualization, so I downloaded kernel 2.6.24-4, enabled it and - WOOHA, this vm with paravirtualized kernel is fabulous! The idle CPU usage of the guest as shown in the host is about just 0-1%.



      Up to now I haven't got enough time to put or move a real high-load production server into this second Linux guest machine - but I'll probably clone the configuration and data from the first guest (FreeBSD) to this one - then will do some tests and compare the the load and performance when serving real life load.






      Nevertheless, I did one comparable test in both guest machines. Thats what I did for both machines:



      • same virtual HW configuration for both virtual machines (as shown above);

      • power on just one guest machine, kill all unnecessarily processes and ran the UNIXbench.



      I know that UNIXbench isn't the most modern and best tool to measure performance, but it can give me some idea of how fast the basic UNIX operations are doing in host (Linux), guest 1 (FreeBSD) and guest 2 (Linux paravirtualized).



      The final scores of UNIXbench are (see details in attachments):



      host (Linux): 442.2




      guest 1 (FreeBSD): 47.6

      guest 2 (Linux paravirtualized): 219.8








      OK, now let me finish the story. So, VMware Server 2.0 beta2 is really great and ready to do some real life, high-load production servers. Especial if you are planing to use paravirtualized Linux guests.



      BUT! Beware! Those cool performances WILL NOT come out-of-the box, oh no, many optimizations will have to be done before.



      • Rule no.1: HOST: optimize disks: spread the load through many disk fields and set writeback and noatime parameters for Ext3 partitions - I can not stress how important this is - it really does impact on whole guest performance;

      • Rule no.2: HOST: use /dev/shm for tmp dir: (/etc/vmware/config: tmpDirectory = "/dev/shm", make sure you have mounted /dev/shm as big as your RAM is);

      • Rule no.3: GUEST - all: do standard optimizations:







      sched.mem.pshare.enable = "FALSE"

      mainMem.useNamedFile = "FALSE"

      MemTrimRate = "0"

      MemAllowAutoScaleDown = "FALSE"

      • Rule no.4: GUEST - all: install vmware-tools, enable time synchronization;

      • Rule no.5: GUEST - all: use vmxnet or at least e1000 for virtual network;

      • Rule no.6: GUEST - UNIX: reduce timer interrupt demand (100 Hz);

      • Rule no.7: GUEST - Linux: use tickless timer option (CONFIG_NO_HZ);

      • Rule no.8: GUEST - Linux: compile paravirtualized kernel.






      Well, that would be all. I just can not stress hard enough how well performing the Server 2.0 beta2 is - just make sure you will optimize it properly. Like from almost all software you can not get best results just out of the box - this is even more obvious in the world of virtualization.



      Now all what is left for me is to wait for the GA release of Server 2.0 where I do not expect many improvements, but I do expect for VMware to add official support for additional guests (FreeBSD!, Debian...).






      Kind regards, Marko Kobal.

        • 1. Re: Beta 2 in *Production* server - it works and it Rocks!
          sej7278 Enthusiast

          most of the places i've worked you'd be fired for using beta products in production - especially ones that are going to expire!


          nice to see freebie guest support and linux 2.6.24 host support is working well, 2.6.23 worked fine for me - without the need to patch like 1.0.5


          i certainly haven't seen the performance you have - especially with paravirtualisation which made little difference for me. 1.0.5 is still faster than 2beta2 (with debugging off, optimisations etc.)

          • 2. Re: Beta 2 in *Production* server - it works and it Rocks!
            markokobal Enthusiast



            sej7278, where I work, using beta products in production is welcome as this is the only way to keep up with the newest technologies and trends. However - it is quite obvious you can not play with production without backup. For the server I've been talking about I do have two backup machines in different locations with full replication of data - so if the machine running beta2 would die or cause me problems I could switch to backup machine instantly.


            As of performance - yes, I agree to you - 1.0.5 is still faster than 2beta2 (the only exception is Linux with paravirtualization which is truly faster). So I do hope VMware will put some extra effort in polishing performance issues before GA or later revisions of GA.


            So, VMware, keep up the good work and do not let us down - we want Serever2 GA to be a true performer!


            Kind regards, Marko Kobal.

            • 3. Re: Beta 2 in *Production* server - it works and it Rocks!
              wizdude2 Novice


              Hi Mark,



              I'm testing out vmware 2.0b2 under slackware 12.1 with some huge performance issues to do with guests when they are performing base operating system installation.



              i'm keen to know if there are any specific options in your host kernel configuration which may be affecting my performance.



              are you aware of specific kernel options that you know affect the performance of vmware server?



              i'm more than happy if you wish to post your kernel .config for me and I'll happily compare it to mine



              my host kernel is i was forced to go for 2.6.25 since the kernel revisions lower than this do not support the Realtek 8168B PCI Express Gigabit Ethernet which is built onto my motherboard. i'm happy, however, to install an intel gigabit nic and bring the kernel revision down lower if this is also a known issue.



              feedback appreciated



              cheers, wizdude



              • 4. Re: Beta 2 in *Production* server - it works and it Rocks!
                wizdude2 Novice


                just a side note to anyone that cares.



                after having many problems with linux guests running slow under 2.0b2 and doing a lot of research, i ended up doing performance tests with ESX 3.5.



                i was very disappointed with the paravirtualization. it gave me virtually no speed increase at all.



                i eventually tracked the problem down to my guest kernel. i was running version and it appears that the paravirtualization support and/or VMI support is broken. i downgraded to a kernel and paravirtualization worked.



                hope this might help someone running under various vmware environments. 2.6.25 appears to have broken support for vmware paravirtualization.



                cheers, wizdude2