1 2 Previous Next 18 Replies Latest reply: Jul 29, 2008 7:50 AM by getut RSS

    Tips for Improving Performance On Linux Host

    gpc Lurker

       

      In my first trails of VMWmare server (my first real use of a VMware procuct on a Linux host) I have experienced VERY poor performance of Windows XP and Vista guests with >512MB memory. Guests would lock up for >60seconds as a time, even doing simple things like opening a command prompt.

       

      Searching these forums, I found this seems to be an issue many users have experienced with VMWare products on linux hosts. I found many recommendations for tweaking settings, but none solved the issue for me.

       

      However, the advice and experiences I found did help me greatly with tracing the problem, and finding a solution that has significantly improved the speed of hosts for me. Therefore I wanted to share my solution, in the hope it can also be of help to others.

       

      Before going further, I should note this solution has worked for me on the following host configuration. It may or may not be of use with different configurations (in particular 64bit host OS's)

       

      Host System:

       

          - Dell T740, Quad core Xeon (E5420), 4GB RAM, 4x SATA disks

          - Gentoo Linux, 32 bit, Kernel 2.6.24-gentoo-r4 with HIGHMEM64G enabled 

          - One disk is dedicated to virtualk machines, to avoid other processes causing delays in disk access.

          - XFS Filesystem for VMs

          - All VM's configured  with debug and logging disabled (running vmware-vmx not vmware-vmx-debug)

       

      Symptoms:

       

          - Windows guest performance inversly proportional to the amount of RAM allocated (the opposite of what one might expect!) More memory resulted in >60s hangs of the guest OS, with no system load on host or guest, and no other guests running - just opening a windows command prompt to >60s for example.

          - Excessive disk thrashing (on the vmware disk) whenever the guests were paused.

          - Top showing high iowaits.

          - During pauses in guest VMs the host system showed no noticeable slowdown, only the guest VM process was affected.

       

      The solution (for the impatient - more detail on why I think this works below):

       

          Set the following values in /proc/sys/:

              vm/swappiness = 0

              vm/overcommit_memory = 0           

              vm/dirty_background_ratio = 2

              vm/dirty_ratio = 100                              * This is the real key

              vm/dirty_expire_centisecs = 9000          * Not recommended if you have an unreliable power source!

              vm/dirty_writeback_centisecs = 3000

       

      The following did not appear to make a difference alone, but I have left them set anyway:

       

          In /etc/vmware/config:

              prefvmx.minVmMemPct = "100"

              prefvmx.allVMMemoryLimit = "3200"

              prefvmx.useRecommendedLockedMemSize = "TRUE"

              svga.enableOverlay = "FALSE"

              MemTrimRate = "0"

              sched.mem.pshare.enable = "FALSE"

              MemAllowAutoScaleDown = "FALSE"

              mainMem.useNamedFile = "TRUE"          * Not FALSE as recommended by others!

       

          Run at boot:

              blockdev --setra 16384 /dev/sdb               * This is the disk with my VMs on. Other drives are set differently for my requirements.

              echo anticipatory > /sys/block/sdb/queue/scheduler               * Again, sdb is the disk with my VMs. Other drivers are using CFQ.

       

       

      Personally, I am not happy with just changing settings without some understanding of what I am changing and why. So for those like myself, here is a brief explanation of how and why I reached the above solution.

       

      Further investigation and other posts on the forums led me to the conclusion that the pauses were a result of iowaits while the disk backed memory file (VMEM files) were updated. I saw many recommendations to set:

          mainMem.useNamedFile = "FALSE"

      The justification of this from many posts was that it turned off the VMEM file. However this is not strictly true, it just uses an unlinked (deleted) file in $TMPDIR instead. I learned this the hard way when my /tmp partition filled up:-(  However even setting $TMPDIR to a directory on my VMWare disk, this still did not give me a great performance boost, and the VM's still had the pauses. Other users investigation this had reported that Linux seemed to delay writing back changes when the file was unlinked. I'm not sure if this has changed in the new Kernels, is different in the XFS kernel driver than ext3 used by others, or is related to my 32 bit, HIGHMEM64G config, but it doesn't seem to make a difference for me. To avoid delays suspending and resuming guests I set this back to TRUE.

       

      I followed recommendations for tweaking host performance with the other /etc/vmware/config options above, though alone they did not help. Other forum posts document the purpose of these well, so I will not repeat it here.

       

      Next I found recommendations to set the following in /proc/sys:

          vm.swappiness=0

          vm.overcommit_memory=1

          vm.dirty_background_ratio=5

          vm.dirty_ratio=10

          vm.dirty_expire_centisecs=1000

       

      These values did not help me, in fact the vm.dirty_background_ratio and vm.dirty_ratio values were the same as the defaults for my kernel anyway. However it did lead me to investigate these options further. I discovered:

          vm.dirty_background_ratio:

              Defines the percentage of memory that can become dirty before a background flushing of the pages to disk starts.

              Until this percentage is reached no pages are flushed to disk.

              When the flushing starts, then it's done in the background without disrupting any of the running processes in the foreground.

          vm.dirty_ratio

              Defines the percentage of memory which can be occupied by dirty pages before a forced flush starts.

              If the percentage of dirty pages reaches this number, then all processes become synchronous,

              they are not allowed to continue until the io operation they have requested is actually performed and the data is on disk.

       

      Now, by my understanding (and I am by no means an expert on this, so please correct me if I have this wrong), every VMs RAM is mapped (with mmap in 1MB blocks) to a file on disk.

      <aside> I cannot claim to understand the reason for this - my host machines RAM is not disk-backed (No the hosts swap does not count - the guest OS's can have swap working in the same way as the hosts), so why when I tell VMWare to give a guest 2GB of my RAM, and not to put any in swap, does it back it with a file on disk. I have yet to see a convincing explanation of this apparent madness (speeding up suspend/resume is not a good reason in my book, I can live without this.)</aside>

       

      So, now imagine the scenario, my Windows guest thinks it has 2GB of RAM, which it merrily accesses as it sees fit, running apps, caching disk etc. However every time the guest RAM is written, the Linux host sees a new dirty page that must then be written back to the mmap'ed file on disk. Now 2GB of RAM in the VM is a lot of pages on the host, and therefore the dirty page count can increase rapidly even when the guest does simple task (eg. caching disk reads, which the guest kernel believes is improving performance, when in fact it is actually slowing things down by causing the Linux host to perform slower disk writes as a result!)

       

      OK, so we have the idea, the guest is merrily changing RAM and creating dirty pages. Now the Linux host kernel needs to decide when to commit these dirty pages back to disk. This can happen in 2 ways, in the background by a pdflush process, which other processes still run, or synchronously, causing any processes dirtying RAM (eg. vmware-vmx) to wait for the dirty page writes to complete. Clearly the synchronous option is going to have severe consequences for the guest VM process. This, to me, completely explains the huge and regular pauses in guests with large memory allocations.

       

      This conclusion reached, I reached the opinion the best way to manage writing back the vmem files was:

       

      1) To avoid synchronous writes, and have all writing done by the background pdflush process

      2) Not to accumulate so many dirty pages, as a long synchronous write becomes unavoidable.

      3) Not update the file too often. Guest RAM could change fast, we do not want to cause a disk write for every change.

       

      Now, in my understanding, each of these seems to relate directly to  a /proc/sys parameter:

       

      1) This calls for a high value for vm.dirty_ratio

            I am sure there a good reasons why this is set as low as 10% in recent kernels, however, for the configuration discussed above, a 2GB RAM host very quickly causes 10% of dirty pages. Therefore I maxed this out at 100%, of the belief that all suspending processes for disk writes should be a last resort. Maybe somewhere a bit lower would be better, but 100% has been working well for me so far. I'd welcome others thoughts on this.

      2) This calls for a low value for vm.dirty_background_ratio

            From my tests the background pdflush process does not appear to adversely affect the performance of the system, so I set this down to 2%. My theory is that the earlier background writes are started, the lower the chances of reaching vm.dirty_ratio and forcing synchronous writes.

      3) This can be controlled by vm/dirty_expire_centisecs

            I set this up at 9000, causing Linux to wait 90seconds before committing changes to disk. It is important to remember that this affects the whole system, not just VMWare vmem, and caching writes for 90 seconds could well be considered a 'bad thing', especially if you are value data integrity and are not running off a good UPS. From what I can tell this value is ignored once vm.dirty_background_ratio is reached, and thus this does not counter objective number 1.

       

      As I said before, this works for me. I am not an expert, and there may be good reasons for not using these settings (please let me know!). What I do know is that the above configuration has restored my confidence in VMWare server on Linux (I was for a brief period considering having to try a Windows host OS!)

       

      I hope this information is of use to others.

       

      Regards,

      Graham

        • 1. Re: Tips for Improving Performance On Linux Host
          LKPDsmith Novice

           

          I too have noticed periods when vm's appear to lock up.

           

           

          I have refrained from modifying these settings in the belief that there are set to reasonable levels.

           

           

          I would be very interested in hearing a technical response to these issues, and the recommended settings that gpc has used, by the vmware staff.

           

           

          • 2. Re: Tips for Improving Performance On Linux Host
            chort Enthusiast

            These are some great tips.  I was experiencing abysmal performance with an Oracle Enterprise Linux 5.1 host and CentOS 4.x guest.  Both host and guest were running at 2.00-5.00 load average with nearly no activity.

             

            After setting these values on the host:

            dirty_background_ratio=4

            dirty_ratio=64

            swappiness=16

             

            I have noticed a dramatic improvement in guest performance, and it's down to around 1.30 load.  The host has almost no load what so ever now (well under 0.30).  The only changes I made were the above.

            • 3. Re: Tips for Improving Performance On Linux Host
              getut Enthusiast

               

              I have tried all of the above and am still getting huge iowaits that bring my VM's to a stall.

               

               

              I'm between a rock and a hard place. I first set up this monster server (dual quad core xeon with 32GB of RAM and an 8 drive Perc 5/I RAID array set up with 3 volumes. 2 drive RAID 1, 2 drive RAID 1 and a 4 drive RAID 5 with the last volume a 2TB volume) as an Ubuntu Gutsy server and VMWare Server 1.0.4 and ran into performance issues with the IOWait thing. See this thread for all that was done.  http://communities.vmware.com/message/798400#798400

               

               

              I finally downgraded to Feisty 64bit and the performance issues went away. Well here is where the rock and the hard place starts. I got clock drift so bad and couldn't resolve it that it was causing kerberos issues and problems with Domino server so I started testing Hardy and VMWare Server 2.0 Beta 2. It solves the clock drift issues but now the iowait issue is back.

               

               

              I am forced to run this beta in a production server because the clock drift was causing breakage of services where this iowait is just causing severe hangs in performance. Slow is better than broken, but I've still got to get this fixed before my users riot.

               

               

              What is cauing VMWare such performance issues when paired with RAID on later kernels?  Does ANYONE have any additional ideas on steps that can be taken to make this better? I really hoped that the /proc/sys/vm changes were going to be the key for me, but they didn't even slow it down. My munin graphs for processor usage still look like their bleeding from the top down because of the huge iowaits and the same load didn't even phase this server under Feisty 64bit, but does under both Gutsy and Hardy.

               

               

              • 4. Re: Tips for Improving Performance On Linux Host
                kingneutron Master

                 

                #1 Rule: You do NOT EXPERIMENT on a *PROD* SERVER.

                 

                 

                 

                 

                 

                If I were you, I'd go all the way back and install ubuntu 6.06 LTS64 on that thing (supported for 5 years on the server), and buy a support contract from Vmware.  Server 2.x BETA is NOT anywhere near production-ready!

                 

                 

                 

                 

                 

                Also - what are your kernel parms?  Have you tried using " clock=pit " or the like?

                 

                 

                 

                 

                 

                ./. If you have appreciated my response, please remember to apply Helpful/Correct points. TIA

                 

                 

                • 5. Re: Tips for Improving Performance On Linux Host
                  getut Enthusiast

                  Thanks for the reply, but I think you missed my rock and a hard place comment. I wasn't running VMWare Server 2 Beta when I was on Feisty. I was running 1.0.4 and still had the same performance issues.

                   

                  I had good disk performance on down level kernels BUT I got unacceptable clock drift (slow clock... after 5 minutes when VMWare Tools recorrects, the clock would have only ticked off about 45 seconds). It was causing kerberos errors on all my clients and on the Sametime server it would cause service disconnects when the time was corrected by VMWare Tools. I tried all the usual clock drift fixes and nothing made a difference, So I couldn't run with the above scenario. The slowness I'm experiencing now at least isn't causing complete service failures but the slowness is proving very hard to fix also.

                   

                   

                   

                  But to answer your question... no I have not tried clock=pit on Hardy. I did however try it on Gutsy in the thread I mentioned above and it didn't fix the slowness. Right now I am using pure stock install kernel options on Hardy. I'm getting <better> performance than I did on Gutsy, but not by a tremendous amount.

                  • 6. Re: Tips for Improving Performance On Linux Host
                    kingneutron Master

                     

                    --What disk scheduler are you using?

                     

                     

                     

                     

                     

                    ' dmesg|grep sched '

                     

                     

                     

                     

                     

                    --I was having disk-I/O issues with CFQ (default for Ubuntu 7.04) and things got better when I added to grub:

                     

                     

                    " elevator=deadline "

                     

                     

                     

                     

                     

                    --This is what my grub menu.lst kernel parms entry looks like for 64-bit xUbuntu 7.10:

                     

                     

                    "  ro vga=ext clock=pit nohpet elevator=deadline "

                     

                     

                     

                     

                     

                    ./. If you have appreciated my response, please remember to apply Helpful/Correct points. TIA

                     

                     

                    • 7. Re: Tips for Improving Performance On Linux Host
                      getut Enthusiast

                       

                      Sorry for the long reply KingNeutron. I just found your response today.

                       

                       

                      This is Ubuntu server 8.04 which evidently is defaulting to deadline since I checked and that is what it is already running. BUT I am trying other scheduling options.

                       

                       

                      I just changed the all VMWare volumes to NOOP and will run that way for a couple of day (if improved) and then try CFQ and finally anticipatory. If I see immediate poor performance I will go to the next option immediately.

                       

                       

                        I have left SDA alone which is the OS volume and 2 very low disk usage VMWare images, so it is still set to the default of deadline. I will post back in a few days if any one of these options makes a noticeable difference.

                       

                       

                      • 8. Re: Tips for Improving Performance On Linux Host
                        ian_tk Lurker

                         

                        This is really addressed to getut and his first post regarding huge io waits,  I am posting here as this is where the focus seems to have switched.

                         

                         

                        I am definately still a Linux newbie, certainly when it comes to vmware, and this is my first foray into creating a fully fledged vmware server.  I started with Workstation 6 on Vista Ultimate, which impressed me a great deal with it's ability to run my old xp install and thereby circumvent all the downside of Vista with it's lack of support for anything remotely yesterday such as vb6 and my HP scanner/printer. I added a fedora scalix mailserver and a 2-3 other test distributions with no noticable performance impact (My vista machine has a quad intel processor, 1.2T hardware adaptec raid and 7GB ram - should have been 8 but don't go there!)

                         

                         

                        Accordingly when asked to build a new server for our dealership to replace the 4 individual linux servers we had I put together the following:

                         

                         

                        Supermicro X7DVA motherboard

                        twin Xeon quad 5410 processors

                        LSI Megaraid 300-8XLP hardware raid

                        8 x 750GB sata disks

                        8GB Memory

                         

                         

                        I initially developed the virtual machines on my Vista box while waiting for various parts, they consisted of:

                         

                         

                        Postfix incoming mailserver for screening email on Ubuntu 7.10 32 bit (1 processor - 512MB ram)

                        Scalix 11  mailserver on Fedora 7 32 bit (1 processor - 754MB ram

                        Apached Webserver on Ubuntu 8.04 64 bit (2 processors 1GB ram)

                         

                         

                        The fourth machine is due from a third party after installation,  All machines ran on the Vista box without a problem, though I did not load them particularly.  I then installed them on the new server which had been configured with Ubuntu server 8.04 64 bit, and VMware Server 1.7. This had two raid 5 arrays configured, The first used 4 disks and had two logical drive on it, sda was 47GB and intended for the host system, sdb was 2T and intended for the Virtual Machines, the second raid 5 array used 3 disks with 1 logical drive, sdc was 1.5T and intended for backup and future expansion. The eighth disk was configured as a hot spare.

                         

                         

                        I loaded them all up and everything worked, however I noticed that whenever I transferred large amounts of data to the host system the virtual machines would slow to a crawl, and when we started to test the webserver we noticed intermittant performance problems, this when the server is not in production use.  Eventually, with all the other software development issues coming to a close I started to look more closely at the performance and realised we were going to have a serious problem once we went live.

                         

                         

                        After a day of googling to find potential solutions I had tried most of the optimization tricks covered in these posts, including upgrading to vmware server 2 RC1, and I was about to downgrade to Feisty 64 bit in line with getut's previous thread when I came across this thread.

                         

                         

                        To summarise what my understanding of the problem is:

                         

                         

                        There is an issue, particularly with later distro's, using (hardware?) raid with VMware

                        This manifests itself with huge iowaits (64s not uncommon)

                        The underlying cause seems to be vmware writing all dirty memory to disk even though everything is configured so that no swapping should take place.

                        This disk writing is configurable using  mainMem.useNamedFile = "FALSE" which can control where this writing takes place. (I have not pursued this particular avenue yet)

                         

                         

                        Downgrading to Feisty 64 bit solves these issues but raises time keeping issues instead.

                         

                         

                        If I am wrong on these points could someone please advise.

                         

                         

                        The question is, getut, have you got any further during the last month?

                         

                         

                         

                         

                         

                         

                         

                         

                         

                        • 9. Re: Tips for Improving Performance On Linux Host
                          ian_tk Lurker

                          I thought I would check back with my results for the day, I fitted a standalone small sata harddrive to one of the motherboard SATA ports rather than the raid card, I then set mainMem.useNamedFile = "FALSE" in each vmx file for the clients. Finally I mounted the new harddrive on /tmp and made sure vmware was using this as its tmp store. Result: not absolutely perfect, but nearly so, no iowaits reported above 2 seconds and then only on one client. Not tested in production yet but file transfers on the host which were causing lock-ups now have little effect. I have loaded three more clients and the server is still responding as it should.

                          • 10. Re: Tips for Improving Performance On Linux Host
                            Peter_vm Guru

                            Now you only need to provide explanation for this behavior, and everyone will be happy.

                            • 11. Re: Tips for Improving Performance On Linux Host
                              ian_tk Lurker

                               

                              I can only provide an explanation for the solution, that is I observed from earlier comments that the IOWAIT errors moved from disk to disk as the parameters were changed controlling where VMware stored it's memory images, and that everyone that was having this problem was using hardware raid of one flavour or another. Using any form of ramdrive solution would not only reduce ram available but is also discouraged by VMware for technical reasons.  Therefore taking the problem off raid and onto a cheap sata drive stood a good chance of working. 

                               

                               

                              I know nothing of the technical side of how VMware writes to disk, but there were comments that for these memory writes it was using a different method, could it therefore be that if standard disk reads/writes are in progress that the 'other method' is not getting sufficient priority, either from the raid software or more likely the Linux kernel?

                               

                               

                              I am way out of my depth here, so I will look forward to reading a more informed explanation from more experienced vm'ers!  Meantime I am delighted to have a server that seems deployable at last!

                              • 12. Re: Tips for Improving Performance On Linux Host
                                Expert
                                getut wrote:

                                Thanks for the reply, but I think you missed my rock and a hard place comment. I wasn't running VMWare Server 2 Beta when I was on Feisty. I was running 1.0.4 and still had the same performance issues.

                                 

                                I had good disk performance on down level kernels BUT I got unacceptable clock drift (slow clock... after 5 minutes when VMWare Tools recorrects, the clock would have only ticked off about 45 seconds). It was causing kerberos errors on all my clients and on the Sametime server it would cause service disconnects when the time was corrected by VMWare Tools. I tried all the usual clock drift fixes and nothing made a difference, So I couldn't run with the above scenario. The slowness I'm experiencing now at least isn't causing complete service failures but the slowness is proving very hard to fix also.

                                 

                                I'm curious as to what fixes you tried? I assume you read the VMware Timekeeping whitepaper?

                                http://www.vmware.com/pdf/vmware_timekeeping.pdf

                                 

                                I would've thought that switchign to NTP would have taken care of any issues but since I don't administer any production Linux servers using Server, or work on the timekeeping code, I'm not a qualified expert on the matter.

                                • 13. Re: Tips for Improving Performance On Linux Host
                                  getut Enthusiast

                                  In reponse to ian_tk... I am still fighting the issue.

                                   

                                  My Lotus Domino software simply would not function at all with the time drift that I was hit with in Feisty 64bit even though the disk performance was very good.

                                   

                                   

                                   

                                  So I upgraded to Hardy and am still there. I'm getting better disk performance with Hardy than I did with Gutsy, but not by much. It is still pretty horrible and my users are complaining alot. But its better running slowly than crashing because of time discrepancies.

                                   

                                   

                                   

                                  In response to rrdharan... In attempts to fix the time drift in Feisty, I tried all the different boot options for different timekeeping mechanisms...such as clock =pit and a couple of others. Its been quite a while so I can't remember, but I pretty much documented all of my attempts in the first thread. Also the vmware tools time reset did not help much. I was getting clock drift of about 5-7 minutes IN BETWEEN vmware tools resetting the clock. Then Domino would freak when the clock got reset back in time 5-7 minutes. At times, the clock would be running 7-10 times real speed and it seemed to vary with load.

                                   

                                   

                                   

                                  Running the exact same VM's under Server Beta and RC1 has them keeping time very well and running stably OTHER than the disk performance. I would LOVE to get this resolved once and for all and I've seen indications that running something other than RAID 5 or 10 in the newer kernels MAY resolve the issue.. but I simply can't afford to lose the additional disk space by running RAID 0+1.

                                   

                                   

                                   

                                  I would really be satisfied just seeing some indication that SOMEONE with the power to fix it from either the Linux side or VMWware side is aware of the problem and knows the cause and is working on getting a patch integrated into whatever is broken with VMWare or the newer kernels.

                                   

                                   

                                   

                                  So far, with all of the posts and threads, quite a few people are having the problem.... no one that I am aware of yet has fixed it. No one with powers to fix it has chimed in to say they are aware of it and working on the fix. ian_tk has worked around the  issue but still hasn't truly resolved the issue of the bad disk performance when all 3 pieces of the puzzle are thrown into the pot. 1) VMWare, 2) Latest generations of Linux Kernels and 3) Hardware RAID 5.

                                  • 14. Re: Tips for Improving Performance On Linux Host
                                    kingneutron Master

                                     

                                    I would try filing a bugreport with vmware Support; but not sure if you have a paid support plan in place.

                                     

                                     

                                    If you are indeed using Vmware in a PRODUCTION ENVIRONMENT, then I would definitely recommend buying a support/maintenance plan.  It should help in situations like this where multiple end-users are being affected.

                                     

                                     

                                     

                                     

                                     

                                    ./. If you have appreciated my response, please remember to apply Helpful/Correct points. TIA

                                     

                                     

                                    1 2 Previous Next