Skip navigation
1 2 3 Previous Next

Virtualization Edge

81 posts


One of the things that did not surprise me was the popularity of virtualization at VMworld.  Everybody who came by the VMware booth knew who VMware was and that we were the "Virtualization Guys" at the very least.  It also seemed like everybody was using VMware for something - although that didn't always include their Oracle environment. 



After being at VMworld just two weeks before, the contrast between the OOW and VMworld attendees was interesting.  I think that is best illustrated by the information handouts we were giving away at the VMware booth at OOW this year.  We had a handout on vCloud Director (which had just been officially announced / released at VMworld), a whitepaper on Oracle performance on vSphere, and a virtualization 101 guide.  We ran out of the virtualization 101 guides on the second day.



I think that this again reflects on the huge popularity of virtualization at OOW, but also points out that many are new to virtualization and just started using or learning about it recently.  This reminds me that not everybody has been following virtualization closely for the past few years and I (or more broadly we) need to be sure to revisit topics from time to time.  Which gives me lots of ideas for more blogs!







After attending Oracle Open World for five years running, I had missed the last three.  Instead I had been focusing my attention on VMworld.  But I got the chance to return to OOW this year to present in the VMware booth and spend time as a performance expert in the booth talking to customers.



There were two questions that came up over and over again and were asked much more than anything else:



Question number 1 - Is Oracle supported on VMware vSphere? The short answer is yes, everything is supported except for RAC.  We have put up a website that has many more details which I recommend that you read if you want to know more.



Question number 2 - Do I have to license Oracle for all of my vSphere ESX servers?  I have to be careful answering this question becuase I am not a licensing expert or a lawyer.  Further I'm commenting on another company's software licensing.  So I'm simply going to tell you what our customers have been able to work out in regards to this specific question around licensing.  We have many customers who have setup their vSphere environment so that all VMs with Oracle software running them are on a specific cluster.  The Oracle VMs never leave this cluster of hosts and so our customers have only licensed Oracle for that set of ESX servers.  There are other aspects of licensing, but this one was the question that kept comming up.



The concerns about performance were very few.  I spent most of my time talking about how to improve, optimize, or monitor performance. 



I also presented some early performance results of some testing I've done with Oracle RAC.  I'm still working on that testing and will be putting some blogs up on that soon.  It was fun to go back to OOW and I'm planning to return next year.






As part of my recent testing with Exchange 2010 on vSphere 4.1, I created a Database Availability Group (DAG) and used it to create copies of the mailbox databases on a VM. To keep things simple I only used a two Exchange 2010 mailbox server VMs for this test: a primary VM and a copy VM. The primary VM was where the active mailbox databases were running handling all of the users requests. The copy VM was only supporting the replicated copy that was created within the DAG.


I used the same VMs and server as in my previous Exchange 2010 tests. In the configuration for these tests each VM was configured with 2 vCPU and 24 GB of RAM. The server was a Dell R710 with Intel X5570 processors and 96GB of RAM.



There were three test scenarios. The first was with both VMs in the DAG, but no database copies created. The second was with only one of the eight mailbox databases replicated to the copy VM and the third test was with all eight mailbox databases being replicated. In all tests 4000 LoadGen 2010 users with the Outlook 2007 Online Heavy profile were run for a full 10 hour test run.



Of primary concern was if the addition of the DAG group with the replicated databases would affect the performance as seen by end users. The graph below shows the SendMail response time for all three tests with a very small increase in latency.



It was also interesting to see what effect on CPU and IOPS would be with the DAG group and active database copies. The graph below shows that the IOPS on the primary VM remained flat and the CPU only increased slightly. It also shows that the work being done on the VM just maintaining the copy is much lower than the primary VM.



Overall, the effect of a DAG on the Exchange VM performance was very small with only slight increases in CPU, IOPS, and response time. This must be tempered by the configuration used here, as both the primary and copy VM were running on the same physical host.




In my VROOM! blog post on Scale-Out Performance of Exchange 2010 on vSphere 4, I published some data that showed how performance only degraded slightly as the same number of users was run on multiple VMs.  This was because for the tests I kept the amount of RAM and vCPUs constant, and just divided that same amount of resources across the virtual machines.  For example - 8000 users were run on one VM with 8-vCPUs and 48GB, then on two VMs with 4-vCPUS with 24GB each, and finally on four VMs with 1-vCPU and 12GB each.



This is a good way to run the test and keep resources constant.  The reason that there is a slight decrease in performance is that the amount of resources available for Exchange becomes less with each additional VM.  There is a certain amount of RAM that the OS and Exchange executables use that is replicated in each VM, leaving less total RAM available for the Exchange Servers to use for caching data.  This then results in more IOPS which will increase response time.  Additionally, Exchange 2010 seems to do a better job of managing and using a large amount of RAM than previous versions of Exchange.  The end result was shown in my tests where a single 8vCPU VM with 48GB of RAM was able to provide better response time than two 4vCPUs with 24GB or four 2vCPU VMs with 12GB.



The 4000 user scale-out tests showed a very small increase in response times.  In the 8000 user tests a larger increase than expected was found when moving from two VMs to four VMs.   While the absolute performance of the four VMs was still very good, I investigated to determine why the larger increase. I found that IOPS had begun to increase to the point where disk latencies were beginning to creep up as well.  So in addition to more requests being serviced from disk instead of from memory, each disk access was beginning to take slightly longer. 



I did another test and increased the amount of RAM assigned to each of the four VMs from 12GB to 20GB.  Everything else was left the same.  The graph below shows how IOPS decreased and performance improved after adding RAM to the VMs. 



These results are not surprising (or life changing).  It is basically a combination of the IOPS and RAM article and the scale-out article showing that increasing the amount of RAM improves performance with lower response times and fewer IOPS.  These test results didn't really fit in the scale-out test article, but I thought that it would be fun to post them here. 



  Thanks - Todd



Putting together links for my recent blog on Exchange 2010 Scale-Out Performance I discovered that Exchange 2010 Load Generator (aka LoadGen or LoadGen2010) is no longer a beta tool. The site says that this current version was released yesterday - June 15th - and it is no longer referred to as beta.


I hope that some of the problems that I ran into with the beta version are fixed, but I won't be able to try it out for a few days. I've got to finish the current round of tests that I'm doing with the Beta version of LoadGen before switching over.


There is a note on the web site that states the following:  "There are now three OWA modules: OWA2007Module (Exchange 2007), OWA2010Module (Exchange 2010 RTM) and OWAModule (Exchange 2010 SP1 or latest). Please use the correct version for your environment."


I would guess from this that they were waiting to release this final version of LoadGen with the module to support the soon to be released Exchange 2010 SP1. This would explain the long delay between the beta and final releases.


One of the big areas of improvement with Exchange 2010 was with how it handled I/O.  Specifically the individual I/Os are now bigger.  In my blog post on VROOM! about Exchange I/O performance, I looked at how adding more RAM decreases IOPS and improves performance.  But it didn't look specifically at the size of the I/Os - which is what this blog is about.



Using the vscsistats tool that comes with ESX it is possible to get snapshots of storage performance.  One of the things that it show is the size of the I/Os for the interval sampled.  This size info is detailed down to individual virtual disks for individual VMs.  This makes it possible to see the size of reads and writes for the data and log disks on our Exchange 2010 Mailbox Server VM.



I sampled vscsistats data while running LoadGen 2010 beta with an 8000 user Outlook Online 2007 Heavy Profile across four Exchange 2010 Mailbox server VMs (Gabe has a good summary of how to use vscsistats).  I collected the data against all four Mailbox VMs, and the data was essentially the same, so I'm just reporting for one VM.  This data is based on an approximately 5 minute time period for 2000 users on a single Exchange 2010 Mailbox server VM.  The chart below shows the number I/Os for each size, but also denoted by color for data disk reads, data disk writes, log disk reads, and log disk writes.!!



I/O operations on the data disk were all 32K or higher, with a large majority of them at 32K.  I/O operations to the log disk were of a completely different profile with exclusively writes and the vast majority of them being less than 32K.  This shows that Microsoft Exchange 2010 does not use the smaller 8K and 4K I/Os for the database as in previous versions. 










I recently put together a blog on VROOM! that is the first in a three part series on Exchange 2010 performance. This first one was on scale-up performance within a single Mailbox Server VM. I was able to easily get 8000 users in a 4-vCPU VM running on an Intel X5570 (Nehalem) based server.



On VROOM! we keep things pretty formal. This is a good thing (for the most part), but I really like to use my Virtualization Edge blog here in the VMware Communities side to be less formal.



In testing Exchange 2010, the biggest problem I had was getting LoadGen 2010 Beta running consistently. I posted a couple of posts here about things that I learned about LoadGen 2010 Beta. The installation, configuration, and performance of Exchange 2010 was great. I found great performance with most of the tests that I did.



When I started I wasn't sure if I would be able to get great performance with 8000 users and a moderate amount of RAM. I found that 8000 users easily fit in a VM with 4vCPU and a range of RAM settings. In addition to a Mailbox VM running 8000 users I was also able to run the Client Access Server (CAS) VM and Hub Transport VM on the same physical server. Even in this configuration with all three VMs running on the same ESX host, there was still lots of processing power left over in the system. The chart below shows the CPU utilization for all three roles:



In the 8000 user case, all three VMs are still consuming less than 4 CPUs worth of processing meaning that more than half of the CPU capacity is still available.



More on Exchange 2010 Performance to come.



Edited 4/19/2010 - After actually seeing an iPad in action, connecting to a VMware View based desktop I must admit that I was surprised how well it worked. I can see it being very useful for many people. For example I think that a doctor could carry around an iPad while seeing patients and find it very useful. The original blog below is really from my personal perspective as a performance engineer who uses two big flat panel screens all day and spends alot of time writing and creating documents. I just can't imagine myself using an iPad in replacement of my current setup, although I must admit that I almost bought one after seeing it cobnnected to a View desktop.



Original Post from 2/10/2010

There has been a discussion on twitter starting just after the big Apple iPad announcement about using the iPad as an endpoint or client device for virtual desktops. Probably not a good idea - and here's a few reasons why.


While the iPad looks really cool, that won't make it a good device to connect to your desktop running the virtual cloud. The current desktop computing environment assumes a mouse and keyboard. The iPad doesn't have either. The touch screen can easily substitute for a mouse, but the keyboard is more difficult to truly replace. The screen of the iPad is way bigger than any phone, but it isn't nearly as big or high resolution as a standard monitor from 3 years ago. This limits the amount of screen real estate for your desktop and leads to a lot of resizing and or scrolling. Additionally, if you have to use an on screen touch keyboard this just gets worse.


The cost of the iPad is much higher than a Netbook and is equivalent to a low end notebook, while not having some of the things that make these devices better desktops. Namely a keyboard and more screen real estate.



I do think that the iPad could be a great device for reading books, checking out Facebook, reading blogs, and catching up on your favorite sites on the internet. I'm just not so sure about connecting to your virtual desktop and getting alot of work done.






As a quick preview of the Exchange 2010 on vSphere work that I am doing I put together this blog post to talk about the CPU utilization of the various server roles. Exchange 2010 changes things a bit and now that the Client Access Server (CAS) role is now more involved several people have asked me about the impact.


I have tested across a wide range of memory configurations with a few different vCPU configurations with Exchange 2010 on vSphere. In a future post on VROOM! I will go into all the details of those tests. Here I'm just going to show the CPU utilization with 1000, 2000, 4000, and 8000 users. Regardless of the number of vCPUs or RAM assigned the CPU utilization stayed within a very narrow range for given number of users for the Mailbox, CAS, and Hub Exchange 2010 server roles. The table below shows the average CPU utilization for the number of users, with each vCPU equal to 100. So a 2vCPU VM would max out at 200 and a 4vCPU VM would max out at 400. This is the same way that esxtop reports CPU usage, which is what these numbers were gathered with. The results show a linear increase in CPU usage across all three roles:

Number of Users


Hub Transport

Client Access Server

















The LoadGen 2010 beta VeryHeavy User profile with 100MB mailbox size was used for these tests. The system the tests were run was a Dell R710 with 2x Intel Xeon x5570 quad core processors.


In a previous post I talked about my experiences in getting the Exchange LoadGen 2010 beta to work.  I recently had to increase the number of users again and I learned a couple of more things in the process. 



The first is that you need to be sure and check the Pre-Test Login box in the user profile definition screen.  This is on the same screen as where you select the load profile of Light, Heavy, Very Heavy, etc.  For smaller numbers of users it doesn't matter, but for larger numbers it is importation to give the system a chance to get all the users logged in before starting tasks.  If you don't check this box with a large number of users, then the task queue will exceed the max allowed of 1.5x the number of users configured during the initial part of the testing.  This will cause LoadGen to exit. 



The second tip is that if your first initialization of the mailboxes fails, simply do the initialization again until you get no errors.  Once you get a clean initialization you can start a test.







I have been testing with the Microsoft Exchange 2010 Load Generator tool for the past month or so.  I initially had some problems getting LoadGen 2010 beta to work well, so I thought that I would put together a quick blog post with the key things that I learned.



When changing the number of users in the mailbox databases or the size of the mailbox databases, you may need to first delete all the users.  I found that when I tried to increase the number of users that had from an initially small number (about 100) to the number I was going to start testing with (4000) that I connection errors.  These errors would occur when I tried to start the initialization of the users.  I deleted all the users two different ways.  I deleted and recreated the mailbox databases (hard way) and I set the number of users to 0 in LoadGen for all groups and let LoadGen delete the users (easy way).



Run the LoadGen 2010 Beta directly on the mailbox server VM for the initialization.  This may actually be related to why I was having problems with changing the number of users.  I initially started by doing the initialization from the load driver system I was also planning to run my Load Gen 2010 tests from.  I found that the initialization went much faster by also installing the LoadGen 2010 Beta program directly on the Exchange mailbox server VM.  I do the initialization on the mailbox server VM, but then switch to the load driver system to run the actual test.



Just reinitialize the users between each test instead of backup and restore of the mailbox databases.  In previous tests with Exchange 2007 and LoadGen I initialed the mailbox database once it was initialized it, and then simply restored it before every test.  The first initialization of users takes quite a bit of time, but subsequent initializations are much faster.  I'm able to reinitialize 4000 users in about 30 minutes, which is faster than a restore in my case.



The documentation included with LoadGen 2010 beta is actually for the Exchange 2007 version of LoadGen.  This is true for the included readme as well as the online help.



The best place that I have found with information on LoadGen 2010 beta is this post in the Exchange Team Blog - LoadGen 2010 (Beta Preview) now available.



I clearly cannot provide support for LoadGen 2010 beta, but I would be happy to try to answer questions based on my experiences. 







Last week, I published a new blog post on VROOM! detailing some results from testing I did with an SAP three-tier configuration. After a really cool graph (featuring some great trendlines!) it concludes that SAP shows good scaling in a three-tier environment spread across multiple hosts.  On the VROOM! blog I didn't really talk about how amazing this is, so I wanted to emphasize a couple of points here.


A large and sophisticated enterprise applicaiton can be run on top of a virtualization layer on x86 based servers and achieve excellent performance that scales as additional hosts are added. Even more cool is that the performance increase came with no changes to the software running inside the VMs. Ninety percent more performance was added without making any changes to the OS or application.



Additionally, VMotion allows these same enterprise VMs to move from one physical host to another with no downtime. (Here's a video demo that I did with VMotion and some SAP VMs)  This means that the performance increase that comes from spreading an SAP three-tier setup across multiple hosts can be done dynamically with no downtime.



The most impressive thing about all of this is that it has become what is expected. These new results with an SAP three-tier setup are exactly what VMware customers expect from vSphere. 







The VMware Austin office is moving from the Mopac/183/Hwy 360 intersection a few miles down the road to just south of the Pennybacker bridge. In the old office I had a cube right next to the window, with what was a pretty nice view from the 6th floor. In the new office the layout is different and there aren't as many windows for engineers this time around. I've only been at VMware for a little over a year, so I won't be next to a window anymore. I couldn't be happier that it worked out this way.


A few weeks ago I was sitting a couple of cubes over in @jpschnee's cube talking about something. I had my back to the window and while he was in mid sentence he became distracted for moment. Looking back I think that I saw a shadow in the corner of my eye, but I didn't think anything about it at the time. I then noticed that one of the sales guys from a few rows over had rushed to the windows and was raising all the mini blinds. He was saying that he couldn't believe what he just saw. I then looked out my window and saw that an office building was on fire, with a hole in it.


He said that he saw a plane just miss the Embassy Suites hotel and then hit the building. The whole office came over to the area just outside my cube to look out the window. I called my wife and told her that it wasn't my building and that I was OK.



The building burned with thick black smoke for several hours. We later found out that a crazed man had done this, mourned the loss of life, and were thankful that no more were hurt.



Yesterday, when I packed up my office, I looked out of my window and that ruined building is still there. Not having a window view in the new building is really no big deal and I won't miss the old view at all.






I have received many requests for performance data for Microsoft Exchange 2010 on vSphere. While there still isn't a ton of information out there, it is beginning to come out. In a blog post put up recently, some initial Exchange 2010 data for a single VM with 500, 1000, and 2000 user configurations has been published. It shows that performance is good and includes graphs showing CPU utilization, IOPS, and memory usage. (I know that the graphs will be a big draw for geeks everywhere.)


I worked with Kong a bit on the ESX side of the setup for these tests, but he was the one who was able to get these tests done and early numbers out. We are both continuing work on Exchange 2010 testing and additional results will be published, but we felt it was important to get something out as soon as possible.





I've been working on some performance testing with SAP on vSphere with @KongY_Dell in the Dell TechCenter over the past few months. While my testing is targeted to produce some performance results, it was relatively easy to record a quick demo video using the same setup.


This cool video demo was the result. It shows an SAP VM being moved wtih VMotion to another physical host while under load, with no errors or interruption of service. At the beginning of the video there are four SAP VMs (1 Database Server and 3 Application Servers) on a single Dell PowerEdge M710 with Intel Nehalem x5570 processors. The SAP Sales and Distribution benchmark was used to put this 3-Tier SAP setup under load, and then the Database VM was moved to another host with VMotion.


Just a short demonstration of how VMotion allows for VMs under load to be moved without interruption to users - including things like SAP.



A couple of side notes - 1) I tried to get Kong to do the voice narration with me, but couldn't convince him to get behind the microphone. 2) I pulled off the entire voice narration in one take with no edits necessary!