sjsotelo
Contributor
Contributor

ESXi 3.5 slow?

Jump to solution

VMware seems to be slow, compared to a physical windows server.

We have two brand new dell PE 1950 III servers, 2.6 quad core, 32gigs of ram, 15k SAS drives. One sever I built with just windows 2003 enterprise, and the other I build with ESXi 3.5. and installed a windows 2003 enterprise guest.

Keeping the default VMware settings, when I do a robocopy from one of our other servers to the VMware server I get about 4.2mb/sec and then if copy the same file to the physical server with just windows 2003 on it, I get 13mb/sec.

Why is the windows OS so much slower when running on VMware?

-Stephen

0 Kudos
1 Solution

Accepted Solutions
AdamSnow
Enthusiast
Enthusiast

In our case, we have a lot of metrics. We did a number of tests on a physical server, and then the same tests on a VM after we virtualized it.

  • The physical server on average copied files over the network 30-40% faster. Also when doing so, the physical server only used 12% of its CPU, whereas the VM used about 35-40% CPU (the physical server has a 3.0 Xeon, and the ESX sefver has a 3.0 Xeon. The ESX server is also 2 years newer, therefore the processor technology is better). In addition, the physical server would peak out at about 12% network utilization. The VM never went above 5%.

  • When launching a remote desktop connection, the physical server came up with the login screen almost instantly. with the VM, it took about 2-3 seconds. After typing in the credentials, the physical server took 10 seconds to login. The VM took 22 seconds, sometimes longer.

  • In the physical server, right-clicking on C:\window and clicking properties takes 6 seconds to determine the amount of files and their sizes. In a VM, it takes 15 seconds.

We are heavily invested in VMWare, so I want it to work. As I said before, native VMs are much faster than converted VMs, but they're is still some performance hit. The converted VMs are the big ones though.

View solution in original post

0 Kudos
19 Replies
Dave_Mishchenko
Immortal
Immortal

Your post has been moved to the ESXi 3.5 forum

Dave Mishchenko

VMware Communities User Moderator

0 Kudos
kjb007
Immortal
Immortal

Do you have vmware tools installed? You will get better performance with the enhance network driver from the tools.

-KjB

vExpert/VCP/VCAP vmwise.com / @vmwise -KjB
0 Kudos
sjsotelo
Contributor
Contributor

Yes, they are installed.

0 Kudos
AdamSnow
Enthusiast
Enthusiast

We notice that VMWare is noticeably slower as well. Converted VMs seem to be much slower than native/newly created VMs, but they all take a performance hit. Remote desktop is slower, file transfers are slower, network throughput is slower, accessing the local disk of the gurest OS is slower, etc. We have a ticket open with VMWare but nobody seems to be willing to admint what a lot of us already know - VMWare is a big performance hit.

0 Kudos
dominic7
Virtuoso
Virtuoso

Do you have any metrics, or does it just give you the heebie-jeebies?

0 Kudos
AdamSnow
Enthusiast
Enthusiast

In our case, we have a lot of metrics. We did a number of tests on a physical server, and then the same tests on a VM after we virtualized it.

  • The physical server on average copied files over the network 30-40% faster. Also when doing so, the physical server only used 12% of its CPU, whereas the VM used about 35-40% CPU (the physical server has a 3.0 Xeon, and the ESX sefver has a 3.0 Xeon. The ESX server is also 2 years newer, therefore the processor technology is better). In addition, the physical server would peak out at about 12% network utilization. The VM never went above 5%.

  • When launching a remote desktop connection, the physical server came up with the login screen almost instantly. with the VM, it took about 2-3 seconds. After typing in the credentials, the physical server took 10 seconds to login. The VM took 22 seconds, sometimes longer.

  • In the physical server, right-clicking on C:\window and clicking properties takes 6 seconds to determine the amount of files and their sizes. In a VM, it takes 15 seconds.

We are heavily invested in VMWare, so I want it to work. As I said before, native VMs are much faster than converted VMs, but they're is still some performance hit. The converted VMs are the big ones though.

0 Kudos
dominic7
Virtuoso
Virtuoso

Thanks for providing more details. Have you migrated your converted VMs to a uniprocessor HAL ( assuming they were using a multiprocessor HAL when they were physical )?

0 Kudos
AdamSnow
Enthusiast
Enthusiast

Yes, I have done everything imaginable to the VMs to increase performance.

  • Change to 1 vCPU and change the windows HAL to uniprocessor

  • Remove all serial, floppy, cd-rom, and usb devices

  • change from BusLogic to LSILogic scsi driver

  • Remove all manufacturer agents, drivers, apps, utilities, etc.

  • Use devcon to remove all hidden devices (old drivers from when it was a physical server)

  • Change hardware acceleration to Full

0 Kudos
AdamSnow
Enthusiast
Enthusiast

I also just noticed that this thread is about ESXi. My situation was to do with regular ESX version, not the embedded one. so sorry I hijacked the thread. Also, we've seen the issue withe very version we've tried - 3.0.2, 3.0.2 Update 1, 3.5, and 3.5 Update 1.

Thanks.

0 Kudos
sjsotelo
Contributor
Contributor

I'm also getting the same results with ESXi 3.5, vmware just cant compare to our physical server.

0 Kudos
AdamSnow
Enthusiast
Enthusiast

Are you using newly created VMs or converted VMs? What type of storage are you using?

0 Kudos
kastlr
Expert
Expert

Hi,

how did you compare your VM with a physical host?

Did you use memory and cpu limitations and reservations?

How many VM's did your ESX server hosts while running your tests?

Did you use raw disk mapping or a VMFS datastore for your VM?

Did you align the NTFS filesystem?

How many ESX hosts are sharing the LUN's you use for your VM?

Did you use a dedicated NIC for your VM and configure it to use a static mode instead of using autosense?

Don't get me wrong, I believe you that your VM is slower than a physical server.

But as in windows, you could also increase the speed of a VM when adjust some settings.

Even when I don't work for VMWare, I think this is a fantastic tool which is worth using it.

Without it, I wouldn't be able to run our storage LAB with a few physical server.

But in our environment we're focused on functions instead of performance.



Hope this helps a bit.
Greetings from Germany. (CET)
0 Kudos
sjsotelo
Contributor
Contributor

These are local data stores on 15k SAS drives, no san. We are comparing these directly to other physical servers that are setup the exact same way, just without vmware. We tested these with just one guest running and with two guests running, and one guest running vs a physical server the performance of the physical server was definitely much larger. I think VMware is great for servers that are not critical for performance but rather functions. But we where testing these for web servers and database servers so in this case VMware doesn't seem to perform as well for this type of environment.

0 Kudos
AdamSnow
Enthusiast
Enthusiast

Ghost, here are the answers to the questions you asked me:

We compared it by running a set of tests on the physical server, and then virtualizing it, and then comparing the same tests.

No limitations and reservations. I have since played around with them but have not seen any effect.

For the purpose of the tests, I had a dedicated ESX server with 2 dual core CPUs and 12 GB of memory. This was the only VM on there.

I used VMFS for my datastores

I'm not sure what you mean by aligning the NTFS filesystem, can you explain? I've heard of aligning the VMFS file system, which I've read is done for you when you create it through VI Client.

For the purpose of the test, this was the only ESX server with access to that LUN.

What is static mode for my NIC? for our tests, we tired it 2 ways - once with both NICS teamed, and the other time with just 1 NIC.

Thanks.

0 Kudos
kastlr
Expert
Expert

Hi,

by default, MS misalligned a partition to use an offset of 31,5 kB.

To allign a NTFS partition to your storage array, you need to use diskpart.exe from MS.

There are severall articles available about the how to.

VMWare does recommend to configure your physical NIC (and the LAN switch port) to a static speed instead of using autosensing.

When running only one VM per ESX host, using reservations and limits doesn't provide a performance win.

Same with the LUN, if only one host is accesing it no locking condition could occur.



Hope this helps a bit.
Greetings from Germany. (CET)
0 Kudos
AdamSnow
Enthusiast
Enthusiast

I will research into aligning the partition for NTFS. This is the first I've heard of this, thanks.

Everything I've read in VMWare's documentation says that with gigabit NICs, it should be left as auto-negotiate and not set to 1000 Full. I have tried both and seen no difference.

0 Kudos
tumbleweed
Contributor
Contributor

I think the important point regarding the NIC configuration is that you have the same identical settings on your interface as what is configured on the switch. Disabling autonegotiation and using fixed settings avoids the situation where the negotiation process is unreliable. I have seen autonegotiation fail and have a NIC set at half duplex while the switch is in fullduplex. This can cause big performance problems. Also, 1000 fullduplex is great on a decent switch, but cheap switches don't always perform well even though they may claim to support fullduplex gigE.

Nataraj

0 Kudos
jhanekom
Virtuoso
Virtuoso

There are a couple of assumptions here that I would like to point out in order to make the tests more representative. Most prominently, I would not consider Robocopy to be a performance benchmarking tool. (What exactly are we benchmarking here? The disk subsystem? The networking subsystem? Network transmission speed? The server we're copying from? The antivirus software sitting somewhere? Current load conditions?)

(My other main reason for posting is that you should not see a 50% performance hit on VMware on the same hardware, so I consider this to be a problem somewhere, with proper troubleshooting steps to be taken.)

Firstly, you would need to be certain that the conditions are exactly the same. That is, you need to make sure that they are using the same types of network links, ideally the same switch ports and cables (or some other measure that rules out bad ports and cables) and you will need to make sure that the disk subsystems are set up exactly the same (same RAID type, either both systems must have write cache or none must have write cache, etc.) Other obvious things are to have the same BIOS levels (both system, RAID controller and - if applicable - network adapter).

Next, you should use proper benchmarking tools to verify performance on the systems, rather than introducing a third variable (that external server you're copying from) when testing your platform. Use iperf to verify networking performance ("iperf -w 128K -s" on the one side, "iperf -w 128K -c " on the other side) and IOmeter to test disk throughput (see disk performance threads on these forums for IOmeter input files. This will give you a reasonably accurate view of the performance of the system.

On networking: For Gigabit devices, autonegotiation must work. If it doesn't work, the device is flawed and does not comply with IEEE 802.3 (http://en.wikipedia.org/wiki/Autonegotiation)

On disk: ESX-created disks are, by default, created in "zeroed-thick" mode. This means that, the first time a sector is accessed, that sector will be zeroed out, reducing performance. To avoid this during the performance testing, either manually create "eager-zeroed-thick" disks from the command line using vmkfstools (not sure how this is done with 3i, but I assume you can use the RCLI appliance), or run the benchmark multiple times.

Once we know where the bottleneck is (or isn't), we can make a more concerted effort at figuring out exactly how to address it.

0 Kudos
rlund
Enthusiast
Enthusiast

I will have to test this my self, I am just trying ESXi, but we have some VM's running on SELS 10.2 on VMWare Severs.

I can tell you that those VM's on SLES are very fast, and a reboot of a windows 2003 server, takes less than 20 sec's, and a remote desktop connect is instant.

FYI.

Roger Lund Minnesota VMUG leader Blogger VMware and IT Evangelist My Blog: http://itblog.rogerlund.net & http://www.vbrainstorm.com
0 Kudos