Terrible VDI Performance

[jcosta]

ok, so I have two different VDI Environments.  One is a test environment made up of low end Supermicro gear and a homebuilt DSM NAS from an old core 2 duo machine.  Then I have a R620 with H710 cached controller, dual E5 cpus and 32 gigs of ram.  I ran some tests using passmark between the two vdi systems and the production environment came back with some really bad results.  Can anyone answer why this is?  As expected the disk passmark scores on the R620 were awesome compared to even SSD on the host of the super micro but the memory and cpu are terrible on the R620.  All were done with Win 7 Linked Clones

Supermicro Setup

VSphere 6 - Horizon 6

ESXI 6

MB: MBD-X10SLL-F-O

32 GIGs of Ram

Single E3-1231 v3 # 3.40GHz

VDI: Windows 7 64 bit - 8 gigs or ram, 2 CPU 2 CORE

Hard drive: DSM NAS - 5400rpm Reds

CPU Passmark Score: 7396

Memory Passmark Score: 1956

R620 Setup

VSphere 6 - Horizon 6

ESXI 5.1

Dual E5-2620 v0 @ 2.00GHz

192 GIGs of Ram

H710 RAID Controller

Hard drives: 10,000rpm SAS

VDI: Windows 7 64 bit - 8 gigs or ram, 2 CPU 2 CORE

CPU Passmark Score:  2923

Memory Passmark Score: 772

All tests were ran with multiple VDIs on both hosts so there was load on both servers.

The R620 VDIs came back with a score worse than a Core2duo according to passmark. 

Can anyone tell me what I should check?    Something must be going on.  Is it the difference from ESXI 5.1 to 6?

Thanks for any insight.

J

[nzorn]

What IOPS are you getting with each of your setups?

[jcosta]

Haven't test IOPS.  I will setup some performance metrics on the Windows boxes and let you know what they show.

[jcosta]

So I have a support request into VMware for this but could really use some suggestions.

I noticed some disk write slowness to the main R630 server running all the Server 2012 servers and discovered it was because they put a H310 with no cache and it was supposed to be a H710.  I upgraded it over the weekend to a H720P but the slowness continues on the other host running the vdis.

I read some posts where people say that the desktops are slow because they set them up with non-persistent hard drives.  The virtual desktops are linked clones with non-persistent hard drives but they are assigned the same desktop every time and I never refresh or recompose the desktops.  They act just like regular desktops and all windows updates, patches and applications are installed right to the individual desktops.

I know this isn't best practice so I set up a test pool with a linked clone and remove the non-persistent hard drives and ran a few tests and the performance is still really bad.  Even on a single stand alone Windows 7 desktop through the console the performance is bad.

The CPU's on the performance test rates like it is a core 2 duo from 15 years ago.

Like I said in my original post I have a test environment running old E3 Super micro servers and the same setup is blazing fast.  Linked Clones - Persistent hard drives running view persona.    I feel like something is wrong with the dell R620 server.  It has newer E5 2Ghz chips in it and it is dog slow with 10 desktops.

Any help would be appreciated.

Thanks,  J

[nzorn]

So what what IOPS are you getting out of your storage?  If you have low IOPS your experience will not be good.

[joshopper]

I would start by using esxtop - the following article will show you how your CPU and storage are performing by looking directly in to the hosts.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200100...

[RyanGoldstein]

You have 10 Desktops each with 8 gig of ram?  But you have only 32 gigs of RAM on your host...  + i would imagine you have a couple view servers maybe a file server... 

How does one single server perform on the R620 with no other servers/desktops turned on.  

[jcosta]

No my test server in the office only has 32 gigs of ram which actually performs great.

The trouble server has 192 gigs of ram (just realized my original post was wrong and updated it) and only runs the 10 virtual desktops and there is a second R630 server that runs the Server 2012 instances with the view, vcenter, domain controller...etc...

I just got done talking with the dell and they said there are several firmware updates that have been know to help with bad performance issues.

I am going to run the updates tonight and see if that fixes the issue.

Thanks for all the suggestions and I will let you know the outcome.

Thanks,

J

[jcosta]

ok, I spent about 2 hours updating all the firmware on the server last night and it helped a little bit but not much.

The performance tests ran still show the equivalent of a core 2 duo.

I ran ESXI top and the CPU Load average right now as I look at it is .77, .92, .98 so it is under 1 per the KB article.  It jumps above 1 every now and then but for the most part it stays under 1.

I don't see a memory over commit in that output but looking at the vsphere client it isn't even close to using all the memory on the server.

The %RDY for most of the VMs are sitting under 5.  There are a few sitting at 6.5 and one at 8 but as I watch it some of the VMs kick well up to 40+ and then back down to normal.  I just saw all of the VMs go up to 30+ all at once and then back down.  As I type this one just went to 18% and held there for about 8 seconds.

Maybe I am over thinking this and this is the performance they should be getting but I just can't get over the fact my older E3 supermicro with 1 cpu and 4 cores far out performs a hefty and more expensive R620 with 2 cpus and 8 cores.

I don't have resource pools setup.  I don't limit the resources to the CPUs.  I have power management features turned off.

I keep reading about HP and Dells with poor performance and in some cases it was something wrong with the hardware and the server limiting the speed of the processors due to it stuck in power saving mode.  I can't help to think this is my issue.

Thanks again for the suggestions!

[joshopper]

Hmm...

a couple of housekeeping type items - is Virtualization set in the BIOS? hyper-threading? how did your %CSTP look in esxtop?

Sometime more vCPUs != better performance depending on what you are running and how procs handles the time slices. Can you try these tests on a pool where the desktops only have 1 vCPU and see how your performance looks?

[jcosta]

hyper-threading is enabled on the cpus.  I checked through IDRAC.

Looking at these ESXTOP results I see one thing really off and maybe you guys can verify.  This is the result on the trouble server:

The %used for the idle is crazy high.  When looking at any other ESXI server including my test system the %used for idle is 0.

Do you think there is an issue with the ESXI image I got from Dell?  I just realized it is running ESXI 5.1.  Maybe upgrade it to 5.5?

Thanks,

J

[Bleeder]

No mention of power management settings, so I'll throw that out there as something to check.  There are articles on both sides (Dell / VMware) with regard to that.

[jcosta]

In the bios it is set to performance and in vsphere it is set to performance as well.

Here is a screen shot of my test system supermicro box running a 4 CPUs x 3.2999 E3-1230 with 32 gigs of ram and it is running like a top with 14 vms.

Look at the CPU load averages.  They are a fraction of the R620.

This is the CPU performance test single thread on my supermicro Win 10 VDI:

This is the same test on the vdi on the R620 running 13 machines:

[Bleeder]

Windows 10?  Your original post mentioned Windows 7 for everything.   Just clarifying as that may make a difference.

[Bleeder]

I think you may want to go all the way to 6.0 (if supported on your hardware) unless this KB has been fixed for 5.5:

http://kb.vmware.com/kb/2127782

[jcosta]

yea, sorry...didn't mean to confuse. 

The Clients machines are Windows 7 on the problem R620.   The Windows 10 box is just in my office on my test system.  I get the same performance marks on my Windows 7 test system on my test environment.

[larjona]

Hi

Try to check in BIOS any parameters like energy power saver. Disable all that options.

Set CPU Maximum performance, or something like that always on all your ESXi servers.

Regards,

[ksliger_lexon]

Bump to all the posts about BIOS performance settings. Read the Dell performance tuning PDF's linked at the bottom of this KB. Maxing out all of these performance settings on our R720's really helped out a lot on our WIn 7 View desktops.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=101820...

[joshopper]

So here is the question of the day is your "terrible performance" just that single threaded CPU test? or are you actually seeing performance problems. Looking at the results I would say that is 'as expected' you are comparing a v0 2.0 GHz proc from 2012 for a 3.4Ghz proc from 2014 and single threaded won't show the benefit of the additional cores on the older E5-2620.

[jcosta]

So I know the Bios is set to Performance Per Watt which apparently leaves some options enabled that ESXI has an issue with.  Something that has to do with CE and CE1.  I am going to switch it to just performance which apparently turns that off. 

I understand what your'e saying joshopper.  I am just really hoping that isn't the case.  The performance is pretty bad all around.    I was over there and one of the users hit ctrl-alt-del and it took over a minute for it to pull up.  They had a few programs open but nothing to crazy intensive.   I just can't believe that a E5 even an older 2GHZ one would rank below a Core 2 Duo from 2007.

I just feel that with a server decked out like this even though it is a little older it should still be able to run 20-30 desktops no problem.

I will change the power setting and upgrade them to ESXI 5.5 and if none of that helps then I will just chock it up to my expectations being to high for an older processor like these and look at a replacement server with a new E5.

Thanks again for all your help!

J