Poor performance on Windows Server 2016

[nickstafford]

Hello,

I have Windows Server 2016 running on VMWare esxi 6.7.0 and have been having really poor performance. The instance runs on a single, Dell R610 with 2 cpu's w/ 6 cores each (Using 8 due to license limitations) and has Windows Server 2016 along with 32 GB of RAM. It has 800 GB of hard drive space and I run a RAID 1 implementation for backups.

This server is designed to allow users to connect over a VPN and also have remote desktop access. I built the environment myself and have access to all equipment (tp-link archer ax6000 router, netgear gs105se switch and I did all the cabling with CAT6 cables). I have a 1 gbps, uncapped connection and all other appliances work fine.

Where I've run into an issue is with the Windows Server 2016 environment. There is a noticeable lag on the desktop with a mouse along with video playback being frozen with anything above 480p (twitch for example). I have vmware tools installed along all necessary routing done in my router/switch. Users have been successful in connecting to the vpn along with accessing remote desktop using their credentials, there is just poor performance.

Any help would be greatly appreciated as I'm looking to expand this environment and I feel like I've overlooked something simplistic or just haven't dug into performance issues enough to find if others have a similar problem (this is a side project so it doesn't necessarily get the love it deserves).

Thanks,

Nick Stafford

[scott28tt]

@nickstafford 

Moderator: Please create threads in the cares for the product used - moved to vSphere Discussions

[nickstafford]

@scott28tt 

Apologies, thank you for moving this into the correct area.

[bryanvaneeden]

Hi @nickstafford ,

So the windows Server 2016 server is a virtual machine right? And you have VMware Tools installed. You might want to check the following:

• Are the VMware Tools up to date?
• Do you have the latest VM Hardware version on the VM?
• Is the VM created with the a VMXNET3 driver and a paravirtual scsi controller? If not this shouldn't be an issue but it might.

It would be wise to also check the following to actually see where the performance issue is coming from:

• Can you visibly notice any performance hits on the VM from the vSphere Client -> Monitor -> Performance tab? Things like Memory Contention, CPU contention (CPU READY %), or storage latency (Read/Write)? 

I think if you know where the performance issue is coming from, i.e. the above values are high, you can start to pin-point the problem here.

Let me know if you have any other questions.

[nickstafford]

@bryanvaneeden 

Thank you for the response.

1) VMware tools is currently sitting on version 10.3.5, which I'm updating right now to 11.2.

2) ESXi is currently sitting on 6.7.0, which will be updated to 7.0 once I finish the vmware tools update

3) VMXNET3 driver is currently being used. In regards to paravirtual scsi controller, it is setup to use lsi logical SAS. I had read something about paravirtual scsi being used, but when I switched it I got a BSOD on boot, so I reverted while I read more about it.

In terms of performance:

The CPU usage hasn't gone above 35% in the various things I've done (setup the server, run some video streaming for testing and windows updates)

HDD I/O hasn't gone above 2.2 MB/s

Consumed RAM has been a constant 13.54 GB

I'll update once I've gone through these upgrade steps. Thank you again

[nachogonzalez]

Hello, hope you are doing fine:
let me ask you a few questions:

What type of Scsi Controller are you using?
What type of virtual nic are you using?

How many cores does your VM have? As I read you had assigned 8 vCPU on a 2 sockes x 6 cores systems. Can you lower it to six vCPU? You are spanning two numa nodes and that has a performance penalty. 
How much memory does you VM have?

What kind of storage are you using? SSD? HDD?  
How are you backing your storage? vSAN, SAN, iSCSI, Etc? Is this VM the only one performing bad?

Have you checked performance charts and compared them with the operating system?

[nickstafford]

VM will be on paravirtualization shortly. But it had been on lsi logical SAS until today.

VNIC uses VMXNET3

I've played with different core configurations but I will change it to 2x3 to see if that makes a difference.

My storage is all HDD. My backups are done locally by another set of HDD's that are running a RAID 1 configuration.

My performance charts are not showing anything serious, as identified by other response.

[nickstafford]

UPDATE:

So the system currently sits on VMware Tools 11.2 and HDD was migrated over the paravirtualization successfully. Also, cores have been changed to run 2 sockets with 3 cores per socket.

I still see slight lag in VMWare workstation, but I'm able to get video streaming up to 720p now and it is noticeably better.

Something I noticed is that Windows Server 2016 recommends a 3.0 ghz processor. The two processors in my server are rated at 2.40 ghz, which isn't quite what is recommended. Would a combination of these processor speeds along with using older HDD's be the cause of lag in workstation? I can upgrade to SSD but this server was originally just a lab that has transformed over time. I'm starting to think I'll need a better processor along with SSD to see lagless performance with what I'm asking out of the appliance.

[nachogonzalez]

Don't take this bad, but i think you should upgrade your setup.
SSD's AND 10 GBE NIC would make a huge difference in your performance.

[nickstafford]

No offense taken, I'm a pragmatic person and the writing was the on the wall once I started looking up hardware requirements for Windows Server 2016 deployments. I think I've squeezed as good of performance as I can out of the system. Thanks again and now I'll be looking to upgrade!

[bryanvaneeden]

Hi @nickstafford,

Great that the upgrade of the VMware Tools and the PVSCSI adapter helped a bit. Did you also upgrade the VM Hardware? That also might be useful.

As others have said, and what I forgot to ask in my first reaction, the lagg might be due to the fact you are running it on HDD's. However, this should normally be no problem. You can also use something like I/O Meter and run this on the server to check what you can do without it being slow.

During this test you should pay a visit to the charts like I mentioned before and check for known performance bottleneck metrics like CPU/Memory Congestion/CPU Ready % and Storage latencies.

I hope all of this helped you!