vSphere 6.5 Slow network performance on new system

[UniCav]

I have 2 ESXi systems:

Older system running VMWare ESXi 6.0. Intel 2600CWR mainboard, 1x Xeon E5-2603 v4 @ 1.70GHz, 128gb RAM, 4tb RAID 10 with SATA HDD's

New system running ESXi 6.5.0 5969303 (U1), Intel 2600CWR (same mainboard), 2x Xeon E5-2620 v4 @ 2.10GHz, 256Gb RAM, with 6tb RAID 10 on SSD's

I'm having terrible network performance through the new system.

Initial setup went fine, Shut down my older vCenter server on VMWare1 (old) and installed new 6.5 vCenter on VMWare2 (new server). Then installed a Win2016 server on it, made the network tweaks in Windows for VMXNET3. My Windows AD/DHCP/DNS is all running great, both ESXi boxes and the vcenter are on the AD and I can directly ping and log into them through the web client using the FQDN and Domain Admin. So everything seems fine. One of the AD's is a 2016 VM on VMWare1 and the new one is on VMWare2.  Office network is running on a Netgear GS748T (gigabit) switch.

I rolled in the other night to migrate our accounting server (Win2012) from VMWare1 to VMWare2. Found out I can't use Migrate because I only have an Essentials license. So I shut down the server and tried copying the folder to VMWare2. Wasn't seeing any progress at all in the task list so I looked on the datastore, Folder was created but there's only a _flat_vdmk file not all the existing files. Did a little Googling and decided that wasn't going to work. Never did figure out why Copy on the folder created the flat file inside instead of copying all the actual files. Next I discovered that Veeam has a Quick Migrate feature. I use Veeam Free version to backup my VM's so I hopped into it and started the migration. After about 15 minutes of nothing I realize that it's showing a processing rate and throughput of about 1.2Mbps and never moved off that so I stopped it.  I did a few tests and verified that I get upwards of 150Mbps doing the same thing on VMWare1 but never anything over 1.2mb on VMWare2. I had an extra Intel NIC card installed in the new server so I swapped the management network to it. I can now get upwards of 80MBps but still nowhere near what I get from the older server. I did some SSD Benchmarks and the disk speed is exponentially faster on the new server as it should be, but the network speed is complete crap.

I'm sure it's some kind of config issue since the old server has the same mainboard and runs great.  Haven't been able to find anything other than the ongoing discussions that lead to https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=21505...  And that's supposed to be fixed in U1. Suggestions?

[Nick_Andreev]

You say you're getting 80MB/s over 1Gb network, which is not that bad if you ask me. What is the speed you're expecting?

[UniCav]

I'm getting a solid 150mb/s and up throughput on the other server with VMWare 6.0 on the same mainboard with same onboard i350 Ethernet and mech hard drives.

I would think it more than reasonable to expect at least the same throughput not half, plus the new server has double the CPU's, RAM and SSD's so it SHOULD be even faster with the higher processing and exponentially higher disk r/w rates.  Something has to be wrong with the networking in 6.5 or there must be something I need to tweak to fix this

[UniCav]

Can anyone else chime in on this? I have tested down to the switch level and the only common denominator here is 6.0 vs 6.5.

Same hardware, same cables, same switch but no matter what I've done I get HALF the throughput on 6.5.

I hate to have to reinstall to 6.0 but that's what's going to happen if I can't get this speed up

[UniCav]

Well after some more testing and searching I figured out that I do not actually have a network problem, it was just a matter of source location

I DID have a disk problem but I didn't know it until I hit this thread

https://communities.vmware.com/thread/554004

Definitely helped my speeds. Before the change I was only getting about twice the Read Rate and not quite twice the Write Rate compared to the system with normal drives.

After disabling the module I'm getting MUCH higher R/W speeds.

Finally I addressed the one test I hadn't performed over the network by putting Veeam on the Windows VM that's on VMWare2 (New server) and sure enough when THAT becomes the source I get that full network speed upwards of 150mb.  So once Veeam is going over the network to the source rather than being on the same VM it drops down to about 75-80mb average.