4 Replies Latest reply on Oct 25, 2017 12:04 PM by UniCav

    vSphere 6.5 Slow network performance on new system

    UniCav Novice

      I have 2 ESXi systems:

      Older system running VMWare ESXi 6.0. Intel 2600CWR mainboard, 1x Xeon E5-2603 v4 @ 1.70GHz, 128gb RAM, 4tb RAID 10 with SATA HDD's

      New system running ESXi 6.5.0 5969303 (U1), Intel 2600CWR (same mainboard), 2x Xeon E5-2620 v4 @ 2.10GHz, 256Gb RAM, with 6tb RAID 10 on SSD's

      I'm having terrible network performance through the new system.

      Initial setup went fine, Shut down my older vCenter server on VMWare1 (old) and installed new 6.5 vCenter on VMWare2 (new server). Then installed a Win2016 server on it, made the network tweaks in Windows for VMXNET3. My Windows AD/DHCP/DNS is all running great, both ESXi boxes and the vcenter are on the AD and I can directly ping and log into them through the web client using the FQDN and Domain Admin. So everything seems fine. One of the AD's is a 2016 VM on VMWare1 and the new one is on VMWare2.  Office network is running on a Netgear GS748T (gigabit) switch.

       

      I rolled in the other night to migrate our accounting server (Win2012) from VMWare1 to VMWare2. Found out I can't use Migrate because I only have an Essentials license. So I shut down the server and tried copying the folder to VMWare2. Wasn't seeing any progress at all in the task list so I looked on the datastore, Folder was created but there's only a _flat_vdmk file not all the existing files. Did a little Googling and decided that wasn't going to work. Never did figure out why Copy on the folder created the flat file inside instead of copying all the actual files. Next I discovered that Veeam has a Quick Migrate feature. I use Veeam Free version to backup my VM's so I hopped into it and started the migration. After about 15 minutes of nothing I realize that it's showing a processing rate and throughput of about 1.2Mbps and never moved off that so I stopped it.  I did a few tests and verified that I get upwards of 150Mbps doing the same thing on VMWare1 but never anything over 1.2mb on VMWare2. I had an extra Intel NIC card installed in the new server so I swapped the management network to it. I can now get upwards of 80MBps but still nowhere near what I get from the older server. I did some SSD Benchmarks and the disk speed is exponentially faster on the new server as it should be, but the network speed is complete crap.

       

      I'm sure it's some kind of config issue since the old server has the same mainboard and runs great.  Haven't been able to find anything other than the ongoing discussions that lead to https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2150591  And that's supposed to be fixed in U1. Suggestions?

        • 1. Re: vSphere 6.5 Slow network performance on new system
          Nick_Andreev Expert

          You say you're getting 80MB/s over 1Gb network, which is not that bad if you ask me. What is the speed you're expecting?

          • 2. Re: vSphere 6.5 Slow network performance on new system
            UniCav Novice

            I'm getting a solid 150mb/s and up throughput on the other server with VMWare 6.0 on the same mainboard with same onboard i350 Ethernet and mech hard drives.

            I would think it more than reasonable to expect at least the same throughput not half, plus the new server has double the CPU's, RAM and SSD's so it SHOULD be even faster with the higher processing and exponentially higher disk r/w rates.  Something has to be wrong with the networking in 6.5 or there must be something I need to tweak to fix this

            • 3. Re: vSphere 6.5 Slow network performance on new system
              UniCav Novice

              Can anyone else chime in on this? I have tested down to the switch level and the only common denominator here is 6.0 vs 6.5.

              Same hardware, same cables, same switch but no matter what I've done I get HALF the throughput on 6.5.

              I hate to have to reinstall to 6.0 but that's what's going to happen if I can't get this speed up

              • 4. Re: vSphere 6.5 Slow network performance on new system
                UniCav Novice

                Well after some more testing and searching I figured out that I do not actually have a network problem, it was just a matter of source location

                I DID have a disk problem but I didn't know it until I hit this thread

                https://communities.vmware.com/thread/554004

                Definitely helped my speeds. Before the change I was only getting about twice the Read Rate and not quite twice the Write Rate compared to the system with normal drives.

                After disabling the module I'm getting MUCH higher R/W speeds.

                Finally I addressed the one test I hadn't performed over the network by putting Veeam on the Windows VM that's on VMWare2 (New server) and sure enough when THAT becomes the source I get that full network speed upwards of 150mb.  So once Veeam is going over the network to the source rather than being on the same VM it drops down to about 75-80mb average.