Cold Cross vCenter vMotion slower than Hot Cross vCenter Motion

[AhhhOK]

I will try to keep this simple though I have done a bit of testing and could describe in more detail if needed.  I am seeing the throughput of of Cold Cross vCenter vMotion (not shared storage) slower than Hot Cross vCenter Motion (not shared storage).   And not just slower, more like 3 or more X's slower. 

This is contrary to what I expected to see.  I expected to see cold vmotions to be faster.  Any simple explanation for this?  

[a_p_]

How did you configure the network/VMkernel port groups?
Maybe https://blogs.vmware.com/vsphere/2019/12/hot-and-cold-migrations-which-network-is-used.html helps.

André

[AhhhOK]

Keep in mind these vMotions include moving the disk data across non shared storage.

Both Source and Destination hosts have Management vmk0 on the same L2 subnet (Call it Subnet A ESXI Management) using Default TCP/IP Stack

Both Source and Destination hosts have vMotion vmk1 on the same L2 Subnet (Call it Subnet B vMotion Network) Default TCP/IP Stack

Separate subnets for the traffic but both hosts using same subnets for each service.  

Test 1

Initially I had vmk0 for the Management Service and vmk1 for vMotion service on both my source and destination hosts.  What I saw was cold vMotion traffic all flowed through vmk0 which was expected as Frank Denneman explained the difference between hot and cold data in this article.

https://frankdenneman.nl/2013/02/07/why-is-vmotion-using-the-management-network-instead-of-the-vmoti...

"vmotion is used for performance-related content (aka hot data/live vmotion) and (cold data/cold vmotion) is migrated via Network File Copy via Management vmk0"

So I was not surprised to see the traffic going out vmk0 when looking at ESXTOP however the speed was slow.  300 - 500 MbTX/s coming from source host and similar MbRX/s speeds on destination host.

Same config with a live vMotion showed the traffic going out the vmk1 (vMotion) adapter at speeds of 2000 - 4000 mbTX/s coming from source host and similar MbRX/s speeds on destination host.

Test 2

Had me turn on the Provisioning service on vmk1 on both hosts so now vMotion and Provisioning are running on vmk1 (Subnet B vMotion Network).

Results there were that now cold vMotion Network File Copy (cold data/cold vMotion) by-passed vmk0 and went out Source host and back in Destination host via vmk1 but again with the same slower speeds than a live vMotion.  

So why?  Being that both Hot and Cold vMotions are pulling all the disk data from Source host storage and migrating it to a destination host storage why is cold migration slower?  Is this expected?