Slow cluster to cluster migration speeds

[bkjackson78]

I am helping a customer migrate multiple sites where they are upgrading hardware. The new hardware is preinstalled and my team is coming in after and migrating the vm’s from old to new hardware. We are doing cluster to cluster cold migrations. Cluster 1 (old hardware) is usually 1 gig mgmt connections and 10 gig iscsi or 8G FC storage with tiered storage. 1 gig (usually 8 link LAG)/10 gig top of rack switches to the router. It depends on the site. Cluster 2 (new hardware) 10 gig network for everything and iscsi all flash storage. 10 gig top of rack switches to the router.  Jumbo is not on anywhere but the new cluster iSCSI network. 1500 MTU on everything else. Old and new management are on different VLANs/subnets. We are seeing 50-60 Mbs speeds with 1 gig and 10 gig . Never saturating the 1 gig links anywhere in the path.

Since the migrations are cluster to cluster and storage is not shared we know that the vm’s are moving through the management network. The first couple sites we did, we were a little surprised to see how long it took to migrate the vm’s. It took 1 hour + for most vm’s. For example a 100 gig vm might take 2 hours to move. We originally thought that since the vm was thick provisioned and on tier 3 (7.2k NL drives) that was the reason for the slowness. We moved it to tier 1 (15k SAS drives) storage and and migrated it forcing thin provision. This took some time off but still not what we were expecting.

So we finally get a site that is 10 gig backbone and all connections were 10 gig. So we thought this should be a breeze and we still saw the same slowness. Any ideas on what could be wrong? Let me know if you need more info.

[dbalcaraz]

Hi,

I don't know but, are you sure that you have both networks isolated?

Otherwise, you could create a separate TCP/IP stack just for that and then removed it (or not): Place Traffic for Cold Migration, Cloning, and Snapshots on the Provisioning TCP/IP Stack

You must check which VMkernel are you using for this migrations and the physical layer in order to find what's happening there.

[bkjackson78]

The mgmt networks are on public IP's. I read through that link and it looks like it is requiring that both clusters be on 6.x? Cluster 1 for my team is 5.5 and cluster 2 is 6.5.

[dbalcaraz]

Well, mgmt network can have public IPs but, it must be secured and not open to the Internet of course.

Oh, thought that both clusters were at least 6.x .

It seems that you already test the speed but don't know why isn't be used for the migration, isn't it?

If you are just doing cold migration ("vMotion" with powered-off VMs) the traffic would be through the management network so, check the connection between both clusters.