NSX-T Backup takes more than 4mins.

[BlackRamIT]

I've been struggling with something in my nested Home Lab that is driving me to drinking!  I deployed VLC and have VCF 4.2.1 up and stable.  My NSX-T Manager is running 3.1.1.  I'm at the point with SDDC LCM, that if I could get NSX-T upgraded to 3.1.2, I can move on to VCF 4.3.

I can't seem to get the NSX-T Manager to complete it's backups under the 4min. SDDC time limit for completion.  It consistently takes about 10-15mins to finish.  I have been banging on this for quite sometime; building backup linux boxes and researching setting up an SFTP store.  But I'm stuck!

Anyone run into this or have any ideas on how to determining why the backups are taking so long?

[CyberNils · Accepted solution]

Hi,

Try increasing the size of your NSX-T Manager.

[engyak]

How big is the backup file? Have you taken a look inside of it to see what might be taking so long?

[Sarfa_J]

Few things to consider :

-- Since it's nested lab, is the NSX Manager getting enough CPU cycles ? Check for CPU contention %rdy value by running esxtop. 

-- Depends on the inventory size of NSX-T. 

-- Is the backup server also in the nested environment ? If so apply the first point here too.

-- Also, check the Network latency between the NSX manager and SFTP / SCP server. 

[BlackRamIT]

Thanks for the feedback folks.  My backup sizes are about 10Mb.  My Manager is sized with 12vCPU & 24Gb of RAM.  The (4) nested ESXi Hosts are running 12vCPU & 96Gb of RAM, so my single NSX-T Manager is sized to pretty much utilize one ESXi Host. 

Now, my physical vSAN platform is built on three 24vCPU and 256Gb RAM Hosts.  As you can see, I'm sizing this so a single physical host can support two nested ESXi Hosts.  So I'm pretty limited on CPU.

I've built my backup Linux box to be on the same VDS (sddc-vds01-mgmt) as the NSX-T Manager, so there's no crazy routing going on.  Again my backup times are 12-15mins and it looks like SDDC v4.3.0 has increased the timeout from 4mins to 10mins, but it still isn't enough to get the NSXT Cluster backed up and upgraded.

[BlackRamIT]

I do have some latency hits from time to time.  I just moved the Backup Box onto the same host as the NSX-T Manager and the latency looks a little better.  I'll try another upgrade attempt and see if this helps.  (EDIT: It failed again, I'm going to see if I can temporarily push the vCPU/RAM of the NSX-T Manager and see how that behaves).

[BlackRamIT · Accepted solution]

I bumped the nested ESXi Host up to 24vCPU and pinned it to a Physical Host.  I only have the NSX-T Manager running on that Physical/Nested Host.  I pushed the vCPU up to 18vCPU for the Manager and went from 24Gb RAM all the way up to 64Gb.

I was able to complete the backup in 6mins, which has allowed me to get through that part of the upgrade.  I'll edit this post if I get all the way through the v3.1.3 upgrade.  

EDIT:

• UC has been updated!  So far so good!
• NSX Edge Cluster updated! 2 down, 2 to go!
• Transport Nodes Updated, but failed initially.  I had to decrease the NSX-T Manager back down to 12vCPU so that it could be migrated to the other ESXi Hosts (which only have 12vCPU assigned) during the upgrade.  Since the NSX-T Manager Backup had already been done as part of the UC upgrade, the decrease didn't seem to impact anything. 
• NSX-T Manager Upgrade completed successfully.

Bottom-line is to get through the backup, I had to increase from 12->18vCPU to speed up the backup and get in under the 10min limit in SDDC v4.3.0.