tomvr
Contributor
Contributor

Backup Server for SDDC Manager and NSX Manager

Jump to solution

For File-Based-Backup of our VCF (SDDC manager and NSX Manager) I am trying to configure an external SFTP server (Ubuntu with OpenSSH). The 'Validate Backup Location Details' keeps failing with error: Validation failed for directory path /mnt/backups on server xx.xx.yy.yy. Please make sure backup directory is intact and sftp server has write permissions on backup path.

SFTP and SSH from command line of SDDC manager works and I am able to create and delete folders. Also am I using the same SFTP server for File-Based-Backup of our vCenter Server without these issues.

Any ideas?

Thanks in advance,

Tom

Labels (5)
0 Kudos
1 Solution

Accepted Solutions
tomvr
Contributor
Contributor

Hi ksagona, thank you for your reply. The openSSH version is indeed above 8.8. But as I had also opened a case with VMware and as they were able to solve the issue by temporary lowering the security settings, I did not make any changes to the openSSH version.

View solution in original post

0 Kudos
4 Replies
michelev
Contributor
Contributor

Hi Tom


**Possible Cause**

The SFTP account was expired on the backup server.


** Resolution **

To resolve this issue, refresh the expired account on the backup server.


** Other checks list **

[] - Verify that the folder exists on the SFTP backup server.

[] - Verify that the SFTP permissions on the folder allow you to write to it.

[] - Check the user account being used to connect to the Backup server is valid and working.

[] - Try manually connecting to the same SFTP using either an FTP utility or CLI.

Example: sftp username@IPAddress

[] - Manually clear and readd all the Backup parameters and save the configuration again in SDDC Manager.

[] - To obtain the SSH Fingerprint of the target system to verify, connect to the SDDC Manager Appliance over ssh and run the following command:

ssh-keygen -lf <(ssh-keyscan -p 22 -t rsa sftp_server_fqdn 2> /dev/null) | cut -d' ' -f2

[] - Testing connectivity between NSX-T manager and the SFTP server

Login to an NSX-T manager appliance via root or if you do not have root login enabled you can login via the ‘admin’ account and then type ‘st en’ and then provide your root password.

nsxt-mgr> st en
Password:
***************************************************************************
NOTICE TO USERS

WARNING! Changes made to NSX Data Center while logged in as the root user
can cause system failure and potentially impact your network. Please be
advised that changes made to the system as the root user must only be made
under the guidance of VMware.
***************************************************************************
root@nsxt-mgr:~#

From the root CLI of an NSX-T manager appliance we will run the following command to try and connect to our SFTP server and login using the built in OpenSSH package on the NSX-T appliance.

sftp svc_backup-nsxt@sftp-backup
The authenticity of host 'sftp-backup (10.0.1.27)' can't be established.
ECDSA key fingerprint is SHA256:tm1qkfHb19sx5qzwJABAtojOd4cVywRCjmkoGeHeR6E.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sftp-backup' (ECDSA) to the list of known hosts.

Authorized Personnel Only
--------------------------------------------------------
WARNING: Unauthorized access to this system is forbidden
and will be prosecuted to the fullest extent of the law.
--------------------------------------------------------

svc_backup-nsxt@sftp-backup's password:
Connected to sftp-backup.


Now that we are connected to our SFTP server we have successfully validated

We have network connectivity from the NSX-T manager to the SFTP server
The SFTP server service is running on the SFTP server listening for connections
The service account created has permission to connect via the SFTP protocol

Now we need to verify the service account has permission to navigate to the destination backup folder structure we created earlier and create a folder to verify we have ‘write’ permissions

cd /mnt/backups
mkdir test
ls
test
rmdir test
ls
bye

We have now successfully deployed a backup SFTP server that meets the security requirements of NSX-T and validated end to end network connectivity, the SFTP service is running/listening and we have the necessary permissions.

HTH
Michele V.

tomvr
Contributor
Contributor

Hi MIchele,

As stated in my post: 

"SFTP and SSH from command line of SDDC manager works and I am able to create and delete folders. Also am I using the same SFTP server for File-Based-Backup of our vCenter Server without these issues."

I even used the same sftp-account and directory as i have used for the vCenter Server backup.

0 Kudos
ksagona
Contributor
Contributor

Tom, please check to see if your OpenSSH on your SFTP server is at or above version 8.8. I was having the exact same issue with the exact same symptoms that you describe with both a fresh Windows and Linux SFTP server. In the /var/log/vmware/vcf/operationsmanager/operationsmanager.log, I kept seeing errors regarding ssh-rsa keys. If you check out https://www.openssh.com/txt/release-8.8 and https://ikarus.sg/rsa-is-not-dead/, you'll see that ssh-rsa has been deprecated in OpenSSH 8.8 and above this could possibly cause errors like we're seeing.

I tried building a Windows SFTP server with OpenSSH 8.6 using (https://github.com/PowerShell/Win32-OpenSSH/releases/tag/V8.6.0.0p1-Beta), and set it up exactly the way I had set my other Windows SFTP server, and it connected instantly. So it seems like there may be a bug in SDDC manager where it's still looking for ssh-rsa keys instead of bouncing to the ecdsa-sha2-nistp256 keys. Will have to follow up with VMWare to see how we can use the latest version of OpenSSH with SDDC manager as I don't want to use the old standards.

tomvr
Contributor
Contributor

Hi ksagona, thank you for your reply. The openSSH version is indeed above 8.8. But as I had also opened a case with VMware and as they were able to solve the issue by temporary lowering the security settings, I did not make any changes to the openSSH version.

0 Kudos