VMware Cloud Community
tomvr
Contributor
Contributor
Jump to solution

Backup Server for SDDC Manager and NSX Manager

For File-Based-Backup of our VCF (SDDC manager and NSX Manager) I am trying to configure an external SFTP server (Ubuntu with OpenSSH). The 'Validate Backup Location Details' keeps failing with error: Validation failed for directory path /mnt/backups on server xx.xx.yy.yy. Please make sure backup directory is intact and sftp server has write permissions on backup path.

SFTP and SSH from command line of SDDC manager works and I am able to create and delete folders. Also am I using the same SFTP server for File-Based-Backup of our vCenter Server without these issues.

Any ideas?

Thanks in advance,

Tom

Labels (5)
Reply
0 Kudos
1 Solution

Accepted Solutions
tomvr
Contributor
Contributor
Jump to solution

Hi ksagona, thank you for your reply. The openSSH version is indeed above 8.8. But as I had also opened a case with VMware and as they were able to solve the issue by temporary lowering the security settings, I did not make any changes to the openSSH version.

View solution in original post

Reply
0 Kudos
8 Replies
michelev
Contributor
Contributor
Jump to solution

Hi Tom


**Possible Cause**

The SFTP account was expired on the backup server.


** Resolution **

To resolve this issue, refresh the expired account on the backup server.


** Other checks list **

[] - Verify that the folder exists on the SFTP backup server.

[] - Verify that the SFTP permissions on the folder allow you to write to it.

[] - Check the user account being used to connect to the Backup server is valid and working.

[] - Try manually connecting to the same SFTP using either an FTP utility or CLI.

Example: sftp username@IPAddress

[] - Manually clear and readd all the Backup parameters and save the configuration again in SDDC Manager.

[] - To obtain the SSH Fingerprint of the target system to verify, connect to the SDDC Manager Appliance over ssh and run the following command:

ssh-keygen -lf <(ssh-keyscan -p 22 -t rsa sftp_server_fqdn 2> /dev/null) | cut -d' ' -f2

[] - Testing connectivity between NSX-T manager and the SFTP server

Login to an NSX-T manager appliance via root or if you do not have root login enabled you can login via the ‘admin’ account and then type ‘st en’ and then provide your root password.

nsxt-mgr> st en
Password:
***************************************************************************
NOTICE TO USERS

WARNING! Changes made to NSX Data Center while logged in as the root user
can cause system failure and potentially impact your network. Please be
advised that changes made to the system as the root user must only be made
under the guidance of VMware.
***************************************************************************
root@nsxt-mgr:~#

From the root CLI of an NSX-T manager appliance we will run the following command to try and connect to our SFTP server and login using the built in OpenSSH package on the NSX-T appliance.

sftp svc_backup-nsxt@sftp-backup
The authenticity of host 'sftp-backup (10.0.1.27)' can't be established.
ECDSA key fingerprint is SHA256:tm1qkfHb19sx5qzwJABAtojOd4cVywRCjmkoGeHeR6E.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'sftp-backup' (ECDSA) to the list of known hosts.

Authorized Personnel Only
--------------------------------------------------------
WARNING: Unauthorized access to this system is forbidden
and will be prosecuted to the fullest extent of the law.
--------------------------------------------------------

svc_backup-nsxt@sftp-backup's password:
Connected to sftp-backup.


Now that we are connected to our SFTP server we have successfully validated

We have network connectivity from the NSX-T manager to the SFTP server
The SFTP server service is running on the SFTP server listening for connections
The service account created has permission to connect via the SFTP protocol

Now we need to verify the service account has permission to navigate to the destination backup folder structure we created earlier and create a folder to verify we have ‘write’ permissions

cd /mnt/backups
mkdir test
ls
test
rmdir test
ls
bye

We have now successfully deployed a backup SFTP server that meets the security requirements of NSX-T and validated end to end network connectivity, the SFTP service is running/listening and we have the necessary permissions.

HTH
Michele V.

tomvr
Contributor
Contributor
Jump to solution

Hi MIchele,

As stated in my post: 

"SFTP and SSH from command line of SDDC manager works and I am able to create and delete folders. Also am I using the same SFTP server for File-Based-Backup of our vCenter Server without these issues."

I even used the same sftp-account and directory as i have used for the vCenter Server backup.

Reply
0 Kudos
ksagona
Contributor
Contributor
Jump to solution

Tom, please check to see if your OpenSSH on your SFTP server is at or above version 8.8. I was having the exact same issue with the exact same symptoms that you describe with both a fresh Windows and Linux SFTP server. In the /var/log/vmware/vcf/operationsmanager/operationsmanager.log, I kept seeing errors regarding ssh-rsa keys. If you check out https://www.openssh.com/txt/release-8.8 and https://ikarus.sg/rsa-is-not-dead/, you'll see that ssh-rsa has been deprecated in OpenSSH 8.8 and above this could possibly cause errors like we're seeing.

I tried building a Windows SFTP server with OpenSSH 8.6 using (https://github.com/PowerShell/Win32-OpenSSH/releases/tag/V8.6.0.0p1-Beta), and set it up exactly the way I had set my other Windows SFTP server, and it connected instantly. So it seems like there may be a bug in SDDC manager where it's still looking for ssh-rsa keys instead of bouncing to the ecdsa-sha2-nistp256 keys. Will have to follow up with VMWare to see how we can use the latest version of OpenSSH with SDDC manager as I don't want to use the old standards.

tomvr
Contributor
Contributor
Jump to solution

Hi ksagona, thank you for your reply. The openSSH version is indeed above 8.8. But as I had also opened a case with VMware and as they were able to solve the issue by temporary lowering the security settings, I did not make any changes to the openSSH version.

Reply
0 Kudos
DougBaer
Commander
Commander
Jump to solution

For those who arrive here and want to know more, the required "lowering" of security is permitting SHA-1 RSA keys on your SSH/SFTP server. This was deprecated as of OpenSSH v8.8, as called out in the release notes - https://www.openssh.com/txt/release-8.8 

This release disables RSA signatures using the SHA-1 hash algorithm
by default. This change has been made as the SHA-1 hash algorithm is
cryptographically broken

The changes I made to my /etc/ssh/sshd_config file to resolve the issue (VCF 4.5.0) were the addition of two lines:

HostKeyAlgorithms=+ssh-rsa
PubkeyAcceptedAlgorithms=+ssh-rsa

followed by a restart of the sshd service.

Doug Baer, Solution Architect, Advanced Services, Broadcom | VCDX #019, vExpert 2012-23
RajeevVCP4
Expert
Expert
Jump to solution

You need to change permission on sddc of backup user

 

go inside cd  /nfs/vmware/vcf/nfs-mount/

ls -lrth 

check is backup folder is there 

if yes

run this command 

 chmod g+w backup

then try

 

Rajeev Chauhan
VCIX-DCV6.5/VSAN/VXRAIL
Please mark help full or correct if my answer is use full for you
Reply
0 Kudos
ksagona
Contributor
Contributor
Jump to solution

Thanks, I tried adding those two lines 

HostKeyAlgorithms=+ssh-rsa
PubkeyAcceptedAlgorithms=+ssh-rsa

to the sshd_config on my Windows server and a quick restart of the service and it worked perfectly. Hopefully it will get addressed in a future update, but for now, that works. Thank you!

Reply
0 Kudos
GregNew
Enthusiast
Enthusiast
Jump to solution

I've just run into the same issue on a fresh build of vCF 5.1.0 backing up to a Photon OS VM.

Thanks for posting the issue @tomvr and to @DougBaer for the solution 🙂

Cheers,
Greg

Reply
0 Kudos