VMware Cloud Community
rbolgerTrace3
Enthusiast
Enthusiast

File-based VCSA backup over SCP not working against non-Linux SSH server

I've been attempting to test out the SCP option in the new File-based VCSA backup feature in 6.5 (specifically, I'm testing against 6.5.0e which is the latest as of today). My backup target is currently a Windows box running Bitvise SSH Server using a dedicated local account that is limited to file transfers only (no shell) and a virtual root directory.  The user config looks like this if anyone wants to try and reproduce locally.

bitvise-account-details.png

I have verified that SCP file transfers to and from my VCSA to this SSH server using the account I configured work just fine. That is, I can SSH to vCenter, drop to the shell, and "scp filename backup@myserver:/" or "scp backup@myserver:/filename ./" and everything transfers successfully.

However, when I attempt to configure the backup job from the VCSA web UI, I get the following error.

Access to the remote server is denied. Check your credentials and permissions.

For reference, here are the values I used in the wizard. I also pre-created an empty "myfolder" in the virtual root.

backup-gui.png

When I check the logs on the SSH server, I see that the session was denied because it was attempting to execute a command, "Command execute request rejected." This is exactly what I wanted it to do when I set the "No shell access" option. But why is VCSA trying to open a shell when it's supposed to be using SCP?

So let's dig into the log on the VCSA side.  I see this in /var/log/vmware/applmgmt/backupRestoreAPI.log:

[MainProcess:PID-2311] ERROR: Failed to validate SSH login at 10.10.10.10:22/myfolder; sshCmd:['sshpass', '-p', '****', 'ssh', '-o', 'ConnectTimeout=10', '-o', 'StrictHostKeyChecking=no', '-p', '22', '-q', '-l', '****', '10.10.10.10', 'exit'], rc: 255, stdOut: , stdErr:

That definitely verifies that it was trying to open a shell. But why? Just to validate credentials in the wizard? Why not just SCP a zero byte file or something?  Again, I've already verified that plain old scp works just fine with the current configuration and the GUI and Docs all refer to this option specifically as "SCP" and not "SSH".

In any case, now I'm curious what it will take to get it to work. So I modify the settings for the backup user to allow a shell. Bitvise has a few options for shells. But the only one that respects the virtual filesystem root folder is their BvShell option. From their help docs:

BvShell is a bash-like shell provided by the SSH Server. This is the only shell type which respects the virtual filesystem layout configured for the client in SSH Server settings. BvShell does not permit execution of arbitrary programs. It is intended to be enabled for clients limited to file transfer access that may wish to use commands (e.g. file copy) that their SFTP or SCP client might not provide. (emphasis mine)

Sounds great! The backup user should still be tied to the virtual root folder and limited to commands that are associated with file transfers. I run through the wizard again and make it through to where it seems to be starting the backup job. But it eventually errors out with the message,

BackupManager encountered an exception. Please check logs for details.

So, back to the logs. backupRestoreAPI.log is no help. It just confirms the job failed. There's now a /var/log/vmware/applmgmt/backup.log though which has more details.

[MainProcess:PID-60326] ERROR: Failed to create dir /myfolder/ at 10.10.10.10; sshCmd:['sshpass', '-p', '****', 'ssh', '-o', 'ConnectTimeout=10', '-o', 'StrictHostKeyChecking=no', '-p', '22', '-q', '-l', u'****', u'10.10.10.10', u'mkdir -p /myfolder/'], rc:2, stdOut:, stdErr:mkdir: Unexpected command parameter: -p.

So it's trying to create the "myfolder" which I had already pre-created using "mkdir -p" and apparently the mkdir command in BvShell doesn't support the "-p" option. So now we know not only does this "SCP" transfer method require a shell to execute commands. It is also expecting to have access to (presumably) GNU compatible binaries.

None of our other shell choices in Bitvise respect the virtual filesystem root. So at this point, we're not going to satisfy the security folks even if we do manage to make this work. But in the interest of science, let's see if we can go further and make it work.  The "Command Prompt" and "Powershell" options are going to be no good without installing Windows ports of those utilities or resorting to Cygwin. "Git access only" won't work either.  But we do have a "bash" option that just needs to be pointed to an executable and this server already has Git for Windows installed which comes with Git Bash that might just do the trick. At the very least, it has a GNU mkdir that supports "-p".

So we change the backup user's shell to "bash" and point it to "C:\Program Files\Git\bin\bash.exe". We also change the virtual filesystem layout to "Allow full access" and update the Location we're using in the Backup GUI to "10.10.10.10/c/temp/backup/myfolder".  We make it through the Backup wizard GUI again and the job starts...and then fails with,

Error at process ComponentScriptsBackup; rc:1. For details please check the log file: /var/log/vmware/applmgmt/backup.log.

*sigh* Back to the log file where we find a lot more going on but there's some sort of error with cat following a tar command?

[LotusBackup:PID-6322] INFO: BackupLotus: Dispatching files ['lock.mdb', 'data.mdb']

[LotusBackup:PID-6322] INFO: tarCmd = tar -cz -C /tmp/backup_lotus/ --ignore-failed-read --warning="no-file-ignored" "lock.mdb" "data.mdb"

[ComponentScriptsBackup:PID-6324] ERROR: rc: 1, stderr: cat: '>': No such file or directory

cat: /c/temp/backup/myfolder/imagebuilder.gz.enc: No such file or directory

[ComponentScriptsBackup:PID-6324] ERROR: Component imagebuilder backup dispatch failed.

At this point I'm spent and I keep coming back to the same question. Why is this backup option labeled "SCP" if it's not actually going to use SCP???  There's nothing in the documentation that references any requirements for a GNU compatible shell or even a shell at all. The only notes about SCP at all are reminding you to use the absolute path to the folder. Why should SCP work any differently than FTP where it is clearly able to just transfer the backup files???

14 Replies
AishR
VMware Employee
VMware Employee

Reply
0 Kudos
rbolgerTrace3
Enthusiast
Enthusiast

Sadly, those walkthroughs have zero value beyond demonstrating the basics of how the feature is supposed to work. They also don't say anything specific about SCP other than that it's supposed to be supported.

virtualDD
Enthusiast
Enthusiast

Your post was a very interesting read, thanks for that, even though I have no real solution to your problem.

What I can tell you is that there are different ways to make the VCSA backup scheduled.

You can either use the method from this blog: VMware vCSA 6.5 Scheduled Backup – The vGoodie-bag  that will leverage the new backup functionality. Or you can try the way I've done this starting with VCSA 6.0 (still works with 6.5 though): Automating VCSA Backup & Restore – virtualfrog – daily virtual life

Either way works. The benefit that comes from doing it my way is that you can restore your DB without deploying a new VCSA. If you have to deploy a new VCSA you can still do the db restore after deploying it or use the db backup on a clone of the original vcsa.

Reply
0 Kudos
rbolgerTrace3
Enthusiast
Enthusiast

Scheduling the backup is actually the easy part thanks largely to Brian's script which I've even contributed code to via github. But your first link is still just using FTP rather than SCP. And it's the SCP implementation on the VCSA side that is the problem here.

For the time being, I'm working around the problem by making my target SCP server the VCSA itself.  So it's literally backing up to itself in /tmp over SCP and then I'm separately SCP'ing the resulting files down to my backup server (all in one script). But it's a frustrating hack for something that seems so simple and should just work.

Reply
0 Kudos
Mnemonic
Enthusiast
Enthusiast

Hi, I am glad that my article is found useful.

I have updated it with information on how to store backups using SCP.

VMware vCSA 6.5 Scheduled Backup – The vGoodie-bag

Reply
0 Kudos
rbolgerTrace3
Enthusiast
Enthusiast

Forgive me. When I referred to "Brian's script", I meant Brian Graf's PowerShell script here: https://www.brianjgraf.com/2016/11/18/vsphere-6-5-automate-vcsa-backup/

Your (Brian Knutsson) article is fine. But it doesn't really address the issues I'm having with SCP. It's largely just the same as everyone else who just says change the protcol to SCP and everything works like magic which is totally wrong unless your SCP server happens to be a standard Linux distribution with shell access for the account you're using.

Unfortunately, upgrading to vCenter 6.7 doesn't seem to fix this either. Though they did tweak the shell commands it's using somewhat.

Reply
0 Kudos
rbolgerTrace3
Enthusiast
Enthusiast

LOL. Apparently, they've updated the docs for 6.7. From Considerations and Limitations for File-Based Backup and Restore

If you use SCP protocol, you must use the Linux backup server

I guess that's one way to do it. Or you know, you could just use the SCP protocol like you say you're going to instead of some weird shell piping tar gzip nonsense. /sigh

HFMudd
Enthusiast
Enthusiast

I have been beating my head on my desk for the past few days and not figuring out why SCP backup is not working with my Windows Server when it works for other things and there it is the smoking gun... not a fan of this backup

Reply
0 Kudos
HendersonD
Hot Shot
Hot Shot

After spending about two hours trying to get VCSA to backup to my Windows server running Bitvise, I stumbled across your post. Apparently either VCSA or Bitvise does not comply with the standards for SCP. I am using SCP to backup my Juniper switches to this same server with no difficulties. Unfortunately, Bitvise does not support FTPS yet or I would try that. I do not have a Linux box spun up to use it as a target for SCP. Not exactly sure what I am going to do now

Reply
0 Kudos
rbolgerTrace3
Enthusiast
Enthusiast

Again, based on my research it's not that Bitvise isn't compatible with SCP. It's that vCenter is just not using SCP despite the label. It's using Linux shell commands and piping data streams across a standard SSH terminal session. And it makes no sense to me why they'd do that when the other options clearly have the ability to just transfer files using things like FTP and WebDAV without a shell.

Reply
0 Kudos
mfarell31
Contributor
Contributor

/edit: I acknowledge this doesn't fix the problem, I'm afraid I dont have a solution for that - I merely thought it was worth providing an alternative that might meet your needs

I feel your pain, I struggled getting a reliable backup going in our environment too and the easiest/most reliable method I've found is webdav via a simply python library which is available to both Linux & Winblows users.

1) Install python3, and pyton3-pip

2) command: pip3 install pywebdav

3) Run your new webdav server: python3 /usr/local/lib/python3.5/dist-packages/pywebdav/server/server.py --directory /tmp/vcsa/ --host <BackupHost / DST IP> --port 8080 --user vcsa --password vcsa

Optional (My shortcut to launch):
alias vcsa_dav='(sudo iptables-save | fgrep '\''<VCSA Source IP>'\'' 1> /dev/null || sudo iptables -A INPUT -s <VCSA Source IP> -p tcp --dport 8080 -j ACCEPT ; [ -d  /tmp/vcsa ] ||  mkdir /tmp/vcsa ;  python3 /usr/local/lib/python3.5/dist-packages/pywebdav/server/server.py --directory /tmp/vcsa/ --host <BackupHost / DST IP> --port 8080 --user vcsa --password vcsa)'

*** Note: iptables is only necessary in my case because I have my rules auto-revert to a known state every 60 minutes ***

I hope this helps, cheers,

Mike

M.F.
Reply
0 Kudos
mgiannoni
Contributor
Contributor

I had the same problem, so I ssh to the the vcsa and tried to scp to the same linux server where I had the issue backing up with scp from the VCSA GUI .

after I typed the command scp <somefile> anaccount@mioserver/home/anaccount/backup I got the ssh mismatch error that the key for the ssh exchange was wrong...

The solution is to vi the ~/.ssh/known_host file and delete all the record related to the server where I was trying to SCP the backup file.

After this the VCSA backup via SCP start working like a charm from the web gui interface.

I hope this helps.

Thanks

Marcello

Reply
0 Kudos
SAGCH
Contributor
Contributor

Hi there, similar problem here, and I can confirm it is not working with the openssh implementation from Microsoft either (Releases · PowerShell/Win32-OpenSSH · GitHub ), as sshpass has not been implemented. scp alone tested from VCSA to the Windows server works fine of course.

sshCmd:['sshpass', '-p', '****', 'ssh', '-o', 'ConnectTimeout=10', '-o', 'StrictHostKeyChecking=no', '-l', 'Edited:Username', 'Edited:Hostname', 'test -d /'], rc:1, stdOut:, stdErr:'test' is not recognized as an internal or external command, operable program or batch file.

Reply
0 Kudos
loungehostmaste
Enthusiast
Enthusiast

yeah, great cinema that vmware don't manage such basics like ask to accept a new host key.... had the same issue now because i removed any hostkeys which are not ed25519 weeks ago and after i smelled that could be the reason vreated a new ecdsa one..... hell - software which don't support ed25519 in 2020 can be found where?

Reply
0 Kudos