Hi,
so I'm trying to copy folder with it's content recursively
using LucD's script from Here but instead of copy 1 item
I'm trying to copy a folder with its content
and its super slow, its copying 1 file every 1-2 seconds, no matter the file volume.
while if I copy using WinSCP its done within seconds... so I'm guessing its not a firewall thing
or any network issues between the 2 ends.
is there faster way to copy folders and files?
this is what I use:
New-PSDrive -Location $ds -Name DS -PSProvider VimDatastore -Root "\" > $null
if(!(Test-Path -Path "DS:/$($tgtFolder)")){
New-Item -ItemType Directory -Path "DS:/$($tgtFolder)" > $null
Copy-DatastoreItem -Item $content -Destination "DS:/$($tgtFolder)" -Recurse -Force
}
else{
Write-Host "DS Folder Exists"
}
Remove-PSDrive -Name DS -Confirm:$false
This seems to work for me.
$localFolder = 'TestFolder'
$dsName = 'MyDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'
$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)
$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
Write-Error "SSH service is not running on $esxName"
}
$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')
$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;mkdir $folder" | Out-Null
}
else{
Get-ChildItem -Path $folder |
ForEach-Object -Process {
Invoke-SSHCommand -SSHSession $session -Command "cd $($destRoot + '/' + $folder); rm -f $($_.Name)" | Out-Null
}
}
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
ComputerName = $esx.Name
Credential = $cred
Path = $localFolder
Destination = $destRoot
AcceptKey = $true
Force = $true
}
Set-SCPItem @sUpload
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
We could leave that test out altogether, and just delete the folder every time (with the -f option that will not return an error when the folder doesn't exist).
$localFolder = 'TestFolder'
$dsName = 'MuDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'
$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)
$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
Write-Error "SSH service is not running on $esxName"
}
$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')
$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -rf $localFolder; mkdir $localFolder" | Out-Null
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
ComputerName = $esx.Name
Credential = $cred
Path = $localFolder
Destination = $destRoot
AcceptKey = $true
Force = $true
}
Set-SCPItem @sUpload
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
If you can stick with SCP.
An alternative can be the HTTP(S) access. See also URL Syntax for HTTP Access
An example Solved: Re: PowerCli : Read from URL - VMware Technology Network VMTN
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
is there a way to "stick with SCP" via powershell commands?
I just need to copy folders with files in them from local\shared folders to the DS
I use the Posh-SSH module which has a Get-SCPItem and Set-ScpItem cmdlet.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
can you show me an example in script how one can use the Set-SCPItem to upload from local folder recursively to Datastore destination please?
also if possible to delete the folder recursively after some commands finished successfully?
Thanks
Sure.
The following snippet copies a complete folder from local storage to a Datastore folder.
The target folder on the Datastore will be created if it doesn't exist yet.
Note1: when one or more of the files you want to copy already exist on the Datastore in the target folder, the Set-ScpItem cmdlet will error out.
You can remove existing files, but then you would need a more complex script that checks for the existence of a file in the destination folder on the Datastore, and eventually remove it. And that for every file in the source folder.
Note2: when you want to upload to a non-root folder on the Datastore, the intermediate folders will have to be created recursively.
The Set-ScpItem cmdlet expects that the target folder(s) exist.
$localFolder = 'TestFolder'
$dsName = 'MyDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'
$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)
$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
Write-Error "SSH service is not running on $esxName"
}
$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')
$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;mkdir $folder" | Out-Null
}
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
ComputerName = $esx.Name
Credential = $cred
Path = $localFolder
Destination = $destRoot
AcceptKey = $true
Force = $true
}
Set-SCPItem @sUpload
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
its saying no such file or directory
Set-SCPItem : scp: /vmfs/volumes/testFolder: No such file or directory
EDIT:
actually I can fix it with the script above, just to create the folder there...
unless its easy to fix in yours new one
What kind of Datastore is that?
VMFS, NFS, VSAN, ...?
There normally is a UUID in that path (after volumes)
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I think I fixed that just with creating the folder first with
if(!(Test-Path -Path "DS:/$($tgtFolder)")){
New-Item -ItemType Directory -Path "DS:/$($tgtFolder)" > $null
}
and then continue to copying
got another 2 questions though:
1. incase I have multiple DS assigned on the said Host, how would I pick the first one which ever it will be? without giving the DS name...
tried to check with array picking but couldn't make it work.
lets say Host1 have 3 DS - DS1, Datastore2, HDD3
how can I pick the first that comes up in the Get-Datastore (without stating the name of the DS) so it would pick DS1
2. you said it would be more complex to remove the folder with files from the DS after some commands are done?
how complex are we talking? cant it be like if($someCommands -ne $null (or $error) ){do the commands that remove the folder }
?
1. Just do
Get-Datastore -RelatedObject $esx | Select -First 1
2. You would need a foreach loop over each files in the source folder.
In the loop you would need an IInvoke-SSHCommand to test if the file exist, and if it does remove it.
The bash command could be something like
rm -f <filename>
The -f option will not produce an error when the file doesn't exist, and will remove it when it exists
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
This seems to work for me.
$localFolder = 'TestFolder'
$dsName = 'MyDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'
$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)
$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
Write-Error "SSH service is not running on $esxName"
}
$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')
$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;mkdir $folder" | Out-Null
}
else{
Get-ChildItem -Path $folder |
ForEach-Object -Process {
Invoke-SSHCommand -SSHSession $session -Command "cd $($destRoot + '/' + $folder); rm -f $($_.Name)" | Out-Null
}
}
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
ComputerName = $esx.Name
Credential = $cred
Path = $localFolder
Destination = $destRoot
AcceptKey = $true
Force = $true
}
Set-SCPItem @sUpload
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
ok thanks!
about 2. I'll try to come up with the right script.
but why I need a loop to check for every file?
I need just to check if the folder exists, and remove it with it's content, I don't need to check for content of the folder for removing the folder
Correct, but I assumed that there could be some files in there that should not be overwritten.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
could you help me out with script for deleting the folder too please?
That is basically doing a rm but with -r parameter
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -r $folder" | Out-Null
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
so I tried few different ways and still not getting it...
I got this and its still not working for me.
obviously I'm doing it wrong...
help 🙂
if($dsPath -ne $null){
$cred = New-Object -Type PSCredential -ArgumentList $esxiUser,(ConvertTo-SecureString -String $esxiPass -AsPlainText -Force)
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')
$session = New-SSHSession -ComputerName $hosts -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -r $folder" | Out-Null
}
Remove-SSHSession -SSHSession $session | Out-Null
}
In the IF you are testing if the folder does NOT exist (exitcode is 1), and then you try to remove that folder.
You should test for exitcode equal to 0, and then remove the folder
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
yes I did that too before with
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 0){
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -r $folder" | Out-Null
}
Remove-SSHSession -SSHSession $session | Out-Null
it didn't work as well.
We could leave that test out altogether, and just delete the folder every time (with the -f option that will not return an error when the folder doesn't exist).
$localFolder = 'TestFolder'
$dsName = 'MuDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'
$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)
$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
Write-Error "SSH service is not running on $esxName"
}
$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')
$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -rf $localFolder; mkdir $localFolder" | Out-Null
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
ComputerName = $esx.Name
Credential = $cred
Path = $localFolder
Destination = $destRoot
AcceptKey = $true
Force = $true
}
Set-SCPItem @sUpload
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
your the best!