VMware Cloud Community
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

Copy-DatastoraItem so very slow

Hi,

so I'm trying to copy folder with it's content recursively 

using LucD's script from Here but instead of copy 1 item

I'm trying to copy a folder with its content

and its super slow, its copying 1 file every 1-2 seconds, no matter the file volume.

while if I copy using WinSCP its done within seconds... so I'm guessing its not a firewall thing

or any network issues between the 2 ends.

is there faster way to copy folders and files?
this is what I use:

 

New-PSDrive -Location $ds -Name DS -PSProvider VimDatastore -Root "\" > $null
if(!(Test-Path -Path "DS:/$($tgtFolder)")){
    New-Item -ItemType Directory -Path "DS:/$($tgtFolder)" > $null
    Copy-DatastoreItem -Item $content -Destination "DS:/$($tgtFolder)" -Recurse -Force
}
else{
    Write-Host "DS Folder Exists"
}
Remove-PSDrive -Name DS -Confirm:$false

 

Reply
0 Kudos
2 Solutions

Accepted Solutions
LucD
Leadership
Leadership
Jump to solution

This seems to work for me.

$localFolder = 'TestFolder'
$dsName = 'MyDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'

$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)

$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
  Write-Error "SSH service is not running on $esxName"
}

$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')

$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
  Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;mkdir $folder" | Out-Null
}
else{
  Get-ChildItem -Path $folder |
  ForEach-Object -Process {
    Invoke-SSHCommand -SSHSession $session -Command "cd $($destRoot + '/' + $folder); rm -f $($_.Name)" | Out-Null
  }
}
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
   ComputerName = $esx.Name
   Credential = $cred
   Path = $localFolder
   Destination = $destRoot
   AcceptKey = $true
   Force = $true
}
Set-SCPItem @sUpload


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

View solution in original post

LucD
Leadership
Leadership
Jump to solution

We could leave that test out altogether, and just delete the folder every time (with the -f option that will not return an error when the folder doesn't exist).

$localFolder = 'TestFolder'
$dsName = 'MuDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'

$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)

$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
  Write-Error "SSH service is not running on $esxName"
}

$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')

$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -rf $localFolder; mkdir $localFolder" | Out-Null
Remove-SSHSession -SSHSession $session | Out-Null

$sUpload = @{
   ComputerName = $esx.Name
   Credential = $cred
   Path = $localFolder
   Destination = $destRoot
   AcceptKey = $true
   Force = $true
}
Set-SCPItem @sUpload

 


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

View solution in original post

19 Replies
LucD
Leadership
Leadership
Jump to solution

If you can stick with SCP.

An alternative can be the HTTP(S) access. See also URL Syntax for HTTP Access

An example Solved: Re: PowerCli : Read from URL - VMware Technology Network VMTN


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

is there a way to "stick with SCP" via powershell commands?

I just need to copy folders with files in them from local\shared folders to the DS

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

I use the Posh-SSH module which has a Get-SCPItem and Set-ScpItem cmdlet.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

can you show me an example in script how one can use the Set-SCPItem to upload from local folder recursively to Datastore destination please?

also if possible to delete the folder recursively after some commands finished successfully?

Thanks

 

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Sure.
The following snippet copies a complete folder from local storage to a Datastore folder.
The target folder on the Datastore will be created if it doesn't exist yet.

Note1: when one or more of the files you want to copy already exist on the Datastore in the target folder, the Set-ScpItem cmdlet will error out.
You can remove existing files, but then you would need a more complex script that checks for the existence of a file in the destination folder on the Datastore, and eventually remove it. And that for every file in the source folder.

Note2: when you want to upload to a non-root folder on the Datastore, the intermediate folders will have to be created recursively.
The Set-ScpItem cmdlet expects that the target folder(s) exist.

$localFolder = 'TestFolder'
$dsName = 'MyDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'

$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)

$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
  Write-Error "SSH service is not running on $esxName"
}

$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')

$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
  Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;mkdir $folder" | Out-Null
}
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
   ComputerName = $esx.Name
   Credential = $cred
   Path = $localFolder
   Destination = $destRoot
   AcceptKey = $true
   Force = $true
}
Set-SCPItem @sUpload




Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

its saying no such file or directory 

 

Set-SCPItem : scp: /vmfs/volumes/testFolder: No such file or directory

 

EDIT:

actually I can fix it with the script above, just to create the folder there...

unless its easy to fix in yours new one

 

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

What kind of Datastore is that?
VMFS, NFS, VSAN, ...?
There normally is a UUID in that path (after volumes)


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

I think I fixed that just with creating the folder first with 

if(!(Test-Path -Path "DS:/$($tgtFolder)")){
        New-Item -ItemType Directory -Path "DS:/$($tgtFolder)" > $null
}

and then continue to copying

got another 2 questions though:

1. incase I have multiple DS assigned on the said Host, how would I pick the first one which ever it will be? without giving the DS name...

tried to check with array picking but couldn't make it work.
lets say Host1 have 3 DS - DS1, Datastore2, HDD3
how can I pick the first that comes up in the Get-Datastore (without stating the name of the DS) so it would pick DS1

 

2. you said it would be more complex to remove the folder with files from the DS after some commands are done?

how complex are we talking? cant it be like if($someCommands -ne $null (or $error) ){do the commands that remove the folder }

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

1. Just do

Get-Datastore -RelatedObject $esx | Select -First 1

 

2. You would need a foreach loop over each files in the source folder.
In the loop you would need an IInvoke-SSHCommand to test if the file exist, and if it does remove it.
The bash command could be something like

rm -f <filename>

The -f option will not produce an error when the file doesn't exist, and will remove it when it exists


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

This seems to work for me.

$localFolder = 'TestFolder'
$dsName = 'MyDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'

$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)

$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
  Write-Error "SSH service is not running on $esxName"
}

$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')

$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
  Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;mkdir $folder" | Out-Null
}
else{
  Get-ChildItem -Path $folder |
  ForEach-Object -Process {
    Invoke-SSHCommand -SSHSession $session -Command "cd $($destRoot + '/' + $folder); rm -f $($_.Name)" | Out-Null
  }
}
Remove-SSHSession -SSHSession $session | Out-Null
$sUpload = @{
   ComputerName = $esx.Name
   Credential = $cred
   Path = $localFolder
   Destination = $destRoot
   AcceptKey = $true
   Force = $true
}
Set-SCPItem @sUpload


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

ok thanks!

about 2. I'll try to come up with the right script.

but why I need a loop to check for every file?
I need just to check if the folder exists, and remove it with it's content, I don't need to check for content of the folder for removing the folder

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Correct, but I assumed that there could be some files in there that should not be overwritten.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

could you help me out with script for deleting the folder too please?

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

That is basically doing a rm but with -r parameter

Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -r $folder" | Out-Null




Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

so I tried few different ways and still not getting it...

I got this and its still not working for me.

obviously I'm doing it wrong...

help 🙂

if($dsPath -ne $null){
    $cred = New-Object -Type PSCredential -ArgumentList $esxiUser,(ConvertTo-SecureString -String $esxiPass -AsPlainText -Force)
    
    $destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')

    $session = New-SSHSession -ComputerName $hosts -Credential $cred -AcceptKey -Force
    if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 1){
        Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -r $folder" | Out-Null
    }
    Remove-SSHSession -SSHSession $session | Out-Null
}

 

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

In the IF you are testing if the folder does NOT exist (exitcode is 1), and then you try to remove that folder.
You should test for exitcode equal to 0, and then remove the folder


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

yes I did that too before with 

if((Invoke-SSHCommand -SSHSession $session -Command "ls $($destPath + '/' + $folder)").ExitStatus -eq 0){
        Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -r $folder" | Out-Null
    }
    Remove-SSHSession -SSHSession $session | Out-Null

 it didn't work as well. 

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

We could leave that test out altogether, and just delete the folder every time (with the -f option that will not return an error when the folder doesn't exist).

$localFolder = 'TestFolder'
$dsName = 'MuDS'
$esxName = 'MyEsx'
$user = 'root'
$pswd = 'VMware1!'

$cred = New-Object -Type PSCredential -ArgumentList $user,(ConvertTo-SecureString -String $pswd -AsPlainText -Force)

$esx = Get-VMHost -Name $esxName
if(-not (Get-VMHostService -VMHost $esx).Where{$_.Key -eq 'TSM-SSH'}.Running){
  Write-Error "SSH service is not running on $esxName"
}

$ds = Get-Datastore -Name $dsName -RelatedObject $esx
$destRoot = $ds.ExtensionData.Info.Url.Replace('ds://', '')

$session = New-SSHSession -ComputerName $esx.Name -Credential $cred -AcceptKey -Force
Invoke-SSHCommand -SSHSession $session -Command "cd $destRoot;rm -rf $localFolder; mkdir $localFolder" | Out-Null
Remove-SSHSession -SSHSession $session | Out-Null

$sUpload = @{
   ComputerName = $esx.Name
   Credential = $cred
   Path = $localFolder
   Destination = $destRoot
   AcceptKey = $true
   Force = $true
}
Set-SCPItem @sUpload

 


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Vlad_Belo
Enthusiast
Enthusiast
Jump to solution

your the best!

Reply
0 Kudos