Bulk SFTP download of files specified by URL preserving directories

Question

I have a list of several hundred files I need to pull via SFTP. They are located in various directories and I need them to be in the same directory setup on my local machine. It is important the files on the list located in these directories are not included in the download.

I asked this question for downloading URLs here: Bulk Download Images with Organization. I need to have the option of SFTP download available to me as well. Any recommendations on going about doing this would be appreciated.

Answer 1

I do not know of any software that can do it our of the box.

I even suppose you won't find any, as using an HTTP URL as SFTP download specification is not common and there's not even any direct mapping from the URL to an SFTP path.

If you have a URL like

https://www.example.com/sample1/image1.jpg

The file as presented using the SFTP will be in /sample1/usage1.jpg only if the website SFTP account is chrooted. If not, the file can be in path like /home/user/httpdocs/sample1/usage1.jpg or any other.

So I believe you have to script it somehow.

Also you didn't specify if the hostname in the URL changes or is constant. If it changes, where do you get the host credentials from? Or are they included in the URLs too?

Below, see an example of PowerShell script using WinSCP .NET assembly.

Configure the $remoteRoot accordingly.

try
{
    # Load WinSCP .NET assembly
    Add-Type -Path "WinSCPnet.dll"

    # Setup session options
    $sessionOptions = New-Object WinSCP.SessionOptions
    $sessionOptions.Protocol = [WinSCP.Protocol]::Sftp
    $sessionOptions.HostName = "example.com"
    $sessionOptions.UserName = "user"
    $sessionOptions.Password = "mypassword"
    $sessionOptions.SshHostKeyFingerprint = "ssh-rsa 2048 xxxxxxxxxxx...="

    $session = New-Object WinSCP.Session

    $remoteRoot = "/home/user"

    try
    {
        # Connect
        $session.Open($sessionOptions)

        foreach ($line in [System.IO.File]::ReadLines("list.txt"))
        {
            if ($line -Match "http\://[a-z.]+(/(.*)/[a-z0-9.]+)$")
            {
                $remotePath = $matches[1]
                $remoteDir = $matches[2]
                $localDir = $remoteDir -Replace "/", "\"

                if (!(Test-Path $localDir))
                {
                    Write-Host "Creating directory $localDir"
                    New-Item $localDir -Type directory | Out-Null
                }

                Write-Host "Downloading $remotePath"
                $session.GetFiles(($remoteRoot + $remotePath), ($localDir + "\")).Check()
            }
            else
            {
                Write-Host "$line does not have expected URL format"
            }
        }        
    }
    finally
    {
        # Disconnect, clean up
        $session.Dispose()
    }

    exit 0
}
catch [Exception]
{
    Write-Host $_.Exception.Message
    exit 1
}

(I'm the author of WinSCP)