0

I need a scripting solution to download several (more then 100) websites by FTP for backup purposes. For each website I want a scripting solution which supports 10 parallel connections to only one website.

  • Each website has his own FTP address.
  • Splitting the complete task in 10 different batches is ok but still there will be 1 connection to 1 server which cause very a very slow download. Create tars on the server is not a option and is out of this question :-)

I'm using Windows 8.1, 64-bit.

I tried Windows FTP but that supports only 1 parallel connections to 1 server.

Also WinSCP is not supporting more then 1 parallel connection to 1 FTP server command line mode. I tried several options and maybe I'm doing something wrong here???

download.bat > this file is started with admin rights

md n:\backupftp\ithaka.im\public_html
winscp.exe /script="ithaka.txt"  

ithaka.txt

open ftp://xxx:xxxxx@ftp.ithaka.im/domains/ithaka.im/public_html
lcd n:\backupftp\ithaka.im\public_html
get *

The winscp.exe and winscp.com are portable executables in the same directory as the download.bat and ithaka.txt

As a result of this I don't see the GUI interface. Only a task in the task manager. If I turn on logging I see only 1 connection to 1 FTP server at a time. No parallel downloads from 1 FTP server.

If I change the download.bat into (notice that I change .exe to .com):

md n:\backupftp\ithaka.im\public_html
winscp.com /script="ithaka.txt"  

The result will be:

modules                   |            0 B |    0,0 KB/s | binary |   0%
php                       |            0 B |    0,0 KB/s | binary |   0%
php.test                  |           4 KB |    6,9 KB/s | binary | 100%
php.module                |           7 KB |    6,6 KB/s | binary | 100%
php.install               |           1 KB |    1,8 KB/s | binary | 100%
php.info                  |          274 B |    1,3 KB/s | binary | 100%
block                     |            0 B |    1,3 KB/s | binary |   0%

etc etc etc.. these downloads are not parallel and not supported by WinSCP.

So:

  1. Do I something wrong with WINSCP
  2. Is there a FTP scripting solution which support this on Windows? I looked into several alternatives but they all have graphical user interfaces which I don't want..
  • Yes, there are several but WinSCP should work just fine. You will need to define the connection to the multiple FTP servers within WinSCP... I use this for hundreds of FTP connections just fine using the plain old batch script and dynamic WinSCP files. You're likely just not using it correctly. Are the websites accessed via FTP or do you access those via SMB and then upload the files via FTP to an FTP server. Does each website have it's own correlated FTP to each site or how exactly is all that configured on your side? More detail will likely be needed. – Pimp Juice IT Feb 3 '16 at 15:22
  • 1
    Also, if you can provide what you see for whatever is indicating that you cannot get more than one connection to one server to work, that may be helpful as well. – Pimp Juice IT Feb 3 '16 at 15:27
  • 1
    Cannot you just split the task to say 10 batches, running them in parallel? – Martin Prikryl Feb 3 '16 at 16:25
  • 1
    I believe hardly anyone can understand from your question that you connect to a single FTP site, but you are looking for parallel transfers from that single site. Your question here is as vague as the question that you most probably posted recently to WinSCP forum – Martin Prikryl Feb 3 '16 at 16:31
  • 1
    You could have one batch script with a simple loop up top that one after the other calls a routine changing different WinSCP connection names but I agree with Martin that your detail is vague. I would suggest WinSCP over the Windows built-in FTP if you can for even plain old insecure FTP as it is much more robust and simple to understand and works quite reliably from my years using it at the batch script level. – Pimp Juice IT Feb 3 '16 at 17:22
1

WinSCP scripting does not support parallel transfers on its own.

But you can run multiple WinSCP scripts in parallel.

You just have to split the set of files/directories to batches.

There's actually a ready-made PowerShell script based on WinSCP .NET assembly that does this.

See Automating download in parallel connections over SFTP/FTP protocol.

You can run the script (say parallel_download.ps1) like:

powershell.exe -File parallel_download.ps1 -sessionUrl ftp://xxx:xxxxx@ftp.ithaka.im/ ^
    -remotePath /domains/ithaka.im/public_html/ ^
    -localPath n:\backupftp\ithaka.im\public_html\ ^
    -batches 3

The above command can be directly used in a batch file (.bat).
Or executed from a command-line, if merged into a single line with all ^ removed.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.