1

Is there a command line tool which supports both multiple simultaneous connections and multiple network protocols (ftp, http etc)?

Here is a practical example:

I have a large backup file on an ssh server I want to transfer to my current machine. The file download is taking too long, so I want to use multiple simultaneous network connections to speed it up. Assume the ssh server is not heavily loaded.

Downloads of multiple files/URLs using simultaneous connections are not relevant to this question.

I do not want to have to write any code to solve this.

2 Answers 2

2

The lftp program can handle both multiple network protocols and simultaneous connections.

Protocol support:

Lftp supports the following protocols:

  • ftp
  • sftp
  • http and https
  • bittorrent

Further information on ftps etc here.

Parallel downloads:

To download a single large file located at <URI> using 2 simultaneous connections:

lftp -c "pget -c -n 2 <URI>"

The first -c option specifies the command(s) to run. pget establishes a file transfer using several parallel connections. The second -c option means continue transfer if this transfer had previously been interrupted. The -n option limits the maximum number of connections to use.

There is more information on commands and options in the online man page.

Warning: This can speedup transfers, but loads the network and servers heavily. Speedups are contingent on network and server conditions.

To address the practical example in the question:

lftp -c "pget -c -n 2 sftp://[email protected]/~/backups/large.tar.gz"

This example downloads a large backup file over ssh (sftp) with 2 simultaneous connections. It assumes you have password-less ssh login setup.

Note: Lftp can also be used in an interactive mode; similar to other ftp commands.

2
  • 1
    As you've answered your own question - could you give figures on how beneficial parallel transfers is? i.e: transfer rate of a single connection vs. multiple (2 or 4)
    – Attie
    Jun 30, 2020 at 13:52
  • Easier said than done. Rigorous benchmarking would require multiple runs with multiple files sizes and multiple connections etc and I cannot control network conditions. Benchmarks do not necessarily transfer from test to production environments. Anecdotally, I see a bit less than 2 times speed increase when using 2 connections on an active internal work network while transferring multi-GB files. Diminishing returns set in with large numbers of connections as you'd expect. Jun 30, 2020 at 15:48
1

aria2 supports multiple connections and FTP, HTTP and SFTP (amont others).

For example:

aria2c -j ${num_concurrent} "${uri}"

I want to use multiple simultaneous network connections to speed it up

Unless the server is configured to throttle individual connections, running multiple parallel downloads may not give any benefit... in fact, it may negatively affect your throughput.

Using standard tools (i.e: rsync, scp, curl) will typically give you what you're after.

If dropped connections / partial transfers are an issue, then a utility like this can be beneficial, but note that rsync --partial will also help.

2
  • I don't remember ever seeing a dropped connection with lftp, but I mostly use it on an internal network. Jun 30, 2020 at 13:49
  • Interesting to note that aria2 can download from multiple sources like so: aria2c a/f.iso b/f.iso Jun 30, 2020 at 13:55

Not the answer you're looking for? Browse other questions tagged .