Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to use cURL to upload some files to an SFTP server, it has to be using cURL because I'll be later using cURL_multi etc to transfer files simultaneously (unless anyone knows a command line library which supports FTP, SFTP using password, SFTP using key, and supports up to 10 simultaneous transfers).

The problem I have is cURL is running painfully slowly. Uploading using the standard SFTP client on Ubuntu 12.04, I'm getting around 70Kb/s (this is fine, that's just the speed of the server), but when using cURL, it's around 35Kb/s, half the speed of SFTP :/

Does anyone know of any ways I can speed up cURL, preferably without having to recompile cURL, but I will if I need to.

Here's the command for SFTP:

/usr/bin/sftp -oProtocol=2 -oIdentityFile=/path/to/key.pri -oPort=5000 -oStrictHostKeyChecking=no username@thedomain.com

then just put filename.wav when it's connected...

and with cURL:

curl -v -u username: --key /path/to/key.pri --pubkey /path/to/key.pub -T /path/to/filename.wav sftp://thedomain.com:5000/folder/TEST/filename.wav --insecure

This is the cURL version info:

curl 7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 libssh2/1.2.8 librtmp/2.3 Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp Features: GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP

Surely it can't be this slow for everyone?

share|improve this question
add comment

1 Answer

This is unfortunately a known issue with curl. It seems it can be mitigated by recompiling with a larger CURL_MAX_WRITE_SIZE; see this entry from their mailing list.

Since you're trying to achieve multi upload in the end, you may still succeed in filling your outgoing pipe with SFTP over curl, which would give you the best possible throughput anyway :)

share|improve this answer
    
The problem we've got is there's an added complexity of using it with PHP... if I have to recompile cURL, then I think I also have to recompile PHP, and I just can't risk that when it's being used on a live production site! –  BT643 May 24 '13 at 10:34
    
The way we need to send the files doesn't fully utilise the "multi" nature either... we have to send a block of files at a time, and can only send more when they've finished, sometimes this means we can only send one at a time, and other times we can send 10 at once :( think I'll have to use a different method :( –  BT643 May 24 '13 at 10:40
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.