0
votes
0answers
18 views

curl won't shut up (--silent does nothing)

I'm writing a quick shell script, and I cannot for the life of me get curl to suppress the progress bar. The relevant line looks like this: curl http://website -o $temp >/dev/null 2>&1 ...
1
vote
2answers
94 views

How to stop shell script if curl failed

I have script that used curl when i pass wrong parameters to script curl failed but script continue executing. I have tried use curl -f/--fail parameter but problem does not solved. What is the best ...
0
votes
0answers
66 views

Error uncompressing file when piping curl to gunzip

I need to download a gzipped xml feed to parse in my Ruby app. It may well be quite large, 30Mb compressed, 300Mb uncompressed. So I do this: curl -s http://example.com/file.xml.gz | gunzip and read ...
0
votes
1answer
55 views

A Script That is Aware of Remote File Change

curl -z pci.ids --remote-time --silent -f -L -O http://pciids.sourceforge.net/v2.2/pci.ids This Code Checks For The Remote File Change and Download If Remote File Has Been Modified but Is There a ...
2
votes
1answer
44 views

Shell commands executed from Vim behave differently

I have mapped a key in Vim to run curl-based tests on my API with :map ,t :w\|:!quizzical tests.sh <CR>. The tests run as expected when I press the key; however, the tests which take longer to ...
0
votes
2answers
282 views

SFTP - Recursively fetch new files

I have a remove file system structure like this: + /measure + / 2013-09 + / 2013-09-04 + / fooinstrument + result03343445845.csv + ...
3
votes
1answer
631 views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
1
vote
1answer
2k views

How to split the HTTP error code from the contents in cURL?

Yes, this is related to Getting curl to output HTTP status code? but unfortunately not the same. In a script I would like to run: curl -qSfsw %{http_code} URL where the -f option ensures that the ...
0
votes
1answer
128 views

Get Directory/file structure without payload

I need to recursively retreive the file/directory structure of an FTP server, getting only file names (without payload). Can you help me how to do this (curl, wget, ...)? The result should be the ...
0
votes
1answer
409 views

Downloading and extracting a .tar to a specific directory

I want to download a tar of a git repo and extract its contents to another folder. The following line (broken up for clarity) works: curl -L -0 nt-dotfiles.tar.gz ...
0
votes
4answers
115 views

Simple script parsing text, what is wrong here?

I'm a big user of https://www.grc.com/passwords.htm to get strong passwords. However, having to go to the site and manually copy the password every time gets old fast, so I decided to do a little ...
0
votes
2answers
400 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
3
votes
1answer
569 views

How to make alias/function in .bash_profile for `curl` that takes an argument?

I found this tip super helpful, for using curl to resume an interrupted file copy. The full syntax is: curl -C - -O file:///Volumes/path/to/file I would like to add an alias or function called ...
0
votes
1answer
391 views

How to make curl reconnect after 60"?

Sometimes, curl freezes: $ curl -O 'http://www.site.com/index.html' www.site.com/index.html --> index.html % Total % Received % Xferd Average Speed Time Time Time Current ...
0
votes
2answers
1k views

How can I curl the output of another command

I want to pass curl the output from awk ./jspider.sh http://www.mypage.com | grep 'resource' | awk '{print $4}' | curl OUTPUT_FROM_AWK | grep myString How can I achieve this?!
2
votes
2answers
812 views

curl not uploading file to ftp

I am using curl to upload a file to ftp server. The same script is working on another system, but on this one after connecting to ftp it is not doing anything and connection times out. The detailed ...
0
votes
2answers
346 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...
2
votes
2answers
620 views

bash: source from URL

In addition to my own computer, I sometimes use an Ubuntu cluster at my school. Rather than manually keep my .bashrc's in sync, I would like to make the school cluster's .bashrc source my personal ...
14
votes
4answers
12k views

cURL: how to suppress strange output when redirecting?

I'm trying to print just the verbose sections of a cURL request (which are sent to stderr) from the bash shell. But when I redirect stdout like this: curl -v http://somehost/somepage > /dev/null ...