I am using curl through Cygwin on Windows 7 to download content from Apache Solr in CSV format. This is my command which is running in a for loop

curl -o ret-key-data-$i.csv $URL

As the content is very large, I am downloading 100k rows at a time. The file size is approx. 4mb each. However, if I increase the row count to 1 million, curl is not writing any content to disk.

What other option do I have to set to ensure curl can handle larger content?

EDIT: The command gives no error. It does not hang. It waits for a period of time and ends successfully but generates a zero kb file. Looks like the network is timing out.

  • Please be more specific as to what happens. How is it "not writing any content"? Does the command wait forever and there is no output file, or an empty one? Do you see any messages output to terminal? – slhck May 12 '14 at 5:52
  • could the web server be timing out and severing the connection? – rob May 12 '14 at 8:05
  • I use cURL on a daily basis to download files on the hundreds of GB's and have not had an issue with limits to write to file. Can you post the results using the verbose switch -v? It would be helpful to see the details of the request and the response from the server. – MikeBoss Mar 14 '17 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.