Tag Info

New answers tagged

1

Though none may exist for your country there are national archives for some countries. E.g., see list of Web archiving initiatives. That web page lists software used in such initiatives for various countries. Some, such as Australia, use HTTTrack. Others use Heritrix, which is free under an Apache license, and has both a web interface and a command-line ...


0

Its your proxy. Your environment would be useful here (linux? Windows?) but I'm guessing linux. You need to find your actual proxy settings (which firefox is picking up) - a little guesswork would be needed here but you can use curl and test each known automatic proxy url or possible variations. Then you need to set a http_proxy environment variable which ...


0

Sorry, didn't catch the point initially. So, a complete new answer (quick and dirty - and since it's bash programming, it works only with linux): use cURL to determine if there's a redirect (using -w parameter); depending on the result, use wget to get either the original url or the redirect target: url=www.smugmug.com/photos/1235566-S.jpg; redirurl=`curl ...


0

This will crawl a site looking for gif,png,jpg,jpeg, then grep those results for any strings. The resulting list is then retrieved to the specified directory by wget. You need to adjust '/path/to/save/files', as well as the url to grab the files from. Adjust the final grep statement to define your search parameters. It would be: grep "191\|282" --instead ...


0

When you use - as the parameter of the option -O it represent the standard output. You are using a pipeline so both commands are executed in parallel and the standard input of the tar command is bound to the standard output of the wget command. The tar command is broking the pipeline so wget can not write to standard output. When this happens, check the ...



Top 50 recent answers are included