0
votes
1answer
21 views

Trace web redirection in bash script

I want to make a script to download the image at best resolution (not the preview image) from a deviantART link, as if I clicked the "Download" button. However, it seems that deviantART redirect the ...
0
votes
0answers
23 views

OpenWRT wget in background

I start first script /etc/rc.local : sh /root/script1.sh & In file /root/script1.sh I connect by wget to server (while for tries) And after it I run : # say hello tries=1 while [ 1 ]; do ...
0
votes
1answer
40 views

WGET/cron downloading webcamera file

our new town doctor put a web camera to his waiting room for people to see how much crowded it is. As my mom and grandparents go to this doctor, I wanted to download the files periodically and if they ...
1
vote
1answer
44 views

Firefox - Download files using external bash script

Is possible to issue external bash script when files are being downloaded in Firefox? My goal is high customization. E.g. scripting something that will save files to specified folder based on ...
0
votes
1answer
95 views

How can I encode an url for wget?

I am looking for a way to convert a string to a clean url. For example : wget http://myurl.com/toto/foo bar.jpg This is going to download http://myurl.com/toto/foo and http://bar.jpg. I want to ...
0
votes
1answer
35 views

Use tmux for managing multiple downloaders as a supervisorctl service?

I have many data-servers I need to download data from via http as soon as it is available. For each server I start a bash "while true"-loop and within that a wget to poll the server for new data. To ...
2
votes
1answer
799 views

How would I `wget` files and then save them by date downloaded rather than filename?

My goal: To download 131 JPEGs and save them in a file name format that is relative to the date/time format rather than their file name. I have already tried things that involve changing the files' ...
1
vote
0answers
67 views

Ncat throws errors while being run from crontab

I have a simple script set up on my OpenWrt box to check if one of the computers in a local network is streaming, and if it does, the script starts wget | madplay to play the stream. Script works ...
1
vote
1answer
551 views

wget execute script after download

I need every time you complete a transfer program wget this run a script. this is my full command line: wget -i urls-to-downloads.txt My problem is that I want to go subtracting it downloads the ...
3
votes
2answers
222 views

Dowloading all Urls accessible under a given domain with wget without saving the actual pages?

Hi trying to determine all the valid urls under a given domain without having the mirror the site locally. People generally want to download all the pages but I just want to get a list of the direct ...
0
votes
2answers
202 views

Script for CRON (FTP get files EVERY MINUTE)

I want to create a script that must run with a CRON every minute (Q: Is it wise to run CRON FTP/wget every minute?) compares local directory with remote directory (Q: do I use FTP or wget and how ...
3
votes
1answer
583 views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
4
votes
3answers
261 views

Spawning multiple parallel wgets and storing results in a bash array to be pretty printed when all wgets are done

I have a long list of urls on my own website listed in a carriage return seperated text file. So for instance: http:/www.mysite.com/url1.html http:/www.mysite.com/url2.html ...
2
votes
1answer
175 views

How do I disable my internet using curl or wget to hit my router in a script?

I want to be able to run some kind of shell script/curl commands to toggle my TV services by accessing a form on my router page using curl or wget. I have been able to get past authentication using ...
0
votes
1answer
126 views

Get Directory/file structure without payload

I need to recursively retreive the file/directory structure of an FTP server, getting only file names (without payload). Can you help me how to do this (curl, wget, ...)? The result should be the ...
2
votes
3answers
57 views

mirror full website1 and all links to specific website2

I want to mirror a website that will be taken down soon, the problem im facing at the moment is simple -> mirror whole website1 and all links (will be files/images/similiar) to website2 too, so i got ...
0
votes
2answers
395 views

cURL or xargs: show progress

I am using cURL to download many files and concatenate them to STDOUT. About 100,000 small files. I would like to see progress against the 100,000. Is this possible with curl or by using curl into ...
0
votes
1answer
223 views

wget timestamp alternatives

I have a script that works in a loop and checks for mysqlbackup copies on another server. couple of times a day wget downloads the file even though no change has been made. what I've tried is using ...
1
vote
2answers
48 views

wget : how to get a directory to a specify directory

The command I am using is: wget -r -nH ftp://some.domain.com/home/work/test/* -P ./my The output directory becomes: ./my/home/work/test/* How can I make it to just ./my/*?
2
votes
2answers
423 views

Variable parsing with Bash and wget

I'm attempting to use wget in a simple bash script to grab a jpeg image from an Axis camera. This script outputs a file named JPEGOUT, instead of the desired output, which should be a timestamp jpeg ...
1
vote
3answers
1k views

How can I continue a loop in bash after an application returned an error?

I save images with: #!/bin/bash for i in {1..30000} do wget "http://services.runescape.com/m=itemdb_rs/3809_obj_sprite.gif?id="$i done Is there a way to speed up the process? Sometimes a URL ...
3
votes
5answers
1k views

Why does wget'ing an image give me a file, not an image?

I do wget http://services.runescape.com/m=itemdb_rs/3809_obj_sprite.gif?id=2 and it downloads the following file: 3809_obj_sprite.gif?id=2 How can I make it save as the following? 2.png
3
votes
2answers
1k views

Wget - if / else download condition?

I want wget to prefer a certain filetype over another, if the files have the same basename. For example: if foo.ogg available, don't download foo.mp3 the way i use wget so far to ...
3
votes
3answers
230 views

How to pass a file that changes name to another command in Bash?

I frequently use wget to download tarballs and zip files from the web, then either untar then or gunzip them. I do: wget google.com/somefile.zip unzip somefile.zip rm somefile.zip Is there a way ...
0
votes
2answers
345 views

remote backup with curl/wget

I need to backup a clients server (source) from our mediatemple server (destination) and then add it into a git repository while ssh'd in. What is the most efficient/fastest way to do so? wget? or ...
7
votes
5answers
23k views

Using Wget to Recursively Crawl a Site and Download Images

How do you instruct wget to recursively crawl a website and only download certain types of images? I tried using this to crawl a site and only download Jpeg images: wget --no-parent --wait=10 ...
0
votes
2answers
651 views

wget error: bad header line

Why doesn't this work?: wget -q -U Mozilla "http://rss.slashdot.org/Slashdot/slashdot" -O - | tr '\n' ' ' p.s.: and under openwrt 10.03: # wget "http://slashdot.org/" -O - Connecting to slashdot.org ...
4
votes
3answers
5k views

How to download this webpage with wget?

I want to download this webpage: http://forum.ubuntu-it.org/ but it require username and password. So I have used this wget --save-cookies cookies.txt --post-data ...
3
votes
1answer
697 views

Logging into webpage via script

I'm trying to automate the extraction of some information from a website that first requires me to log in. I have done this in the past (years ago) using wget, but that method no longer seems to work ...
7
votes
2answers
3k views

Is it possible to set wget to show the transfer times in milliseconds?

Generally, wget shows transfer times in seconds. Is there a way I can to make it show the time in milliseconds?
3
votes
8answers
5k views

How to recursively download an entire web directory?

i have a web directory that has many folders and many sub folders containing files. i need to download everything using wget or bash.
1
vote
1answer
540 views

What is wrong in the Bash script about parameters and Wget?

Runnning the command gives wget_exam -h Usage: wget_exam2 <fileType> <source> exit // then immediately Terminal shut dows The code # example wget_exam2 java ...
2
votes
1answer
896 views

Scripts that extend wget --page-requisites?

I have posted a very similar question today but I would like to ask again with a wget/linux focus. I hope that's all right. I need to create offline copies of web pages programmatically on a LAMP ...