up vote 1 down vote favorite
share [g+] share [fb]

I'm trying to crawl a directory on a website and basically download everything in it. The structure is simple enough(but there are also multiple folders), but there is one thing that makes wget choke up. The source looks something like this:

<a href="index.php?path=/blah/whatever&download=resource.txt"><img... /></a> .... <a href="/blah/whatever/resource.txt">resource.txt"</a>

Both of the links work, but they are both the same thing. So wget will download the same file twice. How can I make wget ignore the first one? Or if you have a better crawler suggestion that works in Linux(or OpenBSD) then suggest it instead

I've tried using the reject list as so:

wget -r -R index.php\*\&download\* http://url

but this doesn't seem to actually do anything. It will still download the duplicate URLs

link|improve this question
64% accept rate
feedback

1 Answer

up vote 0 down vote

can you do something like... just -R index.php* ?as that sounds like the other links will still work fine?

link|improve this answer
Actually I figured out my problem is a caveat to wget. My pattern was correct(except for a missing \* at the beginning) but wget will download even files in it's reject list. It will just delete them right after downloading them for whatever reason. – Earlz Mar 19 '11 at 6:23
oh, haha... gotcha. Inefficient but good enough I guess – CreeDorofl Mar 19 '11 at 6:58
From the wget manual: Note, too, that query strings (strings at the end of a URL beginning with a question mark (‘?’) are not included as part of the filename for accept/reject rules, even though these will actually contribute to the name chosen for the local file. It is expected that a future version of Wget will provide an option to allow matching against query strings. – p.vitzliputzli Mar 19 '11 at 7:54
Well, thankfully even for the horrible feature of downloading every file twice, wget is handling the download correctly in the final file structure – Earlz Mar 19 '11 at 15:52
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.