Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm downloading a site with wget and a lot of the links have queries attached to them, so when I do this:

wget -nv -c -r -H -A mp3 -nd http://url.to.old.podcasts.com/

I end up with a lot of files like this:

1.mp3?foo=bar
2.mp3?blatz=pow
3.mp3?fizz=buzz

What I'd like to end up with is:

1.mp3
2.mp3
3.mp3

This is all taking place in ubuntu linux and I've got wget 1.10.2.

I know I can do this after I get everything via a script to rename everything. However I'd really like a solution from within wget so I can see the correct names as the download is happening.

Can anyone help me unravel this?

share|improve this question
    
Post your question at www.stackoverflow.com. –  Deniz Zoeteman Oct 26 '09 at 19:42
2  
@TutorialPoint why? question is looking for a within-wget-way-to-do-it, SO would just migrate it back here. –  quack quixote Oct 26 '09 at 19:57
    
Well, there is no within-wget-way-to-do-it –  ayrnieu Oct 26 '09 at 20:32
1  
@ayrnieu: not in one command, no. and not without a helper. but you can certainly do it with as few as n+1 wget commands (if not fewer). –  quack quixote Oct 26 '09 at 20:36
add comment

3 Answers 3

If the server is kind, it might be sticking a Content-Disposition header on the download advising your client of the correct filename. Telling wget to listen to that header for the final filename is as simple as:

wget --content-disposition

You'll need a newish version of wget to use this feature.

I have no idea how well it handles a server claiming a filename of '/etc/passwd'.

share|improve this answer
add comment

I think, in order to get wget to save as a filename different than the URL specifies, you need to use the -O filename argument. That only does what you want when you give it a single URL -- with multiple URLs, all downloaded content ends up in filename.

But that's really the answer. Instead of trying to do it all in one wget command, use multiple commands. Now your workflow becomes:

  1. Run wget to get the base HTML file(s) containing your links;
  2. Parse for URLs;
  3. Foreach URL ending in mp3,
    1. process URL to get a filename (eg turn http://foo/bar/baz.mp3?gargle=blaster into baz.mp3
    2. (optional) check that filename doesn't exist
    3. run wget <URL> -O <filename>

That solves your problem, but now you need to figure out how to grab the base files to find your mp3 URLs.

Do you have a particular site/base URL in mind? Steps 1 and 3 will be easier to handle with a concrete example.

share|improve this answer
add comment

so I can see the correct names as the download is happening.

OK. Use wget as you normally do; use the post-wget script that you normally use, but process wget's output so that it's easier on the eyes:

#! /bin/sh
exec wget --progress=bar:force $* 2>&1 | \
  perl -pe 'BEGIN { $| = 1 } s,(?<=`)([^\x27?]+),\e[36;1m$1\e[0m, if /^Saving/'
cgi-cut # rename files

This will still show the ?foo=bar as you download, but will display the rest of the name in bright cyan.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.