0

I have a list of URIs in a file uris.txt:

http://example.com/file1.pdf
http://example.com/file2.pdf
http://example.com/Folder/file3.pdf
http://example.com/Folder/Subfolder/file4.pdf

The resulting directory structure on my HDD is supposed to be as follows:

|
|-file1.pdf
|-file2.pdf
|-Folder
  |-file3.pdf
  |-Subfolder
    |-file4.pdf

My current command is: wget -i uris.txt -P downloads

Unfortunately, wget flattens all directories, i.e. the filename is determined by concatenating the base directory (given by -P on the command line) and the last part of the URI's path.

Maintaining the directory structure a server serves is possible, albeit only with the recursive mode, which only operates on HTML and CSS as opposed to a given list of URIs.

  • You could use some regex-fu to convert the file list to more specific wget command lines then execute the file. – Neil Smithline Sep 27 '15 at 15:00
  • @NeilSmithline I actually generate the list by means of a Node.js script I wrote. I hoped wget has some built-in, ready-to-use functionality including dealing with special characters not allowed/preferred in directory names. – ComFreek Sep 27 '15 at 15:08
1

You can get this with the extra options -x or --force-directories and -nH or --no-host-directories to avoid the example.com top directory.

0

Suggested reading that will provide the answer in a future-proof way:

man wget

Look for --no-host-directories and --cut-dirs=number

--no-host-directories
will make wget skip creating a dir named "example.com" to put all data in.

--cut-dirs=number
Will do the same, for number subdirs, counting from the top.

  • Correct me if I am wrong, but don't these options do the opposite, i.e. reducing the amount of directories? As a matter of fact, calling wget with the parameters --no-host-directories --cut-dirs=10 (10 being an arbitrarily chosen value) results in no difference compared to the execution without them. – ComFreek Sep 27 '15 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.