1

i used wget on a website and all files were downloaded without errors. the pages that has this type of link:

  • website.com/english/apple
  • website.com/english/dog
  • website.com/english/baby

were downloaded and working properly. however other files with link:

  • website.com/japanese/おと
  • website.com/japanese/ょうき

were downloaded and they all have the default 404 response page from the site. (it doesn't literally say 404, it just says something like "you typed the link wrong")

i tried using the command "--restrict-file-names=ascii" but the same thing happened.

note:after using the command the downloaded files had bad utf-8 codes. (one of them is %C3%A5%C2%A4%E2%80%A2 when decoding it, it will give gibberish)

i did some searching and someone mentioned that: "

"wget url-encodes bytes 128-159 (which are control characters in some 8-bit encodings). This is wrong"

is that related to my problem?

  • 1
    Are you downloading the files by specifying their direct URLs on the command line, or by letting wget follow links in recursive mode? – user1686 Oct 3 '19 at 18:30
  • no, wget is downloading links automatically. i just typed the name of website – problem404 Oct 3 '19 at 20:48
0

i found a solution on a random Japanese blog. found it in the second page on google!!

just add --local-encoding=UTF-8

the full command that i used and worked for me is:

wget -m -np -c -R "index.html*" --restrict-file-names=ascii --local-encoding=UTF-8 "https://www.website.com/"
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.