1

Since IMDb is planning to remove all Message Boards in 2 weeks, I'd like to fetch some pages, however wget fails to do so.

This is what I did:

  1. Logged as user at http://www.imdb.com/boards/ page.
  2. Exported cookies into cookies.txt file using cookies.txt Chrome extension.
  3. Run wget as below (as suggested in cookies.txt file):

    $ wget -x --load-cookies cookies.txt --keep-session-cookies -e robots=off -A Mozilla http://www.imdb.com/boards/
    --2017-02-09 15:19:45--  http://www.imdb.com/boards/
    Resolving www.imdb.com... 54.239.23.73
    Connecting to www.imdb.com|54.239.23.73|:80... connected.
    HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
    2017-02-09 15:19:46 ERROR 503: Service Temporarily Unavailable.
    

    This should even work without cookies.txt, since the page is accessible for public.

What I am missing?

  • I got the same error... I found a comment on another site saying "they don't like the wget useragent" – Manachi Feb 19 '17 at 11:42
1

It seems there's a few extra settings you need to add including the user agent. I added the following to my ~/.wgetrc file and it seems to do the job. Although occasionally I'm getting a 500 or 503 error. I wonder if that might be a throttling/security mechanism.

header = Accept-Language: en-us,en;q=0.5
header = Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
header = Connection: keep-alive
user_agent = Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:53.0) Gecko/20100101 Firefox/53.0
referer = /
robots = off

Got the basis for this answer from Stack Overflow.

1

Not a direct answer to your question, but there is a common initiative to backup IMDB by the ArchiveTeam: everyone interested can help downloading to a common archive which eventually ends up in archive.org (see the archiveteam collection).

Instructions are provided. Basically, you can run a VirtualBox/VMWare appliance called ArchiveTeam Warrior, but I decided to go with the Linux scripts.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.