I need a program to get all the web pages under a website. The website is Chinese, I want to get all those English words out. Then I can extract all the information I need. Any ideas for this? Is there any software for this purpose?

If NO, I would like to write one. Any suggestions?

Thanks much.


Use eg wget -r http://site.to.copy.com to recursively retrieve all the web pages to your local machine (hope it's not too big...), then you can search or do whatever with the files afterwards.

| improve this answer | |
  • What I was going to suggest. Why bother building yet another mousetrap? – Carl Smotricz Nov 4 '09 at 21:55
  • 2
    You may want to consider using "--convert-links" flag also so that you can browse locally... – AJ. Nov 4 '09 at 22:09
  • Depending on how many pages you're intending to download, you might also make sure to specify the --limit-rate option to avoid overloading the server. – David Nov 6 '09 at 2:43

wget (manpage here) can also serve well as a crawler, look at its --recursive option.

| improve this answer | |

Your pretty much describing a web crawler (something that takes a page, looks for all the links, follows them etc). There are crawler implementations already out there, tool that act like crawlers (such as wget), and questions relating to them here on Stack Overflow. For example...


Once you have something that can visit each page you then need code that'll parse the page and look for the text you're interested in.

| improve this answer | |

Not a PHP solution, but you can use the Lynx text-only Web browser with the -crawl and -dump options to visit all the pages on a site and dump them as text files. You can then use a script to extract the information you want from them.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.