Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need a program to get all the web pages under a website. The website is Chinese, I want to get all those English words out. Then I can extract all the information I need. Any ideas for this? Is there any software for this purpose?

If NO, I would like to write one. Any suggestions?

Thanks much.

share|improve this question

migrated from Oct 4 '12 at 1:18

This question came from our site for professional and enthusiast programmers.

4 – jeffamaphone Nov 4 '09 at 21:50 if no wget available – DFectuoso Nov 4 '09 at 21:51

Use eg wget -r to recursively retrieve all the web pages to your local machine (hope it's not too big...), then you can search or do whatever with the files afterwards.

share|improve this answer
What I was going to suggest. Why bother building yet another mousetrap? – Carl Smotricz Nov 4 '09 at 21:55
You may want to consider using "--convert-links" flag also so that you can browse locally... – AJ. Nov 4 '09 at 22:09
Depending on how many pages you're intending to download, you might also make sure to specify the --limit-rate option to avoid overloading the server. – David Nov 6 '09 at 2:43

wget (manpage here) can also serve well as a crawler, look at its --recursive option.

share|improve this answer

Your pretty much describing a web crawler (something that takes a page, looks for all the links, follows them etc). There are crawler implementations already out there, tool that act like crawlers (such as wget), and questions relating to them here on Stack Overflow. For example...

Once you have something that can visit each page you then need code that'll parse the page and look for the text you're interested in.

share|improve this answer

Not a PHP solution, but you can use the Lynx text-only Web browser with the -crawl and -dump options to visit all the pages on a site and dump them as text files. You can then use a script to extract the information you want from them.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .