0

I need a program to get all the web pages under a website. The website is Chinese, I want to get all those English words out. Then I can extract all the information I need. Any ideas for this? Is there any software for this purpose?

If NO, I would like to write one. Any suggestions?

Thanks much.

2

4 Answers 4

10

Use eg wget -r http://site.to.copy.com to recursively retrieve all the web pages to your local machine (hope it's not too big...), then you can search or do whatever with the files afterwards.

3
  • What I was going to suggest. Why bother building yet another mousetrap? Nov 4, 2009 at 21:55
  • 2
    You may want to consider using "--convert-links" flag also so that you can browse locally...
    – AJ.
    Nov 4, 2009 at 22:09
  • Depending on how many pages you're intending to download, you might also make sure to specify the --limit-rate option to avoid overloading the server.
    – David
    Nov 6, 2009 at 2:43
3

wget (manpage here) can also serve well as a crawler, look at its --recursive option.

3

Your pretty much describing a web crawler (something that takes a page, looks for all the links, follows them etc). There are crawler implementations already out there, tool that act like crawlers (such as wget), and questions relating to them here on Stack Overflow. For example...

https://stackoverflow.com/questions/102631/how-to-write-a-crawler

Once you have something that can visit each page you then need code that'll parse the page and look for the text you're interested in.

1

Not a PHP solution, but you can use the Lynx text-only Web browser with the -crawl and -dump options to visit all the pages on a site and dump them as text files. You can then use a script to extract the information you want from them.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .