Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need a program to get all the web pages under a website. The website is Chinese, I want to get all those English words out. Then I can extract all the information I need. Any ideas for this? Is there any software for this purpose?

If NO, I would like to write one. Any suggestions?

Thanks much.

share|improve this question

migrated from stackoverflow.com Oct 4 '12 at 1:18

This question came from our site for professional and enthusiast programmers.

4  
gnu.org/software/wget –  jeffamaphone Nov 4 '09 at 21:50
    
spidersoft.com if no wget available –  DFectuoso Nov 4 '09 at 21:51

4 Answers 4

Use eg wget -r http://site.to.copy.com to recursively retrieve all the web pages to your local machine (hope it's not too big...), then you can search or do whatever with the files afterwards.

share|improve this answer
    
What I was going to suggest. Why bother building yet another mousetrap? –  Carl Smotricz Nov 4 '09 at 21:55
2  
You may want to consider using "--convert-links" flag also so that you can browse locally... –  AJ. Nov 4 '09 at 22:09
    
Depending on how many pages you're intending to download, you might also make sure to specify the --limit-rate option to avoid overloading the server. –  David Nov 6 '09 at 2:43

wget (manpage here) can also serve well as a crawler, look at its --recursive option.

share|improve this answer

Your pretty much describing a web crawler (something that takes a page, looks for all the links, follows them etc). There are crawler implementations already out there, tool that act like crawlers (such as wget), and questions relating to them here on Stack Overflow. For example...

http://stackoverflow.com/questions/102631/how-to-write-a-crawler

Once you have something that can visit each page you then need code that'll parse the page and look for the text you're interested in.

share|improve this answer

Not a PHP solution, but you can use the Lynx text-only Web browser with the -crawl and -dump options to visit all the pages on a site and dump them as text files. You can then use a script to extract the information you want from them.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.