I'm looking for a robust web spider/web crawler to crawl webpages in the background.
It needs to run in the background and spider pages continuously.
It needs to be fast and memory efficient.
can anyone send me good recommendations please ?
It can be coded in any programming language
Heard about a python script which is quite efficient but not sure where to find it though ?
