Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Does anyone know of any (free) command-line web scraper?

I'm looking for something like Beautiful Soup for Python, but for the command line in Windows XP.


C:> scraper www.siteexample.com\abcde.html -find("div", {"id": "articlebody"})

Does anyone know of any program like that (or not like that, but that does the job)?

share|improve this question

closed as off-topic by Kevin Panko, Tog, Heptite, random Mar 8 at 13:15

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. Here are a few suggestions on how to properly ask this type of question." – Kevin Panko, Tog, Heptite, random
If this question can be reworded to fit the rules in the help center, please edit the question.

How about ... write a Python script using BeautifulSoup? –  grawity Mar 2 '11 at 10:55
I'm having trouble working with python and BeautifulSoup in Windows (linux man here ;) ). This was for me to get some work done on M$ Windows machines. –  Jhonny Goodlife Mar 2 '11 at 11:04
If you have programming experience, you can rig up something really quickly with C# or with AutoIt under Windows. –  Breakthrough Mar 2 '11 at 11:53
What is M$? Some old joke? :-/ –  Tom Wijsman Mar 2 '11 at 11:55
add comment

3 Answers

You can install Python on Windows. Alternatively, use Portable Python if "command-line" means "I don't want to install anything."

share|improve this answer
add comment

Try Xidel


Xidel is a command line tool to download Web pages and extract data from them. It can download files over HTTP/S connections, follow redirections, links, or extracted values, and process local files. The data can be extracted using XPath 2.0, XQuery 1.0, and JSONiq expressions, CSS 3 selectors, and custom, pattern-matching templates that are like an annotated version of the processed page. The extracted values can then be exported as plain text/XML/HTML/JSON, or assigned to variables to be used in other extract expressions or be exported to the shell. There is also an online CGI service for testing.

share|improve this answer
add comment

HTTrack is the classic web scraper, and it has a windows port.

share|improve this answer
HTTrack is nice software, but it's not helping with this problem (see example; finding elements from pages). Of course, it's possible to use HTTrack and then do some custom software, but alone it's not a solution. –  Olli Mar 2 '11 at 13:01
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.