Questions tagged [screen-scraping]

Screen scraping, also known as web scraping or data scraping, is a software technique used to collect and parse information from user interfaces. Questions about using programming languages to do screen scraping are off topic and should be asked at Stack Overflow Q&A.

Filter by
Sorted by
Tagged with
0 votes
0 answers
29 views

how to test bot detection [closed]

I want to test cloudflare bot detection. in the site I'm working with after solving it once it never showed again. but how can i force it to show again notes: I'm using puppeteer for Web scraping. I ...
Fire Gamer's user avatar
0 votes
0 answers
38 views

How to scrape website protected area using Win HTTtrack

I need to scrape information from a website that will need to be logged in to see all the data I need. I was trying using puppeteer but I've noticed that it will be not able to login so I've done a ...
ICTDEV's user avatar
  • 101
0 votes
0 answers
38 views

How can I extract two or three "parallel" XPaths on a web page to/in a format that can be imported into a spreadsheet?

How can I extract two or three "parallel" XPaths on a web page to/in a format that can be imported into a spreadsheet? Here is an example: on https://www.amazon.co.uk/s?rh=n%3A20606777031&...
d-b's user avatar
  • 814
0 votes
1 answer
52 views

Accessing items in a web directory which is not directly accessible

E.g., I want to download an image of each of the US state trees. I find Alabama's longleaf pine at: https://statesymbolsusa.org/sites/statesymbolsusa.org/files/primary-images/longleafpine.jpg More ...
lowndrul's user avatar
  • 365
0 votes
1 answer
83 views

Excel web queries - how do I add the data I get to a running list rather than just updating the latest number?

I have a web query that pulls data from a website. Each day the website publishes an updated value for an index. And I can pull that value into my spreadsheet. But I'd like to make a running list of ...
user348514's user avatar
0 votes
1 answer
286 views

VBA - IBAN validation not working

I have a list of IBANs in Excel and wish to validate them via scraping data through VBA from this website: https://www.iban.com/iban-checker. In column A (cells A2:A3000), I have the IBANs and in ...
Goro Shikeda's user avatar
0 votes
0 answers
185 views

Why won't web page won't give data to Excel

I would like to get this nice table-like data from a website (https://app.zerion.io/0x6ef9dca82362509cd878051d1fdc6db12dda2989/overview). (In particular, the table with xpath is /html/body/div[1]/div[...
J. Win.'s user avatar
  • 145
0 votes
1 answer
232 views

Add domain url to wget result page

i'm trying to make a list of websites into static websites, and i'm using this wget command wget --quiet -E -H -k -nd -K -p -e robots=off #{url} the result is mostly what i want, but all the urls on ...
Eduardo Veras's user avatar
1 vote
1 answer
197 views

Web crawling/scraping a list of all files (which are publicly available) instead of going via 5 clocks from home page for each file

Summary of the problem I am trying to create a list (and potentially download) of all files (100s) from the following address: https://eba.europa.eu/regulation-and-policy This is something I would use ...
Wadzer's user avatar
  • 11
0 votes
0 answers
1k views

Downloading videos from multiple web pages

I'm trying to download videos from a list of web pages. There is 1 video per page. The page is rendered using JavaScript, and is totally blank with JS disabled. I've tried youtube-dl but I believe ...
Duncan Felix's user avatar
2 votes
0 answers
274 views

How to paste HTML headings into Excel

In a previous answer (vba - html table to excel worksheet) about parsing/pasting HTML table contents into an Excel sheet, wbeard2 shared this very helpful, illustrative piece of code. He/she notes ...
Reservist72's user avatar
0 votes
1 answer
203 views

Can a website distinguish between a web scraper and a legitimate user on a basis other than the speed of traversal? [closed]

If I reduce the speed of a web scraper to match the speed of a human, will a third party be able to distinguish between them? What other criteria do websites use to distinguish between users and ...
user239457's user avatar
3 votes
4 answers
9k views

I am seeing animated PNG files on some sites in lieu of GIFs. How can I save/download them?

I know how to save a GIF, it's super easy. Just right click and save. Voila! But with these new animated PNG's, I haven't the foggiest. Without using some sort of screen scrape where I'm grabbing the ...
Braden Dodge's user avatar
0 votes
0 answers
728 views

How to scrape email from 'websites using cloudflare' ? which gives [email protected]

i am trying to scrape website and all of its content. The website i am trying to scrape uses cloudflare, so if there are email addresses in the website, it gets obfuscated and it shows [email ...
user avatar
0 votes
0 answers
389 views

Webpage to paginated, text PDF

I have been looking for a simple way to convert a website to a paginated PDF with selectable text. I've tried many browser extensions but they only allow for PNG, full-page screenshots and print with ...
kirkas's user avatar
  • 101
4 votes
2 answers
5k views

How to automatically copy text from different websites

I want to know how to automatically copy text from different websites. I am building a database of companies which belong to certain associations. The website has a list of companies with the ...
BDstat's user avatar
  • 41
-2 votes
3 answers
2k views

How to implement anti-scraping mechanisms for my Amazon S3 based site?

I have a few static web pages hosted on amazon S3 that are very frequently updated. I want to implement few anti-scraping mechanisms like banning IP's which are making too many requests or making ...
Avinash's user avatar
  • 51
0 votes
0 answers
35 views

How to tell if multiple sites contain a keyword

I have a list of 9k web addresses, and a couple of keywords. I would like to know if a webpage contains that word. Ideally, automated google search with some search operators would work great, for ...
jaded's user avatar
  • 1
3 votes
1 answer
875 views

What is the name for google's information summary box (pic inside)? Is there an api to access it from a google search?

Example of what I'd like to access: Just wondering if I could write something that'll query google with a search string and if the summary box returns, fetch only that, otherwise fetch the first few ...
Daz C's user avatar
  • 201
0 votes
1 answer
416 views

Web Scraping Macro error

I am trying to scrap multiple pages that are a search result of a key work. I wrote this code but I am getting an error when I run it. Error: Application \-defined or object-defined error. When I ...
Haytham's user avatar
  • 111
0 votes
1 answer
2k views

Httrack filter links with certain pattern

I am trying to use httrack to download an entire webarchive from archive.org. The idea is to download only the archive links (as many as possible), but only the links that are really from the archive ...
user1812076's user avatar
1 vote
0 answers
988 views

Recommendation on Web scraping and data flow

I have a solar panel unit, and the company that set it up (Fronius) has a website where I can live stream the data collected from the solar panel (Current power, Energy today, monthly, and yearly data)...
trueCamelType's user avatar
0 votes
1 answer
263 views

Automatically go through pages download filtered links on a site

With DownThemAll! I can download all the links on a given page and I can filter them. lets say the site is www.google.com and I want to grab all the search results. I set up a filter: /search?q= ...
Kwestion's user avatar
  • 126
0 votes
1 answer
915 views

Web scraping / crawling a particular website

I'm trying to scrape 1265 html files at once to get name and descriptions of items I have on a website. I have permission from the wholesaler to copy there data but I don't want to spend days just ...
Jamie's user avatar
  • 1
0 votes
0 answers
62 views

How to backup a bulletin board hosted by a 3rd party?

A friend has asked me to back up this forum: http://oldforums.comicbookresources.com/ It's powered by vBulletin. Something bad had happened there and the hosts no longer want to host, and are ...
Ruby's user avatar
  • 287
3 votes
0 answers
274 views

wget put all prerequisites in flat subdirectory, but not root page?

I'm trying to get wget to save a page + prerequisites in a format resembling that of a web browser: article.html article_files/img.jpg article_files/script.js I am able to get almost this behavior, ...
Sridhar Sarnobat's user avatar
2 votes
2 answers
4k views

Web scraping / crawling a particular Google book

For my work, I need to scrape the text from a large book on Google Books. The book in question is a very old book and is out of copyright. The book is a Gazetteer of the World. We will be putting the ...
user2661243's user avatar
4 votes
4 answers
229 views

Save report from Windows checking removable disk?

Is there any way to save a report of the errors that Windows found and fixed in a USB key? Windows presented me this dialog, btw that is not resizable: . Some problems were found and fixed. When I ...
JohnC's user avatar
  • 669
1 vote
1 answer
3k views

How do I use AutoHotKey to read the text at the mouse location?

I can read the entire window text using WinGetText() but I am trying to get the text at the current mouse location. I've found several examples on the AutoHotKey forums but they are all very old (...
shaneknysh's user avatar
1 vote
1 answer
759 views

Is it legal to screen scrape your own bank statements in the US? [closed]

I want to automatically download my bank statements. My bank charges monthly for OFX access so have considered gathering the data points by other means (scripting, screen scraping), but I want to know ...
Matt's user avatar
  • 121
1 vote
0 answers
42 views

igoogle style page with content scraped from other pages

I'm looking for an online homepage or Chrome extension that will display an iGoogle type home page but that lets me select content from other pages. In the same way AdBlock will allow me to select a ...
ChrisB's user avatar
  • 1,596
0 votes
1 answer
194 views

IRobot Scraping tool

I am using IRobot scraping tool to obtain some data from website but for some reason it is not comming I think the problem must be of proxy settings can anyone help me to how to change the proxy ...
user avatar
4 votes
1 answer
229 views

Extract data from an online atlas

There is an online atlas that I would like to extract values from. The atlas provides a tool ('Query') to extract values when you click a location or enclose a region on the map, or you can specify ...
KAE's user avatar
  • 1,859
0 votes
0 answers
115 views

Programs for reading data off interactive web charts

This question is in the same vein as this one: Program for reading data off scanned graphs Recently more and more interactive charts show up on the Net. Have a look at the following one: http://www....
vonjd's user avatar
  • 115
9 votes
2 answers
7k views

Get Current HTML Of Page Built With AJAX Requests

So, I'm using the Chrome/Chromium browser (put could use Firefox, if need be). I'm viewing webpages which are constructed "on the fly" with (presumably) AJAX (think the how you scroll down on ...
Richard's user avatar
  • 3,441
1 vote
1 answer
2k views

Alternative, more efficient scraping method for a noncoder, than Google doc's importxml and xpath?

I've searched throughout the net for a simple solution, but it seems everyone has their own unique method (coding language) of achieving this. I'm only just beginning to learn Linux, and my coding ...
binarybunny's user avatar
1 vote
1 answer
3k views

Web scraper for Outlook Web App 2010?

I've been using weboutlook for years with great success, but recently we switched from the 2007 to the 2010 version of Outlook Web App (OWA). Does anyone know of a scraper that works with the newer ...
ladenedge's user avatar
  • 516
1 vote
4 answers
7k views

Is there a Firefox or Chrome plugin to automatically save images from webpage

A website has a slideshow of images, for example:This link Since the slideshow is automatic, is there anyway to save those images automatically into a folder? either by a browser plugin or by some ...
McBear Holden's user avatar
12 votes
4 answers
7k views

How "legal" is site-scraping using cURL? [closed]

Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
Chetan Sharma's user avatar
1 vote
2 answers
219 views

Data Mining Software [closed]

I want to harvest some data like this http://www.newcardealers.ca/en/Dealers/List-A.aspx And insert the name, address, phone number, email, etc. into a database. Is there some software I can use that ...
mpen's user avatar
  • 12.1k
29 votes
7 answers
20k views

How do I copy text from a dialog box? [closed]

In Windows sometimes I get an error dialog with long text. Manually typing out the exact details of an error message can be an annoying and lengthy process. Is there way to copy text from a Windows ...
Tony_Henrich's user avatar
  • 11.7k
1 vote
3 answers
1k views

Save parts of a website as pure text

I hope I may ask this here. I need to extract the contents of an existing website (in charge of the website owner) to Word (or text) documents. For this, I only need the content from one DIV with a ...
Martin's user avatar
  • 3,932