Questions tagged [screen-scraping]
Screen scraping, also known as web scraping or data scraping, is a software technique used to collect and parse information from user interfaces. Questions about using programming languages to do screen scraping are off topic and should be asked at Stack Overflow Q&A.
42
questions
0
votes
0
answers
29
views
how to test bot detection [closed]
I want to test cloudflare bot detection.
in the site I'm working with after solving it once it never showed again.
but how can i force it to show again
notes:
I'm using puppeteer for Web scraping. I ...
0
votes
0
answers
38
views
How to scrape website protected area using Win HTTtrack
I need to scrape information from a website that will need to be logged in to see all the data I need. I was trying using puppeteer but I've noticed that it will be not able to login so I've done a ...
0
votes
0
answers
38
views
How can I extract two or three "parallel" XPaths on a web page to/in a format that can be imported into a spreadsheet?
How can I extract two or three "parallel" XPaths on a web page to/in a format that can be imported into a spreadsheet?
Here is an example: on https://www.amazon.co.uk/s?rh=n%3A20606777031&...
0
votes
1
answer
52
views
Accessing items in a web directory which is not directly accessible
E.g., I want to download an image of each of the US state trees. I find Alabama's longleaf pine at:
https://statesymbolsusa.org/sites/statesymbolsusa.org/files/primary-images/longleafpine.jpg
More ...
0
votes
1
answer
83
views
Excel web queries - how do I add the data I get to a running list rather than just updating the latest number?
I have a web query that pulls data from a website. Each day the website publishes an updated value for an index. And I can pull that value into my spreadsheet. But I'd like to make a running list of ...
0
votes
1
answer
286
views
VBA - IBAN validation not working
I have a list of IBANs in Excel and wish to validate them via scraping data through VBA from this website: https://www.iban.com/iban-checker. In column A (cells A2:A3000), I have the IBANs and in ...
0
votes
0
answers
185
views
Why won't web page won't give data to Excel
I would like to get this nice table-like data from a website (https://app.zerion.io/0x6ef9dca82362509cd878051d1fdc6db12dda2989/overview). (In particular, the table with xpath is /html/body/div[1]/div[...
0
votes
1
answer
232
views
Add domain url to wget result page
i'm trying to make a list of websites into static websites, and i'm using this wget command
wget --quiet -E -H -k -nd -K -p -e robots=off #{url}
the result is mostly what i want, but all the urls on ...
1
vote
1
answer
197
views
Web crawling/scraping a list of all files (which are publicly available) instead of going via 5 clocks from home page for each file
Summary of the problem
I am trying to create a list (and potentially download) of all files (100s) from the following address: https://eba.europa.eu/regulation-and-policy
This is something I would use ...
0
votes
0
answers
1k
views
Downloading videos from multiple web pages
I'm trying to download videos from a list of web pages.
There is 1 video per page. The page is rendered using JavaScript, and is totally blank with JS disabled.
I've tried youtube-dl but I believe ...
2
votes
0
answers
274
views
How to paste HTML headings into Excel
In a previous answer (vba - html table to excel worksheet) about parsing/pasting HTML table contents into an Excel sheet, wbeard2 shared this very helpful, illustrative piece of code. He/she notes ...
0
votes
1
answer
203
views
Can a website distinguish between a web scraper and a legitimate user on a basis other than the speed of traversal? [closed]
If I reduce the speed of a web scraper to match the speed of a human, will a third party be able to distinguish between them? What other criteria do websites use to distinguish between users and ...
3
votes
4
answers
9k
views
I am seeing animated PNG files on some sites in lieu of GIFs. How can I save/download them?
I know how to save a GIF, it's super easy. Just right click and save. Voila! But with these new animated PNG's, I haven't the foggiest. Without using some sort of screen scrape where I'm grabbing the ...
0
votes
0
answers
728
views
How to scrape email from 'websites using cloudflare' ? which gives [email protected]
i am trying to scrape website and all of its content.
The website i am trying to scrape uses cloudflare, so if there are email addresses in the website, it gets obfuscated and it shows [email ...
0
votes
0
answers
389
views
Webpage to paginated, text PDF
I have been looking for a simple way to convert a website to a paginated PDF with selectable text.
I've tried many browser extensions but they only allow for PNG, full-page screenshots and print with ...
4
votes
2
answers
5k
views
How to automatically copy text from different websites
I want to know how to automatically copy text from different websites. I am building a database of companies which belong to certain associations. The website has a list of companies with the ...
-2
votes
3
answers
2k
views
How to implement anti-scraping mechanisms for my Amazon S3 based site?
I have a few static web pages hosted on amazon S3 that are very frequently updated. I want to implement few anti-scraping mechanisms like banning IP's which are making too many requests or making ...
0
votes
0
answers
35
views
How to tell if multiple sites contain a keyword
I have a list of 9k web addresses, and a couple of keywords.
I would like to know if a webpage contains that word. Ideally, automated google search with some search operators would work great, for ...
3
votes
1
answer
875
views
What is the name for google's information summary box (pic inside)? Is there an api to access it from a google search?
Example of what I'd like to access:
Just wondering if I could write something that'll query google with a search string and if the summary box returns, fetch only that, otherwise fetch the first few ...
0
votes
1
answer
416
views
Web Scraping Macro error
I am trying to scrap multiple pages that are a search result of a key work.
I wrote this code but I am getting an error when I run it. Error:
Application \-defined or object-defined error.
When I ...
0
votes
1
answer
2k
views
Httrack filter links with certain pattern
I am trying to use httrack to download an entire webarchive from archive.org. The idea is to download only the archive links (as many as possible), but only the links that are really from the archive ...
1
vote
0
answers
988
views
Recommendation on Web scraping and data flow
I have a solar panel unit, and the company that set it up (Fronius) has a website where I can live stream the data collected from the solar panel (Current power, Energy today, monthly, and yearly data)...
0
votes
1
answer
263
views
Automatically go through pages download filtered links on a site
With DownThemAll! I can download all the links on a given page and I can filter them.
lets say the site is www.google.com
and I want to grab all the search results. I set up a filter: /search?q=
...
0
votes
1
answer
915
views
Web scraping / crawling a particular website
I'm trying to scrape 1265 html files at once to get name and descriptions of items I have on a website.
I have permission from the wholesaler to copy there data but I don't want to spend days just ...
0
votes
0
answers
62
views
How to backup a bulletin board hosted by a 3rd party?
A friend has asked me to back up this forum:
http://oldforums.comicbookresources.com/
It's powered by vBulletin.
Something bad had happened there and the hosts no longer want to host, and are ...
3
votes
0
answers
274
views
wget put all prerequisites in flat subdirectory, but not root page?
I'm trying to get wget to save a page + prerequisites in a format resembling that of a web browser:
article.html
article_files/img.jpg
article_files/script.js
I am able to get almost this behavior, ...
2
votes
2
answers
4k
views
Web scraping / crawling a particular Google book
For my work, I need to scrape the text from a large book on Google Books. The book in question is a very old book and is out of copyright. The book is a Gazetteer of the World. We will be putting the ...
4
votes
4
answers
229
views
Save report from Windows checking removable disk?
Is there any way to save a report of the errors that Windows found and fixed in a USB key? Windows presented me this dialog, btw that is not resizable:
.
Some problems were found and fixed.
When I ...
1
vote
1
answer
3k
views
How do I use AutoHotKey to read the text at the mouse location?
I can read the entire window text using WinGetText() but I am trying to get the text at the current mouse location.
I've found several examples on the AutoHotKey forums but they are all very old (...
1
vote
1
answer
759
views
Is it legal to screen scrape your own bank statements in the US? [closed]
I want to automatically download my bank statements. My bank charges monthly for OFX access so have considered gathering the data points by other means (scripting, screen scraping), but I want to know ...
1
vote
0
answers
42
views
igoogle style page with content scraped from other pages
I'm looking for an online homepage or Chrome extension that will display an iGoogle type home page but that lets me select content from other pages. In the same way AdBlock will allow me to select a ...
0
votes
1
answer
194
views
IRobot Scraping tool
I am using IRobot scraping tool to obtain some data from website but for some reason it is not comming I think the problem must be of proxy settings can anyone help me to how to change the proxy ...
4
votes
1
answer
229
views
Extract data from an online atlas
There is an online atlas that I would like to extract values from. The atlas provides a tool ('Query') to extract values when you click a location or enclose a region on the map, or you can specify ...
0
votes
0
answers
115
views
Programs for reading data off interactive web charts
This question is in the same vein as this one: Program for reading data off scanned graphs
Recently more and more interactive charts show up on the Net. Have a look at the following one:
http://www....
9
votes
2
answers
7k
views
Get Current HTML Of Page Built With AJAX Requests
So, I'm using the Chrome/Chromium browser (put could use Firefox, if need be).
I'm viewing webpages which are constructed "on the fly" with (presumably) AJAX (think the how you scroll down on ...
1
vote
1
answer
2k
views
Alternative, more efficient scraping method for a noncoder, than Google doc's importxml and xpath?
I've searched throughout the net for a simple solution, but it seems everyone has their own unique method (coding language) of achieving this. I'm only just beginning to learn Linux, and my coding ...
1
vote
1
answer
3k
views
Web scraper for Outlook Web App 2010?
I've been using weboutlook for years with great success, but recently we switched from the 2007 to the 2010 version of Outlook Web App (OWA). Does anyone know of a scraper that works with the newer ...
1
vote
4
answers
7k
views
Is there a Firefox or Chrome plugin to automatically save images from webpage
A website has a slideshow of images, for example:This link
Since the slideshow is automatic, is there anyway to save those images automatically into a folder?
either by a browser plugin or by some ...
12
votes
4
answers
7k
views
How "legal" is site-scraping using cURL? [closed]
Recently I was experimenting with the cURL, and I found lot is possible with it. I built a small script that crawls a musical site, which plays online songs. On the way of my experiment, I found that ...
1
vote
2
answers
219
views
Data Mining Software [closed]
I want to harvest some data like this http://www.newcardealers.ca/en/Dealers/List-A.aspx
And insert the name, address, phone number, email, etc. into a database. Is there some software I can use that ...
29
votes
7
answers
20k
views
How do I copy text from a dialog box? [closed]
In Windows sometimes I get an error dialog with long text. Manually typing out the exact details of an error message can be an annoying and lengthy process.
Is there way to copy text from a Windows ...
1
vote
3
answers
1k
views
Save parts of a website as pure text
I hope I may ask this here.
I need to extract the contents of an existing website (in charge of the website owner) to Word (or text) documents. For this, I only need the content from one DIV with a ...