4

I want to know how to automatically copy text from different websites. I am building a database of companies which belong to certain associations. The website has a list of companies with the description of each of them which I am manually copying. Is there a way to create a macro and make this automatically since it is repeating the same process over and over.

Or anybody know in which language I could develop something similar? Thanks,

2 Answers 2

1

You are looking for a technique called web scraping. You can use this to gather data from websites which offer no APIs for getting the data. There are plenty of tools for it and there are also many libs for different languages to do it (e.g. there is BeautifulSoup for python). Beware, you may run into legal issues when scraping/hammering a site constantly without permission.

1
  • Maybe look into Powershell and Invoke-WebRequest. E.g. $a = Invoke-WebRequest -Uri http://superuser.com/questions/1182298/how-to-automatically-copy- text-from-different-websites $a | get-member So you could get a list of links with $a.links, etc... Feb 24, 2017 at 9:42
1

Over at "Software Recommendations" there is list of good web scraping applications, they can all do what you need.

If you are just starting with screen scraping, tools with a recorder might be best. This avoids creating all the selectors manually. But there is no solution that magically works with every website, even the tools with a recorder require often some tweaking for each page.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .