The tag has no usage guidance.

learn more… | top users | synonyms (3)

0
votes
0answers
13 views

Bashscript using curl to crawl a website that needs a login

I am looking to crawl a website that has kind of a newsfeed with elements hat have a timestamp. There's only a certain amount of elements in the feed and a "Previous" button at the end of the feed. It ...
2
votes
1answer
46 views

Finding pages on a webpage that contain a certain link

Google does a good jobs finding relevant information. Say I google: FDA's opinion on ISO-9001 Then it finds a link to a PDF on fda.gov ...
0
votes
0answers
10 views

Indexing file resources from a website

How can I get a list of all (image) resources from a website? Is there a better wait than using wget and parsing the log stream in order to get the list of all images? Worst is, I'm pretty use I ...
1
vote
0answers
55 views

How to extract text from websites

I am looking for a way to automate extracting text from multiple websites to a word document. When pasted into the word doc I need it to paste using "merge formatting". This is how I would like the ...
0
votes
0answers
39 views

Best way to check google results number for 1000,000 search words?

I have 1 million species in a data file and i wish to rewrite the file using only species which have more than 300 search results in Google, Bing, or another broad search engine. Instead of doing 1 ...
1
vote
0answers
48 views

How to allocated different IP while crawling web pages

How can i assign different ip address while crawling I am crawling billions of page everyday(allocating different ip will help to handel blockages)
1
vote
1answer
34 views

save web page with all the related content

I am trying to figure how I can save a webpage with all the related files, for example : http://docs.oasis-open.org/ubl/os-UBL-2.0/xsd/ I want to save all the files in the directory kinda like a ...
0
votes
1answer
38 views

How could I crawl all the files in file server recursively

There are thousands of files in a file server http://xxxx.com I tried to crawl it with the tool httrack It doesn't work, is there any alternative tool can download the whole files recursively base ...
1
vote
2answers
24 views

Web crawler with converte links option

I would like to crawl a backup site I lost access to. The site is backed up in subdomain.somesite.com while the links on the web page are www.subdomain.com this leads to the following situation: ...
0
votes
0answers
32 views

Multipage search using DEiXTo web data extraction

I am trying to use the DEiXTo tool to extract a sequence of moves from a chess website. The extraction tool works fine and pulls out the data I need from a single page. However, I cannot get the tool ...
-3
votes
2answers
295 views

How can I scrape only word data from a website? [duplicate]

I want to download all the word content from a particular website. Save the results in MS Word, Excel or Notepad and check which words are repeated most often and how many times.
2
votes
0answers
45 views

wget - limit following to specific links

I want to do mirror of website but limit following only to specific links. Command: wget --recursive --page-requisites --html-extension --convert-links --domains linuxmagazine.pl ...
-1
votes
2answers
124 views

The “smart” way to crawl the web [closed]

I have a project in which I have to crawl the web for lyrics, I am implementing depth-first search and I am making a database in a list (which contains nested lists which will later be saved in a ...
0
votes
1answer
98 views

Why is my personal web site getting visitors at mysterious URLs? [closed]

I recently made an update to my site's visitor logging and when checking that it was working properly noticed that I'm getting visitors at URLs that do not belong to me. I was surprised to even see ...
1
vote
0answers
38 views

Access to all links on a domain(no hyperlink available)

I need to have access to all links like: http://www.thewebsitename.com/random_alphanumeric_code1.asp http://www.thewebsitename.com/random_alphanumeric_code2.asp etc.for a given website: ...
0
votes
0answers
27 views

web scraping import to local website

I'm looking for a way to search for a specific product on a website and extract data from it, but without going directly to this website, like from a local website that queries information and ...
2
votes
3answers
286 views

Extracting links from a numeric range of web pages

I would like to extract links from a numerical sequence of pages like this: http://example.com/page001.html http://example.com/page002.html http://example.com/page003.html ... ...
-1
votes
1answer
103 views

List all links of one website on other website [closed]

There are two websites. 1) www.aa.com 2) www.bb.com Now there are so many link of aa.com and its webpages on bb.com like aa.com/blhahhaa.html aa.com/beautifulday.html aa.com so i want to find ...
1
vote
1answer
321 views

Mirroring a web site having pages that uses simple JavaScript [duplicate]

There are simple and great web site download tools that allow to create a local mirror of simple sites having no JavaScript infrastructure behind. However, sometimes there cases when a site looks ...
-1
votes
1answer
68 views

Compiling a list of links on a website and their validity [closed]

I would like to crawl my website and generate a list of internal and outbound links, and what their final destination is and HTTP Code (at least for internal links). How can I do that? Please note: I ...
1
vote
1answer
73 views

How would I scrape text from a site? [closed]

Schemer is closing down, and I'm trying to preserve the text because there are many good ideas on the site. I'd like to get the text from within each of the links listed on the sitemap. Is there a ...
4
votes
1answer
3k views

How to crawl using wget to download ONLY HTML files (ignore images, css, js)

Essentially, I want to crawl an entire site with Wget, but I need it to NEVER download other assets (e.g. imagery, CSS, JS, etc.). I only want the HTML files. Google searches are completely useless. ...
1
vote
0answers
24 views

Scan all available requests on website [closed]

I would like to scan some website for all available requests/api requests that can be done on this server, is there any way to do that? Thanks
15
votes
2answers
490 views

Why is @ in email address sometimes written as [at] on webpages?

Why is @ sometimes in webpages written as [at]? Does it have any specific reason ?
0
votes
1answer
150 views

Wget getting responce 403

I am using API that has some limit of requests in an hour. But my script does all at one time so I lose about 1/3 of requests because I get 403. Is there any way to check response of wget and if I ...
1
vote
2answers
1k views

Web scraping / crawling a particular Google book

For my work, I need to scrape the text from a large book on Google Books. The book in question is a very old book and is out of copyright. The book is a Gazetteer of the World. We will be putting the ...
0
votes
1answer
658 views

How to crawl your own website to save to cache

I'm using Squid, a caching program, to cache my website. However to do so it appears that each page has to be accessed at least once before Squid can cache it. My question is, is there a program which ...
1
vote
1answer
443 views

recursively downloading all the folders and subfolder from webpage

I would like to download a directory from webpage. i am trying to do using wget and crawl but i am not able to download the whole directory as it was there in web-page or in any-other way. suppose the ...
0
votes
1answer
285 views

Windows - Crawl URL and grab links

I have recently signed up to a large video site as a premium member and I want to generate a list of the video files on the site so that I can put them into my download manager and download the ...
0
votes
2answers
564 views

Streaming Video Bulk Download

There is a video series I have been watching: http://english.cntv.cn/program/learnchinese/growingwithchinese/index.shtml I have tried a few plugins and utilities to download the episodes with varying ...
2
votes
2answers
9k views

How can I scrape specific data from a website

I'm trying to scrape data from a website for research. The urls are nicely organized in an example.com/x format, with x as an ascending number and all of the pages are structured in the same way. I ...
0
votes
5answers
9k views

Website crawler/spider to get site map [closed]

I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ ...
2
votes
1answer
947 views

Is there a graphical web crawler that indexes a site in excel?

I am trying to recreate the product catalogue for an ecommerce site as an excel table. Basically i want to fetch product name, brand name, price, and any associated tags and store it in an excel ...
2
votes
1answer
262 views

wgt downloads all files except for the images i want

i have trouble using wget to download images from a gallery. as start i use the overwiew page. it has thumbs that link to the individual pages with the large images. here is the script i use: wget ...
4
votes
1answer
118 views

Extract data from an online atlas

There is an online atlas that I would like to extract values from. The atlas provides a tool ('Query') to extract values when you click a location or enclose a region on the map, or you can specify ...
1
vote
1answer
870 views

Spider/crawl a website and get each URL and page title in a CSV file

I am moving from an old ASP shopping cart site to a Drupal/Ubercart site. Part of this move is to ensure that old links will redirect to the new ones. To do that all I need is some way to get a list ...
0
votes
1answer
954 views

Scan and map website and log all links that have “particular-string” in them

Is there a tool that would scan a website and log all links that have a particular string in them? I see Teleport Pro, but it seems like it does make a copy of the site, but I only need links.
0
votes
0answers
51 views

Looking to backup content from online class [duplicate]

Possible Duplicate: How can I download an entire website? Basically, I took an online class, and now I want to spider the content, and back it up locally. Any Suggestions on ...
2
votes
3answers
5k views

Is it possible to discover all the files and sub-directories of a URL?

I wonder if there is a software I can use to discover all the files and sub-directories given a URL? For example, given www.some-website.com/some-directory/, I would like to find all the files in ...
1
vote
2answers
195 views

Which sites reject crawler requests?

Is there any site which can be rejected by a crawler? I am using the Burp Suite crawler for the time being to crawl the sites. I want to know when and in which cases a crawler fails to retrieve the ...
0
votes
1answer
35 views

How can I search the Internet for sites containing keywords in HTML (not text)?

I want to search the Internet for sites using a certain widget. Google has already crawled the Internet but their standard search trims any pieces of code and leaves only human readable text to ...
5
votes
0answers
3k views

wget: recursively retrieve urls from specific website

I'm trying to recursively retrieve all possible urls (internal page urls) from a website. Can you please help me out with wget? or is there any better alternative to achieve this? I do not want to ...
1
vote
2answers
5k views

Firefox addon to download a whole site and one step more

Do you know any Firefox addon that could download a whole website and download all the sites from the links on the first website? I mean also all images and so on.
0
votes
2answers
1k views

Can storing 300k files in one folder cause problems?

I'm crawling a large website (over 200k pages) using wget (is there a better tool btw?). Wget is saving all the files to one directory. The partition is HFS (I think), will it cause problems if I ...
10
votes
4answers
31k views

Using Wget to Recursively Crawl a Site and Download Images

How do you instruct wget to recursively crawl a website and only download certain types of images? I tried using this to crawl a site and only download Jpeg images: wget --no-parent --wait=10 ...
1
vote
3answers
204 views

Extracting information from web page in given interval

I have a problem. I need to extract something like 800 records and it would take days to do that manually. The information may be taken by inserting few fields into form (always the same) and then a ...
1
vote
1answer
550 views

Crawling a large directory with wget with two links pointing at the same thing

I'm trying to crawl a directory on a website and basically download everything in it. The structure is simple enough(but there are also multiple folders), but there is one thing that makes wget choke ...
0
votes
0answers
176 views

web spidering/crawling, can i do it or just search engines?

i already had a question answered about web-scraping with wget. but as i read a little more, i realize i may be looking for a web-crawling program. particularly the part about web-crawlers being able ...
9
votes
4answers
12k views

Convert web pages to one file for ebook

I want to download HTMLs (example: http://www.brpreiss.com/books/opus6/) and join it to one HTML or some other format that i can use on ebook reader. Sites with free books don't have standard paging, ...
0
votes
1answer
1k views

Extract text from web

I'm looking for an easy toll which can crawl the web I give it to and extract all text elements from that. It will be nice if it can do: plain text, alt and title for images, header section. All of ...