Can Googlebot Crawl or Pull URLs / URIs From Apache index / fancy index Sites? [closed]

Question

That's it,- that's all I am wanting to know with this. - I did a search on Google and it does not even mention such situations (crawling websites with Apache index / fancy index layout); but I know that I can send a spider to a site and find all kinds of things that are not linked, so can Googlebot do likewise with sites like mine [insurgent.info]?

Answer 1

Can Googlebot crawl or pull URLs/URIs from Apache index/fancy index sites?

Yes.

I did a search on Google and it does not even mention such situations (crawling websites with Apache index/fancy index layout).

Likely because there really isn't anything remarkable to mention. =)

The inability to parse some or all of a page typically only arises with JavaScript since, excluding Google, most web crawlers don't execute JavaScript code. Apache returns indexes (even the fancy versions) as simple HTML pages (no JavaScript).

There is likely a conversation to be had about search engine optimization with index-only sites, but that's probably about it. That said, googling "firefox ftp" returns https://ftp.mozilla.org/pub/firefox/releases/ as its first result.

I know that I can send a spider to a site and find all kinds of things that are not linked...

Everything a spider (or what-have-you) typically accesses is linked as a URL/URI somewhere in the code for the pages which are parsed (even if it isn't visible to "normal" visitors).

The only exceptions to this might be:

• Links that require JavaScript to access (e.g. as in lazy loading), which more advanced bots can sometimes do anyhow, at least in conjunction with headless browsers.

• Links based on brute force URL guessing (most often practiced by malicious scripts/tools, etc.).

• Links mistakenly created by some server configuration error or security vulnerability, which lead to files outside the web root.

... so can Googlebot do likewise with sites like mine?

Regarding Apache index/fancy index pages, Googlebot can index everything a regular (or even savvy) visitor could access.