Can Regular Expressions be used in URLs while browsing?

Question

I am aware of Google Dorks that use advanced operators to search for keywords on Web, and Google Dorks is not Regex by anyhow.

Out of curiosity I tried putting wildcard characters in URLs mostly for images on various sites that didn't worked out. Most of them showed various HTTP Error Codes, with exception of 2-3 that redirected to some other page within the website.

Some examples that didn't worked :

• Searching for PDFs. Got 404 : Page Not Found

• Searching for Images. Got 500 : Internal Server Error

• Searching for all HTML pages on Website. Simply Error page

• Youtube converts characters to Safe Hexadecimal characters.

An example that worked (No Idea how) :

• https://superuser.com/questions/tagged/windows*

• https://superuser.com/questions/tagged/lin*

See the Question Tagged section on the right of page to see all searches for the word lin*

I apologize If question has too much details, but I find it appropriate giving some examples. So, here I conclude my question :

1. Can Regular Expressions be used in any way in URLs while browsing ?

2. If so what are rules or methods of using them ?

3. Also, Uses Section under Wikipedia page for Regular Expressions quote this :

   While regexes would be useful on Internet search engines, processing them across the entire database could consume excessive computer resources depending on the complexity and design of the regex. Although in many cases system administrators can run regex-based queries internally, most search engines do not offer regex support to the public. Notable exceptions: Google Code Search, Exalead. Google Code Search has been shut down as of January 2012.[39] It used a trigram index to speed queries.[40]

   So, Is it like we can do Regex Search on Search Engines but not on Webpages ?

Answer 1

Generally, no. (Also wildcards and regular expressions are not the same thing.)

Each website decides completely on its own how the path and query should be interpreted. Paths might correspond to actual files, or they might map to abstract resources like "questions" or "posts". (e.g. SuperUser doesn't actually have a folder called questions full of numbered subfolders – the path just tells it to what response to generate.)

This also means each website decides whether it'll support wildcards (or regular expressions, or any other filtering method) or not. Usually, adding this feature would need additional programming to the website.

As you've discovered, Super User did implement wildcards for tag browsing – it's useful to find all Windows-related questions (and probably was easier to implement than hierarchical tags).

But, for example, YouTube video IDs are fully random, so there is no use whatsoever in trying to list all videos whose IDs begin with RgK... or any other prefix. So YouTube didn't implement that.

"Plain" webservers, which just serve ordinary .html files, could implement this feature as they often already have automatic directory listings. However, many website owners wouldn't want this – they might have "unlisted" pages which are only known to a few people, for example.

(There is a similar feature in Apache httpd, which lets you obtain a full directory listing even if an index.html file is present. But it has to be disabled by default, because many people actually rely on index.html as a reliable method to disable those listings.)