0

I am aware of Google Dorks that use advanced operators to search for keywords on Web, and Google Dorks is not Regex by anyhow.

Out of curiosity I tried putting wildcard characters in URLs mostly for images on various sites that didn't worked out. Most of them showed various HTTP Error Codes, with exception of 2-3 that redirected to some other page within the website.

Some examples that didn't worked :

  • Searching for PDFs. Got 404 : Page Not Found

PDFs Search

  • Searching for Images. Got 500 : Internal Server Error

Image regex Search

  • Searching for all HTML pages on Website. Simply Error page

HTML Pages Search

  • Youtube converts characters to Safe Hexadecimal characters.

Youtube Search

An example that worked (No Idea how) :

Windows Tagged Questions

Linux Tagged Questions

See the Question Tagged section on the right of page to see all searches for the word lin*

I apologize If question has too much details, but I find it appropriate giving some examples. So, here I conclude my question :

  1. Can Regular Expressions be used in any way in URLs while browsing ?

  2. If so what are rules or methods of using them ?

  3. Also, Uses Section under Wikipedia page for Regular Expressions quote this :

    While regexes would be useful on Internet search engines, processing them across the entire database could consume excessive computer resources depending on the complexity and design of the regex. Although in many cases system administrators can run regex-based queries internally, most search engines do not offer regex support to the public. Notable exceptions: Google Code Search, Exalead. Google Code Search has been shut down as of January 2012.[39] It used a trigram index to speed queries.[40]

    So, Is it like we can do Regex Search on Search Engines but not on Webpages ?

  • While you made a nice example of question, your question itself still confusing. Basically URL is just a string and how it would be parsed on receiving side it depend how it handled on receiving side, so the answer in most cases - NO. If your particular question is regarding how to use modifiers on particular search engine - it a different story and AFAIK google don't support full feature regex – Alex Jul 21 '18 at 13:37
1

Generally, no. (Also wildcards and regular expressions are not the same thing.)

Each website decides completely on its own how the path and query should be interpreted. Paths might correspond to actual files, or they might map to abstract resources like "questions" or "posts". (e.g. SuperUser doesn't actually have a folder called questions full of numbered subfolders – the path just tells it to what response to generate.)

This also means each website decides whether it'll support wildcards (or regular expressions, or any other filtering method) or not. Usually, adding this feature would need additional programming to the website.

As you've discovered, Super User did implement wildcards for tag browsing – it's useful to find all Windows-related questions (and probably was easier to implement than hierarchical tags).

But, for example, YouTube video IDs are fully random, so there is no use whatsoever in trying to list all videos whose IDs begin with RgK... or any other prefix. So YouTube didn't implement that.

"Plain" webservers, which just serve ordinary .html files, could implement this feature as they often already have automatic directory listings. However, many website owners wouldn't want this – they might have "unlisted" pages which are only known to a few people, for example.

(There is a similar feature in Apache httpd, which lets you obtain a full directory listing even if an index.html file is present. But it has to be disabled by default, because many people actually rely on index.html as a reliable method to disable those listings.)

  • Your answer covers my example scenarios and I really need to know difference between Regex and WildCards. – C0deDaedalus Jul 21 '18 at 13:53
  • I wonder how do you get to know that SuperUser doesn't actually have a folder called questions ? Did you use dirb tool. – C0deDaedalus Jul 21 '18 at 13:57
  • @C0deDaedalus lookup .htaccess files and redirects. It'll show you how you can redirect anything based on different patterns. I do believe some kind of regex can be used in .htaccess though. – LPChip Jul 21 '18 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.