How do file system scans (like CHKDSK or fsck) really work? I know there are full applications with lots of options, but how do they scan for "bad" sectors? What are they doing when scanning? How do they know they found a "bad" sector?
Filesystem checking has nothing to do with scanning for "bad" sectors. "bad" sectors in this context usually means sectors that can't be read reliably or can't be written reliably. Although a filesystem check can certainly stumble upon a bad sector since it attempts to read (and maybe write) blocks all over the filesystem.
What a filesystem check actually does is really up to each individual filesystem and up to the implementer of the filesystem checking utility. For some types of filesystems (for example, NFS and tmpfs), the concept is meaningless, for others it is redundant, because the filesystem continuously checks itself in the course of normal operation, and for others it is vital that to check the filesystem once in a while.
In general, filesystem checking utilities are designed to check for corruption, inconsistencies, and violated invariants in the filesystem's data structures. If you want more detail than that, then you will have to ask about a specific type of filesystem.
CHKDSK does quite a few things behind the scenes. They are represented in the different phases of CHKDSK.
NTFS has something called an MFT (or Master File Table). This MFT is a list of all files on the hard drive. To check for file consistency, CHKDSK reads the MFT entry by entry, and then it goes up and looks up the corresponding file in the HDD.
For example CHKDSK finds MyPhoto1.jpg in the MFT. The MFT says it is located in sector 230, and is 30 sectors long. CHKDSK then goes and reads sector 230 to 260, and sees that it is in fact populated by data, and is one file. This is how CHKDSK checks for file system consistency. Files inconsistency doesn't necessarily mean a bad sector, it could mean the computer was powered off during a write to the HDD.
With the /r flag (recover), CHKDSK checks for file inconsistencies and attempts to repair them. Without this flag, chkdsk performs a "dry run"
With the /f flag (fix), CHKDSK attempts to repair the bad sector. In other words, it attempts to recover the data on the bad sector, and attempts to rewrite to that sector. If CHKDSK reads what it wrote the sector in question, then it can mark the sectors as repaired.