The files in
\Windows\System32\config\ are registry hives. Their binary format is technically directly readable with third-party software but by far the easiest way to load them in Windows and access them either through
regedit.exe or third-party software that uses the registry APIs. SYSTEM (and SAM, which is actually where most of Windows' auth stuff lives) are under the
HKEY_LOCAL_MACHINE root key. You can also extract the registry hive from a computer to load it on another computer.
regedit has an option to open and mount a registry hive.
Note that on any booted Windows machine, the SYSTEM and SAM registry hives will be mounted by the OS and locked to prevent access via the file system. You can access the files directly if you mount the hard disk in another machine (or boot from DVD or flashdrive or something). You can also theoretically unmount registry hives, but the system-critical ones like SYSTEM and SAM would make the computer stop working quite dramatically if they were unmounted, so the OS won't allow it.
Windows passwords can be stored in a few different ways, but most common by far are NTLMv2 hashes (specifically, the outputs of the NT One-Way Function v2, or NTOWFv2). These are relatively easy to break; they use the obsolete MD4 and MD5 hash algorithms, and do not contain anything to limit the speed of hash computation (such as the way PBKDF2 requires repeating the same hash process many times) to slow down brute-forcing. Nonetheless, you're not going to see the actual passwords in plain text. Even when the OS is configured in such a way that it's possible to get the plain text of the passwords (which it isn't, be default), they are stored at rest under symmetric encryption so you'd need to get the encryption key. Almost everybody just uses the password hashes, though, and usually only the v2 form (which, although bad by modern standards, is much better than NTLMv1 and pre-NT LANMAN one-way functions).