The file extension is often either random or not enough to identify the format.
- PEM files with ASN.1 data, encoded with DER
- PEM files with data encoded in some other format
- Non-PEM formats
PEM files wrap Base64 between
-----END----- "tags". They are also commonly used to contain both private key and SSL certificate (-chain). Use an online ASN.1 decoder to check the Base64 contents of a PEM file.
PKCS#1 / OpenSSL: id_rsa, *.pem, *.der, *.key, ...
-----BEGIN RSA PRIVATE KEY-----
PuTTY Key Generator calls this "OpenSSH SSH-2 private key (old PEM format)" (?). The "SSLeay" or "traditional" format, according to this answer. Base64 starts with
MII.... ASN.1 content. More info.
PKCS#8: *.pem, *.der, *.key, ...
-----BEGIN PRIVATE KEY----- or
-----BEGIN ENCRYPTED PRIVATE KEY-----
Base64 of the unencrypted variation starts with
MII...IBADAN. ASN.1 content, basically PKCS#1 plus version info. More info.
OpenSSH: *.??? (don't know what a typical file extension would be)
-----BEGIN OPENSSH PRIVATE KEY-----
PEM on the outside, but non-ASN.1 content. Apparently a somewhat undocumented format.
PuTTY Private Key: *.ppk
Content also contains human readable words identifying it as a putty private key.
PKCS#12 / PFX: *.p12, *.pfx
PFX is a Microsoft format, later released in cleaned-up form as PKCS#12. The content is binary, and can contain not only a private key, but also an SSL certificate (-chain).