# What's up with different hexadecimal encoding schemes?

So I've been working with cryptography for about a year now, learning as much as I can about all I can. A lot of things that I've come across (like the hexadecimally [not a word, but shh] encoded master key of a LUKS volume, or the modulus of an RSA key) are encoded in hexadecimal. I understand that decimal is a fancy way of saying base 10, and hexadecimal is a fancy way of saying base 16, and I get how it works in terms of numbers, but there are a few things that I can't quite wrap my head around:

1. How can ASCII or UTF-8 (or general plaintext) be encoded in a number-based system?
2. I have noticed 2 different kinds of encoding schemes in my ventures. Are there more versions? Why?

The two that I have noticed (while decoding LUKS master keys and decoding RSA moduli) are strange. The LUKS master, key, I echo the key, piping it into "xxd -r -p" and it spits out the actual key, which can be put into a file and used to unlock the volume without the use of keyslots. The other version I've seen is when I pipe an RSA modulus with the attachment "ibase=16; " into "bc". Any other hexadecimal sequence I've sent into BC with "ibase=16; " spits out specifically numbers, no letters. "xxd -r -p" seems to output traditional ASCII.

Why does BC output numbers and xxd output ASCII text?

I apologize is this is such an all-over-the-place question, but like I said earlier, I am way too confused on this topic to be normal.

The "actual key" is the binary characters. However, what you typically put into a file is a hexadecimal representation.

xxd is outputting ASCII text, instead of hexadecimal representations, because of the -r switch. Having actual characters is what the man page means by "binary".

ASCII is basically just a list of characters in a specific order. First a NULL, then a Start of Header character, then an End of Header character, etc. (See: ASCII chart. Code Page 437 (see: Wikipedia's page on CP437 shows these as a blank, a smiley face outline, a filled in smiley face, etc. The 65th character of both of these are the capital letter A.

If the 65th character is the capital letter A, this means we can keep track of characters based on their position in the chart. So the capital letter A is position 65, which is position 41 in hexadecimal (often written in C-style notation, as 0x41). You can see the value "41" in the ASCII chart I mentioned above.

How can ASCII or UTF-8 (or general plaintext) be encoded in a number-based system?

Numbers can be converted into other numbers, such as other bases. e.g., 0x4169 is equivalent to 0x4100 + 0x69. The 41 is equivalent to the capital letter H, and the 69 is equivalent to the lowercase letter i. I'm not sure why you would doubt that numeric conversation is possible. If this is still unclear, please re-explain (possibly by asking another question).

``````I have noticed 2 different kinds of encoding schemes in my ventures. Are there more versions? Why?
``````

Sure. Why? Because there are multiple pieces of software, designed by different people. Just like why there are multiple character standards (EBCDIC, ASCII, UTF-8, Unicode). Some of them are useless (EBCDIC was very popular before 8-bit code pages, using ASCII, became more popular), some may be easier to support by certain software tools. Computers are now powerful enough that there isn't a lot of technical limitations that force multiple standards, but different programmers do different things, sometimes for different reasons.

Conversion tools exist. The lack of standardization is namely because stronger/tighter/better standardization hasn't been required. If there was compelling benefit, more people probably would be more prone to try to align themselves with larger common trends.

Why does BC output numbers and xxd output ASCII text?

`bc` is a calculator. I would expect numeric output. `xxd -r` outputs ASCII text, because that is what it is documented to do. The point of that software is to translate ASCII text to (hexadecimal) numeric strings that are easier to type, and to convert numeric strings into "binary"/ASCII characters that are actually more useful to certain software. Each program is fulfilling it's own, different, purpose.

I hope this collection of answers helps to answer enough of your collection of questions to help things be more clear. I know you alluded to this, but I will point out that this question is a bit disastrous because it's not very a single clear question, so consider making follow-up questions in new questions. Asking one actual question per "question page" on Stack Exchange is generally preferred.

Parts of your question don’t make much sense:

Any other hexadecimal sequence I’ve sent into BC with "ibase=16; " spits out specifically numbers, no letters.

Well, the default output base for `bc` is decimal:

``````\$ echo 'ibase=16; 100'   | bc
256

\$ echo 'ibase=16; 1000'  | bc
4096

\$ echo 'ibase=16; 10000' | bc
65536
``````

so what letters would you expect?  If you want `bc` to output hexadecimal numbers, you have to set its output base to 16.  But if the input base and the output base are the same, its output will be the same as the input (with leading zeroes stripped):

``````\$ echo 'obase=16; ibase=16; 100'  | bc
100

\$ echo 'obase=16; ibase=16; 007F' | bc
7F

\$ echo 'obase=16; ibase=16; CAFE' | bc
CAFE

\$ echo 'obase=16; ibase=16; FACE' | bc
FACE
``````

Note that, if you want to do this rather pointless exercise, you have to say `obase=16; ibase=16` or `ibase=16; obase=10`, because any numbers that are entered after `ibase=16` are interpreted as hex.  If you say `ibase=16; obase=16`, the output base will be set to 22 (16(hex) = 10(hex) + 6(hex) = 16+6 = 22).

"xxd -r -p" seems to output traditional ASCII.

Either you’re giving `xxd` a hexadecimal representation of “traditional ASCII” text, or you have a nonstandard definition of “traditional ASCII”.  As TOOGAM says, `xxd -r` outputs “binary”/ASCII characters, which is peculiar nomenclature.  Note that the “ASCII chart” that TOOGAM linked to goes only up through character 7F, and the Wikipedia page on ASCII says

… ASCII encodes 128 specified characters into seven-bit integers …

also ranging from 00 to 7F.  `xxd -r` outputs eight-bit (“binary”) characters.  This is a superset of standard ASCII.  (The term Extended ASCII refers to eight-bit encodings, but it is probably better to abandon this phrase as Unicode permeates the landscape.)  Since eight-bit (“binary”) characters form a superset of standard ASCII, `xxd -r` can output traditional ASCII characters; for example:

``````\$ echo 464F4F0A | xxd -r -p
FOO
``````

since 46 is the hex representation of the ASCII code for `F` (the 6th letter of the alphabet), 4F is the hex representation of the ASCII code for `O` (the 15th letter of the alphabet, where F is the hex representation of 15), and 0A is the hex representation of the ASCII code for newline.  But

``````echo 000102030405067F | xxd -r -p
``````

will output non-printable ASCII control characters, and

``````echo 80818283A0A1A2A3 | xxd -r –p
``````

and

``````echo E29885E29FB9F09F98BA0A | xxd -r –p
``````

will output bytes with values above 7F.  The behavior (appearance) of these characters (if any) is determined by your terminal and its settings.