I have a text file from a legacy source which contains corrupted characters.

At first I thought the corruption was just gobbledygook, but it appears upon closer examination that some of the corrupted text could probably be reconstructed.

In order to focus my efforts, it would be helpful to understand what the original looked like, even if I cannot reconstruct it fully.

Unfortunately, the document is from a collection which I cannot share freely, but here is a snippet. The message has been converted to UTF-8, but the conversion failed somewhere, so it is mostly illegible. Fragments of text in Czech are visible, where the accented Czech characters have been replaced with Cyrillic characters (which were probably something completely different before the conversion).

0001f80: 33d1 936e 6576 79d1 87d0 bd7a 656e d18c  3..nevy....zen..
0001f90: 6368 7e58 3833 d193 7e58 3945 d19b d0b1  ch~X83..~X9E....
0001fa0: 646f 7374 d0bd 7e58 3833 d193 6e61 7e58  dost..~X83..na~X
0001fb0: 3833 d193 7ad1 87d0 bd7a 656e 20d0 bd7e  83..z....zen ..~
0001fc0: 5838 33d1 936e 6562 6f7e 5838 33d1 9370  X83..nebo~X83..p
0001fd0: d187 656b 6cd0 b164 6b75 7e58 3833 d193  ..ekl..dku~X83..
0001fe0: 7465 6c65 666f 6e6e d0bd 7e58 3833 d193  telefonn..~X83..
0001ff0: 7374 616e 6963 657e 5838 33d1 9376 207e  stanice~X83..v ~
0002000: 5838 33d1 9372 6567 696f 6e75 7e58 3833  X83..regionu~X83
0002010: d193 5072 6168 617e 5838 33d1 9365 7669  ..Praha~X83..evi
0002020: 6475 6a65 7e58 3833 d193 5350 547e 5838  duje~X83..SPT~X8
0002030: 33d1 9354 656c 6563 6f6d 2e7e 5838 33d1  3..Telecom.~X83.
0002040: 934e 617e 5838 33d1 9364 6e65 7e58 2039  .Na~X83..dne~X 9
0002050: 41d1 996e d0bd 7e58 3833 d193 7469 736b  A..n..~X83..tisk
0002060: 6f76 d0b9 7e58 3833 d193 6b6f 6e66 6572  ov..~X83..konfer
0002070: 656e 6369 7e58 3833 d193 746f 7e58 3833  enci~X83..to~X83
0002080: d193 d187 656b 6c7e 5838 33d1 93d1 8765  ....ekl~X83....e

I'm vaguely speculating that the encoding might be related to ISO-2022, but I am not familiar enough with it to really be sure. It has obviously gone through at least one broken filter, possibly multiple, before ending up like this.

Looking at the first line, d1 93 is ѓ and was probably a single 8-bit byte before the conversion. A general pattern seems to be ~XFF followed by a signal byte, where the FF is some hex sequence in plain ASCII (mostly 83 here, but generally from 80 through 9E in the entire sample), and the final byte is now an UTF-8 character. (It could have been multiple bytes in the input as well, of course.) This sequence appears between words (always ~X83ѓ?), and sometimes within words.

Here is the same fragment as just text, as it renders under UTF-8 now.

3ѓnevyчнzenьch~X83ѓ~X9Eћбdostн~X83ѓna~X83ѓzчнzen н~X83ѓnebo
~X83ѓpчeklбdku~X83ѓtelefonnн~X83ѓstanice~X83ѓv ~X83ѓregionu
X 9Aљnн~X83ѓtiskovй~X83ѓkonferenci~X83ѓto~X83ѓчekl~X83ѓчe

I have other samples in other languages so getting the Czech sorted out is not really my focus. Here is the beginning of one in, I don't know, probably some Far Eastern language?

 X1B%0 ~XD0^?~X98^?~XD0^?^?^?~X82^?~XD0^?~XB5^?^?~X80^?^?~X84^?~XD0^?~XB0^?~XD0
^?^?^?~X81^? ~XD0^?~XB1^?^?~X83^?~XD0^?^?~XD0^?~XB2^?~XD0^?~XB0^?~XD0^?^?^?~X8C
^?~XD0^?^?~XD0^?~XBE^? ~XD0^?~XB7^?~XD0^?~XB0^? ~XD0^?^?~XD0^?~XB5^?^?~X81^?~XD
0^?^?~XD0^?~XBE^?~XD0^?^?^?~X8C^?~XD0^?^?~XD0^?~XBE^? ~XD0^?^?~XD0^?~XB8^?~XD0^
?^?^?~X83^?^?~X82^? ~XD0^?~XB4^?~XD0^?~XBE^? ^?~X81^?~

(The ^?:s are literal DEL characters, ASCII 0x7F.)

The space in the place of a tilde at the beginning might be a hint as to what went wrong in the conversion, but this is wild speculation.

ESC % 0 looks like the ISO-2022 code to "designate other coding system", but what does the 0 stand for here? I'm probably too dense to understand the Wikipedia article without further examples, and everything else I could find seems very focused on some subset such as ISO-2022-JP.

Does my analysis so far make sense to you? Can you help me figure out what happened, and perhaps even offer advice on how to revert the corruption?

I have posted hex dumps of extended fragments of these two examples at http://pastebin.com/ffn7CtdG

  • 2
    It would help in the forensic work if you could extract the above fragment using an hex editor, store it as a binary file and post it somewhere. Make the fragment as large as you can, and start it if you can from the first byte of the file. Do you know under which operating system the files were generated? – harrymc Oct 9 '14 at 16:43
  • No, I don't know what sort of system this originally ran on. – tripleee Oct 10 '14 at 5:54
  • Thanks for the feedback! Added a Pastebin link to extended examples. – tripleee Oct 10 '14 at 6:55
  • Thank you, but I really meant the entire beginnings of files as they are, and not the output is from xxd that I can't use in my hex editor. Anyway, one obvious thing I noticed from your example is that the "~X83.." part should rightly be a blank. For example, "SPT Telecom" is well-known in Czech. Any additional information you have about the origin of these files will help. – harrymc Oct 10 '14 at 7:40
  • For licensing reasons, I am uncertain whether I can post a file in its entirety. You can easily turn the fragments into binary with something like perl -lne '@s = m/: ((\X\X ?){16})/; $s = join ("", @s); $s =~ s/ //g; print pack("H*", $s)' – tripleee Oct 10 '14 at 8:20

In this answer I detail my ideas of the source of these files. This is not a complete answer, since a more detailed forensic analysis requires hands-on access to at least some subset of complete files.

Some points that strike me in the fragments that I have seen:

  1. The words are in Czech
  2. There are strange sequences separating the words and they repeat a lot
  3. These strange sequences are composed of UTF-8 characters which make no sense at all, except that some of them are Cyrillic in nature.

My conclusion is that these files were not originally text files, but were erroneously converted to UTF-8 as if they were text, using a code-page that contained Cyrillic characters.

For example, the omnipresent sequence of d193 is the Cyrillic letter small gje whose differing code-page representations are :


This gives us a list of possible encodings of the original files, which depend on the original operating systems. If they were created on a Windows computer, their original code-page was probably Windows-1251, but on a Mac they were probably in Macintosh Cyrillic. Of course, it is also entirely possible that the translation to UTF-8 used the wrong encoding.

For example, we find the sequence SPT~X83..Telecom. The company "SPT Telecom" is none else then the Czech national telecommunications company, founded in 1993, whose presence in a Reuters newswire text is quite logical. However, there is no reason for any separator beside a blank between the two words.

My explanation to these puzzling strings that repeat among the words is that they were not, and couldn't be, a part of the text. I believe that they must have then been binary characters placed between the words, which had probably some connection to the formatting of the files. The conversion program that converted the files to UTF-8 therefore converted them blindly to UTF-8 characters that make no sense.

Even trying to convert this sequences to binary, using any of the code-pages in the above list, I do not get any meaningful sequences. However, I have experience with text files coming from some old text editors that placed "invisible" characters in the text whose purpose was never to be displayed, but rather to control the display.

I believe that this is the explanation to these files, but I do not know this strange file format. It could have been some unknown Czech text editor (at least unknown to me). If the files can be scanned for dates contained in the text, this might help to narrow down the possibilities.

I do not believe in your theory of the original files being well-constructed and encoded in ISO-2022, since these strange sequences don't seem to be (or ever have been) ISO-2022 control sequences.

  • Sorry I wasn't able to give an authoritative answer. If this doesn't lead to a solution, please don't award me the bounty. I just tried to clearly formulate my ideas. – harrymc Oct 13 '14 at 7:36
  • Sorry to see the bounty expired before you got enough upvotes to earn it. I'll try to recover enough rep to post another bounty eventually. – tripleee Oct 17 '14 at 12:20
  • Don't worry about it. – harrymc Oct 17 '14 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.