8

I've created a text file from an application that I developed.

When I send the text file to a SYSTEM validation, they (third-party system) say that the file is invalid and that the file contains three characters in the beginning of the file that are not allowed as well special characters are not correct.

They also say I need to use either ISO 8859-1 or PC850.

Well, I'm using Notepad++, and I can't see that at all! What is the best text file reader for these kind of problems?

Alt text

I also have a Mac and just thought I remembered opening in TextMate ... WOW! Now I know what they are talking about!

How can I have the same in Windows?

Alt text

4
  • 1
    As an aside: you're looking at an UTF Byte Order Mark (BOM) of a UTF-8 encoded document. If an application has other means to tell it's UTF-8, then such BOM is not required for UTF-8 encodes content, but a good editor knows how to handle it. See unicode.org/faq/utf_bom.html#bom2
    – Arjan
    Mar 23, 2010 at 11:47
  • 1
    (On a Mac, also see the file command, to tell if a BOM is present.)
    – Arjan
    Mar 23, 2010 at 11:47
  • 1
    By the way, a quick way to enforce a wrong encoding on any computer: open the file in a browser and explicitly set the encoding to ISO-8895-1 just like you did in TextMate. Next, do a View Source for the file.
    – Arjan
    Mar 23, 2010 at 15:36
  • 1
    @Arjan not just a Mac. Any *nix based system, whether linux or *nix. And anyhow, the xxd -p command shows the actual bytes so is much better.
    – barlop
    May 2, 2016 at 20:56

3 Answers 3

4

Well, I'm using NotePad++ and I can't see that at all! What is the best text file reader for this kind of problems?

The problem is, a ‘good’ text editor should be able to load all text encodings transparently — even stupid broken ones like UTF-8-plus-BOM — which would prevent you from seeing the problem. Sure, a good text editor should save UTF-8 without the bogus-BOM, or at least give you the option to do so, but you won't know to re-save it if you don't see the faux-BOM there.

The reason you see the three high-bytes at the start of the file in TextMate is actually because TextMate has got it wrong and guessed the encoding as Latin-1 instead of UTF-8. This presumably reproduces the behaviour of the service you're sending to which don't know about Unicode, but it's not really a desirable feature in itself. It's also why the æ​s and ø​s haven't come out.

If you want to see every byte in the file explicitly, what you want isn't really a text editor, but a hex editor. There are lots to choose from, eg. xvi32 on Windows.

And then fix your application to not produce bogus BOMs; they have no place in a UTF-8 file anyway, never mind the problems it causes to non-Unicode applications. [I don't know what the application is written in, but a common cause of unwanted BOMs is using .NET's Encoding.UTF8 encoding. A new UTF8Encoding(false) would be preferable.]

Whether the service you're sending to wants UTF-8 or some other encoding is in any case something you'll have to ask the operators of that service. If they're already describing the high-bytes for æ et al in your file as inherently ‘invalid’, you may be facing a situation where they don't support any non-ASCII characters at all, in which case you'll have to consider transliterating characters appropriately for the target language, eg. æ->ae.

3
  • 2
    I don't think TextMate has got it wrong. It looks more like the text encoding was changed manually to Windows Latin-1.
    – Arko
    Mar 23, 2010 at 14:11
  • 2
    bogus BOMs; they have no place in a UTF-8 file anyway -- that's true in most data streams and for XML/HTML files (which provide other means to specify the encoding), and for most other files, but not necessarily for all files. If an application needs to scan the full file to guess if something is UTF-8, then a BOM could be quite useful. Even though it won't define the actual byte order, it does indicate it's UTF-8. And in this very question, the presence is actually quite nice, as it shows it's been explicitly outputted as UTF-8, not the required ISO 8859-1 or PC850. ;-)
    – Arjan
    Mar 23, 2010 at 14:25
  • 1
    xvi32 is a great help... love not having to second-guess even a "good" text editor when what I want is to see the raw - but not too raw :) - data.
    – Mike M
    Nov 15, 2017 at 23:48
6

An easy way to view this kind of stuff in Windows is to use the "type" command.

I would do something like this:

type filename.txt | more
1
  • 1
    -1 that is a terrible way, it will either interpret the hex as ascii when it isnt, Or it suppoers some things like a unicode LE bom and doesn't show it/. e.g. save a file as unicode thats unicode LE in notepad. See the hex with xxd -p (from cygwin or the xxd with vim7) now do type on that file and it will just show the file's contents, not the unicode 16bit LE BOM hex code, not the fffe at the start And look how it shows a unicode BE BOM hex code feff - not that clear
    – barlop
    May 2, 2016 at 20:50
2

Frhed jumps to my mind...it is a very nice tool. And as Arjan pointed out, you're saving the file as UTF-8 encoded document.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .