Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've created a text file from an application that I developed.

When I send the text file to a SYSTEM Validation, they (3rd Party System) say that the file is invalid and that the file contains 3 characters in the beginning of the file that are not allowed as well special characters are not correct.

They also say I need to use either ISO 8859-1 or PC850

Well, I'm using Notepad++ and I can't see that at all! What is the best text file reader for this kind of problems?

alt text


EDITED

I also have a Mac and just thought I remembered opening in TextMate ... WOW! Now I know what they are talking about!

How can I have the same in Windows?

alt text

share|improve this question
    
As an aside: you're looking at an UTF Byte Order Mark (BOM) of a UTF-8 encoded document. If an application has other means to tell it's UTF-8, then such BOM is not required for UTF-8 encodes content, but a good editor knows how to handle it. See unicode.org/faq/utf_bom.html#bom2 –  Arjan Mar 23 '10 at 11:47
    
(On a Mac, also see the file command, to tell if a BOM is present.) –  Arjan Mar 23 '10 at 11:47
    
By the way, a quick way to enforce a wrong encoding on any computer: open the file in a browser and explicitly set the encoding to ISO-8895-1 just like you did in TextMate. Next, do a View Source for the file. –  Arjan Mar 23 '10 at 15:36
add comment

3 Answers

up vote 2 down vote accepted

Well, I'm using NotePad++ and I can't see that at all! What is the best text file reader for this kind of problems?

The problem is, a ‘good’ text editor should be able to load all text encodings transparently — even stupid broken ones like UTF-8-plus-BOM — which would prevent you from seeing the problem. Sure, a good text editor should save UTF-8 without the bogus-BOM, or at least give you the option to do so, but you won't know to re-save it if you don't see the faux-BOM there.

The reason you see the three high-bytes at the start of the file in TextMate is actually because TextMate has got it wrong and guessed the encoding as Latin-1 instead of UTF-8. This presumably reproduces the behaviour of the service you're sending to which don't know about Unicode, but it's not really a desirable feature in itself. It's also why the æ​s and ø​s haven't come out.

If you want to see every byte in the file explicitly, what you want isn't really a text editor, but a hex editor. There are lots to choose from, eg. xvi32 on Windows.

And then fix your application to not produce bogus BOMs; they have no place in a UTF-8 file anyway, never mind the problems it causes to non-Unicode applications. [I don't know what the application is written in, but a common cause of unwanted BOMs is using .NET's Encoding.UTF8 encoding. A new UTF8Encoding(false) would be preferable.]

Whether the service you're sending to wants UTF-8 or some other encoding is in any case something you'll have to ask the operators of that service. If they're already describing the high-bytes for æ et al in your file as inherently ‘invalid’, you may be facing a situation where they don't support any non-ASCII characters at all, in which case you'll have to consider transliterating characters appropriately for the target language, eg. æ->ae.

share|improve this answer
1  
I don't think TextMate has got it wrong. It looks more like the text encoding was changed manually to Windows Latin-1. –  Arko Mar 23 '10 at 14:11
    
bogus BOMs; they have no place in a UTF-8 file anyway -- that's true in most data streams and for XML/HTML files (which provide other means to specify the encoding), and for most other files, but not necessarily for all files. If an application needs to scan the full file to guess if something is UTF-8, then a BOM could be quite useful. Even though it won't define the actual byte order, it does indicate it's UTF-8. And in this very question, the presence is actually quite nice, as it shows it's been explicitly outputted as UTF-8, not the required ISO 8859-1 or PC850. ;-) –  Arjan Mar 23 '10 at 14:25
add comment

An easy way to view this kind of stuff in Windows is to use the "type" command.

I would do something like this:

type filename.txt | more
share|improve this answer
add comment

Frhed jumps to my mind...it is a very nice tool. And as Arjan pointed out, you're saving the file as UTF-8 encoded document.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.