87

This phenomenon has been leaving me questions to ask.

Here is the detailed experiment, my OS is Windows 7 x64 SP1:

  • I changed a picture (JPG) file to TXT by simply changing its extension (or one could just choose to open the JPG with notepad, same thing)

It should look like this, oddly looking sequences of texts, and some of them (very rare) are actually meaningful, like in the screenshot below "creator: dg-jpeg v1.0..."

Sample JPG text

  • I disabled wrapping and selected all the text using Ctrl+A (to make sure nothing's missed)
  • I pasted the copied text to another blank TXT file and saved it as JPG, I compared the new file size with the original JPG. All of them (the original JPG, the converted TXT file and the newly created TXT file) are of the exact same size, to bytes.

When I tried to open, Windows would say "Windows Photo Viewer can't open this picture because the file appears to be damaged, corrupted, or is too large".

I even tried to test it using another method: Opened the JPG with notepad, I cut ONE known character from a location easy to remember (like the first character of the 2nd line) then save the file. The viewer would of course display the same message. Then I opened it again and pasted the character to the EXACT location (Notepad remembers its exit state like windows position, wrapping, fonts size...so I have no problem getting this right)

And still the same error. You can try this to get the idea, remember to choose a small picture else Notepad will act like a old rusty man.

What could have been the cause of this phenomenon?

13
  • 4
    Try the fc command. open a cmd prompt and do- C:\blah>fc file1 file2 It is possible for files to be the same size but different. (though usually some random change doesn't tend to leave a file the same size but it easily could). The fc command will be very useful to you in investigating what is happening. You can also use the xxd command, this is in cygwin, and also comes with vim7. xxd -p file1 That will dump the hex of a file. You can compare the hex of the two files with that and fc. Or even open the hex in notepad and flick between the two notepad windows with alt-tab.
    – barlop
    Jul 13, 2014 at 21:18
  • 24
    You are trying to read a binary file with a simple text editor like notepad. It won't be able to read the ANSI encoding correctly and thus it will convert it. When you save it then the file won't be binary anymore and thus the parser can't read the data inside the file. (Lookup the difference between XML based file saving and Binary file saving it's an interesting topic.) If you would try the same experiment with Notepad++ you'll succeed in what you were trying.
    – woutervs
    Jul 14, 2014 at 10:32
  • 1
  • 3
    For the interested: You can edit images in Vim: However, the trick is, that Vim converts the file in the XPM format, which is plain ASCII.
    – Boldewyn
    Jul 14, 2014 at 15:35
  • 4
    Long story short, Notepad modifies your file before displaying it to you. Jul 15, 2014 at 23:21

6 Answers 6

84

Depending on the encoding used to open the file you might see different behaviour. My Windows 7 notepad allows to open a file in ANSI, UTF-8, Unicode or Unicode big endian.

I've tested this issue with a small 2x2 pixel jpeg image created with gimp and opening and saving the image file with ANSI encoding. Opening both the original and the saved image with an hex editor I see that all 00 sequences (two hex digits, NUL control character) have been converted to 20 (space character).

Replacing back in the hex editor all 20 by 00 restores the image format.

I've googled it a bit and I didn't found any references that explain why it does that. Only a reference to a post that warns about it (google cache link, the page is not available).

If you save/open the file as UTF-8 it seems that it still converts NUL characters to spaces but it also increases the resulting file size due to conversions from single-byte characters to UTF-8 multi-byte sequences.

If you save/open the file as Unicode it seems that it still converts NUL characters to spaces but also adds a byte to the beginning of the file, the BOM.

16
  • 22
    0x00 is a string terminator in C strings. They may have replaced them since a text file should not contain them. Notepad is a very old program.
    – Zonder
    Jul 14, 2014 at 9:07
  • 25
    I doubt that notepad.exe is a .NET executable.
    – knittl
    Jul 14, 2014 at 13:15
  • 11
    @Bakuriu A C string most certainly can exist in a file; I can think of numerous file formats that contain them. And the vast majority of apps that ship with Windows apps are native, not .NET. That said, notepad does not write null-terminated strings to files. Jul 14, 2014 at 14:35
  • 4
    @Bakuriu : Windows programs are usually not written in .Net. It's C/C++ and native at the core. One of the .Net applications developed by microsoft was live writer which is now discontinued.
    – JaDogg
    Jul 14, 2014 at 15:58
  • 5
    @SJuan76 Huh? C++ does not define a data type named byte. Perhaps you're thinking of some other language. And the application developers can deal with binary data however they see fit, including the use of C strings if they so choose. As I said before, I can think of numerous binary file formats that contain C strings. Jul 15, 2014 at 14:27
41

Why it fails :

Notepad create spaces (ASCII code 32) character for characters like NUL (ASCII code 0) because Windows API's text box only allows null terminated char * ASCIIZ (character array, pointer). It gets cut off at the first NUL.

That happens because Windows API is mostly written in C language and null terminated strings are one of the common features. Even when modern Windows and Unicode is considered same null terminated strings occur. So notepad simply replace them with space so you can view the complete file.

So when you save the file it is corrupted.

wikipedia-null terminated strings


How to do further research :

You may use a comparator like beyond compare (commercial,trial) to see the character replacement effect. also see other binary compare tools.

hex comparison

Note : (20)16 = (32)10


Reason for notepad acts slowly on large files

It checks each character and replace special characters with spaces. Other software do not do in-memory conversions (at least not primitive as notepad). They just render special characters differently. And they use advanced buffering techniques.


Looking into Notepad.exe (XP 32 bit)

( I'm assuming its still written in C++ or at least use a comparably similar linker )

notepad

I'm using the PEiD tool (which stopped development with introduction of PE+/64 exes)

PEiD can be found bundled in the bin folder of Universal Extractor

I extracted the notepad. ex_ file from the Windows xp iso obviously. Try it out. It's a cab file extract using 7z.

Warning ! Your virus scanner might detect Universal Extractor/PEiD as hack tools or viruses. Don't Trust it don't download it !!


Further info about windows API

credits:Jason C

It's not just the text box; WM_SETTEXT in general provides no parameter for specifying the string length, and strings are always assumed to terminate at null. You could always create a custom text box with a custom message that specified the string length, but Notepad and most other programs reasonably do not. Also the function SetWindowText does not provide a length parameter as well.

3
  • 1
    It is a little strange that you show the property sheet for a Notepad executable bundled with a version of Windows XP, yet judging by the window theme, you're clearly running some version of Windows 8. That would explain why the executable was linked with version 7.1 of the toolset—that's what they used to compile Windows XP and associated utilities. The Windows 8 version of Notepad will undoubtedly be compiled with a newer version of the SDK tools. Jul 17, 2014 at 9:27
  • 2
    It's not just the text box; WM_SETTEXT in general provides no parameter for specifying the string length, and strings are always assumed to terminate at null. You could always create a custom text box with a custom message that specified the string length, but Notepad and most other programs reasonably do not.
    – Jason C
    Jul 19, 2014 at 14:55
  • @BhathiyaPerera Because I'm satisfied with the level of work that I've done by adding info in a comment. You are welcome to improve your answer with that information if you'd like.
    – Jason C
    Jul 19, 2014 at 16:19
28

Notepad does not preserve all special / extended characters exactly as they are. I don't have a reference for this behaviour immediately at hand but have found this to be the case for example with UNIX-style end of line LF which Notepad will convert into CRLF and null (0x00) which it will ignore. In a binary file such as a JPG there are liable to be random occurrences of the character(s) that Notepad does not preserve. Try your experiment with a HEX-aware editor and it should work then. I'll update my answer if I find a good reference and once I've tested a HEX editor.

Update: I tried a few well known programmers editors but only one of them worked right off the bat, HxD by Maël Hörz. I never used HxD before but found it thanks to an answer to this Stack article, A hex viewer / editor plugin for Notepad++.

The other editors that didn't work after a few minutes effort were Notepad++, Notepad2 and UltraEdit (v17.3, older version). A couple of these had problems with the copy / paste of the first few bytes, the JPEG file signature magic number FF D8 FF. Maybe they would work with a little more fiddling than I have time for at present.

2
  • Sublime Text (2/3) automatically opens a binary file by showing it in hex format. As an example, the start of JPEG file by just clicking "open": puu.sh/aaAVx/bd08dab46e.png
    – tomsmeding
    Jul 14, 2014 at 7:11
  • 3
    Actually, more often than notepad will convert LF to CRLF, it will leave the LF the way it is and display the text as if there was no line break at all!
    – Moshe Katz
    Jul 15, 2014 at 3:18
6

You used to be able to do this with Write back in the day. It was a standard program in Windows 3.1 but I can't remember if Windows 95 included it. Write would allow binary safe editing of any file it could open (probably very limited file size). Notepad is definitely not binary safe (the text remains the same but the actual bytes of non-text characters [e.g. control codes] may change) which is why your JPG example is not working. Try getting a copy of Write (and very old Windows) and try your experiment again!

According to Wikipedia's "Windows Write" article Write was included up to Windows NT 3.5. It was replaced by Wordpad in Windows 95 onwards. write.exe was still present in the Windows directory but was simply a wrapper for opening Wordpad.

5

I think it's not that much a problem of encoding but also of character set. JPG format is basically a byte stream. Thus allowing non-printable characters like NUL, ETX, STX, SOH, DLE, etc.

Microsoft Notepad can't display those non-printable characters. It may display placeholders of some kind like a space for a null-character. So opening the file with Notepad doesn't show the actual content but the content decoded by the selected encoding (utf-8, utf-16, etc) and displayed by a certain character set (unicode, ascii, etc) excluding the non-printable characters.

When selecting all the displayed text and copying the text to the clipboard, you only copy the printable characters including the placeholders. Thus automatically converting null-characters to spaces and ignoring other non-printable characters entirely.

So basically you just lose content doing it this way. If you use a hex-editor instead, it will copy all the content entirely.


Update: Bhathiya Pereras answer is right: https://superuser.com/a/782885/322784 Non-printable characters aren't ignored when copying text to clipboard.

4
  • Every file is "basically a byte stream".
    – Jason C
    Jul 19, 2014 at 15:01
  • 1
    @JasonC I would disagree. While every file can be read as a byte stream. Structured files like XML files are not readable as a stream of data. The content would not be valid until the end of the file has been read. A cut in half jpg is still valid and can be displayed. It's just missing half the picture.
    – sbecker
    Jul 22, 2014 at 7:49
  • There isn't really room for disagreement on that. :) XML is a stream of bytes like anything else, and XML (along with character encoding) defines a format for those bytes. It is certainly readable as a stream of data. Open it in a hex editor, for example. That stream of data just happens to be parseable as XML.
    – Jason C
    Jul 29, 2014 at 2:59
  • @JasonC Can't argue with that actually. :) Touché!
    – sbecker
    Jul 29, 2014 at 12:27
2

The JPEG file contains non text data except for some fields, basically any byte values between 0 and 255 will be found, especially in the area representing the encoded compressed image that contains nearly pseudorandom data.

But Notepad will treat the data as ANSI text by default, so it will do various things that will alter the original data, as:

  • replace bytes mapping special / undefined / forbidden characters as they does not makes sense for a valid ANSI text

  • re encode null characters, end of line and end of file sequences to Windows/DOS conventions

Which means if you edit and save the data as text it will change the jpeg in the best case, and make it unusable in the worst.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .