Why is there a difference between text and binary files in Windows?

Question

In many different programming languages, there are constructs in place to specifically work around the fact that Windows differentiates between text and binary files.

For example, in Ruby:

f = File.open('filename.bin', 'rb')  # read a file in binary mode
f = File.open('filename.txt', 'r')   # read a file in text mode

In Python:

f = open("filename.bin", "rb")  # read a file in binary mode
f = open("filename.txt", "r")   # read a file in text mode

On other operating systems, it seems that there's no difference between a text file and a binary file with the filesystem.

In reality, I'd assume that there's literally no difference between a text and a binary file in reality, as both are simply a collection of bytes. A text file may be easily representable in an editor depending on the encoding whereas a binary file generally won't be, but the underlying representation is the same: a sequence of bytes in a given order.

Why does Windows make this distinction which is seemingly unnecessary?

Answer 1

Why is there a difference between text and binary files in Windows?

Short answer

There isn't.

Long answer

In reality, I'd assume that there's literally no difference between a text and a binary file in reality, as both are simply a collection of bytes. A text file may be easily representable [sic] in an editor depending on the encoding whereas a binary file generally won't be, but the underlying representation is the same: a sequence of bytes in a given order.

As you said, a file is just a bunch of bytes. That’s it. Its contents only take on meaning when interpreted by a program. It is entirely possible for one program to interpret the bytes in a file one way and another program to interpret them another. When you open a “binary” file in a text-editor, it will interpret the bytes as text and display them. If the file is not “plain text”, then the results will likely be gibberish, but the program is still doing its job of interpreting and outputting them.

In many different programming languages, there are constructs in place to specifically work around the fact that Windows differentiates between text and binary files.

Windows does not. What is happening is that most such programming languages evolved on other operating systems like Unix, Linux, etc. and therefore use different line-endings for native plain-text files. It is possible that they also use a different encoding, but it is usually the line-endings that will vary from platform to platform.

Here is a list of common platforms and line-endings:

• Unix, Linux - line-feed
• Windows - carriage-return, line-feed
• Mac (historically) - carriage-return
• (A few, old OSes (e.g., Acorn BBC) - line-feed, carriage return)

Why does Windows make this distinction which is seemingly unnecessary?

Windows is an operating system, it does not distinguish anything itself. The question you should ask is which parts of Windows are distinguishing. In this case, it is the command-prompt that treats text and binary files differently, and even then, it depends on the command being used. For example, the command del foobar.txt is no different than del foobar.bin, however copy a.txt + b.txt c.txt is different from copy /b a.bin + b.bin c.bin Why? Because the Windows command-prompt wants to be helpful and interprets text files as such and copies the lines to the output (adding a newline between files), but copies binary files as is with no interference.

For example, in Ruby:
f = File.open('filename.bin', 'rb') # read a file in binary mode
f = File.open('filename.txt', 'r') # read a file in text mode
In Python:
f = open("filename.bin", "rb") # read a file in binary mode
f = open("filename.txt", "r") # read a file in text mode
On other operating systems, it seems that there's no difference between a text file and a binary file with the filesystem.

Those are all scripting languages, and therefore run from the command-line. When working with text input files, there is usually not much of a problem, but with binary files, you use the binary-mode to avoid the command prompt from pre-processing the file and passing it as raw bytes.

In Linux, when you type or pipe a file, the shell passes all the raw bytes instead of pre-processing it as text like the Windows command-prompt does.

That said, depending on the program and how the input file is passed, then it could easily avoid the pre-processing altogether. For example C:\>pyhton foobar.py baz.bin would pass the name of the input file to the script which would then open it as it likes whereas C:\>type baz.bin | python foobar.py would cause the command-prompt to read the file, then pass each line to the script, which for a binary file is no good.

The different modes simply allow for flexibility and allow you to play it safe and treat files like what you expect them to be.

Answer 2

CP/M (Control Program for Microcomputers) was created by Digital Research in the 1970's. File sizes in CP/M were expressed as a count of 128-byte disk sectors -- in other words, an exact byte count of the file size was not available. This wasn't (as much of) an issue for binary files, since an extra 127 bytes of NULLs (or whatever) typically didn't negatively impact program loading. However, it was problematic for text files, which could be of arbitrary length.

Thus, CP/M distinguished between binary files and text files. By convention, the final byte of a text file was an in-band Control-Z character -- you read the file until you saw ^Z, then stopped. (Binary files didn't require this processing; you just loaded the raw sector count.)

CP/M was very popular, and got hacked on by many people. One of guys who hacked on it was named Tim Paterson, who worked for a small company called Seattle Computer Products. He kluged together a CP/M clone for x86-class hardware called QDOS (Quick and Dirty Operating System), which mimicked a fair amount of CP/M's functional design. Then QDOS was purchased by a spotty college dropout named Bill Gates, who kluged it up even further, mindlessly carrying forward all of its design shortcomings and limitations, and created MS-DOS. And Windows itself started life as a kluge on top of MS-DOS.

Although Microsoft has since learned -- probably by sheer accident -- how to create a filesystem that maintains exact byte counts for files, the distinction between text and binary files remains, even though it no longer serves any useful purpose.

Answer 3

You do realize that python runs on *nix and mac too, right? and that function is the same on those operating systems? It isn't just a Windows thing. Look at the Wikipedia article for binary files - the first line sums it up well enough:

A binary file is a computer file that is not a text file; it may contain any type of data, encoded in binary form for computer storage and processing purposes.

It goes on to state:

Binary files are usually thought of as being a sequence of bytes, which means the binary digits (bits) are grouped in eights. Binary files typically contain bytes that are intended to be interpreted as something other than text characters.

Whereas text files are much simpler:

A text file...is a kind of computer file that is structured as a sequence of lines of electronic text. A text file exists within a computer file system. The end of a text file is often denoted by placing one or more special characters, known as an end-of-file marker, after the last line in a text file

So yes, the answer is that they are both a sequence of bytes, but the method by which they encode the data is very different. A text editor is programmed to read the encoding of a text file and a binary reader is programmed to read the encoding of a binary file. When you call those functions in either python or ruby, you are telling it what encoding to expect from the file so it decodes it properly. This would be the same regardless of what operating system you are on.