Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Occasionally I come across programs that write files that are "0 byte" long during writes, according to the file-system, but the write shrinks the drive's reported free space. Example programs are wget downloading some big file or ffmpeg converting a video.

The write takes long enough, eg. minutes to write hundreds of megabytes, due to some encoding, downloading, or other slower operation. This time I can see that the free disk space shrinks, according to the file system, however the 0 byte file remains 0 byte.

Usually if I abort the operation, the file suddenly gets the 'missing' bytes reported and suddenly grows to the 'right' size.

Also, if I peek inside the file with Total Commander's Lister, I can see that there is actually content in the file.

I wondered, where are these bytes? There is no ".bak" like file next to the 0 byte file. My tmp directory is on another drive. The program's memory usage does not grow, according to Task Manager's Memory (private working set). And also, Lister can display the content!

My method of checking the file system sizes is via Total Commander's displayed infos. I refresh the directory listing regularly which updates also the free space counter. I've also tested with dir, gaves the same result. I'm on NTFS.

Why are the reported size not updated? Where is the data stored that is inside them?

share|improve this question
    
if this is by design, could you please add some reference to a specification in your answer? thanks in advance! reasons of design considerations are also appreciated! –  naxa Mar 12 at 7:25

1 Answer 1

up vote 3 down vote accepted

This is indeed by design. It's long-standing behaviour of Microsoft/IBM operating systems, that one can also find in 32-bit OS/2, 16-bit OS/2, PC-DOS, MS-DOS, and also DR-DOS (a.k.a. Novell DOS a.k.a. OpenDOS).

The behaviour is that a file's directory entry is only updated, showing the correct last modification timestamp, size, and allocation information in a directory listing, when a system file table entry for the file is closed. A SFT entry is closed when all open handles in all processes that reference it are closed. Programs that are in the middle of downloading "hundreds of MiB" haven't yet closed their open file handles. So their SFT entries are not closed. So the directory entries aren't updated.

If one or more additional handles to a file have been created with DosDupHandle, the internal buffers for the file are not written to disk, and its directory entry is not updated, until DosClose has been called for all the handles.
— "Closing Files". OS/2 Warp: Control Program Programming Guide and Reference. IBM Corporation.

On single-tasking MS/PC/DR-DOS, this is somewhat difficult to observe, although one could observe it under DesqView, Windows 3.x in 386 Enhanced Mode, TASKMAX, and so forth. On multi-tasking OS/2 one could regularly and easily observe it.

One of the less welcome features of this behaviour was (on FAT volumes) that a dirty shutdown before the file handle had closed would result in CHKDSK seeing a zero-sized file and truncating everything downloaded thus far.

This was behaviour quite different to UNIX, where one could see a file grow as it was growing with ls -l because the ls command looked at the i-node (in-memory and up-to-date) for file metadata rather than at the directory entry (which on native UNIX file systems does not have size and date information). DOS, OS/2, and Windows NT are growing the file, too. Data blocks are being allocated to the file, the SFT entry is keeping track of the file size, structures like the f-node (on HPFS volumes) are being kept up to date, and as you observed the volume's free space is decreasing. One simply cannot see, with dir or anything else that reports the contents of the directory entry, the file growing.

The situation with Windows NT and NTFS is very similar to the situation with OS/2 and the DOSes. (The nomenclature has changed, which is slightly confusing to the novice. Where DOSes and OS/2 have system file table entries referenced by file handles in individual processes, Windows NT has file objects referenced by handles in individual processes.) It is, ironically, more similar to the older IBM/Microsoft operating systems' behaviours since Windows NT 6.0, where the behaviour was changed. The "new" NTFS behaviour is that directory entries for a file, stored in each of the directories that has a link to the file and used when one is listing a directory from Win32 (with FindFirstFile/FindNextFile), are only updated when a file object for the file is closed, whilst the metadata attached to the file itself, not visible from a directory listing (and only accessible via calls such as GetFileInformationByHandle), are updated as the file is.

To ameliorate the dirty shutdown risk and to see the file size increasing as the file was written to, one employed a simple trick. You too can still employ it.

A file's directory entry is updated whenever a system file table entry for it is closed, and the SFT entry is closed when the last open file handle in any process that references it is closed. That doesn't have to be a handle opened or used by the process writing to the file. One can use anything that opens the file (creating a separate SFT entry) and closes it again. It can be something as simple as the type command running in another console/session:

type file > nul

Ironically, Total Commander's Lister would have done exactly that, every time that you used it to look at the file contents as it was being written. If you'd spotted that, you'd have been here asking why Total Commander's Lister magically "fixed" your directory listing. And the answer would have been that there's nothing special about Total Commander. Anything separately opening and closing the file would have had the same effect.

For the sake of completeness, note that there were system calls that a downloading program could use to update the directory entry as one went along. OS/2 has DosBufReset for example. However, these system calls also flush the disc cache, which is not necessarily what one wants. Opening the file a second time and then closing that file handle does not flush the disc cache.

share|improve this answer
    
Thanks for the great answer! I have a little confusion regarding A SFT entry is closed when all open handles in all processes vs. SFT entry is closed when the last open file handle in any process that references it is closed. If I get it correctly, this means that opening a file will create a new SFT entry, however after this, potentially, multiple processes can obtain multiple handles to that SFT entry; furthermore that a single file can have multiple SFT entries. Is this correct? –  naxa Mar 14 at 10:45
    
Yes. Consider spawning a new process. The SFT stays the same. The new process, however, has its own table of file handles (the process' file handle table or job file table) which reference the same SFT entries as the file handles in the parent process do. Only when all of the process file handles (in all of the processes) go away can the SFT entry that they reference go away. Multiple SFT entries can reference the same file. Similarly, opening the same file multiple times by calling the Win32 CreateFile function multiple times creates multiple file objects all for one single file. –  JdeBP Mar 14 at 11:17

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.