This is indeed by design. It's long-standing behaviour of Microsoft/IBM operating systems, that one can also find in 32-bit OS/2, 16-bit OS/2, PC-DOS, MS-DOS, and also DR-DOS (a.k.a. Novell DOS a.k.a. OpenDOS).
The behaviour is that a file's directory entry is only updated, showing the correct last modification timestamp, size, and allocation information in a directory listing, when a system file table entry for the file is closed. A SFT entry is closed when all open handles in all processes that reference it are closed. Programs that are in the middle of downloading "hundreds of MiB" haven't yet closed their open file handles. So their SFT entries are not closed. So the directory entries aren't updated.
If one or more additional handles to a file have been created with
DosDupHandle, the internal buffers for the file are not written to disk, and its directory entry is not updated, until
DosClose has been called for all the handles.
— "Closing Files". OS/2 Warp: Control Program Programming Guide and Reference
. IBM Corporation.
On single-tasking MS/PC/DR-DOS, this is somewhat difficult to observe, although one could observe it under DesqView, Windows 3.x in 386 Enhanced Mode,
TASKMAX, and so forth. On multi-tasking OS/2 one could regularly and easily observe it.
One of the less welcome features of this behaviour was (on FAT volumes) that a dirty shutdown before the file handle had closed would result in
CHKDSK seeing a zero-sized file and truncating everything downloaded thus far.
This was behaviour quite different to UNIX, where one could see a file grow as it was growing with
ls -l because the
ls command looked at the i-node (in-memory and up-to-date) for file metadata rather than at the directory entry (which on native UNIX file systems does not have size and date information). DOS, OS/2, and Windows NT are growing the file, too. Data blocks are being allocated to the file, the SFT entry is keeping track of the file size, structures like the f-node (on HPFS volumes) are being kept up to date, and as you observed the volume's free space is decreasing. One simply cannot see, with
dir or anything else that reports the contents of the directory entry, the file growing.
The situation with Windows NT and NTFS is very similar to the situation with OS/2 and the DOSes. (The nomenclature has changed, which is slightly confusing to the novice. Where DOSes and OS/2 have system file table entries referenced by file handles in individual processes, Windows NT has file objects referenced by handles in individual processes.) It is, ironically, more similar to the older IBM/Microsoft operating systems' behaviours since Windows NT 6.0, where the behaviour was changed. The "new" NTFS behaviour is that directory entries for a file, stored in each of the directories that has a link to the file and used when one is listing a directory from Win32 (with
FindNextFile), are only updated when a file object for the file is closed, whilst the metadata attached to the file itself, not visible from a directory listing (and only accessible via calls such as
GetFileInformationByHandle), are updated as the file is.
To ameliorate the dirty shutdown risk and to see the file size increasing as the file was written to, one employed a simple trick. You too can still employ it.
A file's directory entry is updated whenever a system file table entry for it is closed, and the SFT entry is closed when the last open file handle in any process that references it is closed. That doesn't have to be a handle opened or used by the process writing to the file. One can use anything that opens the file (creating a separate SFT entry) and closes it again. It can be something as simple as the
type command running in another console/session:
type file > nul
Ironically, Total Commander's Lister would have done exactly that, every time that you used it to look at the file contents as it was being written. If you'd spotted that, you'd have been here asking why Total Commander's Lister magically "fixed" your directory listing. And the answer would have been that there's nothing special about Total Commander. Anything separately opening and closing the file would have had the same effect.
For the sake of completeness, note that there were system calls that a downloading program could use to update the directory entry as one went along. OS/2 has
DosBufReset for example. However, these system calls also flush the disc cache, which is not necessarily what one wants. Opening the file a second time and then closing that file handle does not flush the disc cache.