It seems that you are hit by ext4 optimizations. Some of these are described in Ext4 Disk Layout documentation.
Every time you grow a file, for example with dd, some place is allocated for the new data. Eventually, the file could be fragmented, i.e. data could be spread in several chunks through the disk. Fragmentation is a well known source of slowness (when reading and writing) and therefore file system implementations often try to avoid it. Fragmentation is way more expensive with spinning disks (the head needs to travel to successive chunks) than with SSD, yet most file system implementations use spinning-disk optimized strategies with SSD.
A combination of third, fourth and fifth "tricks" described in the documentation mentioned above might explain why writing to the subdirectory is faster in your case.
The fifth trick is that the disk volume is cut up into 128MB block groups. […] When a directory is created in the root directory, the inode allocator scans the block groups and puts that directory into the least heavily loaded block group that it can find.
As a result, emzed2
was probably created in an empty block of 128MB.
The fourth trick is that all the inodes in a directory are placed in the same block group as the directory, when feasible.
Consequently, /data/testfile
is created in (probably heavily loaded) root block group, while /data/emzed2/testfile
is created in (probably empty) emzed2
block group.
The third trick […] is that it tries to keep a file's data blocks in the same block group as its inode.
For /data/emzed2/testfile
, the file system will first allocate all data blocks in the emzed2
block group. If this block was initially empty, for the first 128MB, this means no fragmentation at all. For /data/testfile
, the file system will first fill the root block group if it was not filled yet, and then look for other places to store data.
Also, you are growing the file 32 bytes at a time. Fortunately, file systems such as ext4 allocate more data than what you request (by chunks of 8kb in the case of ext4) and tries to delay allocation. You probably would see a similar pattern with bs=8196
, but the speed difference might decrease with larger block sizes.