write-to-newfile vs overwriting: performance issue

Question

I observed huge difference between following two scenarios for copying a file:

1. Copying a file into a new file.
2. Copying a file in an existing file and overwriting it.

I expect both operations take the same amount of time to complete. But the first scenario is much faster in practice. I tried this on two different file systems, getting the same result. Any idea why the first scenario is much faster?

Example in Linux:

$ dd bs=1024 count=1000000 if=/dev/zero of=dummyfile.txt
1000000+0 records in
1000000+0 records out
1024000000 bytes (1.0 GB) copied, 7.45639 s, 137 MB/s
$ while [ "1" == "1" ]; do time cp dummyfile.txt dummyfile2.txt ; rm dummyfile2.txt ; done

real    0m0.850s
user    0m0.003s
sys     0m0.847s

real    0m0.778s
user    0m0.000s
sys     0m0.776s

real    0m0.775s
user    0m0.004s
sys     0m0.772s

real    0m0.775s
user    0m0.003s
sys     0m0.770s

real    0m0.776s
user    0m0.008s
sys     0m0.766s
^C
$ rm dummyfile2.txt -f
$ while [ "1" == "1" ]; do time cp dummyfile.txt dummyfile2.txt ;  done

real    0m0.839s
user    0m0.003s
sys     0m0.834s

real    0m6.056s
user    0m0.005s
sys     0m1.683s

real    0m6.614s
user    0m0.002s
sys     0m1.405s

real    0m6.858s
user    0m0.003s
sys     0m1.436s

EDIT: test is performed on an SSD disk. I observed the same trend on HDD, but the gap is lower than SDD (2-3X). Following page explains why SSDs are much slower than HDDs in overwrite:

Trim (computing) Wikipedia page

Answer 1

Actually the physical copying takes the same time. But the ext4 filesystem driver close() waits before data are really written when the operation is done on an existing i-node and it does not wait for write operation if it is done on a new node. I did some experiments and realized that it is the ext4 feature. I have not seen such a difference with copying on btrfs, zfs, ext3, ext.

How can I say that the close is the time consuming operation? strace provides the information:

$ strace -tt -T cp bigfile newfile
...
14:36:41.985437 open("bigfile", O_RDONLY)   = 3 <0.000009>
14:36:41.985466 fstat(3, {st_mode=S_IFREG|0664, st_size=647608649, ...}) = 0 <0.000007>
14:36:41.985495 open("newfile", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4 <0.000086>
14:36:41.985602 fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 <0.000007>
14:36:41.985633 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000008>
... a lot of reads and writes
14:36:43.584223 close(4)                = 0 <0.000009>
14:36:43.584248 close(3)                = 0 <0.000008>
...
$ strace -tt -T cp bigfile existingfile
...
14:36:52.393034 open("bigfile", O_RDONLY)   = 3 <0.000010>
14:36:52.393071 fstat(3, {st_mode=S_IFREG|0664, st_size=647608649, ...}) = 0 <0.000009>
14:36:52.393104 open("existingfile", O_WRONLY|O_TRUNC) = 4 <0.097058>
14:36:52.490211 fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 <0.000007>
14:36:52.490278 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000009>
... a lot of reads and writes
14:36:54.047408 close(4)                = 0 <5.346015>
14:36:59.393466 close(3)                = 0 <0.000011>
...

Note that the close(4) takes more than 5 seconds whereas in case of copying to the existing file and it is immediate when a new file is created.

I run iostat to check what the system doing. The read of the bigfile was executed before the test to avoid file system cache effect.

# iostat sda 1 100
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.00         0.00         0.00          0          0
sda               0.00         0.00         0.00          0          0
--------------- copy to a new file starts here -----------------------
sda              24.00         0.00      8340.00          0       8340
sda             174.00         8.00     86596.00          8      86596
sda             170.00         0.00     86020.00          0      86020
--------------- copy to a new file finishes here ---------------------
sda             177.00         4.00     90112.00          4      90112
sda             176.00         4.00     89600.00          4      89600
sda             166.00         0.00     84992.00          0      84992
sda             161.00         4.00     81920.00          4      81920
sda             157.00         0.00     78888.00          0      78888
sda              52.00         0.00     26224.00          0      26224
sda               0.00         0.00         0.00          0          0
sda               0.00         0.00         0.00          0          0
sda               0.00         0.00         0.00          0          0
--------------- copy to the existing file starts here ----------------
sda              12.00         0.00      4128.00          0       4128
sda             172.00         4.00     87040.00          4      87040
sda             180.00         4.00     91648.00          4      91648
sda             175.00         0.00     89600.00          0      89600
sda             173.00         4.00     88064.00          4      88064
sda             168.00         4.00     83532.00          4      83532
sda             159.00         0.00     81408.00          0      81408
sda             181.00         4.00     92160.00          4      92160
sda              30.00         0.00     14960.00          0      14960
--------------- copy to the existing file finishes here --------------
sda               0.00         0.00         0.00          0          0
sda               3.00         0.00        28.00          0         28

Note the copying to a new file finishes before data is written and the write operation continues some although the cp is done from user point of view.

Answer 2

I believe it's ext4's auto_da_alloc behavior that makes this difference. see ext4 kernel doc

@Zaboj Campula 's answer show us that close() a truncated file would block until written data got "forced to disk". ext4 auto detected this replace-by-truncate use pattern.

likewise, if you rename (mv) your new created file as some existing file, you will find that the rename() syscall will block for some time. that's the replace-by-rename use pattern.

Answer 3

Overwriting takes a seek operation...

The file's permissions must be read in order to know if you can overwrite it(which may or may not be an issue depending on your file system). Also the file must be truncated.

This requires the drive to seek to the position where the file is..

Copying into a new file generates new permissions and uses the closest available free blocks to write and does not need a seek.

Look into "copy-on-write" file systems(Like ZFS) that don't do this..