Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

In the question How can I compress a file on Linux in-place, without using additional disk space?, one answer proposes to simply use

gzip -c file | dd of=file

I tried it (on Debian Linux), and it does appear to work. I don't quite understand why, however.

Doesn't dd truncate its output file before writing? Wouldn't this "pull out the rug" under gzip, thus taking away the data that gzip wants to read?

Or is there some race condition involved, meaning the command will usually work, but may sometimes fail? Or does it somehow depend on the block sizes which the commands use for I/O?

I know that a process that has opened a file can continue reading it even if another process deletes it (the file will be discarded once the process closes it). Is there a similar mechanism if a file is truncated while some process has it open?

share|improve this question
    
It doesn't work for me: dd bs=1M count=1 if=/dev/zero of=file; gzip -c file|dd of=file; gzip -d < file > orginal; ls -l original rsults in -rw-rw-r-- 1 user user 229376 2012-01-18 11:38 file2 where the file size should be 1M –  artistoex Jan 18 '12 at 10:39
add comment

1 Answer

up vote 4 down vote accepted

Experiment shows that this does not work.

I created a 2-megabyte file from /dev/urandom, then tried the above command on it. Here are the results:

% ls -l
total 41008
-rw-r--r-- 1 kst kst 20971520 2012-01-18 03:47 file
-rw-r--r-- 1 kst kst 20971520 2012-01-18 02:48 orig
% gzip -c file | dd of=file
0+1 records in
0+1 records out
25 bytes (25 B) copied, 0.000118005 s, 212 kB/s
% ls -l
total 20508
-rw-r--r-- 1 kst kst       25 2012-01-18 03:47 file
-rw-r--r-- 1 kst kst 20971520 2012-01-18 02:48 orig
$ 

Obviously a 2-megabyte random file won't compress to 25 bytes, and in fact running gunzip on the compressed file yields an empty file.

I got similar results for a much smaller random file (100 bytes).

So what happened?

In this case, the dd command truncated file to zero bytes before starting to write to it; gzip started reading from the newly empty file and produced 25 bytes of output, which dd then appended to the empty file. (An empty file "compresses" to a non-zero size; it's theoretically impossible for any compressor to make all input smaller).

Other results may be possible, depending on the timing of the gzip, dd, and shell processes, all of which are running in parallel.

There's a race condition because one process, gzip, reads from file, while another parallel process, the shell, writes to it.

It should be possible to implement an in-place file compressor that reads and writes to the same file, using whatever internal buffering is necessary to avoid clobbering data. But I've never heard of anyone actually implementing that, probably because it usually isn't necessary and because if the compressor fails partway through, the file will be permanently corrupted.

share|improve this answer
    
Good answer. Now the interesting question would be: What exactly happens here, and why is the result a file of 25 bytes? –  sleske Jan 18 '12 at 15:13
1  
@sleske: In this case, the dd command truncated file to zero bytes before starting to write to it; gzip started reading from the newly empty file and produced 25 bytes of output, which dd then appended to the empty file. (An empty file "compresses" to a non-zero size; it's theoretically impossible for any compressor to make all input smaller). Other results may be possible, depending on the timing of the gzip, dd, and shell processes, all of which are running in parallel. –  Keith Thompson Jan 18 '12 at 21:23
    
@sleske: I've copied and expanded the information from the above comment into my answer. –  Keith Thompson Mar 7 '12 at 19:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.