5

Suppose I want to pipe the output of some command into a file. Is there any difference, however small or nuanced, between simply...well, piping it to a file like

some-command > file

and piping it through dd like

some-command | dd $MAYBE_SOME_OPTIONS_LIKE_BS of=file

This question is primarily to satisfy my curiosity, and unless the latter form has some non-negligible benefit or use case, in practice I would spare my fingers the exercise and go for the former.

0

2 Answers 2

7

Direct access to file

some-command > file
some-command | dd $MAYBE_SOME_OPTIONS_LIKE_BS of=file

In the first case some-command knows it writes to file, so it can retrieve metadata (like mtime or size) for whatever reason. Such metadata have useful meaning for regular files but for a unnamed pipe they are rather meaningless.

some-command may even be interested in opening what its stdout points to for reading, which is rather safe for regular files and obviously not the right thing to do in your second case.

But these scenarios are quite rare, I think.


Seekable vs non-seekable

The behavior of some-command may depend on whether its stdout is seekable or not. It's usually not the case, although it's possible. A regular file is seekable (you can read/write at any location). A pipe is not seekable.

dd itself is a valid example. Consider some-command being dd if=/dev/zero bs=1 count=1 seek=1. Your two examples become:

dd if=/dev/zero bs=1 count=1 seek=1 > file
dd if=/dev/zero bs=1 count=1 seek=1 | dd of=file

The first command will work, it will leave file with two null bytes in it. In the second command the first dd will complain it cannot seek, it will exit; the second dd will get nothing, write nothing and exit without error.

I admit with seek this is not exactly plain "piping the output of some command into a file". But in general some-command may detect if its stdout is seekable and then stream different output without actually seeking. This shell function kinda does it as a proof of concept:

is-seekable() {
  if </dev/null dd bs=1 count=0 seek=1 conv=notrunc 2>/dev/null; then
    echo seekable
  else
    echo non-seekable
  fi
  }

Try these and examine the file after each one:

is-seekable > file
is-seekable | dd of=file

Knowing all the above, there's a scenario in which I would consider your second command: if some-command indeed behaves differently in two cases (like our is-seekable function) and I want it to "think" it's writing to a pipe but I want the output in a file. Then yes. Although I would rather use cat instead of dd (it probably wouldn't matter). E.g. I may want to get non-seekable from is-seekable and write it to a file:

is-seekable | cat > file

But only if it makes any difference. Otherwise it would be (in)famous "useless use of cat". Or in your case "useless use of dd"; or even "harmful use" if you handle the exit status poorly.


Exit status

Now try these two examples:

false > file
false | dd of=file

If you check $? after the second one, you will discover it's 0 (unless dd failed). Your second case discards the exit status of some-command (usually; e.g. in Bash investigate set -o pipefail, which is not portable).

So in the second case some-command may fail for whatever reason and you (or rather the logic of your script) will not know it. This is enough to avoid unnecessary piping to dd (or unnecessary piping at all).


Permissions

Another nuance: some-command > file makes the (sub)shell open the file before some-command is spawned (exec-ed to); dd of=file opens the file on its own behalf.

There are ways to grant more access rights to dd than to any other "regular" command. In other words there are ways to make plain dd behave more like sudo dd without password. You shouldn't do it and hopefully nobody does, yet it's possible.

What you can do is granting access rights on demand: sudo dd. It can be useful when trying to write to a restricted file. Analyze this:

some-command > restricted_file               # doesn't work
sudo some-command > restricted_file          # doesn't work
some-command | sudo dd > restricted_file     # doesn't work
sudo sh -c 'some_command > restricted_file'  # works, but it runs some_command as root
                                             #  you may not want this
some-command | sudo dd of=restricted_file    # works
some-command | sudo tee restricted_file      # works, more common, possibly with
                                             #  > /dev/null to suppress output to tty

I think this is very close to "non-negligible benefit or use case".


Maybe some options

Obviously $MAYBE_SOME_OPTIONS_LIKE_BS can alter dd behavior. I understand that options like conv=swab are out of scope because you want file to receive same data in both cases.

Still there are few options that can make a difference. Examples:

  • status=progress will print the transfer rate and volume statistics on stderr, when processing each input block;

  • oflag=noatime will not update the file's access timestamp;

  • oflag=nolinks will fail if the file has multiple hard links;

  • and more.

The above examples are GNU extensions, not portable. Portable dd is characterized here.


KISS

Finally there's the KISS principle: keep it stupid simple. Your first case is stupid simple.

1
  • A very detailed answer for a small question! One more thing I'm curious about is the way the file gets allocated (if there is a difference at all). I know on filesystems like ext4, it doesn't really matter, but anywhere it would or could possibly matter, what if I did something like, say, give dd a block size (which is why I hinted to it)? Obviously, if I really wanted to pre-allocate a file, I would use fallocate or something. Aug 13, 2019 at 12:24
1

No user should notice any differences. From a performance standpoint, doubt there will be a difference too. If you think of both scenarios, there's pros and cons but these depend heavily on what you're trying to accomplish.

For example: cat file | dd of=file2 vs cat file > file2. The difference here is, cat will open a file, dd will inherit the stream unless you specify a block size or some sort of writing flag, it'll start writing to the fs stack immediately.

If you don't use any features from dd, you're pretty much just forking a middle man, adding a process, this double fork I assume would have minimal performance loss vs just letting dd handle if=file of=newfile. The advantage, in this case, is, you can seek/skip in dd and well, there's only one process.

If you mean strictly just file redirection vs dd, I'd say there's no difference if you don't supply dd with any flags/parameters. You could use in this case as a middle man buffer, but there are way better alternatives, for example: cat file | buffer | gzip > gzipped content. For example, the above-stacked call, let's say you have a super quick source, and a slow destination, with the buffer in between you can make sure the receiving side always has a busy input queue, and if you're compressing you can make sure to keep your CPU from waiting for IO.

Again, all edge cases, that would require separate questions, but as you see, if we speak strictly from one vs the other one, even if there was a difference, doubt it'd be noticeable except for extreme edge cases.

Hope this helps, this question was quite interesting I have to say, took me a while to think about it.

5
  • +1 for the mention of buffer, because I had forgotten that was a thing. Aug 13, 2019 at 12:24
  • Speaking of which, on a system without buffer, can dd with a block size act as a temporary hacky solution? Aug 13, 2019 at 12:32
  • Actually yes, you should be able to use the block size as a really hacky buffer. I'd suggest against it since it won't really add that much performance and buffer is a better-suited solution. I loved the question lol really made me think a lot. The best way's I've seen this type of piping used is in conjunction with pv to see progress. Also when compressing streams, buffer and pv become super useful.
    – ddemuro
    Aug 13, 2019 at 18:07
  • What makes buffer faster? Some kind of low-level optimization quirk? Aug 13, 2019 at 19:47
  • Correct, dd has more control structures than buffer does. That being said, if anyone can notice the difference, I'll be incredibly shocked. I guess I just prefer the Linux motto of having specific applications for specific usage. Getting used to buffer and dd for their purpose I think is best. Don't forget that with buffer you get really useful parameters to manage exactly that. With dd you're kind of hammering a nail with a screw driver.
    – ddemuro
    Aug 13, 2019 at 19:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .