2

I am executing a script via ssh in this way:

 ssh user@host 'bash -s' < ./script.sh

the problem is that, sometimes, the output that I get is not correct, the lines are mixed.

In my case the script executes notmuch new, the normal output is something like:

...
Note: Ignoring non-mail file: foobar
Note: Ignoring non-mail file: foobar
Note: Ignoring non-mail file: foobar
Processed 93 total files in almost no time.
No new mail.

but sometimes the output is something like:

...
Note: Ignoring non-mail file: foobar
Note: Ignoring non-mail file: foobar
Processed 93 total files in almost no time.
No new mail.
Note: Ignoring non-mail file: foobar
Note: Ignoring non-mail file: foobar

and for sure this is not the real output from notmuch new, the command ends with No new mail but it's like it gets the output via ssh not line by line.

Why this happen?

  • 1
    My guess: some of the output is to stdout, some other output is to stderr. If going non interactive, buffering might differ, thus changing the order. – A.B Feb 4 '18 at 15:48
5

Buffering. If we search the source code for notmuch

$ find . -name \*.c -exec grep 'Ignoring non-mail file' {} +
./notmuch-new.c:    fprintf (stderr, "Note: Ignoring non-mail file: %s\n", filename);
$ find . -name \*.c -exec grep 'No new mail' {} +
./notmuch-new.c:    printf ("No new mail.");
$ 

Some of those messages use standard error (which is unbuffered by default) and some use standard out (which is line- or block-buffered by default, depending on whether standard out is to a terminal or to a file). This behavior comes from the standard C library, see setvbuf(3) for details. Thus stderr messages get written immediately, while the printf calls to stdout will show up...well, it depends.

Buffering is typically configured by each application individually, though maybe can be monkeyed around with using utilities such as stdbuf (though some consider the LD_PRELOAD tricks used by stdbuf to be quite horrible...).

The difference is easy to reproduce locally; for example writing to the terminal (line-based buffering for stdout):

$ perl -E 'for (1..4) { say "out"; warn "err\n" }'
out
err
out
err
out
err
out
err
$ 

while instead if that exact same code is instead redirected to a file (block-based buffering for stdout):

$ perl -E 'for (1..4) { say "out"; warn "err\n" }' >x 2>&1
$ cat x
err
err
err
err
out
out
out
out
$ 

ssh adds an additional layer of complication, as one may also have to figure out how it is collecting, buffering, and sending bytes, in addition to notmuch and then what ssh is wired up to on the client system...

  • I understand your point but why this behavior is shown only executing the command remotely? – res1 Feb 4 '18 at 17:25
  • it's easy to show a problem locally, in addition to the complications that ssh adds. – thrig Feb 4 '18 at 17:39
  • P.S. things can go even worse if you try concurrent reads/writes to the same socket - unix.stackexchange.com/questions/364396/… - as in possible data corruption – thrig Feb 4 '18 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.