9

I have tested on CygWin doing "ls -R" with a large directory and sending the output to a Test.txt file, so the process takes about 30 seconds to complete; the file is created, but it is empty; it will only be filled when the "ls -R" command finishes, and I need to see the file content meanwhile new data is entering.

The problem is supposed to come from the buffering that the operating system makes before writing into the file.

This is what I have tested in order to write into the file without buffering:

  • The unbuffer command from the expect package: where is it? Modern expect version has no such unbuffer command (see below):
    ./unbuffer -p ls /cygdrive/y/Repositorio/ -R > Test.txt
    ./unbuffer ls /cygdrive/y/Repositorio/ -R | tee Test.txt
    ./unbuffer -p ls /cygdrive/y/Repositorio/ -R | tee Test.txt
    ./unbuffer ls /cygdrive/y/Repositorio/ -R | ./unbuffer -p egrep "" | tee Test.txt
    ./unbuffer -p ls /cygdrive/y/Repositorio/ -R | ./unbuffer -p egrep "" | tee Test.txt
  • Installing the Expect package and naming "unbuffer" this script:
    #!/usr/bin/expect --
    # Description: unbuffer stdout of a program
    # Author: Don Libes, NIST
    eval spawn -noecho $argv
    set timeout -1
    expect
  • Installing the Expect package and naming "unbuffer" this other script:
    #!/usr/bin/expect --
    # Description: unbuffer stdout of a program
    # Author: Don Libes, NIST
    if {[string compare [lindex $argv 0] "-p"] == 0} {
        # pipeline
        set stty_init "-echo"
        eval spawn -noecho [lrange $argv 1 end]
        close_on_eof -i $user_spawn_id 0
        interact {
        eof {
            # flush remaining output from child
            expect -timeout 1 -re .+
            return
        }
        }
    } else {
        set stty_init "-opost"
        set timeout -1
        eval spawn -noecho $argv
        expect
    }
  • The script command:
    script -c "ls /cygdrive/y/Repositorio/ -R" | tee Test.txt
    script -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | tee Test.txt
    script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | tee Test.txt
    script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | grep "" --line-buffered | tee Test.txt
    script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | egrep "" --line-buffered | tee Test.txt
  • The egrep command with unbuffering option:
    cat BigFile.txt | egrep "" --line-buffered | tee Test.txt
  • The stdbuf: it does not exist on modern CygWin, or so I think.

  • Another methods:

    ls /cygdrive/y/Repositorio/ -R 1>&2 | tee Test.txt
    ls /cygdrive/y/Repositorio/ -R 1>&2 |& tee Test.txt

Results are always the same: the output file Test.txt is only filled at the end of the "ls -R" command.

Any more ideas, please?

1
  • 1
    I'm not sure this is solely due to buffering. Buffers nowadays are usually 64 kb and 30 seconds is a long time; I'd expect ls to fill 64 kb faster than that. If this is still an issue for you, could you tell us the size of the resulting Test.txt? Jun 15, 2015 at 10:02

1 Answer 1

0

Use stdbuf. It is on cygwin in the coreutils package.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .