6

I have tested on CygWin doing "ls -R" with a large directory and sending the output to a Test.txt file, so the process takes about 30 seconds to complete; the file is created, but it is empty; it will only be filled when the "ls -R" command finishes, and I need to see the file content meanwhile new data is entering.

The problem is supposed to come from the buffering that the operating system makes before writing into the file.

This is what I have tested in order to write into the file without buffering:

  • The unbuffer command from the expect package: where is it? Modern expect version has no such unbuffer command (see below):
    ./unbuffer -p ls /cygdrive/y/Repositorio/ -R > Test.txt
    ./unbuffer ls /cygdrive/y/Repositorio/ -R | tee Test.txt
    ./unbuffer -p ls /cygdrive/y/Repositorio/ -R | tee Test.txt
    ./unbuffer ls /cygdrive/y/Repositorio/ -R | ./unbuffer -p egrep "" | tee Test.txt
    ./unbuffer -p ls /cygdrive/y/Repositorio/ -R | ./unbuffer -p egrep "" | tee Test.txt
  • Installing the Expect package and naming "unbuffer" this script:
    #!/usr/bin/expect --
    # Description: unbuffer stdout of a program
    # Author: Don Libes, NIST
    eval spawn -noecho $argv
    set timeout -1
    expect
  • Installing the Expect package and naming "unbuffer" this other script:
    #!/usr/bin/expect --
    # Description: unbuffer stdout of a program
    # Author: Don Libes, NIST
    if {[string compare [lindex $argv 0] "-p"] == 0} {
        # pipeline
        set stty_init "-echo"
        eval spawn -noecho [lrange $argv 1 end]
        close_on_eof -i $user_spawn_id 0
        interact {
        eof {
            # flush remaining output from child
            expect -timeout 1 -re .+
            return
        }
        }
    } else {
        set stty_init "-opost"
        set timeout -1
        eval spawn -noecho $argv
        expect
    }
  • The script command:
    script -c "ls /cygdrive/y/Repositorio/ -R" | tee Test.txt
    script -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | tee Test.txt
    script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | tee Test.txt
    script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | grep "" --line-buffered | tee Test.txt
    script -q -c "ls /cygdrive/y/Repositorio/ -R" /dev/null | egrep "" --line-buffered | tee Test.txt
  • The egrep command with unbuffering option:
    cat BigFile.txt | egrep "" --line-buffered | tee Test.txt
  • The stdbuf: it does not exist on modern CygWin, or so I think.

  • Another methods:

    ls /cygdrive/y/Repositorio/ -R 1>&2 | tee Test.txt
    ls /cygdrive/y/Repositorio/ -R 1>&2 |& tee Test.txt

Results are always the same: the output file Test.txt is only filled at the end of the "ls -R" command.

Any more ideas, please?

  • 1
    I'm not sure this is solely due to buffering. Buffers nowadays are usually 64 kb and 30 seconds is a long time; I'd expect ls to fill 64 kb faster than that. If this is still an issue for you, could you tell us the size of the resulting Test.txt? – André Chalella Jun 15 '15 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.