I'm using cat to pipe different files into one big file. The number of different files varies, from two files up to ten, but the total size of all files is always the same (a couple of GB).

My problem: Whenever I get to the case where I have a total of six files, the time it takes to concatenate them peaks (i.e significantly more than with five or seven), and I have no idea why.

Anyone has an idea?

The files (all same size)



cat outputTEMP* >> output && rm -f outputTEMP*

Currently, the Machine has to perform some calculations, but I will update later when new measurements are available.

  • What is the exact command line you are using? – innaM Dec 4 '09 at 15:09
  • I added the command line. – brandstaetter Dec 4 '09 at 15:21
  • This is definitely odd. I can't tell you why it acts this way, but perhaps you should file a plain text bug report to bug-coreutils@gnu.org. – Reynolds Dec 4 '09 at 16:33
  • Measure it! And be sure that you are not caching when you measure! – Davide Dec 4 '09 at 16:34

One way to debug this problem is to use strace.

strace -tt -e trace=open,close -o /tmp/strace.cat.log cat apt.list authors.txt >/tmp/t.test
cat /tmp/strace.cat.log 

23:12:08.022588 open("apt.list", O_RDONLY|O_LARGEFILE) = 3
23:12:08.023451 close(3)                = 0
23:12:08.023717 open("authors.txt", O_RDONLY|O_LARGEFILE) = 3
23:12:08.025403 close(3)                = 0

-tt option logs the time stamp of system call to milli-seconds resolution. -e trace=open,close log only open,close API. Try remove them and you will see a very noisy log file.

| improve this answer | |

So Davides comment is spot on. We need two things here, to do an accurate assessment:

  1. assurance caching isn't part of the scenario
  2. actual measurement of the time it's taking.

Assuming you have the disk space I'll describe a test scenario that'll more accurately determine if this is a real issue. If so, the supporting evidence from this approach WILL help the developers to know it's real and be able to reproduce it.

To help with problem isolation let's not do the rm part here at all. let the TEMP files sit around afterward. You can then repeat the tests doing the 'rm' part later, if you wish.

Here's the test scenario:

  • make 9 directories - one for each quantity of files ( 2 3 4 5 6 7 8 9 and 10) - if you don't have space, maybe just do 2, 5, 6, 7, and 10.
  • ensure you are putting DIFFERENT files into each of these directories; NO duplicates anywhere
  • use the time command like this:

    time (cat outputTEMP* >> output)

Capture the real, user, and sys numbers reported for each test you run.

I agree with Reynolds; if this is real, you should definitely email details to bug-coreutils@gnu.org.

| improve this answer | |
  • Another thought: To ensure you're copying the same TOTAL amount of data into the output file. So, if it's 1GB total, in the '2' directory you would have files that were 1/2 GB large, and in the '10' directory you would have files that are 1/10th of a GB large, etc. – pbr Dec 4 '09 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.