Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm using cat to pipe different files into one big file. The number of different files varies, from 2 files up to 10, but the total size of all files is always the same (a couple of GB).

My problem: Whenever I get to the case where I have a total of 6 files, the time it takes to concatenate them peaks (i.e significantly more than with 5 or 7), and I have no idea why. Anyone has an idea?


files (all same size):

output
outputTEMP1
outputTEMP2
outputTEMP3
outputTEMP4
outputTEMP5

command:

cat outputTEMP* >> output && rm -f outputTEMP*


Currently, the Machine has to perform some calculations, but I will update later when new measurements are available.

share|improve this question
    
What is the exact command line you are using? –  innaM Dec 4 '09 at 15:09
    
I added the command line. –  brandstaetter Dec 4 '09 at 15:21
    
This is definitely odd. I can't tell you why it acts this way, but perhaps you should file a plain text bug report to bug-coreutils@gnu.org. –  Reynolds Dec 4 '09 at 16:33
    
Measure it! And be sure that you are not caching when you measure! –  Davide Dec 4 '09 at 16:34
add comment

2 Answers 2

So Davides comment is spot on. We need two things here, to do an accurate assessment:

  1. assurance caching isn't part of the scenario
  2. actual measurement of the time it's taking.

Assuming you have the disk space I'll describe a test scenario that'll more accurately determine if this is a real issue. If so, the supporting evidence from this approach WILL help the developers to know it's real and be able to reproduce it.

To help with problem isolation let's not do the rm part here at all. let the TEMP files sit around afterward. You can then repeat the tests doing the 'rm' part later, if you wish.

Here's the test scenario:

  • make 9 directories - one for each quantity of files ( 2 3 4 5 6 7 8 9 and 10) - if you don't have space, maybe just do 2, 5, 6, 7, and 10.
  • ensure you are putting DIFFERENT files into each of these directories; NO duplicates anywhere
  • use the time command like this:

    time (cat outputTEMP* >> output)

Capture the real, user, and sys numbers reported for each test you run.

I agree with Reynolds; if this is real, you should definitely email details to bug-coreutils@gnu.org.

share|improve this answer
    
Another thought: To ensure you're copying the same TOTAL amount of data into the output file. So, if it's 1GB total, in the '2' directory you would have files that were 1/2 GB large, and in the '10' directory you would have files that are 1/10th of a GB large, etc. –  pbr Dec 4 '09 at 18:11
add comment

One way to debug this problem is to use strace.

strace -tt -e trace=open,close -o /tmp/strace.cat.log cat apt.list authors.txt >/tmp/t.test
cat /tmp/strace.cat.log 

23:12:08.022588 open("apt.list", O_RDONLY|O_LARGEFILE) = 3
23:12:08.023451 close(3)                = 0
23:12:08.023717 open("authors.txt", O_RDONLY|O_LARGEFILE) = 3
23:12:08.025403 close(3)                = 0

-tt option logs the time stamp of system call to milli-seconds resolution. -e trace=open,close log only open,close API. Try remove them and you will see a very noisy log file.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.