Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

In a project with thousands of files, I wanted to compare total lines-of-code, to lines-of-code just in PHP (discarding CSS, JS, etc).

When I run

find . -type f | xargs wc -l

I the total on the last line is lower than when I run

find -E . -regex '.+\.(php|inc)' -type f | xargs wc -l

Considering the second find has to be a smaller list of files than (is a strict subset of) the first find, how could wc report a higher total in the second case?

share|improve this question
add comment

3 Answers

xargs can only pass ARG_MAX bytes of arguments to wc.

On my Mac, ARG_MAX is smaller than the full file names and relative paths of the entire project's files, so in the first command, xargs dumped the results of the find to wc in two batches, which meant that wc put out two totals, surrounded by thousands of file names. But ARG_MAX happened to be bigger than the second find output, so the second, smaller find all showed in one wc total.

The fix was to use these commands, so I could see all the totals without the (boring) individual file count lines:

find . -type f | xargs wc -l | grep total
find -E . -regex '.+\.(php|inc)' -type f | xargs wc -l | grep total

Then sum the several "total" rows by hand.

share|improve this answer
    
Or do find . -type f | xargs cat | wc -l –  Alan Shutko Jan 9 '13 at 22:28
add comment

There are many ways to do this and xargs is not the best. Here are a couple:

  1. The simplest, is to cat each of the files found by find and count the lines. Careful, this only works of your file names have no spaces or strange characters:

    find . -type f | while read n; do cat $n; done | wc -l
    find -E . -regex '.+\.(php|inc)' -type f | while read n; do cat $n; done | wc -l 
    

    If your file names are likely to contain weird characters (slashes, spaces etc) use this instead:

    find . -type f | while IFS= read -r n; do cat $n; done | wc -l
    find -E . -regex '.+\.(php|inc)' -type f | while IFS= read -r n; do cat $n; done | wc -l 
    
  2. A better way is to use find's -exec option:

    find . -name "*.pep" -exec cat {} \; | wc
    find -E . -regex '.+\.(php|inc)' -type f -exec cat {} \; | wc
    
share|improve this answer
    
Given that Jeremy’s actual problem was that he had too many filenames to fit on a command line, your first approach is unlikely to work. –  Scott Jan 9 '13 at 20:02
    
@Scott yup, good point, answer updated. –  terdon Jan 9 '13 at 20:28
    
Can even IFS= read –r n handle filenames containing newlines? Maybe the ideal solution is a hybrid: find … –print0 | xargs –0 cat | wc. –  Scott Jan 9 '13 at 21:30
    
Darn it @Scott, good point again :). I thought that it could handle newlines because of a comment from a very experienced user on UL.SE which, I now realize, I must have misunderstood. -print0 also fails on newlines. I removed the reference to newlines to avoid confusion. –  terdon Jan 9 '13 at 21:50
1  
Yeah, but that’s not what I suggested. The –0 option of xargs and the –print0 predicate of find were designed to work hand-in-hand. –  Scott Jan 9 '13 at 22:21
show 2 more comments

Use awk to sum up the various "total" numbers of the wc -l outputs!

(Note: wc -l returns the number of newline characters, i. e. final "lines" without a final \n character will not be counted - as it is the case with awk or sed.)

export LC_ALL=C
find . -type f -print0 | xargs -0 wc -l | 
    awk '/^ *[[:digit:]]+ total$/{ total+=$1 }END{print total}'


# xargs alternatives using: find ... -exec <wc|awk|sed> ... '{}' +
#man find | less -p '{} \+'

# wc
find . -type f -exec wc -l '{}' + 2>/dev/null | 
   awk '/^ *[[:digit:]]+ total$/{ total+=$1 }END{print total}'

# awk
find . -type f -exec awk 'END {print NR}' '{}' + 2>/dev/null | 
    awk '{ total+=$1 }END{print total}'

# sed
find . -type f -exec sed -n '$=' '{}' + 2>/dev/null | 
    awk '{ total+=$1 }END{print total}'
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.