Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I was trying to collect all of the Message-ID: headers (lines) in a directory with 200K .eml (plain text) files. A bit naively, I said:

find -type f -exec grep -Fi "message-id:" {} \; > messageids.txt

I let it run overnight, since I figured it would take a while to grep through that many files. A bit to my surprise this morning, messageids.txt is 1.7TB and my partition is full. I realize that what must have happened is that grep's own output is being picked up as input, but I wouldn't (and still don't, intuitively) expect it to repeat endlessly. Which means that my understanding of the forces at play isn't as strong as it should be.

Can anyone provide a detailed explanation of how the command above works and why this infinite loop should (I assume) be expected? Thanks.

Update: The way I'd expect it to work is that find finds a list of files, and on each one of them grep is called. So at some point grep is called on messageids.txt. If I were to do this on, say, a sort command, messageids.txt would be created as soon as the command executes (possibly whacking it, if it already existed), but it wouldn't be populated until the command completes. In this case, for the loop to be infinite, the file must be getting populated before the output is complete, but in such a way that the input from grep is perpetually keeping up on it. That's the bit that doesn't behave like I'd expect, and I was hoping for a detailed explanation of how this process chain is executing so I can firm up my Linux fundamentals.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

Every time it finds a line with message-id in it, it soon writes it to messageids.txt. And every time it writes a line with message-id in it to messageids.txt, it soon finds it. So this is a trivial endless loop.

share|improve this answer
    
And here's how to circumvent it: find -type f ! -name messageids.txt -exec grep -Fi "message-id:" {} \; > messageids.txt –  Daniel Beck Feb 22 '12 at 17:43
    
I understand that that is -what- is happening, but I don't understand -why- it's happening. Clearly my understanding of either find or grep or redirects isn't perfect, but I'm not sure which one was surprising me. So I was hoping for a detailed explanation of the whole command. Could you elaborate a bit further? –  Hammer Bro. Feb 22 '12 at 18:24
1  
@HammerBro. The output needs to be stored somewhere. RAM is expensive, and temp files are unnecessary if you just write to the file anyway. Unless you actively prevent it from writing immediately, it just does. Why shouldn't it? It's also immediately writing to the terminal if you don't redirect the output. –  Daniel Beck Feb 22 '12 at 19:34
    
@DanielBeck "Why shouldn't it?" That's a good point. I'm still not sure why my intuitions on the matter are so confused; I think I'm mixing pipes with redirects on some level. But that should put the matter to rest well enough. Thanks for the helpful commenting. –  Hammer Bro. Feb 22 '12 at 20:27

I just tested something like this and it worked.

for f in $(find . -type f); do grep -Fi "message-id:" $f > messageids.txt; done

share|improve this answer
    
It probably depends on your output write caching, and how soon in the loop the output file is hit. –  Daniel Beck Feb 22 '12 at 17:46
    
I'm not looking for how to get around the issue -- I can modify find or the output location in a number of ways. I'm looking for a low level explanation as to why this gets picked up as an infinite loop; I'll update the question with greater details. –  Hammer Bro. Feb 22 '12 at 18:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.