Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to get the last 10 million lines of a 100 million lines file.

tail -n 10000000 $in_file>$out_file

however the above command took more than 1 day and is still going. I guess I can use the nice command to assign more priority for the process, but I want to know if there's a fundamentally better way.

share|improve this question
    
Hard disk or SSD? –  Makoto May 6 '13 at 2:05
    
100 million lines is quite a huge file, nah? Couldn't you have the file formerly divided into some couple of shards? This would certainly reduce the EOL searching, thus giving you some speed up. An SSD would also be a great -- darn expansive -- shot (: –  Rubens May 6 '13 at 2:18
    
How long, roughly, is each line (or how big is the file in bytes)? Are the lines reasonably uniform in size, or are there short ones and long ones to worry about? If the standard tool won't do the job timely, then you will probably need to write your own. It is exasperating that you'll probably need to scan the tail end of the file at least twice, but probably unavoidable. –  Jonathan Leffler May 6 '13 at 3:09
add comment

migrated from stackoverflow.com May 6 '13 at 10:17

This question came from our site for professional and enthusiast programmers.

1 Answer

up vote 2 down vote accepted

Apart from splitting up the file into some smaller files, you could simply open the file and seek it to something you may think of as close to the end of the file.

After that, you read as much lines as there may come, and, if you reach EOF without all of your 10000000 desired lines, you just need to make a diff from the first position you guessed, and a new -- prior -- position, and try to read the n = diff lines.

I do not actually know if tail does so, or if there's any available POSIX tool that performs this kind of operation; implementing this shouldn't take more than five minutes, I guess (: This may be of some help.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.