Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a very large file ~400G, and I need to remove the last 2 lines from it. I tried to use sed, but it ran for hours before I gave up. Is there a quick way of doing this, or am I stuck with sed?

share|improve this question
5  
you can give GNU head a try. head -n -2 file –  user31894 Apr 6 '10 at 5:54
    
There were a couple of one line Perl and Java suggestions given in stackoverflow.com/questions/2580335/… –  mtrw Apr 7 '10 at 1:02
add comment

10 Answers

up vote 26 down vote accepted

I haven't tried this on a large file to see how fast it is, but it should be fairly quick.

To use the script to remove lines from the end of a file:

./shorten.py 2 large_file.txt

It seeks to the end of the file, checks to make sure the last character is a newline, then reads each character one at a time going backwards until it's found three newlines and truncates the file just after that point. The change is made in place.

Edit: I've added a Python 2.4 version at the bottom.

Here is a version for Python 2.5/2.6:

#!/usr/bin/env python2.5
from __future__ import with_statement
# also tested with Python 2.6

import os, sys

if len(sys.argv) != 3:
    print sys.argv[0] + ": Invalid number of arguments."
    print "Usage: " + sys.argv[0] + " linecount filename"
    print "to remove linecount lines from the end of the file"
    exit(2)

number = int(sys.argv[1])
file = sys.argv[2]
count = 0

with open(file,'r+b') as f:
    f.seek(0, os.SEEK_END)
    end = f.tell()
    while f.tell() > 0:
        f.seek(-1, os.SEEK_CUR)
        char = f.read(1)
        if char != '\n' and f.tell() == end:
            print "No change: file does not end with a newline"
            exit(1)
        if char == '\n':
            count += 1
        if count == number + 1:
            f.truncate()
            print "Removed " + str(number) + " lines from end of file"
            exit(0)
        f.seek(-1, os.SEEK_CUR)

if count < number + 1:
    print "No change: requested removal would leave empty file"
    exit(3)

Here's a Python 3 version:

#!/usr/bin/env python3.0

import os, sys

if len(sys.argv) != 3:
    print(sys.argv[0] + ": Invalid number of arguments.")
    print ("Usage: " + sys.argv[0] + " linecount filename")
    print ("to remove linecount lines from the end of the file")
    exit(2)

number = int(sys.argv[1])
file = sys.argv[2]
count = 0

with open(file,'r+b', buffering=0) as f:
    f.seek(0, os.SEEK_END)
    end = f.tell()
    while f.tell() > 0:
        f.seek(-1, os.SEEK_CUR)
        print(f.tell())
        char = f.read(1)
        if char != b'\n' and f.tell() == end:
            print ("No change: file does not end with a newline")
            exit(1)
        if char == b'\n':
            count += 1
        if count == number + 1:
            f.truncate()
            print ("Removed " + str(number) + " lines from end of file")
            exit(0)
        f.seek(-1, os.SEEK_CUR)

if count < number + 1:
    print("No change: requested removal would leave empty file")
    exit(3)

Here is a Python 2.4 version:

#!/usr/bin/env python2.4

import sys

if len(sys.argv) != 3:
    print sys.argv[0] + ": Invalid number of arguments."
    print "Usage: " + sys.argv[0] + " linecount filename"
    print "to remove linecount lines from the end of the file"
    sys.exit(2)

number = int(sys.argv[1])
file = sys.argv[2]
count = 0
SEEK_CUR = 1
SEEK_END = 2

f = open(file,'r+b')
f.seek(0, SEEK_END)
end = f.tell()

while f.tell() > 0:
    f.seek(-1, SEEK_CUR)
    char = f.read(1)
    if char != '\n' and f.tell() == end:
        print "No change: file does not end with a newline"
        f.close()
        sys.exit(1)
    if char == '\n':
        count += 1
    if count == number + 1:
        f.truncate()
        print "Removed " + str(number) + " lines from end of file"
        f.close()
        sys.exit(0)
    f.seek(-1, SEEK_CUR)

if count < number + 1:
    print "No change: requested removal would leave empty file"
    f.close()
    sys.exit(3)
share|improve this answer
    
our system is running python 2.4, and I'm not sure if any of our services rely on it, will this work in that? –  Russ Bradberry Apr 6 '10 at 13:28
    
@Russ: I've added a version for Python 2.4. –  Dennis Williamson Apr 6 '10 at 14:38
1  
absolutely amazing! worked like a charm and in less than a second! –  Russ Bradberry Apr 6 '10 at 14:49
    
Get this man some rep! –  th3dude Apr 6 '10 at 14:51
    
@Russ: Glad to help. –  Dennis Williamson Apr 6 '10 at 15:46
add comment

Try VIM...I'm not sure if it will do the trick or not, as I've never used it on such a big file, but I've used it on smaller larger files in the past give it try.

share|improve this answer
    
I do believe vim only loads what's immediately around the buffer when editing, however I've no idea how it saves. –  Phoshi Apr 5 '10 at 23:30
    
vim hangs while it tries to load the file –  Russ Bradberry Apr 5 '10 at 23:38
    
Well if it hangs, ah wait for it. Start it loading, go to work, come home, see if it is done. –  leeand00 Apr 6 '10 at 0:35
1  
See this: stackoverflow.com/questions/159521/… –  leeand00 Apr 6 '10 at 0:38
add comment

What kind of file and in what format? May be easier to use something like Perl dependent on what kind of file it is - text, graphics, binary? How is it formatted - CSV, TSV...

share|improve this answer
    
it is formatted pipe delimeted text, however the last 2 lines are one column each which will break my import so I need them removed –  Russ Bradberry Apr 5 '10 at 23:39
    
is fixing whatever does the "import" to deal with this case an option ? –  timday Apr 6 '10 at 12:22
    
no the import is infobright's "load data infile" –  Russ Bradberry Apr 6 '10 at 13:11
add comment

The problem with sed is that it is a stream editor -- it will process the entire file even if you only want to make modifications near the end. So no matter what, you are creating a new 400GB file, line by line. Any editor that operates on the whole file will probably have this problem.

If you know the number of lines, you could use head, but again this creates a new file instead of altering the existing one in place. You might get speed gains from the simplicity of the action, I guess.

You might have better luck using split to break the file into smaller pieces, editing the last one, and then using cat to combine them again, but I'm not sure if it will be any better. I would use byte counts rather than lines, otherwise it will probably be no faster at all -- you're still going to be creating a new 400GB file.

share|improve this answer
add comment

you can try GNU head

head -n -2 file
share|improve this answer
    
It is the best solution since it is simple. –  xiao Jan 16 '12 at 2:48
1  
This will show him the last two lines of the file, but not remove them from his file..an doesn't even work on my system head: illegal line count -- -2 –  SooDesuNe Apr 14 '12 at 23:07
1  
@SooDesuNe: No it will print all lines from the beginning to 2 lines from the end, as per the manual. However, this would need to be redirected to a file, and then there is the problem with this file being giant, so it's not the perfect solution for this problem. –  Daniel Andersson Apr 25 '12 at 21:38
    
+1 Why isn't this being accepted as the correct answer? It's fast, simple and does work as expected. –  aefxx Sep 12 '12 at 12:27
3  
@PetrMarek and others: The problem was that it concerned a giant file. This solution would require the entire file to be fed through a pipe and rewriting all data to a new location — and the whole point of the question is to avoid that. An in-place solution is needed, such as the one in the accepted answer. –  Daniel Andersson Mar 4 '13 at 7:31
show 1 more comment

If you know the size of the file to the byte (400000000160 say) and you know that you need to remove exactly 160 characters to strip the last two lines, then something like

dd if=originalfile of=truncatedfile ibs=1 count=400000000000

should do the trick. It's been ages since I used dd in anger though; I seem to remember things go faster if you use a bigger block size, but whether you can do that depends on whether the lines you want to drop are at a nice multiple.

dd has some other options to pad text records out to a fixed size which might be useful as a preliminary pass.

share|improve this answer
    
i tried this, but it was going about the same speed as sed. It had written approx 200MB in 10 mins, at this rate it would literally take hundreds of hours to complete. –  Russ Bradberry Apr 6 '10 at 13:35
add comment

If "truncate" command isn't available on your system (see my other answer), look at the "man 2 truncate" for the system call to truncate a file to a specified length.

Obviously you need to know how many characters you need to truncate the file to (size minus the length of the problem two lines; don't forget to count any cr/lf characters).

And make a backup of the file before you try this!

share|improve this answer
add comment

I see my Debian Squeeze/testing systems (but not Lenny/stable) include a "truncate" command as part of the "coreutils" package.

With it you could simply do something like

truncate --size=-160 myfile

to remove 160 bytes from the end of the file (obviously you need to figure out exactly how many characters you need to remove).

share|improve this answer
    
This will be the fastest route since it modifies file in-place, and therefore doesn't require neither copying nor parsing the file. However, you'll still need to check how many bytes to remove... I /guess/ that a simple dd script will do that (you need to specify input offset to get f.e. last kilobyte and then use tail -2 | LANG= wc -c, or sth like that). –  liori Apr 6 '10 at 13:11
    
I'm using CentOS, so no I do not have truncate. However, this is exactly what I am looking for. –  Russ Bradberry Apr 6 '10 at 13:37
add comment
#!/bin/sh

ed "$1" << HERE
$
d
d
w
HERE

changes are made in place. This is simpler and more efficient than the python script.

share|improve this answer
    
On my system, using a text file consisting of a million lines and over 57MB, ed took 100 times as long to execute than my Python script. I can only imagine how much more the difference would be for the OP's file which is 7000 times bigger. –  Dennis Williamson Apr 7 '10 at 4:48
add comment

Modified the accepted answer to solve a similar problem. Could be tweaked a little bit to remove n lines.

import os

def clean_up_last_line(file_path):
    """
    cleanup last incomplete line from a file
    helps with an unclean shutdown of a program that appends to a file
    if \n is not the last character, remove the line
    """
    with open(file_path, 'r+b') as f:
        f.seek(0, os.SEEK_END)

        while f.tell() > 0: ## current position is greater than zero
            f.seek(-1, os.SEEK_CUR)

            if f.read(1) == '\n':
                f.truncate()
                break

            f.seek(-1, os.SEEK_CUR) ## don't quite understand why this has to be called again, but it doesn't work without it

And the corresponding test:

import unittest

class CommonUtilsTest(unittest.TestCase):

    def test_clean_up_last_line(self):
        """
        remove the last incomplete line from a huge file
        a line is incomplete if it does not end with a line feed
        """
        file_path = '/tmp/test_remove_last_line.txt'

        def compare_output(file_path, file_data, expected_output):
            """
            run the same test on each input output pair
            """
            with open(file_path, 'w') as f:
                f.write(file_data)

            utils.clean_up_last_line(file_path)

            with open(file_path, 'r') as f:
                file_data = f.read()
                self.assertTrue(file_data == expected_output, file_data)        

        ## test a multiline file
        file_data = """1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b
1362358458954466,2013-03-03 16:54:18,34.5,3.0,b
1362358630923094,2013-03-03 16:57:10,34.5,50.0,b
136235"""

        expected_output = """1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b
1362358458954466,2013-03-03 16:54:18,34.5,3.0,b
1362358630923094,2013-03-03 16:57:10,34.5,50.0,b
"""        
        compare_output(file_path, file_data, expected_output)

        ## test a file with no line break
        file_data = u"""1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b"""
        expected_output = "1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b"
        compare_output(file_path, file_data, expected_output)

        ## test a file a leading line break
        file_data = u"""\n1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b"""
        expected_output = "\n"
        compare_output(file_path, file_data, expected_output)

        ## test a file with one line break
        file_data = u"""1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b\n""" 
        expected_output = """1362358424445914,2013-03-03 16:53:44,34.5,151.16345879,b\n""" 
        compare_output(file_path, file_data, expected_output)

        os.remove(file_path)


if __name__ == '__main__':
    unittest.main()
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.