Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a large number of file, some of which are very long. I would like to truncate them to a certain size if they are larger by removing the end of the file. But I only want to remove whole lines. How can I do this? It feels like the kind of thing that would be handled by the Linux toolchain but I don't know of the right command.

For example, say I have a 120,000 byte file with 300-byte lines and I'm trying to truncate it to 10,000 bytes. The first 33 lines should stay (9900 bytes) and the remainder should be cut. I don't want to cut at 10,000 bytes exactly, since that would leave a partial line.

Of course the files are of differing lengths and the lines are not all the same length.

Ideally the resulting files would be made slightly shorter rather than slightly longer (if the breakpoint is on a long line) but that's not too important, it could be a little longer if that' easier. I would like the changes to be made directly to files (well, possibly the new file copied elsewhere, the original deleted, and the new file moved, but that's the same from the user's POV). A solution that redirects data to a bunch of places and then back invites the possibility of corrupting the file and I'd like to avoid that...

share|improve this question
    
Deleted my answer… I guess the file size thing in Bytes wasn't too clear, sorry. Maybe you could edit your question and clarify that part (e.g. with an example)? –  slhck Jul 24 '12 at 19:54
    
@slhck: Sorry to see you lose rep just because I was unclear... let me see if I can fix that. –  Charles Jul 24 '12 at 20:58
    
No worries, I should have just asked, sorry :) –  slhck Jul 24 '12 at 21:01
add comment

3 Answers

You can use the linux command sed to remove lines from a file. The following command delete the last line of filename.txt:

sed '$d' filename.txt

With awk or find you can search for pattern that match for your sed command. First you search with awk or find for the files you want to shorten and then you can remove the lines with sed.

share|improve this answer
add comment

The sed approach is fine, but to loop over all lines is not. If you know how many lines you want to keep (to have an example, I use 99 here), you can do it like this:

sed -i '100,$ d' myfile.txt

Explanation: sed is a regular expression processor. With the option -i given, it processes a file directly ("inline") -- instead of just reading it and writing the results to the standard output. 100,$ just means "from line 100 to the end of the file" -- and is followed by the command d, which you probably guessed correctly to stand for "delete". So in short, the command means: "Delete all lines from line 100 to the end of the file from myfile.txt". 100 is the first line to be deleted, as you want to keep 99 lines.

Edit: If, on the other hand, there are log files where you want to keep e.g. the last 100 lines:

[ $(wc -l myfile.txt) -gt 100 ] && sed -i "1,$(($(wc -l myfile.txt|awk '{print $1}') - 100)) d" myfile.txt

What is going on here:

  • [ $(wc -l myfile.txt) -gt 100 ]: do the following only if the file has more than 100 lines
  • $((100 - $(wc -l myfile.txt|awk '{print $1}'))): calculate number of lines to delete (i.e. all lines of the file except the (last) 100 to keep)
  • 1, $((..)) d: remove all lines from the first to the calculated line

EDIT: as the question was just edited to give more details, I will include this additional information with my answer as well. Added facts are:

  • a specific size shall remain with the file (10,000 bytes)
  • each line has a specific size in bytes (300 bytes in the example)

From these data it is possible to calculate the number of lines to remain as " / ", which with the example would mean 33 lines. The shell term for the calculation: $((size_to_remain / linesize)) (at least on Linux using Bash, the result is an integer). The adjusted command now would read:

# keep the start of the file (OPs question)
sed -i '34,$ d' myfile.txt
# keep the end of the file (my second example)
[ $(wc -l myfile.txt) -gt 33 ] && sed -i "1,33 d" myfile.txt

As the sizes are known in advance, there's no longer any need for a calculation embedded to the sed command. But for flexibility, inside some shell script one can use variables.

For conditional processing based on the file size, one can use th following "test"-construct:

[ "$(ls -lk $file | awk ' {print $5}')" -gt 100 ] &&

which means: "if the size of $file exceeds 100kB, do..." (ls -lk lists the file size in kB at position 5, hence awk is used to extract exactly this).

share|improve this answer
    
The OP wants to cut the file based on a certain byte size — not just length in terms of lines. I deleted my answer involving head -n. –  slhck Jul 24 '12 at 19:55
    
@slhck Thank you for the notification. Yes, the OP just edited his question to make the intention more clear. As he has means to calculate how many bytes each line has, my answer remains valid in principle -- as he can calculate the number of lines to remain, and then use my approach to handle the files. Maybe I make a short remark on that within my answer. –  Izzy Jul 24 '12 at 22:08
    
No -- the sizes are not known in advance. That was an example. Each file will have a different size and lines are of irregular length. Some files don't need to be truncated at all. –  Charles Jul 25 '12 at 0:45
    
Oh, again... Well, some things are hard to explain clearly (too many facettes). As for the files which need no truncate, that's probably based on file size? That can be covered. But if there's not even an average line size known, this part gets hard -- I cannot think of an easy solution (without too much overhead) at the moment. –  Izzy Jul 25 '12 at 6:23
    
All I can come up with currently would involve to e.g. get the first n lines, calculate an average length based on them, and use this value. Would that help you? –  Izzy Jul 25 '12 at 6:30
show 2 more comments
up vote 0 down vote accepted

Failing to find a command to do this, I wrote a quick script (not tested):

#!/bin/sh

# Usage: $0 glob.* 25000
# where glob.* is a wildcard pattern and 25000 is the maximum number of bytes.

limit=20000
tmp=/tmp/trim
[[ "$2" == +([0-9]) ]] || limit=$2
limit=`expr $len + 1`
for file in $1;
do
    [[ `wc -c $file` -lt $limit ]] && continue
    head -c $file > $tmp
    sed '$d' $tmp
    $tmp > $file
done
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.