Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there a simple way to truncate a file in a pipe? Specifically, I want to chop of the last four bytes of a file before feeding it into another process.

Ideally, I'd be able to write something like:

cat input.txt | some-process | truncate --size=-4 | another-process > output.txt

but it seems that the truncate command only operates "in place" on a file on disk.

share|improve this question
1  
cat input.txt | some-process is better written as some_process < input.txt. –  Benoit Feb 24 '12 at 9:53
add comment

5 Answers

This is like if I told you to raise your hand as soon as I utter the fourth from last word I am about to say. I am not going to tell you beforehand how many words I am about to say.

A pipe is a stream. Its data has no size, it only has operations for getting the next element from it and / or inserting an element into it, and the result is either a piece of data or a signal that there is no more data.

So unless you first retrieve all the data from the stream, place it into a buffer, count its length, "rewind" the stream, and then retrieve four fewer elements, it can't be done.

EDIT: I need to do more thinking things through instead of coming up with clever analogies:) A stream does not say "stop me immediately n elements before the last", but rather "transmit all elements except the last n", and by maintaining a buffer of just n elements, and waiting until the first n elements have been received before transmitting the first one, it is possible. Obviously this won't work in situations like telecommunications where you want data to be sent immediately after being received as you could if you wanted the first n elements. And I assume truncate doesn't do it this way.

(attempted downvoting self -1)

share|improve this answer
    
You have my upvote, because your answer did help me realize that file/stream length is not known at runtime, which prompted me to write the Python script. –  kostmo Feb 24 '12 at 21:28
add comment

I feel silly after writing that Python script.

There is a built-in shell command head to do it:

cat input.txt | some-process | head --bytes=-4 | another-process > output.txt

Edit: The GNU head command has a conceptually similar implementation (i.e. memory-efficient) to my Python implementation below. One difference is that it rounds up the size of the circular buffer (N, the number of elided bytes) to a multiple of some standard buffer size.

share|improve this answer
    
provided head version is not too old that will work. –  Benoit Feb 24 '12 at 10:16
add comment

sed can operate on the last line. This assumes the last 4 chars are on a single line:

printf "%s\n" abcdef ghijkl mnopqr | sed '$s/....$//'

outputs

abcdef
ghijkl
mn
share|improve this answer
add comment

I couldn't find any built-in shell commands to do this, so I guess that means there's no "one-liner" solution. However, I was able to write a Python script to do what I need:

#!/usr/bin/env python
'''
Usage:
pipetruncate.py <N>

Truncates a stream in a pipe at N bytes before the EOF.
Uses memory proportional to N.
'''

import sys

buffer_length = int(sys.argv[1])
circular_buffer = [0]*buffer_length
count = 0
while True:
    ch = sys.stdin.read(1)
    if not len(ch): # EOF
        break

    index = count % buffer_length
    nextchar = circular_buffer[index]
    circular_buffer[index] = ch

    count += 1
    if count > buffer_length:
        sys.stdout.write(nextchar)

sys.stdout.close()

Then I invoke

cat input.txt | some-process | ./pipetruncate.py 4 | another-process > output.txt

share|improve this answer
add comment

Spent part of the morning writing a python script too. Of course, you better use your "head" instead of writing more code. Anyway here is my version. It is ugly but I think it is my first python script ever:

#!/usr/bin/python

# stream_trunc: cut the last n bits of a stream

import sys

if len(sys.argv) <> 2:
    print 'Usage: ' + sys.argv[0] + ' <number>'
    exit(1)

num = sys.argv[1]

if num.isdigit() != True:
    print 'Argument should be a number'
    print 'Usage: ' + sys.argv[0] + ' <number>'
    exit(1)

n = int(num)
buf = sys.stdin.read(n)
c = sys.stdin.read(1)

while c != '':
    sys.stdout.write(buf[0])
    buf = buf[1:] + c
    c = sys.stdin.read(1)
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.