Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a huge log file of around 3.5 GB and would like to sample random sections in the middle of say 10 MB for the purpose of debugging what my application is doing.

I could use head or tail commands to get the beginning or end of the file, how can I grab an arbitrary portion from the middle of the file? I guess I could do something like head -n 1.75GB | tail -n 10MB but that seems clumsy and I'd need to determine line numbers for the midpoint of the file to get 1.75GB and 10MB line counts.

share|improve this question
add comment

3 Answers

up vote 6 down vote accepted
$ dd if=big_file.bin skip=1750 ibs=1MB count=10 of=big_file.bin.part

You might want to spend some time reading and understanding dd.

share|improve this answer
    
As dd is not aware of lines, you may want to trim of the first and last one, so as only complete lines are included, e.g.: dd ... | head -n-1 | tail -n+2 or with sed: dd ... | sed '1d; $d'. –  Thor Feb 22 '13 at 10:19
add comment

You can use use tail, but by specify a byte offset.

tail -c +$START_BYTE $file | head -c $LENGTH > newfile

That way tail can jump directly to the starting point (without counting new lines) and once head matches the correct length, it stops running.

share|improve this answer
add comment

You just have to write a little program to seek to some random spot and read some amount of lines.

An example in Python (reads one line, but you can modify it):

def get_random_line():
    """Return a randomly selected line from a file."""
    import random
    fo = open("/some/file.txt")
    try:
        point = random.randrange(fo.size)
        fo.seek(point)
        c = fo.read(1)
        while c != '\n' and fo.tell() > 0:
            fo.seek(-2, 1)
            c = fo.read(1)
        line = fo.readline().strip()
    finally:
        fo.close()
    return line
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.