5

I have some text files that are 100 to 300 MB in length that I want to view in Emacs, but my Emacs gets into some performance issues opening and traversing such large files. Therefore, I'm looking for an easy utility to split a file up into manageable chunks, say 50 MB each and to name the chunks based upon the original name plus some suffix to indicate their location in the sequence. Each chunk would pick up where the prior one was truncated and be no longer than 50 MB, with the last chunk possibly being shorter.

Is there any easy tool to do such on a Linux computer, perhaps something like head or tail that gives multiple results, one for each chunk?

e.g., given file test.out that is 120 MB long, break it up into test.out.1 for the first 50 MBs, test.out.2 for the second 50 MBs, and test.out.3 for the remainder 20 MBs at the end of the file.

I could use combinations of head and tail to get the pieces, but I'd like a tool that abstracts this all out, perhaps a Perl script or Python script someone has already created to do such a task?

1 Answer 1

18

There already is a nice tool for that: split

> man 1 split 

NAME
     split -- split a file into pieces

SYNOPSIS
     split [-l line_count] [-a suffix_length] [file [prefix]]
     split -b byte_count[K|k|M|m|G|g] [-a suffix_length] [file [prefix]]
     split -p pattern [-a suffix_length] [file [prefix]]


split --bytes 50M test.out test.out_ would split the file test.out into test.out_xaa, test.out_xab, test.out_xac, ...

A much uglier solution would be to use dd

dd if=test.out of=test.out.part1 bs=1M count=50 skip=0 creates a file named test.out.part1 with the first 50M from test.out. You can increase the value for skip to 1 to get the second chunk, to 2 for the third etc etc. Just make sure to also change the filenames or you will end up overwriting the same output file.

4
  • On at least my Linux, a lowercase 'm' is required for Megabytes.
    – WilliamKF
    Sep 11, 2013 at 5:32
  • Yet another new command learned ☺ Sep 11, 2013 at 5:39
  • @williamKF: Which flavour of Linux was that? I checked on Ubuntu and FreeBSD. FreeBSD needed the -b 50M (No option for --bytes). And Ubuntu accepted both the regular m and capital M.
    – Hennes
    Sep 11, 2013 at 10:40
  • @Hennes split (coreutils) 5.2.1 February 2011 on my Linux centos4 32 bit 2.6.9-103.ELsmp #1 SMP Fri Dec 9 04:31:51 EST 2011 i686 i686 i386 GNU/Linux
    – WilliamKF
    Sep 11, 2013 at 15:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .