Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there a tool to split large text file (9Gb) into smaller files so that I can open it and look through?

Anything usable from command line that comes with Windows (XP)?

Or what's the best way to split it? Can I use 7z to create separate volumes and then unzip one of them separately? Will it be readable or does it need all the other parts to unzip into the big file again?

Update

I put together quick 48 lines python script that split the large file into 0.5GB files which are easy to open even in vim. I've just needed to look through data towards the last part of the log (yes it is a log file). Each record is split across multiple lines so grep would not do.

share|improve this question

migrated from serverfault.com Jan 11 '10 at 1:25

This question came from our site for system and network administrators.

    
I see you edited to mention grep. Do you have cygwin or unxutils installed? You could have used grep -n with head and tail to see chunks of the file. Example, grep -n "something" file.txt returns 95625: something. You want to see that line and 9 lines below it for a total of 10 lines: head -n 95635 file.txt | tail -n 10. –  John T Jan 11 '10 at 4:10
    
I notice you solved your problem, If you're still around, could you post the solution so others might benefit? –  Journeyman Geek Oct 19 '12 at 3:30
    
This has been discussed in much detail at Stack Overflow[1] [1]: stackoverflow.com/questions/159521/… –  Rishi Dua Jun 26 '13 at 6:33

4 Answers 4

up vote 24 down vote accepted

There is a freeware Windows file splitter called HJSplit

Available here. The website claims it can split files of any type and size, but 9GB is a big file.

share|improve this answer
1  
According to the publisher, it supports splitting of files "over 100 Gigabytes" –  yosh m Apr 11 '13 at 7:57
3  
I was able to use this program to split a large file into smaller pieces, but only by size (kB or MB). It does not respect lines, so it's not very useful for splitting log files. It's also very slow. –  nullability Mar 19 at 17:46

Another is GSplit - according to their site it can split very large files (larger than 4Gb <-- since they crossed the 4Gb limit, I guess they can do 9 Gb as well).

But, another thing - you say you want to split it into smaller parts so you can open it up and look at it. That sounds like a very big perhaps log file.

In any case, for opening large text files, may I recommend EmEditor - they claim themselves it can open very large files (up to cca. 250 Gb), and I've used it in the past for files up to 2 Gb. But in any case, I think it may be a better solution than splitting.

share|improve this answer
    
I've just tried out GSplit. It works, albeit through a confusing interface, but it doesn't offer any way to only split at a linebreak - for splitting ASCII data files, therefore, it's not too useful as the split will be half way through a line. –  Simon W May 10 '13 at 15:09
1  
@Flyto It actually is possible to split by line if you choose "Split After Occurrence Number" under the Pieces > Type and Size menu. You then choose the number of lines and the line delimiter in hex. By default it has Windows format CR+LF (0x0D0x0A). –  nullability Mar 19 at 17:56
    
@nullability thanks - I wouldn't have thought of that. But, I found a much simpler solution through the GNU "split" utility - see my new answer :-) –  Simon W Mar 19 at 22:23

The GNU Core Utils package (available here for windows) includes the Split utility. It works well for me :-)

The --help documentation is as follows:

Usage: split [OPTION] [INPUT [PREFIX]]
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when INPUT
is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -a, --suffix-length=N   use suffixes of length N (default 2)
  -b, --bytes=SIZE        put SIZE bytes per output file
  -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file
  -d, --numeric-suffixes  use numeric suffixes instead of alphabetic
  -l, --lines=NUMBER      put NUMBER lines per output file
      --verbose           print a diagnostic to standard error just
                            before each output file is opened
      --help     display this help and exit
      --version  output version information and exit

SIZE may have a multiplier suffix: b for 512, k for 1K, m for 1 Meg.

For example, to split input.txt into 100Mb chunks, only splitting at the ends of lines,

split input.txt -C 100m

will give you output files named xaa, xab, xac, etc.

share|improve this answer
    
for example git msys download includes the utility as well. –  eis May 20 at 6:55

Check out Large Text File Viewer, it's great for things like this. Most archivers and splitters will separate the file into pieces which cannot be used to read each piece of data independently and properly, you need to extract them all to get the file back.

alt text

Large Text File Viewer is free and portable.

share|improve this answer
1  
The link seems to be broken, this works as of now: softpedia.com/get/Office-tools/Other-Office-Tools/… –  huseyint Jul 11 '12 at 20:24
    
The original (or edited?) link is working now; I'd take that link over softpedia any day! Also, just tried LTFViewer, and it's awesome, opened an 818MB file that even Notepad++ complained about opening. Not much in the way of editing features, but once I'd viewed the logs in LTFViewer, echo. > myfile.txt resolved the 818MB logfile issue :D –  Doktor J Jul 1 '13 at 19:16
    
LTFViewer is realy great ( i'm using it to ), but there's an limit. I tryed to open an 3GB SQL file with it, but it freezes. Mayby i didn't have enough patience or something... –  Mathlight Aug 9 '13 at 6:37
    
LTFViewer worked very well for me. I used it on a (relatively) smaller file, only 750MB, but it opened it in under 5 seconds. –  Mike_OBrien Mar 17 at 14:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.