Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Any ideas on how to delete 16 million lines of a 24 GIG file from line 3 onwards. The traditional UNIX command line text tools, recompiled for windows, won’t handle the Unicode aspect of the file nor the file size. Maybe in stream mode> ‘cat’ fails around 4 GIG.

I have looked at the EmEditor. It has the ability to move to and bookmark specific lines but does not appear to have a command or macro to select all lines between two bookmarks. Paging down while holding down the shift key could take forever to select million of lines.

share|improve this question
1  
If you delete 16 million lines from line 3 onwards, doesn't the document have 16,000,003 lines? And doesn't that mean you only need the first 3 lines? –  slhck Apr 23 at 16:13
    
Not exactly. The file has around 18,066,980 lines in total. So I want the remainder of 18,066,980 - 16,000,000 lines. –  kingchris Apr 23 at 16:42
1  
Ah, I see! Couldn't you use some XML/XPath tools for this? –  slhck Apr 23 at 17:02
    
If you know the # of lines for each part you can grab the ones at the start of the file with head and the ones at the end with tail. –  Brian Apr 23 at 17:09
    
@slhck Maybe I can use XML/XPATH. The xml structure is very simple with each line a subset of the group. So tool that deals in lines will also work. –  kingchris Apr 23 at 19:02
add comment

2 Answers 2

As someone else said (I can't leave a comment yet), use an XML tool for the job. Saxon-HE (http://www.saxonica.com) is probably your best bet - depending on the structure of the XML file, you should be able to use a short XPath expression like this:

//*[count(ancestor::*) lt 3]

on the command line (using the -qs: switch) to output only the part of the tree you're interested in.

edit: by keeping inside the XML world, you'll also have the security blanket of knowing that Unicode is handled properly, and you won't therefore risk losing any data.

share|improve this answer
    
Do you know if this will deal with a 24 gig file. –  kingchris Apr 23 at 18:57
    
Yes - assuming you have enough RAM, it'll be fine. You might have to increase the Java heap space (e.g. "java -Xmx4000m") to avoid "out of memory" errors, but otherwise you should be grand. –  Snewsley Pies Apr 23 at 19:42
    
Thanks. Will download and give the Home edition a try. –  kingchris Apr 23 at 19:43
    
Sorry, but that advice is wrong. Saxon-HE can only process a file that fits in memory. Saxon-EE (the paid-for enterprise edition) has streaming capability; but if you can address the content by line number, then a text-based tool will be faster than an XML tool. I would be tempted to write an 8-line Java program to do the job. –  Michael Kay Apr 23 at 19:57
    
Oh... sorry @kingchris, I got mixed up on what's in which edition! Apologies for the red herring. [edit: on the other hand, if you do happen to have more than 24GB ram... :) ] –  Snewsley Pies Apr 23 at 20:03
show 1 more comment

If you can find out the number of lines you want from the start and end of the file you can use something along the lines of (replace 999 with the desired number):

head -n 999 infile.xml > outfile.xml

tail -n 999 infile.xml >> outfile.xml
share|improve this answer
    
I have found that the versions of head and tail that I have for Windows will not work with Unicode files and won't deal with a file bigger than 4 gig. So I don't think that this would work. –  kingchris Apr 23 at 18:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.