2

Any ideas on how to delete 16 million lines of a 24 GIG file from line 3 onwards. The traditional UNIX command line text tools, recompiled for windows, won’t handle the Unicode aspect of the file nor the file size. Maybe in stream mode> ‘cat’ fails around 4 GIG.

I have looked at the EmEditor. It has the ability to move to and bookmark specific lines but does not appear to have a command or macro to select all lines between two bookmarks. Paging down while holding down the shift key could take forever to select million of lines.

5
  • 1
    If you delete 16 million lines from line 3 onwards, doesn't the document have 16,000,003 lines? And doesn't that mean you only need the first 3 lines?
    – slhck
    Apr 23, 2014 at 16:13
  • Not exactly. The file has around 18,066,980 lines in total. So I want the remainder of 18,066,980 - 16,000,000 lines.
    – kingchris
    Apr 23, 2014 at 16:42
  • 1
    Ah, I see! Couldn't you use some XML/XPath tools for this?
    – slhck
    Apr 23, 2014 at 17:02
  • If you know the # of lines for each part you can grab the ones at the start of the file with head and the ones at the end with tail.
    – Brian
    Apr 23, 2014 at 17:09
  • @slhck Maybe I can use XML/XPATH. The xml structure is very simple with each line a subset of the group. So tool that deals in lines will also work.
    – kingchris
    Apr 23, 2014 at 19:02

2 Answers 2

1

As someone else said (I can't leave a comment yet), use an XML tool for the job. Saxon-HE (http://www.saxonica.com) is probably your best bet - depending on the structure of the XML file, you should be able to use a short XPath expression like this:

//*[count(ancestor::*) lt 3]

on the command line (using the -qs: switch) to output only the part of the tree you're interested in.

edit: by keeping inside the XML world, you'll also have the security blanket of knowing that Unicode is handled properly, and you won't therefore risk losing any data.

5
  • Do you know if this will deal with a 24 gig file.
    – kingchris
    Apr 23, 2014 at 18:57
  • Yes - assuming you have enough RAM, it'll be fine. You might have to increase the Java heap space (e.g. "java -Xmx4000m") to avoid "out of memory" errors, but otherwise you should be grand. Apr 23, 2014 at 19:42
  • Sorry, but that advice is wrong. Saxon-HE can only process a file that fits in memory. Saxon-EE (the paid-for enterprise edition) has streaming capability; but if you can address the content by line number, then a text-based tool will be faster than an XML tool. I would be tempted to write an 8-line Java program to do the job. Apr 23, 2014 at 19:57
  • Oh... sorry @kingchris, I got mixed up on what's in which edition! Apologies for the red herring. [edit: on the other hand, if you do happen to have more than 24GB ram... :) ] Apr 23, 2014 at 20:03
  • Thanks for the tip. That what makes sites like this excellent. I would have spent hours learning the package only to discover that it would not work. I will have to break down and write a program to solve this. Thanks to all contributors to this question
    – kingchris
    Apr 24, 2014 at 4:42
0

If you can find out the number of lines you want from the start and end of the file you can use something along the lines of (replace 999 with the desired number):

head -n 999 infile.xml > outfile.xml

tail -n 999 infile.xml >> outfile.xml
1
  • I have found that the versions of head and tail that I have for Windows will not work with Unicode files and won't deal with a file bigger than 4 gig. So I don't think that this would work.
    – kingchris
    Apr 23, 2014 at 18:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .