Exclude lines based on long list of line numbers

Question

I have a long list of line numbers (35389208) that I do not want in my file. By line number I mean the line in my file (e.g. line 277). My list of line numbers I do not want looks like:

277
278
279
280
289
290
291
292
321
322
....

Whats best way to exclude these line numbers from my file? Solution in perl sed or awk (or anything else).

Answer 1

If reading all of the line numbers into memory is an option, you can do it like this with awk:

awk 'FNR == NR { h[$1]; next } !(FNR in h)' line-numbers.txt input.txt

If you have limited memory available and your line-numbers.txt file is numerically sorted, you can do it like this:

delete-lines.awk

BEGIN {  
  lines_file = "line-numbers.txt"
  if(!(getline n < lines_file)) { 
    print "Unable to open lines file " lines_file > "/dev/stderr" 
    exit 
  } 
} 

FNR != n

FNR == n {
  getline n < lines_file
}

Run it like this:

awk -f delete-lines.awk input.txt

Testing where line-numbers.txt contains:

277
278
279
280
289
290
291
292
321
322

and input.txt is represented by seq 325.

First with the line-numbers in memory:

seq 325 | awk 'FNR == NR { h[$1]; next } !(FNR in h)' line-numbers.txt -

then with the line-numbers being read one at a time:

seq 325 | awk -f delete-lines.awk -

Output in both cases (lines 1 through 274 are omitted):

.
.
.
275
276
281
282
283
284
285
286
287
288
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
323
324
325

Answer 2

You could try using a regex with sed:

sed '/^[0-9]*$/d' filename.txt

This will remove the lines which only have numbers in them from your file.

The following Perl script will remove the n-th line from the file input.txt and output the rest to stdout. Line numbers can be specified in line_numbers.txt:

#!/usr/bin/perl

my @lines_to_exclude;

open(my $fh_line_numbers, "<", "line_numbers.txt") or die "Failed to open file: $!\n";
while(<$fh_line_numbers>) { 
  chomp; 
  push @lines_to_exclude, $_;
} 
close $fh_line_numbers;

my $linecounter = 1;

open (my $fh_datafile, '<', 'input.txt') or die "Cannot open $filename: $!";

while ( my $line = <$fh_datafile> ) {

  if ( ! ( $linecounter ~~ @lines_to_exclude ) ) {
    print $line;
  }

  $linecounter++;
}

close($fh_datafile);

( the ~~ operator is only available in perl >= 5.10 )

Answer 3

This might work for you (GNU sed):

sed 's/.*/&d/' line-numbers-to-delete-file | sed -f - file-to-be-shortened

Generate a script from the file containing the lines to delete and feed it to an instance of sed using the file which you want shortened as input.

Answer 4

To delete number in the begining of line

sed 's/^[0-9]*//g' filename.txt

Answer 5

Note that except for the additional code in attempt 2, all the code actually does the opposite of what OP asked. As you can see in attempt 2, it is easy to adapt the commands.

I had a textfile with 1.108.752 lines in it, about 83 MB in size. I wanted to get 46.744 lines from it, ranging between the 15th and the 1.108.716th line, which is about every 24th line on average.

tl;dr;

Second attempt is faster than first. Third only works for fewer lines.

First Attempt (bad)

For every line I want, sed reads lines from the start of the textfile, but not print them (-n). When it reaches the line I want, print it (p), then quit (q) instead of reading to the end of the file. Then do that again for the next linenumber.

Obviously, this takes a little longer each run, because sed has to go through more lines than before each time.

If I calculated that right, in my case that would take about 307332472188 passes through the textfile overall. Oh my.

Note that for this approach the order of lines is irrelevant in the linenumbers file:

while read line; do
    sed -n "${line}{p;q}" "${INFILE}"
done

Timing results: 2568.80s user 256.10s system 92% cpu 51:00.37 total. No good.

Second Attempt (better)

This reads the linenumbers from the file and appends a p (again, for printing this line). This string is piped to the next sed, which reads from a file (-f), which here is STDIN written as -, which each time is the output from the first sed, which is actually the linenumber to be printed:

sed 's/$/p/' "${LINENUMS}" | sed -n -f - "${INFILE}"

Timing results: 146.54s user 0.18s system 100% cpu 2:26.70 total. Pretty good!

If you want to not print the lines from the linefile (like OP wanted to do), slightly change the command so that linenumbers are being deleted instead of printed, and print all other lines instead of deleting them (-n):

sed 's/$/d/' "${LINENUMS}" | sed -f - "${INFILE}"

Third Attempt (badder)

This wasn't working for me at all because I had too many lines I wanted to extract. It should work for (much) fewer lines though, but I don't know the limit to that.

I tried to create a long string for sed, which I expected would lead to sed going through the file only once (!), not printing anything except the linenumbers from the string:

sed -n "12p;15p;24p;345p;...;12345;" ${INFILE}"

but that would result in a string about 420076 characters long, which upon pumping into sed simply led to sed: Argument list is too long. Which is understandable.