I've got a multi-GB file that has elements in 4 lines, and I would like to have every 4 lines randomized in another file, this is, keeping each four lines grouped and randomize those sets. Is there an easy way to do that?
2 Answers
If you're using a reasonable new linux/unix distribution, sort
comes with a -R
flag which randomises lines instead of sorts them. We can use that to create this one-liner solution:
awk '{printf("%s%s",$0,(NR%4==0)?"\n":"\0")}' file.txt | sort -R | tr "\0" "\n" > sorted.txt
First, use awk
to group every 4 lines by replacing \n
with \0
. We then shuffle the lines using sort -R
and finally restore the line breaks with tr
.
-
The
"\n
is unclosed and should be"\n"
. I can't make such a small edit, so I'm just putting it here. Apr 19, 2013 at 19:27 -
Also, for some reason on my system, I couldn't printf the "\0" and get it to replace properly. Instead, I used
printf("%s%c",$0,(NR%4==0)?"\n":0)
. I did not need to change the tr command. Apr 19, 2013 at 19:37
This is in Python. I'm sure someone will post a Perl answer too. ;-)
#!/usr/bin/python import random #Change these to the desired files infile = "/path/to/input/file" outfile = "/path/to/output/file" fh = file(infile) contents = fh.readlines() fh.close() chunked = [contents[i:i+4] for i in xrange(0, len(contents), 4)] random.shuffle(chunked) fh = file(outfile, 'w') for chunk in chunked: for line in chunk: fh.write(line) fh.close()
IANA Programmer so somebody could probably improve this, but I tested it and it works just fine.