Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

We have a data file from a client which is 1,443,777,659 bytes in size.

Sorted output has lines missing and is only 1,269,801,985 bytes in size.

Sample command: sort -k 1,10 -T . -s -i file_to_sort.txt -o out.txt

We've tried on 32-bit Win 7 and XP systems.

We've tried the supplied sort.exe that comes with windows, as well as binaries from UnxUtils and Gnu coreutils.

None give an error, however all result in the exact same output size. I've tried another freeware utility that works but is much slower.

I believe this may be due to a 32-bit limitation, however the file size doesn't seem near any of the usual suspects, and these programs work by writing and merging together smaller files, none of which approach 2 GB in size.

Any tips on how to get to the bottom of this? Thanks.

share|improve this question
    
The best way to verify if this is a 32-bit 64-bit problem is to try it on a 64-bit operating system. Be sure to use a 64-bit tool otherwise it won't matter. –  Ramhound Jul 18 '13 at 11:17

2 Answers 2

up vote 1 down vote accepted

OK so the issue was not related to the size of the file at all. It seems to be that the file is opened in text mode, and contains a 0x1A (^Z or EOF on Windows) character near the end.

Once it hits this character during input, it stops reading. There's no way around this as there is no flag to open the file as binary.

I should have found this quicker, but it's not so easy to dig around a 1.5GB file :)

Related query: http://stackoverflow.com/questions/13582804/why-can-windows-not-read-beyond-the-0x1a-eof-character-but-unix-can

share|improve this answer

You don't want to ignore non-printable characters if the file contains them. Drop the -i option and run with LC_ALL=C.

e.g.

export LC_ALL=C
sort -k 1,10 -s <file_to_sort.txt >out.txt
share|improve this answer
    
We are adding the sort keys in an earlier process, so can guarantee they are all printable chars. I tried this command anyway but got the same result - we're on windows but I just did "set LC_ALL=C" assuming it'll be the same. –  Nick P Jul 18 '13 at 7:54
    
I'm not sure if Windows uses LC_* environment variables. But it does support sort /locale=C. –  grawity Jul 18 '13 at 8:22
    
Yes I've tried that too, we should use it. But the issue is not the sort order. It is that entire lines are missing from the output, over 10% of the file. –  Nick P Jul 18 '13 at 23:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.