Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have two file to join.

FILE 1:

a  A1
a  A2
a  A3
...
c  C1
c  C2
...

FILE 2:

a  feature1_of_a
a  feature2_of_a
...
a  featureN_of_a
...
...
c  feature1_of_c
c  feature2_of_c
...

after join, i could get File like this:

A1  feature1_of_a
A2  feature1_of_a
A3  feature1_of_a
A1  feature2_of_a
A2  feature2_of_a
A3  feature2_of_a
...
A1  featureN_of_a
A2  featureN_of_a
A3  featureN_of_a
...

In order to do that: i wrote shell command join -11 -21 -o1.2,2.2 file1 file2. But the problem is: number N might be huge. So if join read all feautre of a into memory at once, memory might not be enough.

I don't know how join is implemented. WQould the momery become a problem? If so, is there any way to get what I want?

share|improve this question

migrated from stackoverflow.com Dec 19 '12 at 16:01

This question came from our site for professional and enthusiast programmers.

1 Answer 1

How huge do you think N might be?

Unless it is in the 'many millions' range, or you are working in an unusually tightly constrained (memory poor) environment, you are not going to run out of memory.

share|improve this answer
    
I tried in my 1G memory computer like this: make 10 row in file1, all of the first collum are 'a'. Then I set file2 N=2,000,000, the size of file2 is 500M, and all of the first collum of file2 are 'a' too. But then I run the commend "join -11 -21 -o1.2,2.2 file1 file2", it crashed. Do you know why it happened? –  Hancy Dec 19 '12 at 16:37
    
It ran out of memory because you're not being very sensible in your request. You're doing a Cartesian product of the rows in file1 and the rows in file2. I'm a little surprised it was unable to hold all of 500 MB in main memory. I don't know if it makes any difference if you reorder file1 and file2 on the command line (worth a try because it is easy), but I won't be surprised to find it makes no difference. Are you on a 32-bit machine or 64-bit? More saliently, is the join executable 32-bit or 64-bit? Are you sure the output is useful? –  Jonathan Leffler Dec 19 '12 at 16:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.