Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two file to join.


a  A1
a  A2
a  A3
c  C1
c  C2


a  feature1_of_a
a  feature2_of_a
a  featureN_of_a
c  feature1_of_c
c  feature2_of_c

after join, i could get File like this:

A1  feature1_of_a
A2  feature1_of_a
A3  feature1_of_a
A1  feature2_of_a
A2  feature2_of_a
A3  feature2_of_a
A1  featureN_of_a
A2  featureN_of_a
A3  featureN_of_a

In order to do that: i wrote shell command join -11 -21 -o1.2,2.2 file1 file2. But the problem is: number N might be huge. So if join read all feautre of a into memory at once, memory might not be enough.

I don't know how join is implemented. WQould the momery become a problem? If so, is there any way to get what I want?

share|improve this question

migrated from Dec 19 '12 at 16:01

This question came from our site for professional and enthusiast programmers.

How huge do you think N might be?

Unless it is in the 'many millions' range, or you are working in an unusually tightly constrained (memory poor) environment, you are not going to run out of memory.

share|improve this answer
I tried in my 1G memory computer like this: make 10 row in file1, all of the first collum are 'a'. Then I set file2 N=2,000,000, the size of file2 is 500M, and all of the first collum of file2 are 'a' too. But then I run the commend "join -11 -21 -o1.2,2.2 file1 file2", it crashed. Do you know why it happened? – Hancy Dec 19 '12 at 16:37
It ran out of memory because you're not being very sensible in your request. You're doing a Cartesian product of the rows in file1 and the rows in file2. I'm a little surprised it was unable to hold all of 500 MB in main memory. I don't know if it makes any difference if you reorder file1 and file2 on the command line (worth a try because it is easy), but I won't be surprised to find it makes no difference. Are you on a 32-bit machine or 64-bit? More saliently, is the join executable 32-bit or 64-bit? Are you sure the output is useful? – Jonathan Leffler Dec 19 '12 at 16:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .