Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

To compile a software package on a workstation with many CPU cores (say 12), the configuration stage often takes much longer than the actual compilation stage because ./configure does the tests one by one, while make -j runs gcc as well as other commands in parallel.

I feel that it is a huge waste of resources to have the remaining 11 cores sitting idle most of the time waiting for the slow ./configure to complete. Why does it need to do the tests sequentially? Does each test depend on each other? I can be mistaken, but it looks like the majority of them are independent.

More importantly, are there any ways to speed up ./configure?


Edit: To illustrate the situation, here is an example with GNU Coreutils

cd /dev/shm
rm -rf coreutils-8.9
tar -xzf coreutils-8.9.tar.gz
cd coreutils-8.9
time ./configure
time make -j24

Results:

# For `time ./configure`
real    4m39.662s
user    0m26.670s
sys     4m30.495s
# For `time make -j24`
real    0m42.085s
user    2m35.113s
sys     6m15.050s

With coreutils-8.9, ./configure takes 6 times longer than make. Although ./configure use less CPU time (look at "user" & "sys" times), it takes much longer ("real") because it isn't parallelized. I have repeated the test a few times (with the relevant files probably staying in the memory cache) and the times are within 10%.

share|improve this question
1  
It's ridiculous, and a shame, that there are NO good build tools. All the ones that exist are there purely due to inertia. Building binaries is such a fiddly, unpredictable thing. –  Matt Joiner Jun 18 '11 at 11:46
    
It does the tests sequentially because it would be a nightmare to find out how to do parallelism on the particular system one is running on. –  Simon Richter Jun 18 '11 at 13:07

5 Answers 5

There are many types of ./configure scripts. There are popular tools (autconf being one of them) to aide a developer in creating a ./configure script, but there's no rule that says every developer must use these tools, and then even among these tools, there can be wide variations in the way these scripts are constructed.

I'm not aware of any popular ./configure scripts that can be run in parallel. Most of the scripts built by popular tools do at least cache some or all of their results, so if you run it again (without doing a 'make clean' first, anyway), it runs much faster the second time.

That's not to say it couldn't be done... but I suspect there's little motivation for the people working on autoconf, for instance, to do that, since for most packages, the configure phase is very quick relative to the actual compilation and linking phases.

share|improve this answer
2  
There is a good reason for using these tools though: they are mature, and they keep track of a lot of the tiny details. I think Linux wouldn't be in such a great position in the embedded world if you couldn't simply point the configure script to your cross compiler and have it work out of the box 90% of the time. –  Simon Richter Jun 18 '11 at 13:05

The hard drive is the bottleneck in this case. To speed up the build, build on a system with fast drives (read: low access time). There is a lot of fuss about SSD discs, but there was some criticism regarding them not affecting the compilation time in a positive way. That is to say building on SSD was not so much faster than on a decent sata drive. I can't recall where I read this becasue the article is a couple of years old.

Anyways... Untar to ram and build from there.

mkdir /tmp/tmp 
mount -t tmpfs -o size=400M tmpfs /tmp/tmp 
cd /tmp/tmp
tar xjf somesourcetarball-1.1.33.tar.bz2
share|improve this answer
    
Thanks, but I was already compiling on /dev/shm which is a tmpfs :-) –  netvope Jun 18 '11 at 7:08

You have been smart in using ramdrive for the sourcetree to reside in, but think about it twice - what does configure do? It does its job by checking not just your sourcetree, but quite often also the system for availabilities of library, compilers, etc. In this case, the access problem sometimes resides in disk access - You will have it done much quicker if you have for example a SSD-based root file system.

share|improve this answer
    
Very good point that I completely missed! –  netvope Jun 18 '11 at 20:00
    
Unfortunately, it seems that SSDs won't help much. I tried running ./configure repeatedly but subsequent runs take almost as long as the first run. Since there is a lot of free memory in the system, I think the system is running the compilers and libraries from the memory cache without going to the disk. –  netvope Jun 19 '11 at 11:23
    
if you tried to run ./configure repeatedly (and if it is made by autoconf) it should have all the results cached and should do very well. You can post the configure script for us to have a look if you want more help. I'm quite sure there is an abundance of guru here –  bubu Jun 19 '11 at 11:24
    
I actually cleaned it up between runs (./configure is always running in a freshly extracted source tree). I'm going to add more details in the original post (space is limited here). –  netvope Jun 19 '11 at 11:45
    
I just tested without cleaning the folder up (i.e. running ./configure immediately after another ./configure) and the two runs take about the same amount of time. Does it mean caching is not working probably on my system? –  netvope Jun 19 '11 at 12:08

I recall discussions on the Autoconf mailing list about this issue from about 10 years ago, when most people actually only had one CPU core. But nothing has been done, and I suspect nothing will be done. It would be very hard to set up all the dependencies for parallel processing in configure, and do it in a way that is portable and robust.

Depending on your particular scenario, there might be a few ways to speed up the configure runs anyway. For example:

  • Use a faster shell. For example, consider using dash instead of bash as /bin/sh. (Note: Under Debian, dash is patched so that configure doesn't use it, because using it breaks a lot of configure scripts.)
  • If you run builds remotely (through ssh, for example), then I have found that the console output can be pretty slow. Consider calling configure -q.
  • If you repeatedly build the same project, consider using cache file. Call configure -C. See the Autoconf documentation for details.
  • If you build a lot of different projects, consider using a site file (config.site). Again, see the documentation.
  • Build several projects in parallel.
share|improve this answer
    
Could you please explain a little more why make can be parallelized but configure or autoconf cannot? –  netvope Jun 19 '11 at 11:13
    
Thank you for the list of suggestions –  netvope Jun 19 '11 at 11:13
    
It looks like I have some performance problems with the shell. Running sh -c "echo $i" > /dev/null 1000 times takes about 10s on this system, but only 1-2s on my other systems. –  netvope Jun 19 '11 at 11:26
    
GNU make uses quite complicated C code for starting and managing multiple processes. configure scripts are written in portable Bourne shell. It would be possible, but probably very difficult. –  Peter Eisentraut Jun 19 '11 at 20:58
1  
Sorting out the dependencies among the configure tests is actually a low complexity operation (topological sort) and has been solved in the early days of computing. The real problem is that nobody bothered to add the code to autoconf to do it and the fact that many programmers manually modify generated files. The whole system should be revamped so that configuration is no longer done by a shell script but a resident binary reading meta data files. –  billc.cn Sep 17 '11 at 0:26

If you're using the ondemand cpu governor, try using the performance one. This helps on the i7 and a8-3850 by 40-50%. Doesn't make much difference on the q9300.

On a quad core cpu, you might do

for cpu in `seq 0 3`; do sudo cpufreq-set -g performance -c $cpu; done

(The -r option should make it so you don't have to do cpufreq-set for each core, but on my computers it doesn't work.)

The cache option helps even more, though.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.