Thoughts on best UNIX/Linux for python, R, C++

Question

I will be buying or building a new research workstation in the next 2 months. I am currently using a Mac Pro. I do mostly computational finance and financial econometrics with extremely large datasets (market microstructure). I program mostly in python, R, and C++. Can you suggest a great linux distro for scientific programming? What are the trade offs between Mac OS X and linux? For example, does R perform better on one vs. the other? I am leaning towards Ubuntu due to the great packages and large community support. Do you have suggestions for optimizing Ubuntu (or other distro) for high performance? Thanks for your thoughts!

Answer 1

I tend to think that Debian / Ubuntu have the most complete support -- they certainly cover things you asked about in your previous questions like Boost, Armadillo, etc pp. Plus the Python support is good, and I think that our R support is the most complete.

Debian isn't really any harder than Ubuntu in my book and the Debian testing variant gives you current software sooner than Ubuntu with the bi-annual releases. OTOH if you are new to Linux maybe Ubuntu is a little gentler. Neither is perfect and either is a pretty solid choice.

Answer 2

Gentoo!! Great package support, and highest performance possible of any distro. Also support for strange hardware. Plus it's fun to set up.

The C++ performance is almost exactly the same from Linux to OS X. I haven't used R on either platform, but the fundamental systems between OS X and linux are so similar that I wouldn't think the platform would impact performance in any significant way.

You can always download Linux and try it out. You can find a distro that will work on your current Mac.

Answer 3

You can have a good unbiased overview of the main Linux distributions at DistroWatch Major Distributions page. It covers:

• ubuntu
• fedora
• openSUSE
• debian
• Mandriva
• LinuxMint
• PCLinuxOS
• slackware
• gentoo
• CentOS
• FreeBSD (!)

Answer 4

I use Mac OS X and Ubuntu for my daily work with R and Python. I find that anything programming related is easier or as easy on Ubuntu. Installing scientific libraries is definetly easier in Ubuntu, if you don't find the package you need compiling from source is usually easy. I haven't found a significant difference in performance. Otherwise I like working on a Mac more...

You can make linear algebra operations in R significantly faster by using an optimized BLAS, see the R-admin manual http://cran.r-project.org/doc/manuals/R-admin.html for good instructions.

Edit: For R code optimization tips read the entertaining R-inferno

Answer 5

Perhaps try profiling your current apps with an eye on determining where the major bottlenecks occur. If loading/writing large datasets is the problem, then invest in a faster hard drive. If the program requires swap space, then invest in more RAM. And if the program is CPU bound, invest in a faster CPU.

Answer 6

It's a workstation. Although the machine may expend a whole night number crunching I feel safe to assume most of time you're writing new code, testing or tweaking old code, studying, and so on. Select the operating system that's most comfortable for you to work with. The one who will give you less pain administrating. I, personally, believe it matters more than minute differences in running time performance.

Answer 7

If you are not used with Unix/linux I would choose any distribution that is used by your fellow researches and which is used in your institution. If there isn't one, I would sugest Ubuntu/Debian as they have a good community, as you noted. So basicly I agree with Victor, there is not that large of a difference in how different distributions uses the hardware.

To get disk speed, running with a striped RAID would be good. Look up some more information about this, Google is your friend.

To be a bit advanced, you could also look into this:

It could also be a good investment to set up KVM and run your calculations in a virtual machine. That machine could then be stoped during upgrades of your main machine. You could also take a snapshot of your calculation machine, so you can restart from that point, if something happens (like a power out).

If you want to use virtual machines, there could be a good idea to use LVM2 on top of your RAID. Then you could quite easily create and remove logical partitions when you create and destroy virtual machines. You can also put together new RAID pools and disks into your volume if you need to store more data later. LVM on top of RAID is next to ZFS when you want large storages. But ZFS is best. You can find support for ZFS in OpenSolaris, BSD and MacOSX(?))

One important notice though. RAID is not something that can replace backup. There is to many who seems to think that. Data that is not backed up, doesn't really exists.

Answer 8

I don't know of a linux distribution specifically for scientific programming. Most distributions tend to be more generic (aside from the security / forensics ones). Like others in the thread, I'd advise using Ubuntu. I've found this the best trade off between having up to date software and a working system. Gentoo tends to need long compile times as everything is from source. Binary distributions tend to be easier to maintain, albeit with a slight (probably unnoticable) performance hit when running software. Debian unstable (testing) is exactly that! I once experienced my entire x subsystem being removed during an upgrade. Don't bother with anything without a package management system as resolving dependencies can be a sod.

OS X is a paid for product and Linux is free and has evolved into what it is today. I can't really help with the trade off / comparison as I don't use OS X and I'm an avid Linux fan, so anything I put here would be biased! Optimisation is probably more based on hardware choices than distro tweaking.

Regarding hardware, check the linux support first. New hardware probably won't work properly as drivers won't have code to handle it. Always, always check first. As to what hardware to choose, this is always a budget decision. (Solid state raid array anyone!). With large datasets, as much ram as you can cram. Fast disk drives (striped raid, as suggested by Anders).

BTW, have you looked at NVidia CUDA? Recently installed the development drivers to play around with it but haven't actually written any code to run on it yet. Multicore operations on the GPU.

Answer 9

In addition to everything else that's been mentioned, depending on how much performance you need out of your CPU/RAM, unloading a lot of the daemons and desktops will get you a few cycles. It won't get a lot, but if you're worried about squeezing every spare clock cycle, there are a lot of non-essential (GNOME/KDE/GDM, etc.) services that run by default - you can do a lot of stuff without all that overhead, and if you still want a gui while you're running some serious simulations you can use a smaller desktop like fluxbox.

Answer 10

If R's performance is really important, check out OpenCL support/performance on either platform, that is the only thing where you may experience differences...