I am looking for a Mac OS utility that will allow me to use the Compose key sequences known from Linux. Using the Option key is not an alternative as I need it to be consistent with my existing Linux installation.
Based on information from the site LaC linked to I wrote a script that converts an X.org compose table to a cocoa keybindings file. Check my writeup for more information and download links.
The Cocoa text system is one way of implementing the compose key on Mac OS X. The other way is using a dedicated keyboard layout, for instance my U.S. custom keyboard layout (I haven't seen another implementation yet).
When the compose key is implemented with a keyboard layout, it will work all over the OS in any application, not just in Cocoa applications. It can be switched on and off with the UI by selecting another keyboard layout and there is a visual feedback while typing the compose combination.
You don't need a special utility. The Cocoa text system is very configurable, and you can set up key combinations and/or key sequences to perform various actions, including inserting special characters as in compose. See this excellent document.
Of course, you're going to have to spend a bit of time converting the sequences from your Linux compose file to Cocoa dictionary format. A script could probably help here.
This is the best and easiest solution I have seen. It uses a Mac OS keyboard layout with a very simple KeyRemap4MacBook configuration.
Maybe try DoubleCommand?
DoubleCommand is software for Mac OS X (a kernel extension) that lets you remap keys, in other words change the way your keyboard works. Often used to make a PC keyboard more comfortable with a Mac, swapping the Alt (Option) and Windows (Command or Apple) keys, since they are in swapped positions on Mac and PC keyboards. DoubleCommand is a quick and easy way to fix this if you use a PC keyboard with your Mac.