I'd look at JACK. There's a source tarball for OSX on the main project's download page, and a separate OSX-specific port project called JackOSX.
JACK is a system for handling real-time, low-latency audio (and MIDI). [...] It can connect a number of different applications to an audio device, as well as allowing them to share audio between themselves. Its clients can run in their own processes (ie. as normal applications), or can they can run within the JACK server (ie. as a "plugin").
The idea here is that you'd take your audio output from, say, VLC, then run it into a compressor plugin with JACK before sending it to your audio device.
If you want to perform a real time audio processing, you will need a third party application to perform this. I used Audio Hijack.
After installing JACK, open up Audio Hijack. Make sure that you've set it up to "Hijack" the audio for a live capture.
Next setup the audio output for VLC to JACK.
VLC's output goes to Audio Hijack's input.
Audio Hijack's output goes to the system out.
Lastly choose your filters.