OS X “death spiral”? Can't start new processes, eventually becomes unusable

Question

I have this weird situation crop up every once in a while.

My OS X (Snow Leopard) system will simple stop being able to do anything "new". It can't start any new processes (like, for instance, a new app or a new bash process) which it hasn't loaded before. Things that are already running may be able to start again... but if it's out of cache (maybe?) it just sits there hung.

Eventually the whole system degrades, app by app, process by process, to the point where nothing responds (including Finder, UI, etc.) and I have to do a hard power cycle.

Any ideas? When it happens, I can't even start Activity Monitor or top if they aren't already running, so it's hard to pin down.

Answer 1

What do the console log, system log, and kernel log say? Better yet, use sudo syslog | more to dump out your entire syslog message database (warning: lots of output), and look for any messages from around the time the problem was happening.

Also look in /Library/Logs/DiagnosticReports for crashes or spindumps from the time the problem was happening.

Consider updating your Question with this output, or linking to it on Pastebin.com or gist.github.com. Maybe we can help you figure it out.

You don't happen to have either "Mozy" or "Hands Off!" installed, do you? I recently helped solve some problems with similar symptoms that came down to misbehaviors of those packages:

Getting console message: ipc_kmsg_copyout_header: can't grow user ipc space. Any Mac OS X kernel gurus in here?

Some Mac applications crash frequently, with "__THE_SYSTEM_HAS_NO_PORT_SETS_AVAILABLE__" in backtrace

Answer 2

This sounds a lot like the problem I was having that Spiff helped me with in: Getting console message: ipc_kmsg_copyout_header: can't grow user ipc space. Any Mac OS X kernel gurus in here?

I'll share a shell script I wrote to help track down which process might be using up ipc ports. It logs formatted top output to a file periodically and you can check it after the computer locks up and you reboot. Sorry, it's ugly

#!/usr/bin/env bash
# port_profile.sh [output_file_path]
#   use top to profile port usage for processes every 10 minutes (600s)
args=("$@")
sleep_time=600
if [ "$#" -eq 0 ]
    then
        outfile="port_profile.txt"
    else
        outfile="$@"
fi
(while true; do
    data=`top -o ports -stats ports,command,pid,msgsent,msgrecv,user,uid, -l 1 | grep -v 'CPU\|Disks\|LoadAvg\|Load Avg\|MemRegions\|Networks\|PhysMem\|Procs\|Processes\|SharedLibs\|Threads\|Time\|VirtMem\|VM\|Swap\|Purgeable'`
    while read -r line; do
        echo -e "$line"
    done <<< "$data"
    echo -e ""
    sleep $sleep_time
done) >> $outfile &