2

I'm running Ubuntu 12.04, and I am having a major problem. Two separate times today, one of my open gnome-terminal sessions locked up and printed the error

pipe: too many open files

I realize that is likely related to my user limit shown with ulimit -n which, by the way, is 1024. The problem is that once this happens, I can force quit the offending terminal, but then I cannot open anything that will generate a new PID. For example, I can't use xkill to kill anything, I cannot open another terminal to look at top, I cannot run shutdown, I cannot drop to a console only and login. Clicking on any link that should open an X window yields a new window that says:

There was an error launching the application.
Details: Failed to fork (Cannot allocate memory)

This is the same error I get when I try to login after pressing Ctrl+Alt+F1. Because I get completely stuck at this point, I am forced to hardboot my machine. This is making it extremely difficult to debug.

Is there anything I can do to recover once this error is encountered? How should I go about tracking down the error?

As a final note, I have not been running a ridiculous number of things: two terminals, one with a byobu session with 2 tabs, one with a python session; a single instance of emacs, a single google chrome, and several ROS (robot operating system) nodes.

EDIT

Pressing Ctrl+Alt+Backspace to restart my display manager just allowed me to log back in, and start opening a terminal. Using ps revealed that there were still many processes running that had been running with the lockup happened. At the very least, this allows me to reboot my machine from software. How should I proceed to determine what is causing this problem?

EDIT 2

I just opened two terminals that I just set aside, and waited for the crash to happen. Once it happened, I was able to capture a screenshot from the output of top. Don't know if the output will be particularly useful, but I have attached it. The only thing I noticed that was particularly odd was that there were 5 zombie processes.enter image description here

6
  • And increasing your ulimit did what, exactly?
    – user3463
    Sep 6, 2012 at 22:26
  • I never increased my ulimit, it was set at 1024 by default. Sep 6, 2012 at 22:28
  • Can you start a top running in another window before the problem occurs, and then go to it and see what it says? Have you tried increasing your ulimit after the problem occurs (it doesn't fork a new process)? Sep 6, 2012 at 22:32
  • Yes, what happened when you increased it past the default ... ?
    – user3463
    Sep 7, 2012 at 2:35
  • I did not increase it past the default. Sep 7, 2012 at 17:00

3 Answers 3

2

It would appear you have a resource leak somewhere. You can run 'lsof' to get a list of all open files (obviously at some point before your system is starved). Compare the list to active pids and you'll (maybe) find the culprit.

I'd also suggest changing the open file limit in /etc/security/limits.conf (or maybe somewhere else? I've not followed Ubuntu for a while and there seems to be some speculation that limits.conf isn't loaded by default). If you really do have a resource leak, that will only prolong the problem. Look for a leak first.

1
  • This did indeed seem to be the case. After running top in one terminal while very carefully starting up the same processes that caused the crash to happen, I was able to see a process with a definite memory leak. I'm still not sure why I was getting the too many files warning i.e. I'm not sure how it was spawning many child processes. However, fixing the leak appears to have solved the problem. Thanks for the help! Sep 7, 2012 at 23:18
1

The too many open files message is probably process specific, not a system-wide problem. The "cannot allocate memory" could be process specific, but is probably system-wide.

As cydonian.monk pointed out, there is a resource leak of some kind. The most likely culprit is a runaway or misbehaving process that is either spawning children repeatedly, or has a bug or memory leak that is causing it to repeated open files and/or allocate memory to the point that the system is becoming memory starved.

I would focus on what you were doing in the one gnome terminal window that first got the too many files open message. Determine what was running from/in that window and focus on its behavior. Running ps is cron every few minutes or running top -b and saving the output to a file may give you a glimpse into what was going on when the next crash happens.

The top output you provided looks like a normally running system.

0

Have you tried increasing the upper limit for the number of files?

What you need to do is to add these lines into /etc/security/limits.conf

* soft nofile 6000
* hard nofile 6000

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .