1

Disclaimer. First I am NOT a SU. I'm a mid level computer user with a bit of experience. I can write some basic code, have used computers for 20+ years. MSDOS, to Windows through the years now I'm on OS X. I've searched high and low for a few weeks now and so have decided to come to the big guns to ask for assistance. I hope you're OK with this.

Firstly I'm on OS X 10.11.6.

I have been having an issue with my Crashplan backup software and was working through it with their support people. It all came down to the error:

java.io.FileNotFoundException (Too many open files in system)

I read about and found I had to increase the number of open files allowed on my system. I did this creating and changing the values of files located in /Library/LaunchDaemons.

I created two files: limit.maxfiles.plist and limit.maxproc.plist

Within limit.maxfiles.plist I placed:

    <?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"  
        "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">  
  <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
      <string>launchctl</string>
      <string>limit</string>
      <string>maxfiles</string>
      <string>54000</string>
      <string>54000</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
  </dict>
</plist>  

and within limit.maxproc.plist I placed:

    <?xml version="1.0" encoding="UTF-8"?>  
<!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">  
  <plist version="1.0">
    <dict>
      <key>Label</key>
        <string>limit.maxproc</string>
      <key>ProgramArguments</key>
        <array>
          <string>launchctl</string>
          <string>limit</string>
          <string>maxproc</string>
          <string>4096</string>
          <string>4096</string>
        </array>
      <key>RunAtLoad</key>
        <true />
      <key>ServiceIPC</key>
        <false />
    </dict>
  </plist>

I restarted and checked this had taken affect by running launchctl limit maxfiles at the terminal. This showed that it was in effect.

I started crashplan again and it gave the same error. I checked with the crashplan people and they basically said tough, that's the problem, go fix it (which is fair enough I think).

So, I kept upping the number of files allowed to be open until I got to 900000000). Yes, that's a total of 900 million files allowed open. More than I have on my total drive. The error still persists. Checking with launchctl limit maxfiles it shows that it is in effect.

So, that's left me stumped. I then looked around and found that I can run a command like so ulimit -S -n 900000 which will give some other part of the system more files to play with. I'm not sure how that works, but I checked it ulimit -a and it was in effect. Same problem with Crashplan though.

So now I'm not sure what to do so I've come to ask the people who know OS's better than most. Can anyone offer some guidance as to what I can try to resolve this?

Thanks so much.

3
  • 1
    First you need to find out what is opening so many files. From what you've said increasing the max open files is just a bandaid on the root cause. How about going into Activity Monitor and see if there is some application whose memory usage grows continuously? Sort by Memory; note what's at the top and what its memory usage is; go away for a while and see if it has grown. I'm betting it's not CrashPlan. This Q&A might also give some clues: stackoverflow.com/q/20974438
    – jwd630
    Feb 17, 2017 at 4:04
  • Thank you for that. I ran a couple of tests over the weekend with screenshots of the activity monitor and there is basically little to no memory changes for all the open apps. I have 16Gig of RAM and I've consistently had about 10Gig free the whole time.
    – robster
    Feb 20, 2017 at 2:19
  • Then I would try something a bit more low level. From a terminal window while true; do ts=$(date +%H_%M); sudo lsof > open_files_${ts}; wc -l open_files_${ts}; sleep 60; done will capture process that have files open and count how many there once a minute. If the number is growing you can see the differences by diffing the results, e.g.: diff open_files_13_13 open_files_13_14 Don't be fooled since there are bound to be some differences; but if one process stands out consistently you may have a culprit. This will make a new, large file every minute until you interrupt it: Control-C
    – jwd630
    Feb 20, 2017 at 18:29

1 Answer 1

1

Maybe this actually answers the question for you. Paste the following into a Terminal window:

while true; do
    sudo lsof |
       awk '{ files[$1]++ } END { for (key in files) { print key, files[key] } }' |
       sort -n -k 2 | tail -10 
    sleep 60;
done

What that is going to do:

  • while true - Until you interrupt it, e.g., Control-C
  • lsof - List all the open files on the system and the name of the process (command) that has those files open
  • awk - Count the number of the open files by process
  • sort - Sort that list by number of open files
  • tail - Report the 10 processes with the most files open
  • sleep - Wait a minute and do the whole thing again

You should either see the culprit immediately or it will start moving toward the end of the list before too long.

6
  • Thank you for this. Without this help I would be stuck and have no backup. Well, I still have no backup, but I have hope! :-) That's some nice terminal skills indeed. Here is the result after first running your script. Then loading Capsule interface, then letting it sit for a few minutes: mdworker 158 AppleSpel 174 Resilio 177 Telegram 210 com.apple 245 Spotlight 248 UserEvent 278 CrashPlan 331 Dropbox 353 firefox 407 mdworker 151 AppleSpel 174 Resilio 177 Telegram 210 com.apple 245 Spotlight 246 UserEvent 278 Dropbox 354 firefox 389 CrashPlan 581
    – robster
    Feb 21, 2017 at 6:48
  • then... mdworker 151 AppleSpel 174 Resilio 175 Telegram 210 com.apple 245 Spotlight 246 UserEvent 278 Dropbox 354 firefox 377 CrashPlan 727 mdworker 163 AppleSpel 174 Resilio 175 Telegram 210 Spotlight 250 com.apple 269 UserEvent 278 Dropbox 354 firefox 374 CrashPlan 580 mdworker 163 AppleSpel 174 Resilio 175 Telegram 210 Spotlight 250 com.apple 265 UserEvent 278 Dropbox 354 firefox 375 CrashPlan 574 Sorry for the layout, when I added this it was in a nice vertical layout but superuser.com interface has changed it to look messy.
    – robster
    Feb 21, 2017 at 6:50
  • I've also added a pastebin version of another 3 minutes of output here for easier reading pastebin.com/MEpE1FKQ
    – robster
    Feb 21, 2017 at 6:55
  • As you can see CrashPlan is actually reducing the number of open files in that last 3 minute: starts at 638, goes down to 580, then up to 581. I suggest just letting this script run while you go about your business. If/when CrashPlan next has its error see what processes are actually keeping lots of open files at that point. These are not unusual numbers.
    – jwd630
    Feb 22, 2017 at 18:16
  • Thank you again. Unfortunately Crashplan had its error about 6 times in that 3 minute period. So the error was happening whilst the logs were being created. I increased the fidelity to every 10 seconds and this is what I got pastebin.com/S0s5DHVT It's still not massive numbers though. So in that time shown, crashplan gave the error many time.
    – robster
    Feb 23, 2017 at 22:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .