0

I am running multiple terminal cmds using threads in Python (don't make fun of me, there is a reason).

These cmds sometimes overpower the available memory and I have been forced to increase my swap size to avoid Out of Memory (OOA) errors.

My question is: generally, will my (or any) program complete faster by reducing threads and increasing swap space or increasing threads and reducing swap space (or is there a "sweet spot")?.

  • I imagine your operating system writing memory to disk would be the bottleneck, so ideally you want to avoid it all together. How many different cmd processes are you running? How much memory does each one use? How much RAM do you have? What are these processes doing, and what is the need to run them in parallel? – GordonAitchJay Oct 17 at 10:57
  • @GordonAitchJay Hey, thanks for the response. So, just to be clear, you are suggesting reducing the number of threads used? The amount of memory is variable as it depends on what I run the command on (my cmd is a call to a symbolic execution engine over pieces of code) but generally is in the range of 0.4-12% of my 8.3GB RAM per process. I am running a maximum of 15 of these processes (threads). The need to run them in parallel is simply to speed up the process of my output (I am estimating running this on ~3 million programs). – Daniel Connelly Oct 18 at 4:20
  • Yes, reduce the number of threads so you don't run out of RAM, to avoid writing memory to disk. Running out of swap space (or having to increase your swap size) tells me you're running too many processes at once (also, ideally, you want more RAM). I would check the available memory with psutil.virtual_memory().available before starting another process. Having 1GB before starting another seems reasonable. You should do some benchmarking - time how long it takes to complete a variety of different source files, with different values for max processes, and min memory before starting another. – GordonAitchJay Oct 18 at 12:01
  • @GordonAitchJay Wow, I have never heard of psutil, but that looks to be a great resource. Additionally, I really appreciate the tip about benchmarking. I decided to cut off the execution if the symbolic execution engine runs for more than an hour and it seems to be working fine so far. Thank you very much for your help! If I could give the best answer via comments, I would :). – Daniel Connelly Oct 18 at 19:04
  • psutil is great! You can poll the memory usage of processes with psutil.Process(pid).memory_full_info().uss, which might be useful. I don't know anything about symbolic execution engines. I'm curious, how long does it take to complete a particular bit of code? I suppose it depends on the code's complexity, number of possible branches etc. I suppose some of those 3 million programs might only take a few seconds, and others more than an hour? I posted an answer :) – GordonAitchJay Oct 19 at 5:48
0

These cmds sometimes overpower the available memory and I have been forced to increase my swap size to avoid Out of Memory (OOA) errors.

Running out of swap space (or having to increase your swap size) tells me you're running too many processes at once.

My question is: generally, will my (or any) program complete faster by reducing threads and increasing swap space or increasing threads and reducing swap space (or is there a "sweet spot")?.

The two options don't really make sense. If you increase the number of threads (and therefore RAM required) and at the same time reduce your swap size, you'll probably run out of memory (stored in both RAM and swap), causing processes to be terminated by your operating system and/or memory allocation to fail.

Running out of RAM causes your operating system to free some up by writing memory to disk (swap space). This is a relatively time-consuming task, so you really want to avoid using more memory than you have available in your RAM.

I would limit the number of processes so you don't run out of RAM, to avoid writing memory to disk. Check the available memory with psutil.virtual_memory().available before starting another process. Having 1GB before starting another seems reasonable.

You should do some benchmarking - time how long it takes to complete a variety of different source files, with different values for max processes, and min memory before starting another.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.