3

I have a bit of a strange problem with my machines:

Over time, the performance constantly drops.

I am using TensorFlow to train a neural network using the GPU. My Data are xz-compressed float32 arrays that reside on a spinning-disk drive on one machine, and on an SSD on another machine. There are about 400.000 data files. The data files are read continuously in background threads and put into a Queue, which can be at most 1000 items long.

In the training thread, the items are popped of the queue front and passed to the training in batches of 200.

After rebooting, the performance starts out with approximately 4 seconds per Batch. After several hours of training, the performance drops to as low as 16 seconds per batch.

I have timed the training somewhat in detail, and in the beginning is something like:

  • 0.05s waiting for training data to be read
  • 3.8s for processing a batch on the GPU
  • 0.3s for writing summary data.

After training, the timings are highly variable:

  • 0.5 and 4s for reading data
  • 9 to 20 s for processing a batch
  • 0.3s for writing summary data

It should be noted that during the batch processing, I monitored the output of nvidia-smi with a fairly high interval and it seems that the GPU utilization lasts at most 1 second.

This bad performance persists over multiple invocations of the training process, and after logging out and logging in. Rebooting the machine brings the times back down to the original.

Since this question, I acquired another GPU (a GTX 1080) and set up an almost identical system. The slowdown happens on the new machine as well.

Things I tried

I have checked the CPU usage, and at most 1 CPU is utilized, and it's always utilized at 100%, most of the time it's kernel thread utilization.

I have checked the memory usage, and it is 10GB (out of 11GB). This is a bit tight, but the system does not start swapping (swap stays at 30MB).

I have checked the disk usage, and apart from my code doing the data reading, there seems to not be anything strange going on.

I have checked the GPU temperature with nvidia-smi, and it always stays under 60°C.

I have checked the CPU and the Motherboard temperature, and they always stay under 65°C.

I am running out of ideas what the problem could be. Any ideas?

Specs

System 1:

  • Intel(R) Core(TM) i7 930 @ 2.80GHz, 4 Cores with Hyperthreading
  • 11 GB RAM
  • NVIDIA GeForce GTX 960 with 4 GB VRAM
  • Ubuntu 16.04.1 LTS Server, amd64 architecture
  • Proprietary NVIDIA driver, version 361.42
  • Kernel version 4.4.0-31-generic
  • Python 3.5.2
  • TensorFlow 0.9.0

System 2:

  • Intel(R) Core(TM) i7 930 @ 2.80GHz, 4 Cores with Hyperthreading
  • 11 GB RAM
  • NVIDIA GeForce GTX 1080 with 8 GB VRAM
  • Ubuntu 16.04.1 LTS Server, amd64 architecture
  • Proprietary NVIDIA driver, version 367.35
  • Kernel version 4.4.0-31-generic
  • Python 3.5.2
  • TensorFlow 0.9.0

Update

After some more testing, it seems that the slowness is volatile. Sometimes, batches are processed 10x slower than at best, but then it goes back to normal again.

I performed an strace on the process. The summary is this:
strace: Process 7351 attached
strace: Process 7351 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 95.40    0.674470         437      1545       126 futex
  4.37    0.030860        2572        12           munmap
  0.23    0.001627         814         2           madvise
  0.00    0.000000           0        13           write
  0.00    0.000000           0        10           mmap
  0.00    0.000000           0         1           access
------ ----------- ----------- --------- --------- ----------------
100.00    0.706957                  1583       126 total

This, however looks very similar when everything seems to be working normally. In detail, I have uploaded an strace file here:

https://drive.google.com/open?id=0B8TdHRNT7E-0X3F4eF9xWlRsb2s

As far as I can tell, almost all of those syscalls are futex calls. I am not quite sure what to learn from that.

migrated from stackoverflow.com Jun 7 '16 at 22:42

This question came from our site for professional and enthusiast programmers.

  • I guess you should strace/profile, find bottleneck function and return here with results. Another option - restart services one by one, reinitialize gpu device – strangeqargo Jun 7 '16 at 19:50
  • Possible duplicate of How can you profile a Python script? – ivan_pozdeev Jun 7 '16 at 20:26
  • Degradation over time feels like a memory leak. Eleven gigabytes of RAM feels like an odd hardware configuration. It may be worth the time to test with a more typical combination of memory sticks. But with the caveat that I'm just going by feel. – ben rudgers Jun 8 '16 at 1:04
  • @benrudgers: The memory layout is 6 times 2048 MB sticks, that somehow add up to 11GB in linux-land. I don't think this is a memory leak, because the system memory usage stays low. Also, why would memory leaks persist after all the processes have died? – fat-lobyte Jun 8 '16 at 12:16
  • There's data flowing on and off the GPU and that's controlled down at the kernel level. A bug in the driver might explain it. Anyway, it might be worth running sudo dmidecode --type memory to see if there is anything wonky with the RAM. Probably worth trying a different GPU driver also, i.e switching from the opensource to the proprietary or vice versa or running the latest greatest from launchpad. Could always file a bug on Github. – ben rudgers Jun 8 '16 at 17:19
1

For now, my issue seems to have been mitigated.

I have done this by installing the libgoogle-perftools-dev package, and starting every single run with:

LD_PRELOAD="/usr/lib/libmalloc.so"

This has guaranteed a much more stable performance, and I have not had a singe slowdown ever since.

So apparently, the GLIBC allocator is having a hard time collecting garbage over prolonged durations.

As to why my issue seemed to persist across invocations: I do not know. There is a certain chance, that I misinterpreted my results and that the processes slowed down independently of each other.

Either way, running my code for over a week with the new allocator and not having a single slowdown, I would call this problem solved.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.