0

I have 2 machines that are exactly the same hardware-wise. On them, I have 1 Debian, and 1 custom build with buildroot linux image.

The problem I'm experiencing is that:

For my application, I'm getting roughly 2x performance on the Debian machine in the relation to custom build image machine.

I'm trying to understand what could cause this difference.

The cuda version is 7.5
The driver versions are:
Debian: 361.28
Custom: 367.27

Some information that may be relevant:
If I run deviceQuery (from cuda samples) on both machines, the results are almost but not exactly the same:
1st difference is in line:
This is for Debian:
Total amount of global memory: 3069 MBytes
And this is for Custom:
Total amount of global memory: 3008 MBytes
(Though I doubt that this difference can cause the mentioned difference in performance.)
The other difference is:
Debian:
Run time limit on kernels: Yes
Custom:
Run time limit on kernels: No

(There is X server on both systems, but on a Custom one it's a barebones X with twm)

If I run the bandwidthTest from samples, the results are more/less the same for Host to Device Bandwidth, 1 Device(s) and Device to Host Bandwidth, 1 Device(s), but can differ significantly for Device to Device Bandwidth, 1 Device(s)
So for the last entry values on Debian is in the area of 100k MB/s (though sometimes it drops down to around 63k)
For Custom it's consistently on the level of 63k.

I would be glad if you could help me with advice on what should I investigate further and what could be the problem.
Thank you!

UPD: I've tried running X server on a Custom system using nvidia for that. (running X with intel is not influencing perf in any noticeable way). It's not fully starting but also not crashing.
Interesting fact - if I start my application when X is in this half-started state - performance drops even more (more than 4x decrease from the Debian in total)

Upd2: I've just tried updating to CUDA 8 on a custom machine - hasn't changed the described situation.

Upd3: http://cuda-z.sourceforge.net/ Shows 2x or more perf reduction on a Custom system for every type of operation it can measure. (And also almost 2x on device to device memory copy)

2
  • A 100k MB/s line? Wow, I wish I had that. On second thought, I wish anything like that existed at all. Jun 29, 2016 at 6:50
  • Well, that's how the command output looks like :) : Device to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 99241.2
    – Stvad
    Jun 29, 2016 at 7:45

1 Answer 1

0

Ok, what finally helped was updating the way nvidia driver was installed to get a full set of nvidia utils (changing default buildroot nvidia mk). And running application after starting X server. Though this remains the same: Run time limit on kernels: No

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .