Let's say I've a set of deterministic programs that take some data as input. I'm interested to calculate the "time" until a program terminates for different input data. I don't care about the actual time it takes but I need a metric that is
- reproducible (if I run a specific program with specific data twice, I get at least approximately the same result) and
- comparable (if I run e.g. two different programs with the same data and the returned metric for the first program is 100 and the one for the second program is 50, this must allow to claim that the first program takes twice as long to run with this input data).
My setup is a virtual machine that is hosted on a server that has substantially different loads at different times of the day. My limited understanding would suggest that in case the host server is strongly used, the guest virtual machine gets less compute time and runs slower, so using something like wall-time is certainly not going to work. I've full control over the virtual machine. I don't have control over the host server.
Does anyone of you know how I can achieve my goal reliably?