Benchmark with 2 computers and 2 benchmark suites

Question

This is a question I was asked. You have two computers and two benchmarks. First benchmark does well and 2nd benchmark does not on the first computer. The next computer does the opposite by doing bad on the first benchmark and well on the 2nd benchmark. We can assume that this difference is the same.

From this given information is it possible to compare the two computers?

I would say no because we have to know the throughput meaning we don't know the # of processes per time.

Besides that would there be any other reasons and is my reasoning correct?

Answer 1

I think your reasoning has a flaw - you say it's not possible to compare the two machines. But having your description only, we have to assume that the two benchmarks were executed in a comparable fashion - so yes, of course, you can compare the two machines with the benchmarks.

But the only thing you can say is that machine A is better running benchmark A and bad at benchmark B. And vice versa.

If you want to invalidate benchmark results, look for relevancy of the result, relevancy in the way it was executed, and differences among the participants and mode of measuring.

Edit -

for an excellent introduction into techniques and execution of computer benchmarks, have a look at

Jain, Raj: The Art of Computer Systems Performance Analysis (Wiley, 1991)

If I were allowed one, and only one, book about benchmarks, this is it.