Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

This is a question I was asked. You have two computers and two benchmarks. First benchmark does well and 2nd benchmark does not on the first computer. The next computer does the opposite by doing bad on the first benchmark and well on the 2nd benchmark. We can assume that this difference is the same.

From this given information is it possible to compare the two computers?

I would say no because we have to know the throughput meaning we don't know the # of processes per time.

Besides that would there be any other reasons and is my reasoning correct?

share|improve this question

migrated from stackoverflow.com Feb 9 '11 at 1:09

This question came from our site for professional and enthusiast programmers.

2  
So ... you have two applications, one of them gives a bigger number on the first machine than on the second, and vice-versa for the second app, and from no other information than that we're expected to figure out which is the better computer? –  Anon. Feb 8 '11 at 23:58
    
@Anon exactly. Running a IOPS test on a db server and game machine, then run 3dmark on a db server and game machine, what is better? Better for what exactly? –  Byron Whitlock Feb 9 '11 at 0:01
    
@Byron: Even worse than that, we have no idea what each "benchmark" is measuring. One of them (or both!) could be implemented as return rand(); for all we know. –  Anon. Feb 9 '11 at 0:03
    
same as saying "the latest ferrari performs well on track..while the latest mercedes van can transport all my basketball team..what's better" IMHO better is what you really need, gpu power on the game machine, cpu power and hd speed on a server? or maybe I just don't get the point of the question.. –  gpasci Feb 9 '11 at 0:07
    
@Anon, all I know is it's a C++ and Fortran benchmark. –  ranzy Feb 9 '11 at 0:10

1 Answer 1

up vote 1 down vote accepted

I think your reasoning has a flaw - you say it's not possible to compare the two machines. But having your description only, we have to assume that the two benchmarks were executed in a comparable fashion - so yes, of course, you can compare the two machines with the benchmarks.

But the only thing you can say is that machine A is better running benchmark A and bad at benchmark B. And vice versa.

If you want to invalidate benchmark results, look for relevancy of the result, relevancy in the way it was executed, and differences among the participants and mode of measuring.

Edit -

for an excellent introduction into techniques and execution of computer benchmarks, have a look at

Jain, Raj: The Art of Computer Systems Performance Analysis (Wiley, 1991)

If I were allowed one, and only one, book about benchmarks, this is it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.