Given enough detail about a PC is it possible to calculate an accurate benchmark?

Question

If I know enough about a specific computer build is it practically possible to generate a benchmark based on the specification of the components?

For example, I'll know the GPU & CPU speeds, chipset (plus all IO speeds), memory speed / channels etc and I'd like to build benchmarks based on formulas (not software tests).

This seems possible as all of the components should work at the manufacturers stated speed. The downside is that it would be a theoretical benchmark dependent on the build quality of the motherboard and components which can sometimes be variable.

My question is, is this approach feasible and if not, what are the factors preventing an accurate benchmark formula?

Has anyone seen any algorithmic approaches to benchmarking?

Note: This is related to my original post (http://superuser.com/questions/16358/given-a-processor-specification-how-can-i-figure-out-what-ram-it-can-use). Now I have more understanding of the importance of the chipset I am restating the question.

Thanks in advance,

Ryan

Answer 1

How much accuracy for a benchmark do you expect and what type of benchmark do you want to estimate? If 5-10% accuracy will suite you, then you can try. You will have to collect benchmark results from different PC magazines and sites and update your database frequently in order to be able to benchmark different configurations.

In any case you will also have to approximate some results because you will not be able to find tests for any configuration available on the market. For example you found benchmark results for chipsets X and Y for Intel processor 2GHz. Also you have benchmarks for 2.0Ghz and 2.33Ghz processors but for chipset Z. If 2.33Ghz is faster then 2.0Ghz say in 10% on chipset Z, could you expect that it will be faster in 10% on chipsets X and Y, so you can multiply you previous benchmarks on 1.1 ??? This is reasonable, but this will be only an approximate result.

Also please take into account that there is no benchmark that suite all users and each benchmark can depend more on some computer components and less on another. For example if we want to measure floating point operations, we will have better results on more powerful processor. If we will run benchmark test for an Access database, we will see that results depend on HDD performance much more then on processor's and so on.

Theoretical benchmarks are at some point become useless for end users, because what end users want to know is on what PC Vista will start faster or FireFox will open web site faster and it where it will take less time to save huge 20Mb Office document... End users rarely care about numbers of floating point operations.

Answer 2

It's a nice idea, but I don't think it's really realistic. I've seen the exact same hardware generate different benchmark results based on otherwise undetectable differences (presumably related to software).

You may be able to generate a series of expected ranges for particular hardware configurations using an evidence-based approach that aggregates actual benchmarks, but even that wouldn't be particularly accurate.

In short, I believe the answer is no, you can't.

Answer 3

This question reminds me of the popular 'how many piano tuners in a city' problem.
Now, I know a lot of people are going to find faults in that association,
this is how I'd like to explain it.
Please note, I am not saying it is the same problem -- just somewhat similar.

What your are trying to do is
derive a benchmark figure (number of piano tuners)
from various other results that may not correlate together.

You could approximate some derivation rules to arrive at a benchmark,
however, a number of parameters (subjective to each configuration in question)
will cause your computed benchmark figure to deviate arbitrarily from the actual values.

The existing benchmarking tools themselves have a hard time reproducing the results
and like to dictate a lot of things on the test conditions.

Answer 4

You have to keep in mind that there are going to be scenarios that you will not be able to predict... for example, imagine:

• Two processors A and B, where B has been shown to be about 1.5x the speed of A in otherwise identical systems
• A new motherboard X, tested with processor A, giving it a 1.1x boost over its previous score due to optimisations in the chipset

Does this mean you can conclude that B will also be 1.1x faster in X? Not necessarily; it may very well be that although the chipset is theoretically better by that factor, but it may still have some (as of yet) undetected bandwidth limit that prevents it from scaling above 1.5x the performance of A, so the better performance of B already uses up all that headroom.

And this is but one example... there are going to be innumerable limitations like this between various components in the system, and some of them are never detected until just the right combination of components is put together. As a result any algorithmic approximation will undoubtedly end up with many unexpected corner cases that will turn out wildly inaccurate.