[DotGNU]Speed comparison ground rules

[Rhys Weatherley]

j_post wrote:

> My app compiled with cscc  and executed with ilrun runs in 10 seconds. Same
> source compiled with mcs and executed with mono runs in 36 seconds.

OK, some ground rules on speed comparisons, so we don't end up
with a "my engine is faster than your engine" slanging match
with Mono before the week is out ( :-) ):

- Always check the output of the compilers with ildasm to see if
  there is some subtle difference that is contributing to the
  speed difference.  e.g. the "ldlen" thing with "qfractals".

- Try to factor out load-time overhead.  i.e. "time ilrun app.exe"
  may appear to be faster only because our C# library is currently
  smaller and so loads quicker.  Kernel cache mechanisms can also
  play a role during loading.  Measure times after load using
  "DateTime.Now".  Load-time overhead is important, but not on
  benchmark comparisons.

- Report exactly what engine and platform you are using.  e.g.
  "mono" vs "mint"; "unrolled ilrun" vs "switch-loop ilrun".

- Report whether the app is I/O bound, CPU bound, network bound,
  or internalcall bound.

- If you don't supply the source code for the test, we won't
  believe the numbers.

- Realise that discrepancies will occur because there is no such
  thing as the "fastest possible bytecode engine".  The problem
  domain always plays a role.

- Remember that 10 seconds after you post "A is faster than B",
  the developers on B will change the code to make the statement
  false. :-)

Cheers,

Rhys.