|Subject:||Re: benchmarks (was Re: Progress on a Classpath mauve suite?)|
|Date:||Tue, 08 Mar 2005 19:26:23 -0500|
|User-agent:||Debian Thunderbird 1.0 (X11/20050116)|
David P Grove wrote:
I think you need to understand the point of the benchmark suite. The whole goal is reproducible science, so if someone doesn't cite the exact version of the benchmarks, then it isn't useful (in an academic sense). The license forces that plus proper academic credit (ie a citation) for the benchmark suite, which personally I think is quite fair given how much work was put into putting it together (much more than the typical academic paper).
Actually I have the same opinion: we need rigourous benchmark suites in academics. I'm not diametrically opposed to using non-free ones, and I do understand the value (as I said, I use a non-free suite all the time in my work, so politics in practice don't stop me from using non-free software (although it's nice to avoid it)).
Still: why can't there be high quality free Java suites for GNU Classpath VM's to test against? Furthermore, why can't academics use and distribute free software? It would seem to me that they're particularly in a good position to do so.
I know some of the people who put together the benchmark suite fairly well and it was a massive effort (think person-years, not person-weeks; good, usable, portable, benchmark suites are a lot of work). It would be a shame if your politics prevented you from using it, but that's life I suppose.
I don't mean to start a political tirade here, but if you look at all the free VM's and all of GNU Classpath, that's required a *much* larger effort than putting together one benchmark suite (composed of free or at least public domain software at that). So I don't really see why it should be a problem for interested parties to accomplish, although it obviously won't happen overnight.
Perhaps a replacement for DaCapo and SPECjvm98 is not in order: this would make comparison with other papers in the literature difficult and just confuse the issue. So, how about a good *companion* suite then?
|[Prev in Thread]||Current Thread||[Next in Thread]|