Re: ftbench update: make integrated

[Ahmet Göksu]

Thank you Hin-Tak. 
I have checked the makefile of demos and used libs and the includes as there. (it was overriding the ccraw to cc)

about percentages, i runned the bench with -c 200 to have instant results for development process. here in the benchmark file attached, it made more acceptable result when increased the -c flag to 2000.

I changed compiling and the linking process as the demo programs. i would like to continue to another build system if it seem ok.

Best,

Goksu

goksu.in

On 16 Jul 2023 10:44 +0300, Werner LEMBERG <wl@gnu.org>, wrote:

* i modified benchmark program not to report 'time per op’ but
rather 'cumulative time per N iterations'
* changed the table design
* sentence 'smaller values are better’ is present
* embed a small CSS fragment at the top of the page
* linked to the original baseline and benchmark `.txt`
* everything is being created in the build directory

Nice, thanks! Now the next problem: For the same commit IDs, I see
differences in percentage up to 47% in your HTML file! This
essentially means that the delivered numbers are still completely
meaningless – the differences must be at most a few percent or even
smaller, given that the tests are run on exactly the same machine.

Please investigate how to improve that, probably by modifying the
benchmark test options, or probably even by implementing per-test
options so that the single tests can be fine-tuned. Perhaps you
should do some internet research to find how other, similar benchmark
tests are constructed to get meaningful numbers.


Werner

Benchmark Results

Warning: Baseline and Benchmark have the same commit ID

Info

Info	Baseline	Benchmark
Parameters	-c 2000	-c 2000
Commit ID	e9362ecc	e9362ecc
Commit Date	2023-07-14 16:18:00 +0300	2023-07-14 16:18:00 +0300
Branch	GSoC-2023-Ahmet	GSoC-2023-Ahmet

*Smaller values mean faster operation

Results for Roboto_subset.ttf

Test	N	Baseline (ms)	Benchmark (ms)	Difference (%)
Load	240000	1218.299	1146.114	5.9
Load_Advances (Normal)	240000	1253.112	1146.197	8.5
Load_Advances (Fast)	240000	6.242	6.113	2.1
Load_Advances (Unscaled)	240000	5.707	5.780	-1.3
Render	207120 / 197280	785.332	779.617	0.7
Get_Glyph	240000	355.068	347.508	2.1
Get_Char_Index	188000	5.013	4.963	1.0
Iterate CMap	2000	3.994	4.032	-1.0
New_Face	2000	85.614	86.143	-0.6
Embolden	240000	473.296	463.575	2.1
Stroke	55800 / 55200	1595.643	1599.108	-0.2
Get_BBox	240000	237.693	232.396	2.2
Get_CBox	240000	180.251	176.402	2.1
New_Face & load glyph(s)	240000	219.056	213.172	2.7

Results for Arial_subset.ttf

Test	N	Baseline (ms)	Benchmark (ms)	Difference (%)
Load	190000	1584.787	1450.270	8.5
Load_Advances (Normal)	190000	1418.097	1396.258	1.5
Load_Advances (Fast)	190000	4.993	5.029	-0.7
Load_Advances (Unscaled)	190000	4.693	4.711	-0.4
Render	160930 / 159980	605.964	613.998	-1.3
Get_Glyph	190000	311.771	313.342	-0.5
Get_Char_Index	188000	4.883	4.939	-1.1
Iterate CMap	2000	4.004	4.007	-0.1
New_Face	2000	102.681	102.656	0.0
Embolden	190000	419.544	416.402	0.7
Stroke	53200 / 53295	1485.566	1485.318	0.0
Get_BBox	190000	198.476	198.821	-0.2
Get_CBox	190000	143.832	146.177	-1.6
New_Face & load glyph(s)	190000	220.099	218.949	0.5

Results for TimesNewRoman_subset.ttf

Test	N	Baseline (ms)	Benchmark (ms)	Difference (%)
Load	190000	1772.051	1703.292	3.9
Load_Advances (Normal)	190000	1665.753	1689.698	-1.4
Load_Advances (Fast)	190000	5.021	5.053	-0.6
Load_Advances (Unscaled)	190000	4.639	4.657	-0.4
Render	139270 / 135185	585.516	581.286	0.7
Get_Glyph	165015 / 174325	285.711	288.795	-1.1
Get_Char_Index	188000	4.915	4.865	1.0
Iterate CMap	2000	4.008	3.990	0.4
New_Face	2000	106.830	107.081	-0.2
Embolden	150955 / 153995	448.020	461.508	-3.0
Stroke	39710 / 39330	1537.242	1539.127	-0.1
Get_BBox	172710 / 169290	214.260	221.825	-3.5
Get_CBox	165110 / 161310	134.227	137.620	-2.5
New_Face & load glyph(s)	190000	280.419	287.387	-2.5

Results for Tahoma_subset.ttf

Test	N	Baseline (ms)	Benchmark (ms)	Difference (%)
Load	190000	951.933	954.087	-0.2
Load_Advances (Normal)	190000	976.267	977.870	-0.2
Load_Advances (Fast)	190000	5.032	5.043	-0.2
Load_Advances (Unscaled)	190000	4.638	4.688	-1.1
Render	190000	666.053	677.000	-1.6
Get_Glyph	190000	298.770	314.108	-5.1
Get_Char_Index	188000	4.877	4.910	-0.7
Iterate CMap	2000	3.994	3.997	-0.1
New_Face	2000	95.020	98.911	-4.1
Embolden	190000	441.194	439.829	0.3
Stroke	59090	1563.986	1550.612	0.9
Get_BBox	190000	185.814	188.716	-1.6
Get_CBox	190000	137.206	140.941	-2.7
New_Face & load glyph(s)	190000	206.031	210.472	-2.2

Results for Verdana_subset.ttf

Test	N	Baseline (ms)	Benchmark (ms)	Difference (%)
Load	190000	921.440	861.377	6.5
Load_Advances (Normal)	190000	934.869	920.798	1.5
Load_Advances (Fast)	190000	5.066	5.069	-0.1
Load_Advances (Unscaled)	190000	4.670	4.758	-1.9
Render	190000	671.072	679.944	-1.3
Get_Glyph	190000	296.105	305.615	-3.2
Get_Char_Index	188000	4.872	4.895	-0.5
Iterate CMap	2000	3.997	3.998	-0.0
New_Face	2000	95.302	94.055	1.3
Embolden	190000	412.991	415.975	-0.7
Stroke	62035 / 61275	1571.461	1578.974	-0.5
Get_BBox	190000	186.651	190.252	-1.9
Get_CBox	190000	140.051	143.831	-2.7
New_Face & load glyph(s)	190000	202.830	225.799	-11.3

Freetype Benchmark