Hello Werner and Alexei,
> in Chrome, I am encountering issues rolling to latest FreeType ToT, see
> details in
> https://bugs.chromium.org/p/chromium/issues/detail?id=962932
Assuming that you are testing with the current git, please post one or
two images of the pixel differences so that Alexei can have a closer
look.
I appreciate your and Alexei's individual help in looking at failures, but what I really would like to focus on is a different problem here, and it has been discussed on this mailing list before without any immediate outcome.
In my and others' opinion, it is a disproportionate burden for embedders to perform pixel testing and a portion of FreeType QA for cases which should really be caught by FreeType's own continuous integration.
Several aspects:
1) In a case like the rolling problems I am describing above: How would I reproduce and demonstrate this with FreeType's own tools? Does the FreeType repository contain a tool to compute a pixel difference between two revisions? I am not aware of one. This is what makes it hard to report issues like the ones I mentioned and introduces additional friction. And on a larger scope, when pixel differences are not caught by FT's own testing, ending up for embedders to be bisected, analysed and reviewed. Relying on embedders such as Skia and Chromium to do the work of reliable pixel testing increases cost and duplicates efforts. Costs are lower when breakage is caught as early as possible. Also, if commits in FreeType are adding new test baselines, it is much clearer for the embedders why a change caused pixel differences and what was the reasoning behind that.
2) On previous and current arguments recommending against rolling trunk and waiting for releases: I hope we can really all agree on a trunk-based development approach where we aim for trunk to be of release quality. If trunk happens to break, changes need to be reverted and revisited before they get merged to trunk again. Chromium's effort in allowing easier rolling of FreeType have often helped to identify issues early and after single commits, instead of increasing the pain having to bisect a large commit history later to identify issues.
4) From the Chromium embedding point of view commit messages for pixel-difference introducing changes ideally justify why that is required and why such differences were considered necessary. In the list of commits above: 7a81b63abc2b3da0d7f0950f69377d2b3f54b0fb just says:
"Optimize Bézier bisections" - no bug reference, no benchmark results, and no reasonable explanation of how a tradeoff between causing pixel difference vs. presumed performance and readability improvements have been weighed against each other. In 2ea511eed81770f423544525adebf7f954b8be93 the justification is: "The previous implementation is correct but it is too complex." - This is not obvious. How has this been weighted in terms of performance vs. quality and changing pixel results, how was this evaluated?
5) There are quite a few commits in the FreeType git history along the lines of "Oops", "Fix minor after previous commit" etc. - No one's perfect and I am not arguing mistakes can't happen, but if there was continuous integration with reliable pixel level coverage, trybots would help avoid such commits and keep master clean and ready to use, close to release quality.
As a conclusion:
a) With the above, I am envisioning and strongly hoping for development model where master has CI which runs tests, to the pixel level. A change triggering pixel differences should have sufficient justification and evaluation, and bring with it updated rebaselines so it can keep the tree green. In my opinion, this is really a high priority thing to work on, ranked above feature extensions.
b) I would prefer 7a81b63abc2b3da0d7f0950f69377d2b3f54b0fb to be reworked so it doesn't cause pixel differences if possible, or reverted. Similarly with the current reasoning, I do not find 2ea511eed81770f423544525adebf7f954b8be93 a change that is strictly required or justified.
I appreciate the hard work that is going in to FreeType and the readiness to respond to and fix issues often within minutes, this is amazing and it's why it's great to work with the FreeType library. My main focus is thus not to say: The FreeType contributors should do _more now_. My main focus is to build consensus that a) is something we want to do and consider useful, then find resources and engineering time to address this. On the Chromium side, we're looking into how we could be supporting this.
Dominik