[PATCH 1/2] [base] Reject combinations of incompatible FT_OPEN flags

[Oleg Oshmyan]

The three modes are mutually exclusive, and the documentation on the
`FT_OPEN_...` constants notes this. But there was no check to validate
this in the code, and the documentation on `FT_Open_Args` claimed that the
corresponding bits were checked in a well-defined order, implying it was
legal (if useless) to specify more than one. Ironically, this documented
order did not agree with the actual code, so it could not be relied upon;
hopefully, nobody did this and nobody will be hurt by the new validation.

Even if multiple mode bits were allowed, they could cause memory leaks:
if `FT_OPEN_STREAM` and `stream` is set along with either `FT_OPEN_MEMORY`
or `FT_OPEN_PATHNAME`, then `FT_Stream_New` allocated a new stream but
`FT_Open_Face` marked it as an "external" stream, so the stream object was
never released.

* include/freetype/freetype.h (FT_Open_Args): Make it clear that
exactly one of FT_OPEN_{MEMORY,STREAM,PATHNAME} bits must be present.

* src/base/ftobjs.c (FT_Stream_New): Reject incompatible FT_OPEN flags.
---
I would have preferred to use a GitLab merge request, but I keep getting
a 500 error today when trying to sign up for gitlab.freedesktop.org.
Use `git am --keep-non-patch` or `git mailinfo -b` on these patch emails
to keep the `[base]` in the subject.

 include/freetype/freetype.h | 6 ++++--
 src/base/ftobjs.c           | 8 +++++---
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/include/freetype/freetype.h b/include/freetype/freetype.h
index 29a0f7131..9230401a8 100644
--- a/include/freetype/freetype.h
+++ b/include/freetype/freetype.h
@@ -2113,8 +2113,7 @@ FT_BEGIN_HEADER
    *     Extra parameters passed to the font driver when opening a new face.
    *
    * @note:
-   *   The stream type is determined by the contents of `flags` that are
-   *   tested in the following order by @FT_Open_Face:
+   *   The stream type is determined by the contents of `flags`:
    *
    *   If the @FT_OPEN_MEMORY bit is set, assume that this is a memory file
    *   of `memory_size` bytes, located at `memory_address`.  The data are not
@@ -2127,6 +2126,9 @@ FT_BEGIN_HEADER
    *   Otherwise, if the @FT_OPEN_PATHNAME bit is set, assume that this is a
    *   normal file and use `pathname` to open it.
    *
+   *   If none of the above bits are set or if multiple are set at the same
+   *   time, the flags are invalid and @FT_Open_Face fails.
+   *
    *   If the @FT_OPEN_DRIVER bit is set, @FT_Open_Face only tries to open
    *   the file with the driver whose handler is in `driver`.
    *
diff --git a/src/base/ftobjs.c b/src/base/ftobjs.c
index 4a8014542..4783a09c7 100644
--- a/src/base/ftobjs.c
+++ b/src/base/ftobjs.c
@@ -197,6 +197,7 @@
     FT_Error   error;
     FT_Memory  memory;
     FT_Stream  stream = NULL;
+    FT_UInt    mode;
 
 
     *astream = NULL;
@@ -208,13 +209,14 @@
       return FT_THROW( Invalid_Argument );
 
     memory = library->memory;
+    mode = args->flags & ( FT_OPEN_MEMORY | FT_OPEN_STREAM | FT_OPEN_PATHNAME 
);
 
     if ( FT_NEW( stream ) )
       goto Exit;
 
     stream->memory = memory;
 
-    if ( args->flags & FT_OPEN_MEMORY )
+    if ( mode == FT_OPEN_MEMORY )
     {
       /* create a memory-based stream */
       FT_Stream_OpenMemory( stream,
@@ -224,13 +226,13 @@
 
 #ifndef FT_CONFIG_OPTION_DISABLE_STREAM_SUPPORT
 
-    else if ( args->flags & FT_OPEN_PATHNAME )
+    else if ( mode == FT_OPEN_PATHNAME )
     {
       /* create a normal system stream */
       error = FT_Stream_Open( stream, args->pathname );
       stream->pathname.pointer = args->pathname;
     }
-    else if ( ( args->flags & FT_OPEN_STREAM ) && args->stream )
+    else if ( ( mode == FT_OPEN_STREAM ) && args->stream )
     {
       /* use an existing, user-provided stream */
 
-- 
2.32.0