Re: [Qemu-devel] [PATCH v9 32/37] qapi: Rework deallocation of partial struct

[Markus Armbruster]

Eric Blake <address@hidden> writes:

> Commit cee2dedb noticed that if you have a partial flat union
> (such as if an input parse failed due to a missing
> discriminator), calling the dealloc visitor could result in
> trying to dereference the NULL pointer. But the fix it proposed
> requires the use of a 'data' member in the union, which may or
> may not be the same size as other branches of the union
> (consider a 32-bit platform where one of the branches is an
> int64), so it feels fairly dirty.  A better fix is to tweak all
> of the generated visit_type_implicit_FOO() functions to avoid
> dereferencing NULL in the first place, by not visiting the
> fields if the struct pointer itself is not present, at which
> point we no longer even need visit_start_union().  And no one
> was implementing visit_end_union() callbacks.
>
> While rewriting the code, use patterns that are closer to what
> is used elsewhere in the generated visitors, by using 'goto'
> to cleanup labels rather than putting followup code under 'if'
> conditions.  The change keeps the contract that any successful
> use of visit_start_implicit_struct() will be paired with a
> matching visit_end_implicit_struct(), even if intermediate
> processing is skipped.  We are safe in checking *obj alone, as
> as the contract of visit_start_implicit_struct() requires a
> non-NULL obj.
>
> As an example of the changes to generated code:

This could be easier to understand if you show the change to the union
visit (hunks 2+3) before the change to one of its variant members (hunk
1).

> |@@ -1331,10 +1331,16 @@ static void visit_type_implicit_Blockdev
> |     Error *err = NULL;
> |
> |     visit_start_implicit_struct(v, (void **)obj, 
> sizeof(BlockdevOptionsArchipelago), &err);
> |-    if (!err) {
> |-        visit_type_BlockdevOptionsArchipelago_fields(v, obj, errp);
> |-        visit_end_implicit_struct(v);
> |+    if (err) {
> |+        goto out;
> |+    }
> |+    if (!*obj) {
> |+        goto out_obj;
> |     }
> |+    visit_type_BlockdevOptionsArchipelago_fields(v, obj, &err);
> |+out_obj:
> |+    visit_end_implicit_struct(v);
> |+out:
> |     error_propagate(errp, err);
> | }
> ...
> |@@ -1479,9 +1539,6 @@ void visit_type_BlockdevOptions(Visitor
> |     if (err) {
> |         goto out_obj;
> |     }
> |-    if (!visit_start_union(v, !!(*obj)->u.data, &err) || err) {
> |-        goto out_obj;
> |-    }

If v is the dealloc visitor, the condition is !(*obj)->u.data.
Else, it's false.

(*obj)->u.data can be null only for a partially initialized obj.

Spelling out the obvious: partially initialized objects may be visited
with the dealloc visitor only.

So, this basically boils down to "if we're deallocating a partially
initialized object, and the variant part hasn't been initialized, bypass
the switch visiting the variant part.

Your patch changes it to visit the variant part unconditionally.  The
code doing that visit needs to be able to cope with an uninitialized
part.

> |     switch ((*obj)->driver) {

Which variant do we visit?  If the tag hasn't been initialized, we
arbitrarily visit the first one.  Amazingly, this actually works, as we
shall see.

> |     case BLOCKDEV_DRIVER_ARCHIPELAGO:
> |         visit_type_implicit_BlockdevOptionsArchipelago(v, 
> &(*obj)->u.archipelago, &err);

Note that (*obj)->u.archipelago equals (*obj)->u.data by construction of
the union's C data type: the variant members are all stored boxed.

So, when (and only when, I think) we visit an uninitialized variant, the
visit function's obj parameter points to a null pointer.

            break;
        [More cases...]
> |@@ -1570,11 +1627,6 @@ void visit_type_BlockdevOptions(Visitor
        default:
            abort();
        }
> | out_obj:
> |     error_propagate(errp, err);
> |     err = NULL;
> |-    if (*obj) {
> |-        visit_end_union(v, !!(*obj)->u.data, &err);
> |-    }
> |-    error_propagate(errp, err);
> |-    err = NULL;
> |     visit_end_struct(v, &err);

Now let's see how function to visit the variant part changes.  Before:

    static void visit_type_implicit_BlockdevOptionsArchipelago(Visitor *v, 
BlockdevOptionsArchipelago **obj, Error **errp)
    {
        Error *err = NULL;

        visit_start_implicit_struct(v, (void **)obj, 
sizeof(BlockdevOptionsArchipelago), &err);
        if (!err) {
            visit_type_BlockdevOptionsArchipelago_fields(v, obj, errp);
            visit_end_implicit_struct(v);
        }
        error_propagate(errp, err);
    }

After:

    static void visit_type_implicit_BlockdevOptionsArchipelago(Visitor *v, 
BlockdevOptionsArchipelago **obj, Error **errp)
    {
        Error *err = NULL;

        visit_start_implicit_struct(v, (void **)obj, 
sizeof(BlockdevOptionsArchipelago), &err);
        if (err) {
            goto out;
        }
        if (!*obj) {
            goto out_obj;
        }
        visit_type_BlockdevOptionsArchipelago_fields(v, obj, &err);
    out_obj:
        visit_end_implicit_struct(v);
    out:
        error_propagate(errp, err);
    }

If err, no change.

Else if !*obj, i.e. we're visiting an uninitialized part with the
dealloc visitor, we now skip visiting the fields.  We still call
visit_start_implicit_struct() and visit_end_implicit_struct().

Else no change.

Taken together: when we visit a partially initialized union...

* The actual visitor must be the dealloc visitor then.

* We now execute visit_end_implicit_struct() and
  visit_end_implicit_struct().  If the tag has been initialized, we
  execute them for the part belonging to the tag value.  Else, we
  arbitrarily execute them for the part belonging to the value 0.

* The dealloc visitor's qapi_dealloc_start_implicit_struct() stores its
  obj argument for later.

  It's qapi_dealloc_end_implicit_struct() retrieves the stored obj and
  frees *obj.

  For the calls added by this patch, this is a no-op, because *obj is
  null.

* In all the cases, the additionally executed code does nothing.

Works.  The "arbitrarily execute them for tag value 0" bit is quite
ugly, though.

Aside: if we unbox the variant parts, start_implicit_struct() and
end_implicit_struct() should go away, and then things become less ugly.

> Signed-off-by: Eric Blake <address@hidden>
> Reviewed-by: Marc-André Lureau <address@hidden>
>
> ---
> v9: no change
> v8: rebase to 'name' motion
> v7: rebase to earlier context changes, simplify 'obj && !*obj'
> condition based on contract
> v6: rebase due to deferring 7/46, and gen_err_check() improvements;
> rewrite gen_visit_implicit_struct() more like other patterns
> ---
>  include/qapi/visitor-impl.h |  5 -----
>  include/qapi/visitor.h      | 12 ------------
>  qapi/qapi-dealloc-visitor.c | 26 --------------------------
>  qapi/qapi-visit-core.c      | 15 ---------------
>  scripts/qapi-visit.py       | 25 +++++++++----------------
>  5 files changed, 9 insertions(+), 74 deletions(-)
>
> diff --git a/include/qapi/visitor-impl.h b/include/qapi/visitor-impl.h
> index 913f1b0..3b68b7b 100644
> --- a/include/qapi/visitor-impl.h
> +++ b/include/qapi/visitor-impl.h
> @@ -81,11 +81,6 @@ struct Visitor
>
>      /* May be NULL; most useful for input visitors. */
>      void (*optional)(Visitor *v, const char *name, bool *present);
> -
> -    /* FIXME - needs to be removed */
> -    bool (*start_union)(Visitor *v, bool data_present, Error **errp);
> -    /* FIXME - needs to be removed */
> -    void (*end_union)(Visitor *v, bool data_present, Error **errp);
>  };
>
>  /**
> diff --git a/include/qapi/visitor.h b/include/qapi/visitor.h
> index 6e49b51..d7a0110 100644
> --- a/include/qapi/visitor.h
> +++ b/include/qapi/visitor.h
> @@ -274,16 +274,4 @@ void visit_type_any(Visitor *v, const char *name, 
> QObject **obj, Error **errp);
>   */
>  void visit_type_null(Visitor *v, const char *name, Error **errp);
>
> -/**
> - * Mark the start of visiting the branches of a union. Return true if
> - * @data_present.
> - * FIXME: Should not be needed
> - */
> -bool visit_start_union(Visitor *v, bool data_present, Error **errp);
> -/**
> - * Mark the end of union branches, after visit_start_union().
> - * FIXME: Should not be needed
> - */
> -void visit_end_union(Visitor *v, bool data_present, Error **errp);
> -
>  #endif
> diff --git a/qapi/qapi-dealloc-visitor.c b/qapi/qapi-dealloc-visitor.c
> index ede1703..1a54864 100644
> --- a/qapi/qapi-dealloc-visitor.c
> +++ b/qapi/qapi-dealloc-visitor.c
> @@ -171,31 +171,6 @@ static void qapi_dealloc_type_enum(Visitor *v, const 
> char *name, int *obj,
>  {
>  }
>
> -/* If there's no data present, the dealloc visitor has nothing to free.
> - * Thus, indicate to visitor code that the subsequent union fields can
> - * be skipped. This is not an error condition, since the cleanup of the
> - * rest of an object can continue unhindered, so leave errp unset in
> - * these cases.
> - *
> - * NOTE: In cases where we're attempting to deallocate an object that
> - * may have missing fields, the field indicating the union type may
> - * be missing. In such a case, it's possible we don't have enough
> - * information to differentiate data_present == false from a case where
> - * data *is* present but happens to be a scalar with a value of 0.
> - * This is okay, since in the case of the dealloc visitor there's no
> - * work that needs to done in either situation.
> - *
> - * The current inability in QAPI code to more thoroughly verify a union
> - * type in such cases will likely need to be addressed if we wish to
> - * implement this interface for other types of visitors in the future,
> - * however.
> - */
> -static bool qapi_dealloc_start_union(Visitor *v, bool data_present,
> -                                     Error **errp)
> -{
> -    return data_present;
> -}
> -
>  Visitor *qapi_dealloc_get_visitor(QapiDeallocVisitor *v)
>  {
>      return &v->visitor;
> @@ -227,7 +202,6 @@ QapiDeallocVisitor *qapi_dealloc_visitor_new(void)
>      v->visitor.type_number = qapi_dealloc_type_number;
>      v->visitor.type_any = qapi_dealloc_type_anything;
>      v->visitor.type_null = qapi_dealloc_type_null;
> -    v->visitor.start_union = qapi_dealloc_start_union;
>
>      QTAILQ_INIT(&v->stack);
>
> diff --git a/qapi/qapi-visit-core.c b/qapi/qapi-visit-core.c
> index 399256b..3360cdf 100644
> --- a/qapi/qapi-visit-core.c
> +++ b/qapi/qapi-visit-core.c
> @@ -76,21 +76,6 @@ void visit_end_list(Visitor *v)
>      v->end_list(v);
>  }
>
> -bool visit_start_union(Visitor *v, bool data_present, Error **errp)
> -{
> -    if (v->start_union) {
> -        return v->start_union(v, data_present, errp);
> -    }
> -    return true;
> -}
> -
> -void visit_end_union(Visitor *v, bool data_present, Error **errp)
> -{
> -    if (v->end_union) {
> -        v->end_union(v, data_present, errp);
> -    }
> -}
> -
>  bool visit_optional(Visitor *v, const char *name, bool *present)
>  {
>      if (v->optional) {
> diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
> index feef17f..7a44c13 100644
> --- a/scripts/qapi-visit.py
> +++ b/scripts/qapi-visit.py
> @@ -60,10 +60,16 @@ static void visit_type_implicit_%(c_type)s(Visitor *v, 
> %(c_type)s **obj, Error *
>      Error *err = NULL;
>
>      visit_start_implicit_struct(v, (void **)obj, sizeof(%(c_type)s), &err);
> -    if (!err) {
> -        visit_type_%(c_type)s_fields(v, obj, errp);
> -        visit_end_implicit_struct(v);
> +    if (err) {
> +        goto out;
>      }
> +    if (!*obj) {
> +        goto out_obj;
> +    }
> +    visit_type_%(c_type)s_fields(v, obj, &err);
> +out_obj:
> +    visit_end_implicit_struct(v);
> +out:
>      error_propagate(errp, err);
>  }
>  ''',

Matches the diff in the commit message.

> @@ -254,9 +260,6 @@ void visit_type_%(c_name)s(Visitor *v, const char *name, 
> %(c_name)s **obj, Error
>
>      if variants:
>          ret += mcgen('''
> -    if (!visit_start_union(v, !!(*obj)->u.data, &err) || err) {
> -        goto out_obj;
> -    }
>      switch ((*obj)->%(c_name)s) {
>  ''',
>                       c_name=c_name(variants.tag_member.name))

Likewise.

> @@ -293,16 +296,6 @@ void visit_type_%(c_name)s(Visitor *v, const char *name, 
> %(c_name)s **obj, Error
>
>      ret += mcgen('''
>  out_obj:
> -''')
> -    if variants:
> -        ret += mcgen('''
> -    error_propagate(errp, err);
> -    err = NULL;
> -    if (*obj) {
> -        visit_end_union(v, !!(*obj)->u.data, &err);
> -    }
> -''')
> -    ret += mcgen('''
>      error_propagate(errp, err);
>      err = NULL;
>      visit_end_struct(v, &err);

This one's a bit harder to see, but it actually matches, too.