[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Qemu-devel] [PATCH v3 00/58] json: Fixes, error reporting improvements,
From: |
Markus Armbruster |
Subject: |
[Qemu-devel] [PATCH v3 00/58] json: Fixes, error reporting improvements, cleanups |
Date: |
Thu, 23 Aug 2018 18:39:27 +0200 |
JSON is such a simple language, so writing a parser should be easy,
shouldn't it? Well, the evidence is in, and it's a lot of patches.
Summary of fixes:
* Reject ASCII control characters in strings as RFC 7159 specifies
* Reject all invalid UTF-8 sequences, not just some
* Reject invalid \uXXXX escapes
* Implement \uXXXX surrogate pairs as specified by RFC 7159
* Don't ignore \u0000 silently, map it to \xC0\80 (modified UTF-8)
* qobject_from_json() is ridicilously broken for input containing more
than one value, fix
* Don't ignore trailing unterminated structures
* Less cavalierly cruel error reporting
Topped off with tests and cleanups.
If you're into this kind of disaster relief, commit c7a3f25200c
"qapi.py: Restructure lexer and parser" was even funnier.
v3:
* Trivially rebased
* Old PATCH 41,51-52 punted to a future series
* PATCH 01 was PATCH 56 [Eric]
* PATCH 58 is new (but trivial)
* R-bys kept
* PATCH 01,06,11
- Commit message improved [Eric]
* PATCH 08,10,12,14
- from_json_str()'s unusual parameter order tidied up [Eric]
* PATCH 15,18-19,24,26,37
- Refer to RFC 8259 instead of obsolete RFC 7159
* PATCH 37-39,40
- Abbreviate interpolation as INTERP instead of INTERPOL [Eric]
* PATCH 39
- Stale comment updated
* PATCH 45
- Comma after JSON_END_OF_INPUT
v2:
* Rebased
* PATCH 01,11-14,16-18,20,22-23,29-36,41,43,45-50,53-55 otherwise
unchanged
* PATCH 57-60 are new
* R-bys kept unless noted otherwise
* PATCH 02
- Cover unrecognized keyword [Eric]
* PATCH 03
- Cover \r [Eric]
* PATCH 04-05
- Comments touched up [Eric]
* PATCH 06
- Use qmp_fd_send_raw() just for "\xff" [Eric]
* PATCH 07
- Plug memory leak [Eric]
* PATCH 08
- Delay adding coverage for \' until PATCH 09
* PATCH 09
- Cover \\\0
- Drop duplicated test case (editing accident) [Eric]
- Improve surrogate coverage
* PATCH 10
- Don't lose test coverage for \" and \'
- R-by dropped
* PATCH 15,27,38-39
- Cover unkown interpolation specification
- Cover attempt to interpolate into JSON string
- R-by of PATCH 15 dropped
* PATCH 19
- Tweak loop control once more
- R-by dropped
* PATCH 21,26
- Update for tweak to PATCH 19
- I might still drop redundant masking [Eric]
* PATCH 24
- Commit message improved
* PATCH 25
- Comment improvement [Eric]
- Commit message tweaked
* PATCH 28
- Fix error message to show both halves of an invalid surrogate pair
[Eric]
- Fix unpaired leading surrogate followed by \u escape [Paolo]
* PATCH 36
- I might still rename JSON_INTERPOL & friends [Eric]
* PATCH 37
- Document lexing interpolations is now optional [Eric]
- Move deletion of a redundant assignment from PATCH 51 [Eric]
* PATCH 37,42,51-52
- De-duplicate state transitions common to IN_START and
IN_START_INTERPOL [Eric]
* PATCH 38
- Commit message tweaked
* PATCH 39
- More legible commit message [Eric]
- Comment fix [Eric]
* PATCH 40
- Commit message typo [Eric]
* PATCH 44
- Commit message tab damage [Eric]
* PATCH 56
- More on QGA synchronization [Eric]
- I might still move this earlier in the series
Marc-André Lureau (2):
json: remove useless return value from lexer/parser
json-parser: simplify and avoid JSONParserContext allocation
Markus Armbruster (56):
docs/interop/qmp-spec: How to force known good parser state
check-qjson: Cover multiple JSON objects in same string
check-qjson: Cover blank and lexically erroneous input
check-qjson: Cover whitespace more thoroughly
qmp-cmd-test: Split off qmp-test
qmp-test: Cover syntax and lexical errors
test-qga: Clean up how we test QGA synchronization
check-qjson: Cover escaped characters more thoroughly, part 1
check-qjson: Streamline escaped_string()'s test strings
check-qjson: Cover escaped characters more thoroughly, part 2
check-qjson: Consolidate partly redundant string tests
check-qjson: Cover UTF-8 in single quoted strings
check-qjson: Simplify utf8_string()
check-qjson: Fix utf8_string() to test all invalid sequences
check-qjson qmp-test: Cover control characters more thoroughly
check-qjson: Cover interpolation more thoroughly
json: Fix lexer to include the bad character in JSON_ERROR token
json: Reject unescaped control characters
json: Revamp lexer documentation
json: Tighten and simplify qstring_from_escaped_str()'s loop
check-qjson: Document we expect invalid UTF-8 to be rejected
json: Reject invalid UTF-8 sequences
json: Report first rather than last parse error
json: Leave rejecting invalid UTF-8 to parser
json: Accept overlong \xC0\x80 as U+0000 ("modified UTF-8")
json: Leave rejecting invalid escape sequences to parser
json: Simplify parse_string()
json: Reject invalid \uXXXX, fix \u0000
json: Fix \uXXXX for surrogate pairs
check-qjson: Fix and enable utf8_string()'s disabled part
json: Have lexer call streamer directly
json: Redesign the callback to consume JSON values
json: Don't pass null @tokens to json_parser_parse()
json: Don't create JSON_ERROR tokens that won't be used
json: Rename token JSON_ESCAPE & friends to JSON_INTERP
json: Treat unwanted interpolation as lexical error
json: Pass lexical errors and limit violations to callback
json: Leave rejecting invalid interpolation to parser
json: Replace %I64d, %I64u by %PRId64, %PRIu64
json: Improve names of lexer states related to numbers
qjson: Fix qobject_from_json() & friends for multiple values
json: Fix latent parser aborts at end of input
json: Fix streamer not to ignore trailing unterminated structures
json: Assert json_parser_parse() consumes all tokens on success
qjson: Have qobject_from_json() & friends reject empty and blank
json: Enforce token count and size limits more tightly
json: Streamline json_message_process_token()
json: Unbox tokens queue in JSONMessageParser
json: Make JSONToken opaque outside json-parser.c
qobject: Drop superfluous includes of qemu-common.h
json: Clean up headers
tests/drive_del-test: Fix harmless JSON interpolation bug
json: Keep interpolation state in JSONParserContext
json: Improve safety of qobject_from_jsonf_nofail() & friends
json: Support %% in JSON strings when interpolating
json: Update references to RFC 7159 to RFC 8259
MAINTAINERS | 1 +
block.c | 5 -
docs/interop/qmp-spec.txt | 42 +-
include/qapi/qmp/json-lexer.h | 56 --
include/qapi/qmp/json-parser.h | 36 +-
include/qapi/qmp/json-streamer.h | 46 --
include/qapi/qmp/qerror.h | 3 -
include/qapi/qmp/qnum.h | 2 +-
include/qemu/unicode.h | 1 +
monitor.c | 21 +-
qapi/introspect.json | 2 +-
qapi/qmp-dispatch.c | 1 -
qapi/qobject-input-visitor.c | 5 -
qga/main.c | 15 +-
qobject/json-lexer.c | 317 ++++-----
qobject/json-parser-int.h | 54 ++
qobject/json-parser.c | 379 +++++------
qobject/json-streamer.c | 126 ++--
qobject/qbool.c | 1 -
qobject/qjson.c | 31 +-
qobject/qlist.c | 1 -
qobject/qnull.c | 1 -
qobject/qnum.c | 1 -
qobject/qobject.c | 1 -
qobject/qstring.c | 1 -
tests/Makefile.include | 3 +
tests/check-qjson.c | 1058 ++++++++++++++++--------------
tests/drive_del-test.c | 8 +-
tests/libqtest.c | 57 +-
tests/libqtest.h | 13 +
tests/qmp-cmd-test.c | 213 ++++++
tests/qmp-test.c | 252 ++-----
tests/test-qga.c | 3 +-
util/unicode.c | 69 +-
34 files changed, 1484 insertions(+), 1341 deletions(-)
delete mode 100644 include/qapi/qmp/json-lexer.h
delete mode 100644 include/qapi/qmp/json-streamer.h
create mode 100644 qobject/json-parser-int.h
create mode 100644 tests/qmp-cmd-test.c
--
2.17.1
- [Qemu-devel] [PATCH v3 00/58] json: Fixes, error reporting improvements, cleanups,
Markus Armbruster <=
- [Qemu-devel] [PATCH v3 12/58] check-qjson: Cover UTF-8 in single quoted strings, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 34/58] json: Redesign the callback to consume JSON values, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 21/58] check-qjson: Document we expect invalid UTF-8 to be rejected, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 33/58] json: Have lexer call streamer directly, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 39/58] json: Pass lexical errors and limit violations to callback, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 26/58] json: Leave rejecting invalid escape sequences to parser, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 31/58] json: remove useless return value from lexer/parser, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 22/58] json: Reject invalid UTF-8 sequences, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 29/58] json: Fix \uXXXX for surrogate pairs, Markus Armbruster, 2018/08/23
- [Qemu-devel] [PATCH v3 37/58] json: Rename token JSON_ESCAPE & friends to JSON_INTERP, Markus Armbruster, 2018/08/23