[Qemu-commits] [qemu/qemu] 6d49d3: luks: extract qcrypto_block_calculate_payload_offs...

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 6d49d3a859b0f19226dbb0df5e7f50267b42f45c
      
https://github.com/qemu/qemu/commit/6d49d3a859b0f19226dbb0df5e7f50267b42f45c
  Author: Stefan Hajnoczi <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/qcow2.c
    M crypto/block.c
    M include/crypto/block.h

  Log Message:
  -----------
  luks: extract qcrypto_block_calculate_payload_offset()

The qcow2 .bdrv_measure() code calculates the crypto payload offset.
This logic really belongs in crypto/block.c where it can be reused by
other image formats.

The "luks" block driver will need this same logic in order to implement
.bdrv_measure(), so extract the qcrypto_block_calculate_payload_offset()
function now.

Signed-off-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: a9da6e49d8281e65d172ed843807a613651d4ef3
      
https://github.com/qemu/qemu/commit/a9da6e49d8281e65d172ed843807a613651d4ef3
  Author: Stefan Hajnoczi <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/crypto.c

  Log Message:
  -----------
  luks: implement .bdrv_measure()

Add qemu-img measure support in the "luks" block driver.

Signed-off-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: c3673dcf08c7baf900bf572f6d96d8a7bd565c9d
      
https://github.com/qemu/qemu/commit/c3673dcf08c7baf900bf572f6d96d8a7bd565c9d
  Author: Stefan Hajnoczi <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M qemu-img.c
    M tests/qemu-iotests/178
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/178.out.raw

  Log Message:
  -----------
  qemu-img: allow qemu-img measure --object without a filename

In most qemu-img sub-commands the --object option only makes sense when
there is a filename.  qemu-img measure is an exception because objects
may be referenced from the image creation options instead of an existing
image file.  Allow --object without a filename.

Signed-off-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: c13de3b32fdeffb3e53494fd7d5cecb67cb65799
      
https://github.com/qemu/qemu/commit/c13de3b32fdeffb3e53494fd7d5cecb67cb65799
  Author: Stefan Hajnoczi <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    A tests/qemu-iotests/288
    A tests/qemu-iotests/288.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: add 288 luks qemu-img measure test

This test exercises the block/crypto.c "luks" block driver
.bdrv_measure() code.

Signed-off-by: Stefan Hajnoczi <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
[mreitz: Renamed test from 282 to 288]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 7788a319399f17476ff1dd43164c869e320820a2
      
https://github.com/qemu/qemu/commit/7788a319399f17476ff1dd43164c869e320820a2
  Author: David Edmondson <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/curl.c

  Log Message:
  -----------
  block/curl: HTTP header fields allow whitespace around values

RFC 7230 section 3.2 indicates that whitespace is permitted between
the field name and field value and after the field value.

Signed-off-by: David Edmondson <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 69032253c33ae1774233c63cedf36d32242a85fc
      
https://github.com/qemu/qemu/commit/69032253c33ae1774233c63cedf36d32242a85fc
  Author: David Edmondson <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/curl.c

  Log Message:
  -----------
  block/curl: HTTP header field names are case insensitive

RFC 7230 section 3.2 indicates that HTTP header field names are case
insensitive.

Signed-off-by: David Edmondson <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 69135eb30b9c3fca583737a96df015174dc8e6dd
      
https://github.com/qemu/qemu/commit/69135eb30b9c3fca583737a96df015174dc8e6dd
  Author: Eric Blake <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M tests/qemu-iotests/common.rc

  Log Message:
  -----------
  iotests: Fix nonportable use of od --endian

Tests 261 and 272 fail on RHEL 7 with coreutils 8.22, since od
--endian was not added until coreutils 8.23.  Fix this by manually
constructing the final value one byte at a time.

Fixes: fc8ba423
Reported-by: Andrey Shinkevich <address@hidden>
Signed-off-by: Eric Blake <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 4aebf0f0da4a57204f568deed14661cb37b4ac30
      
https://github.com/qemu/qemu/commit/4aebf0f0da4a57204f568deed14661cb37b4ac30
  Author: Pan Nengyuan <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/qcow2.c

  Log Message:
  -----------
  block/qcow2: do free crypto_opts in qcow2_close()

'crypto_opts' forgot to free in qcow2_close(), this patch fix the bellow leak 
stack:

Direct leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f0edd81f970 in __interceptor_calloc (/lib64/libasan.so.5+0xef970)
    #1 0x7f0edc6d149d in g_malloc0 (/lib64/libglib-2.0.so.0+0x5249d)
    #2 0x55d7eaede63d in qobject_input_start_struct 
/mnt/sdb/qemu-new/qemu_test/qemu/qapi/qobject-input-visitor.c:295
    #3 0x55d7eaed78b8 in visit_start_struct 
/mnt/sdb/qemu-new/qemu_test/qemu/qapi/qapi-visit-core.c:49
    #4 0x55d7eaf5140b in visit_type_QCryptoBlockOpenOptions 
qapi/qapi-visit-crypto.c:290
    #5 0x55d7eae43af3 in block_crypto_open_opts_init 
/mnt/sdb/qemu-new/qemu_test/qemu/block/crypto.c:163
    #6 0x55d7eacd2924 in qcow2_update_options_prepare 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1148
    #7 0x55d7eacd33f7 in qcow2_update_options 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1232
    #8 0x55d7eacd9680 in qcow2_do_open 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1512
    #9 0x55d7eacdc55e in qcow2_open_entry 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1792
    #10 0x55d7eacdc8fe in qcow2_open 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:1819
    #11 0x55d7eac3742d in bdrv_open_driver 
/mnt/sdb/qemu-new/qemu_test/qemu/block.c:1317
    #12 0x55d7eac3e990 in bdrv_open_common 
/mnt/sdb/qemu-new/qemu_test/qemu/block.c:1575
    #13 0x55d7eac4442c in bdrv_open_inherit 
/mnt/sdb/qemu-new/qemu_test/qemu/block.c:3126
    #14 0x55d7eac45c3f in bdrv_open 
/mnt/sdb/qemu-new/qemu_test/qemu/block.c:3219
    #15 0x55d7ead8e8a4 in blk_new_open 
/mnt/sdb/qemu-new/qemu_test/qemu/block/block-backend.c:397
    #16 0x55d7eacde74c in qcow2_co_create 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:3534
    #17 0x55d7eacdfa6d in qcow2_co_create_opts 
/mnt/sdb/qemu-new/qemu_test/qemu/block/qcow2.c:3668
    #18 0x55d7eac1c678 in bdrv_create_co_entry 
/mnt/sdb/qemu-new/qemu_test/qemu/block.c:485
    #19 0x55d7eb0024d2 in coroutine_trampoline 
/mnt/sdb/qemu-new/qemu_test/qemu/util/coroutine-ucontext.c:115

Reported-by: Euler Robot <address@hidden>
Signed-off-by: Pan Nengyuan <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: fc124ea1dbfac2a1d5b851596831c9da746b7b66
      
https://github.com/qemu/qemu/commit/fc124ea1dbfac2a1d5b851596831c9da746b7b66
  Author: Pan Nengyuan <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: free memory before re-assign

collect_image_check() is called twice in img_check(), the filename/format will 
be alloced without free the original memory.
It is not a big deal since the process will exit anyway, but seems like a clean 
code and it will remove the warning spotted by asan.

Reported-by: Euler Robot <address@hidden>
Signed-off-by: Pan Nengyuan <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: e7266570f2cf7b3ca2a156c677ee0a59d563458b
      
https://github.com/qemu/qemu/commit/e7266570f2cf7b3ca2a156c677ee0a59d563458b
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/qcow2-threads.c

  Log Message:
  -----------
  block/qcow2-threads: fix qcow2_decompress

On success path we return what inflate() returns instead of 0. And it
most probably works for Z_STREAM_END as it is positive, but is
definitely broken for Z_BUF_ERROR.

While being here, switch to errno return code, to be closer to
qcow2_compress API (and usual expectations).

Revert condition in if to be more positive. Drop dead initialization of
ret.

Cc: address@hidden # v4.0
Fixes: 341926ab83e2b
Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-Id: <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Reviewed-by: Ján Tomko <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 01fe1ca945345d3dc420d70c69488143dc0451b1
      
https://github.com/qemu/qemu/commit/01fe1ca945345d3dc420d70c69488143dc0451b1
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M blockjob.c
    M include/qemu/job.h
    A include/qemu/progress_meter.h
    M job-qmp.c
    M job.c
    M qemu-img.c

  Log Message:
  -----------
  job: refactor progress to separate object

We need it in separate to pass to the block-copy object in the next
commit.

Cc: address@hidden
Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: d0ebeca14a585f352938062ef8ddde47fe4d39f9
      
https://github.com/qemu/qemu/commit/d0ebeca14a585f352938062ef8ddde47fe4d39f9
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/backup.c
    M block/block-copy.c
    M include/block/block-copy.h

  Log Message:
  -----------
  block/block-copy: fix progress calculation

Assume we have two regions, A and B, and region B is in-flight now,
region A is not yet touched, but it is unallocated and should be
skipped.

Correspondingly, as progress we have

  total = A + B
  current = 0

If we reset unallocated region A and call progress_reset_callback,
it will calculate 0 bytes dirty in the bitmap and call
job_progress_set_remaining, which will set

   total = current + 0 = 0 + 0 = 0

So, B bytes are actually removed from total accounting. When job
finishes we'll have

   total = 0
   current = B

, which doesn't sound good.

This is because we didn't considered in-flight bytes, actually when
calculating remaining, we should have set (in_flight + dirty_bytes)
as remaining, not only dirty_bytes.

To fix it, let's refactor progress calculation, moving it to block-copy
itself instead of fixing callback. And, of course, track in_flight
bytes count.

We still have to keep one callback, to maintain backup job bytes_read
calculation, but it will go on soon, when we turn the whole backup
process into one block_copy call.

Cc: address@hidden
Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 9d31bc53facf53d6f817c2472d4d0485d7bc5d8e
      
https://github.com/qemu/qemu/commit/9d31bc53facf53d6f817c2472d4d0485d7bc5d8e
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/block-copy.c

  Log Message:
  -----------
  block/block-copy: specialcase first copy_range request

In block_copy_do_copy we fallback to read+write if copy_range failed.
In this case copy_size is larger than defined for buffered IO, and
there is corresponding commit. Still, backup copies data cluster by
cluster, and most of requests are limited to one cluster anyway, so the
only source of this one bad-limited request is copy-before-write
operation.

Further patch will move backup to use block_copy directly, than for
cases where copy_range is not supported, first request will be
oversized in each backup. It's not good, let's change it now.

Fix is simple: just limit first copy_range request like buffer-based
request. If it succeed, set larger copy_range limit.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 2d57511a88e71485f745ad3dc0afe03b87a8ad5e
      
https://github.com/qemu/qemu/commit/2d57511a88e71485f745ad3dc0afe03b87a8ad5e
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/block-copy.c
    M block/trace-events

  Log Message:
  -----------
  block/block-copy: use block_status

Use bdrv_block_status_above to chose effective chunk size and to handle
zeroes effectively.

This substitutes checking for just being allocated or not, and drops
old code path for it. Assistance by backup job is dropped too, as
caching block-status information is more difficult than just caching
is-allocated information in our dirty bitmap, and backup job is not
good place for this caching anyway.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 17187cb646913356bbd434bebdcddf43f92ce31a
      
https://github.com/qemu/qemu/commit/17187cb646913356bbd434bebdcddf43f92ce31a
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/block-copy.c

  Log Message:
  -----------
  block/block-copy: factor out find_conflicting_inflight_req

Split find_conflicting_inflight_req to be used separately.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: dafaf13593de240724a210e72da66f9d162735c3
      
https://github.com/qemu/qemu/commit/dafaf13593de240724a210e72da66f9d162735c3
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/block-copy.c
    M include/block/block-copy.h

  Log Message:
  -----------
  block/block-copy: refactor interfaces to use bytes instead of end

We have a lot of "chunk_end - start" invocations, let's switch to
bytes/cur_bytes scheme instead.

While being here, improve check on block_copy_do_copy parameters to not
overflow when calculating nbytes and use int64_t for bytes in
block_copy for consistency.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 8719091f9d591595ea6876d25de9c0956951c513
      
https://github.com/qemu/qemu/commit/8719091f9d591595ea6876d25de9c0956951c513
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/block-copy.c
    M include/block/block-copy.h

  Log Message:
  -----------
  block/block-copy: rename start to offset in interfaces

offset/bytes pair is more usual naming in block layer, let's use it.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 5332e5d21060c48511a73bc5dccd15960f7fa395
      
https://github.com/qemu/qemu/commit/5332e5d21060c48511a73bc5dccd15960f7fa395
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/block-copy.c

  Log Message:
  -----------
  block/block-copy: reduce intersecting request lock

Currently, block_copy operation lock the whole requested region. But
there is no reason to lock clusters, which are already copied, it will
disturb other parallel block_copy requests for no reason.

Let's instead do the following:

Lock only sub-region, which we are going to operate on. Then, after
copying all dirty sub-regions, we should wait for intersecting
requests block-copy, if they failed, we should retry these new dirty
clusters.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 397f4e9d83e9c0000905f0a988ba1aeda162571c
      
https://github.com/qemu/qemu/commit/397f4e9d83e9c0000905f0a988ba1aeda162571c
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/backup-top.c
    M block/backup.c
    M block/block-copy.c
    M include/block/block-copy.h

  Log Message:
  -----------
  block/block-copy: hide structure definitions

Hide structure definitions and add explicit API instead, to keep an
eye on the scope of the shared fields.

Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Andrey Shinkevich <address@hidden>
Reviewed-by: Max Reitz <address@hidden>
Message-Id: <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 5931ed5641ca6c4a237e82e4e5d2a08639865ce7
      
https://github.com/qemu/qemu/commit/5931ed5641ca6c4a237e82e4e5d2a08639865ce7
  Author: Peter Maydell <address@hidden>
  Date:   2020-03-11 (Wed, 11 Mar 2020)

  Changed paths:
    M block/backup-top.c
    M block/backup.c
    M block/block-copy.c
    M block/crypto.c
    M block/curl.c
    M block/qcow2-threads.c
    M block/qcow2.c
    M block/trace-events
    M blockjob.c
    M crypto/block.c
    M include/block/block-copy.h
    M include/crypto/block.h
    M include/qemu/job.h
    A include/qemu/progress_meter.h
    M job-qmp.c
    M job.c
    M qemu-img.c
    M tests/qemu-iotests/178
    M tests/qemu-iotests/178.out.qcow2
    M tests/qemu-iotests/178.out.raw
    A tests/qemu-iotests/288
    A tests/qemu-iotests/288.out
    M tests/qemu-iotests/common.rc
    M tests/qemu-iotests/group

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2020-03-11' 
into staging

Block patches for the 5.0 softfreeze:
- qemu-img measure for LUKS
- Improve block-copy's performance by reducing inter-request
  dependencies
- Make curl's detection of accept-ranges more robust
- Memleak fixes
- iotest fix

# gpg: Signature made Wed 11 Mar 2020 13:19:01 GMT
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "address@hidden"
# gpg: Good signature from "Max Reitz <address@hidden>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2020-03-11:
  block/block-copy: hide structure definitions
  block/block-copy: reduce intersecting request lock
  block/block-copy: rename start to offset in interfaces
  block/block-copy: refactor interfaces to use bytes instead of end
  block/block-copy: factor out find_conflicting_inflight_req
  block/block-copy: use block_status
  block/block-copy: specialcase first copy_range request
  block/block-copy: fix progress calculation
  job: refactor progress to separate object
  block/qcow2-threads: fix qcow2_decompress
  qemu-img: free memory before re-assign
  block/qcow2: do free crypto_opts in qcow2_close()
  iotests: Fix nonportable use of od --endian
  block/curl: HTTP header field names are case insensitive
  block/curl: HTTP header fields allow whitespace around values
  iotests: add 288 luks qemu-img measure test
  qemu-img: allow qemu-img measure --object without a filename
  luks: implement .bdrv_measure()
  luks: extract qcrypto_block_calculate_payload_offset()

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/6e8a73e911f0...5931ed5641ca