Re: [Qemu-devel] [PATCH 00/10] bundle edk2 platform firmware with QEMU

[Laszlo Ersek]

On 03/12/19 16:59, Daniel P. Berrangé wrote:
> On Tue, Mar 12, 2019 at 04:40:45PM +0100, Laszlo Ersek wrote:
>> On 03/11/19 11:35, Daniel P. Berrangé wrote:
>>> On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
>>>> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
>>>>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
>>>>>> Repo:   https://github.com/lersek/qemu.git
>>>>>> Branch: edk2_build
>>>>>>
>>>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>>>>> release, and builds and captures platform firmware binaries from that
>>>>>> release. At this point they are meant to be used by both end-users and
>>>>>> by Igor's ACPI unit tests in qtest ("make check").
>>>>>>
>>>>>> Previous discussion:
>>>>>>
>>>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>>>>   http://mid.mail-archive.com/address@hidden
>>>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>>>
>>>> There David raised a concern about "[adding] ~206 MB of binaries to the
>>>> pc-bios directory". I'm also worried.
>>>>
>>>> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
>>>> rather strongly avoid (because we support a wide range of host OS, each
>>>> using a wide types of filesystems).
>>>>
>>>> What about storing those binaries on a file server (http/ftp) altogether
>>>> with a file containing its hashed digest (SHA1/SHA256)? Then we already
>>>> have all the required tools to fetch and verify those blob roms with the
>>>> build system.
>>>> Or we could store the hashes in the QEMU repository too.
>>>
>>> A simple approach would simply be to 'xz' compress all the edk*.fd
>>> files but still store them in git. They're already opaque binary
>>> files, so replacing one binary format with another binary format
>>> is no big deal IMHO
>>>
>>> $ ls -alsh edk2-*
>>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 
>>> edk2-aarch64-code.fd.xz
>>> 2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
>>> 772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 
>>> edk2-i386-secure-code.fd
>>> 528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
>>>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 
>>> edk2-x86_64-secure-code.fd
>>>
>>> This gives a 50% disk space saving over the sparse size:
>>>
>>> $ ls -alsh edk2-*
>>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 
>>> edk2-aarch64-code.fd.xz
>>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
>>>  12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
>>> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
>>> 1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 
>>> edk2-i386-secure-code.fd.xz
>>> 4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
>>>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
>>> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 
>>> edk2-x86_64-code.fd.xz
>>> 1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 
>>> edk2-x86_64-secure-code.fd.xz
>>>
>>> So about 9 MB compressed, instead of 20MB for the uncompressed sparse
>>> files, which is on a par with the existing ROM sizes.
>>>
>>> We would then need a "make" rule that runs  unxz to "build" the firmware
>>> files. Wouldn't need to be more complicated that this:
>>>
>>>    edk2-%.fd: edk2-%.fd.xz
>>>          unzx -c $< > $@
>>>
>>>    CLEANFILES += $(wildcard edk2*.fd)
>>
>> The problem with this idea is that such *.xz files are not directly
>> consumeable by Igor's ACPI regression test cases in qtest. The above
>> approach is suitable for "make install" only, not for "make check", AIUI.
> 
> I was suggesting that this rule to uncompress is wired into the
> default "make" target actually.

Ah, I see.

We might have to mark these "side artifacts" as .SECONDARY then, lest
make remove them in the end.

https://www.gnu.org/software/make/manual/make.html#Chained-Rules

Not entirely sure yet, I'd have to see in practice.

> Probably by just marking them
> as generated files, so they are always built. This will make
> it work for tests, as well as developers running from the source
> tree.

I'm quite lost in the top level Makefile -- this idea is more complex
than just extending INSTALL_BLOBS / BLOBS.

What target should I make dependent on the uncompressed fd files so that
"make" process them for both "make" (which is "make all" I guess) and
"make check"?

Thanks
Laszlo