[Qemu-devel] [PATCH v6 00/33] implement vNVDIMM

[Xiao Guangrong]

This patchset can be found at:
      https://github.com/xiaogr/qemu.git nvdimm-v6

It is based on pci branch on Michael's tree and the top commit is:
commit 6f96a31a06c2a1 (tests: re-enable vhost-user-test).

Changelog in v6:
- changes from Stefan's comments:
  1) fix code style of struct naming by CamelCase way
  2) fix offset + length overflow when read/write label data
  3) compile hw/acpi/nvdimm.c for per target so that TARGET_PAGE_SIZE can
     be used to replace getpagesize()

Changelog in v5:
- changes from Michael's comments:
  1) prefix nvdimm_ to everything in NVDIMM source files
  2) make parsing _DSM Arg3 more clear
  3) comment style fix
  5) drop single used definition
  6) fix dirty dsm buffer lost due to memory write happened on host
  7) check dsm buffer if it is big enough to contain input data
  8) use build_append_int_noprefix to store single value to GArray

- changes from Michael's and Igor's comments:
  1) introduce 'nvdimm-support' parameter to control nvdimm
     enablement and it is disabled for 2.4 and its earlier versions
     to make live migration compatible
  2) only reserve 1 RAM page and 4 bytes IO Port for NVDIMM ACPI
     virtualization

- changes from Stefan's comments:
  1) do endian adjustment for the buffer length

- changes from Bharata B Rao's comments:
  1) fix compile on ppc

- others:
  1) the buffer length is directly got from IO read rather than got
     from dsm memory
  2) fix dirty label data lost due to memory write happened on host

Changelog in v4:
- changes from Michael's comments:
  1) show the message, "Memory is not allocated from HugeTlbfs", if file
     based memory is not allocated from hugetlbfs.
  2) introduce function, acpi_get_nvdimm_state(), to get NVDIMMState
     from Machine.
  3) statically define UUID and make its operation more clear
  4) use GArray to build device structures to avoid potential buffer
     overflow
  4) improve comments in the code
  5) improve code style

- changes from Igor's comments:
  1) add NVDIMM ACPI spec document
  2) use serialized method to avoid Mutex
  3) move NVDIMM ACPI's code to hw/acpi/nvdimm.c
  4) introduce a common ASL method used by _DSM for all devices to reduce
     ACPI size
  5) handle UUID in ACPI AML code. BTW, i'd keep handling revision in QEMU
     it's better to upgrade QEMU to support Rev2 in the future

- changes from Stefan's comments:
  1) copy input data from DSM memory to local buffer to avoid potential
     issues as DSM memory is visible to guest. Output data is handled
     in a similar way

- changes from Dan's comments:
  1) drop static namespace as Linux has already supported label-less
     nvdimm devices

- changes from Vladimir's comments:
  1) print better message, "failed to get file size for %s, can't create
     backend on it", if any file operation filed to obtain file size

- others:
  create a git repo on github.com for better review/test

Also, thanks for Eric Blake's review on QAPI's side.

Thank all of you to review this patchset.

Changelog in v3:
There is huge change in this version, thank Igor, Stefan, Paolo, Eduardo,
Michael for their valuable comments, the patchset finally gets better shape.
- changes from Igor's comments:
  1) abstract dimm device type from pc-dimm and create nvdimm device based on
     dimm, then it uses memory backend device as nvdimm's memory and NUMA has
     easily been implemented.
  2) let file-backend device support any kind of filesystem not only for
     hugetlbfs and let it work on file not only for directory which is
     achieved by extending 'mem-path' - if it's a directory then it works as
     current behavior, otherwise if it's file then directly allocates memory
     from it.
  3) we figure out a unused memory hole below 4G that is 0xFF00000 ~ 
     0xFFF00000, this range is large enough for NVDIMM ACPI as build 64-bit
     ACPI SSDT/DSDT table will break windows XP.
     BTW, only make SSDT.rev = 2 can not work since the width is only depended
     on DSDT.rev based on 19.6.28 DefinitionBlock (Declare Definition Block)
     in ACPI spec:
| Note: For compatibility with ACPI versions before ACPI 2.0, the bit 
| width of Integer objects is dependent on the ComplianceRevision of the DSDT.
| If the ComplianceRevision is less than 2, all integers are restricted to 32 
| bits. Otherwise, full 64-bit integers are used. The version of the DSDT sets 
| the global integer width for all integers, including integers in SSDTs.
  4) use the lowest ACPI spec version to document AML terms.
  5) use "nvdimm" as nvdimm device name instead of "pc-nvdimm"

- changes from Stefan's comments:
  1) do not do endian adjustment in-place since _DSM memory is visible to guest
  2) use target platform's target page size instead of fixed PAGE_SIZE
     definition
  3) lots of code style improvement and typo fixes.
  4) live migration fix
- changes from Paolo's comments:
  1) improve the name of memory region
  
- other changes:
  1) return exact buffer size for _DSM method instead of the page size.
  2) introduce mutex in NVDIMM ACPI as the _DSM memory is shared by all nvdimm
     devices.
  3) NUMA support
  4) implement _FIT method
  5) rename "configdata" to "reserve-label-data"
  6) simplify _DSM arg3 determination
  7) main changelog update to let it reflect v3.

Changlog in v2:
- Use litten endian for DSM method, thanks for Stefan's suggestion

- introduce a new parameter, @configdata, if it's false, Qemu will
  build a static and readonly namespace in memory and use it serveing
  for DSM GET_CONFIG_SIZE/GET_CONFIG_DATA requests. In this case, no
  reserved region is needed at the end of the @file, it is good for
  the user who want to pass whole nvdimm device and make its data
  completely be visible to guest

- divide the source code into separated files and add maintain info

BTW, PCOMMIT virtualization on KVM side is work in progress, hopefully will
be posted on next week

====== Background ======
NVDIMM (A Non-Volatile Dual In-line Memory Module) is going to be supported
on Intel's platform. They are discovered via ACPI and configured by _DSM
method of NVDIMM device in ACPI. There has some supporting documents which
can be found at:
ACPI 6: http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf
NVDIMM Namespace: http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf
DSM Interface Example: http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf
Driver Writer's Guide: http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf

Currently, the NVDIMM driver has been merged into upstream Linux Kernel and
this patchset tries to enable it in virtualization field

====== Design ======
NVDIMM supports two mode accesses, one is PMEM which maps NVDIMM into CPU's
address space then CPU can directly access it as normal memory, another is
BLK which is used as block device to reduce the occupying of CPU address
space

BLK mode accesses NVDIMM via Command Register window and Data Register window.
BLK virtualization has high workload since each sector access will cause at
least two VM-EXIT. So we currently only imperilment vPMEM in this patchset

--- vPMEM design ---
We introduce a new device named "nvdimm", it uses memory backend device as
NVDIMM memory. The file in file-backend device can be a regular file and block 
device. We can use any file when we do test or emulation, however,
in the real word, the files passed to guest are:
- the regular file in the filesystem with DAX enabled created on NVDIMM device
  on host
- the raw PMEM device on host, e,g /dev/pmem0
Memory access on the address created by mmap on these kinds of files can
directly reach NVDIMM device on host.

--- vConfigure data area design ---
Each NVDIMM device has a configure data area which is used to store label
namespace data. In order to emulating this area, we divide the file into two
parts:
- first parts is (0, size - 128K], which is used as PMEM
- 128K at the end of the file, which is used as Label Data Area
So that the label namespace data can be persistent during power lose or system
failure.

We also support passing the whole file to guest without reserve any region for
label data area which is achieved by "reserve-label-data" parameter - if it's
false then QEMU will build static and readonly namespace in memory and that
namespace contains the whole file size. The parameter is false on default.

--- _DSM method design ---
_DSM in ACPI is used to configure NVDIMM, currently we only allow access of
label namespace data, i.e, Get Namespace Label Size (Function Index 4),
Get Namespace Label Data (Function Index 5) and Set Namespace Label Data
(Function Index 6)

_DSM uses two pages to transfer data between ACPI and Qemu, the first page
is RAM-based used to save the input info of _DSM method and Qemu reuse it
store output info and another page is MMIO-based, ACPI write data to this
page to transfer the control to Qemu

====== Test ======
In host
1) create memory backed file, e.g # dd if=zero of=/tmp/nvdimm bs=1G count=10
2) append "-object memory-backend-file,share,id=mem1,
   mem-path=/tmp/nvdimm -device nvdimm,memdev=mem1,reserve-label-data,
   id=nv1" in QEMU command line

In guest, download the latest upsteam kernel (4.2 merge window) and enable
ACPI_NFIT, LIBNVDIMM and BLK_DEV_PMEM.
1) insmod drivers/nvdimm/libnvdimm.ko
2) insmod drivers/acpi/nfit.ko
3) insmod drivers/nvdimm/nd_btt.ko
4) insmod drivers/nvdimm/nd_pmem.ko
You can see the whole nvdimm device used as a single namespace and /dev/pmem0
appears. You can do whatever on /dev/pmem0 including DAX access.

Currently Linux NVDIMM driver does not support namespace operation on this
kind of PMEM, apply below changes to support dynamical namespace:

@@ -798,7 +823,8 @@ static int acpi_nfit_register_dimms(struct acpi_nfit_desc *a
                        continue;
                }
 
-               if (nfit_mem->bdw && nfit_mem->memdev_pmem)
+               //if (nfit_mem->bdw && nfit_mem->memdev_pmem)
+               if (nfit_mem->memdev_pmem)
                        flags |= NDD_ALIASING;

You can append another NVDIMM device in guest and do:                       
# cd /sys/bus/nd/devices/
# cd namespace1.0/
# echo `uuidgen` > uuid
# echo `expr 1024 \* 1024 \* 128` > size
then reload nd.pmem.ko

You can see /dev/pmem1 appears

Xiao Guangrong (33):
  acpi: add aml_derefof
  acpi: add aml_sizeof
  acpi: add aml_create_field
  acpi: add aml_concatenate
  acpi: add aml_object_type
  acpi: add aml_method_serialized
  util: introduce qemu_file_get_page_size()
  exec: allow memory to be allocated from any kind of path
  exec: allow file_ram_alloc to work on file
  hostmem-file: clean up memory allocation
  hostmem-file: use whole file size if possible
  pc-dimm: remove DEFAULT_PC_DIMMSIZE
  pc-dimm: make pc_existing_dimms_capacity static and rename it
  pc-dimm: drop the prefix of pc-dimm
  stubs: rename qmp_pc_dimm_device_list.c
  pc-dimm: rename pc-dimm.c and pc-dimm.h
  dimm: abstract dimm device from pc-dimm
  dimm: get mapped memory region from DIMMDeviceClass->get_memory_region
  dimm: keep the state of the whole backend memory
  dimm: introduce realize callback
  nvdimm: implement NVDIMM device abstract
  docs: add NVDIMM ACPI documentation
  nvdimm acpi: init the resource used by NVDIMM ACPI
  nvdimm acpi: build ACPI NFIT table
  nvdimm acpi: build ACPI nvdimm devices
  nvdimm acpi: save arg3 for NVDIMM device _DSM method
  nvdimm acpi: support function 0
  nvdimm acpi: support Get Namespace Label Size function
  nvdimm acpi: support Get Namespace Label Data function
  nvdimm acpi: support Set Namespace Label Data function
  nvdimm: allow using whole backend memory as pmem
  nvdimm acpi: support _FIT method
  nvdimm: add maintain info

 MAINTAINERS                                        |    7 +
 backends/hostmem-file.c                            |   59 +-
 default-configs/i386-softmmu.mak                   |    3 +
 default-configs/mips-softmmu.mak                   |    1 +
 default-configs/mips64-softmmu.mak                 |    1 +
 default-configs/mips64el-softmmu.mak               |    1 +
 default-configs/mipsel-softmmu.mak                 |    1 +
 default-configs/ppc64-softmmu.mak                  |    1 +
 default-configs/x86_64-softmmu.mak                 |    3 +
 docs/specs/acpi_nvdimm.txt                         |  179 ++++
 exec.c                                             |  114 +-
 hmp.c                                              |    2 +-
 hw/Makefile.objs                                   |    2 +-
 hw/acpi/Makefile.objs                              |    1 +
 hw/acpi/aml-build.c                                |   79 +-
 hw/acpi/ich9.c                                     |   32 +-
 hw/acpi/memory_hotplug.c                           |   26 +-
 hw/acpi/nvdimm.c                                   | 1132 ++++++++++++++++++++
 hw/acpi/piix4.c                                    |   35 +-
 hw/i386/acpi-build.c                               |    6 +
 hw/i386/pc.c                                       |   34 +-
 hw/mem/Makefile.objs                               |    2 +
 hw/mem/{pc-dimm.c => dimm.c}                       |  240 +++--
 hw/mem/nvdimm.c                                    |  142 +++
 hw/mem/pc-dimm.c                                   |  510 +--------
 hw/ppc/spapr.c                                     |   20 +-
 include/hw/acpi/aml-build.h                        |    7 +
 include/hw/acpi/ich9.h                             |    3 +
 include/hw/i386/pc.h                               |   12 +-
 include/hw/mem/dimm.h                              |   95 ++
 include/hw/mem/nvdimm.h                            |  133 +++
 include/hw/mem/pc-dimm.h                           |  104 +-
 include/hw/ppc/spapr.h                             |    2 +-
 include/qemu/osdep.h                               |    1 +
 numa.c                                             |    4 +-
 qapi-schema.json                                   |    8 +-
 qmp.c                                              |    4 +-
 stubs/Makefile.objs                                |    2 +-
 ...c_dimm_device_list.c => qmp_dimm_device_list.c} |    4 +-
 target-ppc/kvm.c                                   |   21 +-
 trace-events                                       |    8 +-
 util/oslib-posix.c                                 |   16 +
 util/oslib-win32.c                                 |    5 +
 43 files changed, 2224 insertions(+), 838 deletions(-)
 create mode 100644 docs/specs/acpi_nvdimm.txt
 create mode 100644 hw/acpi/nvdimm.c
 rename hw/mem/{pc-dimm.c => dimm.c} (65%)
 create mode 100644 hw/mem/nvdimm.c
 rewrite hw/mem/pc-dimm.c (91%)
 create mode 100644 include/hw/mem/dimm.h
 create mode 100644 include/hw/mem/nvdimm.h
 rewrite include/hw/mem/pc-dimm.h (97%)
 rename stubs/{qmp_pc_dimm_device_list.c => qmp_dimm_device_list.c} (56%)

-- 
1.8.3.1