RE: grub causing NVDIMMs to be treated as normal memory

[Elliott, Robert (Persistent Memory)]

> -----Original Message-----
> From: Andrei Borzenkov [mailto:address@hidden
> Sent: Thursday, November 26, 2015 10:55 AM
> To: Elliott, Robert (Persistent Memory) <address@hidden>; The
> development of GNU GRUB <address@hidden>;
> address@hidden; address@hidden
> Subject: Re: grub causing NVDIMMs to be treated as normal memory
> 
> 26.11.2015 09:15, Elliott, Robert (Persistent Memory) пишет:
> ...
> 
> > ...
> > mmap/efi/mmap.c:66: EFI memory region 0x880000000-0xc80000000: 14
> > Unknown memory type 14, considering reserved
> > mmap/efi/mmap.c:66: EFI memory region 0x1480000000-0x1a80000000:
> 14
> > Unknown memory type 14, considering reserved
> >
> >
> >
> > This is booting a new kernel with the EFI boot stub, so I can't
> > confirm that it fixes the issue of exposing the address range as
> > normal, but based on the print it's probably working.  I could
> > add prints in grub-core/loader/i386/linux.c grub_e820_add_region
> > to confirm the e820 contents at exit.
> >
> 
> Thanks. If it results in wrong e820 type we have much larger problem
> so I do not think it necessary.
> 
> I pushed it; as I understand this should serve as stopgap. If I get
> around to implement persistent memory type would you test it? But as
> I cannot tell when it happens, feel free to submit patch in the
> meantime :)

As an experiment, I:
* compiled a linux 4.4rc2 kernel with CONFIG_EFI_STUB disabled
* added some prints in the linux kernel setup_memory_map()
  to print the incoming bios_params e820 entries
* modified the grub patch to set the range to AVAILABLE rather 
  than RESERVED (simulating the bug, to see its impacts on
  the kernel)

Before (CONFIG_EFI_STUB=y, grub mapping to RESERVED):
bp: [mem 0x0000000000000000-0x0000000000092fff] usable
bp: [mem 0x0000000000093000-0x0000000000093fff] reserved
bp: [mem 0x0000000000094000-0x000000000009ffff] usable
bp: [mem 0x0000000000100000-0x000000006affbfff] usable
bp: [mem 0x000000006affc000-0x000000006b5fbfff] reserved
bp: [mem 0x000000006b5fc000-0x000000006b5fcfff] usable
bp: [mem 0x000000006b5fd000-0x000000006b67dfff] reserved
bp: [mem 0x000000006b67e000-0x00000000784fefff] usable
bp: [mem 0x00000000784ff000-0x00000000791fefff] reserved <--b
bp: [mem 0x00000000791ff000-0x000000007b5fefff] ACPI NVS
bp: [mem 0x000000007b5ff000-0x000000007b7fefff] ACPI data
bp: [mem 0x000000007b7ff000-0x000000007b7fffff] usable
bp: [mem 0x0000000100000000-0x000000087fffffff] usable <--a
bp: [mem 0x0000000c80000000-0x000000147fffffff] usable <--a
bp: [mem 0x0000000080000000-0x000000008fffffff] reserved
bp: [mem 0x0000000880000000-0x0000000c7fffffff] reserved <--a
bp: [mem 0x0000001480000000-0x0000001a7fffffff] reserved <--a

After (CONFIG_EFI_STUB=n, grub mapping to AVAILABLE):
bp: [mem 0x0000000000000000-0x0000000000092fff] usable
bp: [mem 0x0000000000093000-0x0000000000093fff] reserved
bp: [mem 0x0000000000094000-0x000000000009ffff] usable
bp: [mem 0x0000000000100000-0x000000006affbfff] usable
bp: [mem 0x000000006affc000-0x000000006b5fbfff] reserved
bp: [mem 0x000000006b5fc000-0x000000006b5fcfff] usable
bp: [mem 0x000000006b5fd000-0x000000006b67dfff] reserved
bp: [mem 0x000000006b67e000-0x00000000784fefff] usable
bp: [mem 0x00000000784ff000-0x00000000788fefff] reserved <--b
bp: [mem 0x00000000788ff000-0x00000000790fefff] type 20  <--b
bp: [mem 0x00000000790ff000-0x00000000791fefff] reserved <--b
bp: [mem 0x00000000791ff000-0x000000007b5fefff] ACPI NVS
bp: [mem 0x000000007b5ff000-0x000000007b7fefff] ACPI data
bp: [mem 0x000000007b7ff000-0x000000007b7fffff] usable
bp: [mem 0x0000000080000000-0x000000008fffffff] reserved
bp: [mem 0x0000000100000000-0x0000001a7fffffff] usable <--a


Note (a): The 088 and 148 ranges were merged into the 
010 and 0c8 usable ranges, as expected for this experiment.

The NVDIMM drivers in the kernel don't load (nmem devices load,
but namespaces fail and no pmem devices show up):
[   27.835319] nd_pmem namespace0.0: could not reserve region 
[0x0x0000000880000000:0x400000000]
[   27.835323]  ndbus1: nd_pmem.probe(namespace0.0) = -16
[   27.835355] nd_pmem namespace2.0: could not reserve region 
[0x0x0000001880000000:0x200000000]
[   27.835356]  ndbus1: nd_pmem.probe(namespace2.0) = -16
[   27.835373] nd_pmem namespace1.0: could not reserve region 
[0x0x0000001480000000:0x400000000]
[   27.835374]  ndbus1: nd_pmem.probe(namespace1.0) = -16

That shows up as normal memory in /proc/iomem and elsewhere:
100000000-1a7fffffff : System RAM

$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 
51 52 53 54 55 56 57 58 59 60 61 62 63
node 0 size: 104614 MB
node 0 free: 103303 MB
node distances:
node   0
  0:  10



Note (b): The internal GRUB_MEMORY_CODE (20) value is 
leaking through to the E820 table.

That appears to be from this patch on 2013-10-14:
        6de9ee86 Pass-through unknown E820 types
which deleted a switch statement mapping the internal codes 
to E820 values and included this:
-    case GRUB_MEMORY_CODE:
-    case GRUB_MEMORY_RESERVED:
-      ctx->cur.type = GRUB_E820_RESERVED;

based on this hope:
+/* GRUB types conveniently match E820 types.  */

That's only true for types 1 through 5:
typedef enum grub_memory_type
  {
    GRUB_MEMORY_AVAILABLE = 1,
    GRUB_MEMORY_RESERVED = 2,
    GRUB_MEMORY_ACPI = 3,
    GRUB_MEMORY_NVS = 4,
    GRUB_MEMORY_BADRAM = 5,
    GRUB_MEMORY_COREBOOT_TABLES = 16,
    GRUB_MEMORY_CODE = 20,
    /* This one is special: it's used internally but is never reported
       by firmware. Don't use -1 as it's used internally for other purposes. */
    GRUB_MEMORY_HOLE = -2,
    GRUB_MEMORY_MAX = 0x10000
  } grub_memory_type_t;


---
Robert Elliott, HPE Persistent Memory