Re: 9 TiB vm memory creation

[Ani Sinha]

On Mon, 14 Feb 2022, David Hildenbrand wrote:

> On 14.02.22 13:36, Igor Mammedov wrote:
> > On Mon, 14 Feb 2022 10:54:22 +0530 (IST)
> > Ani Sinha <ani@anisinha.ca> wrote:
> >
> >> Hi Igor:
> >>
> >> I failed to spawn a 9 Tib VM. The max I could do was a 2 TiB vm on my
> >> system with the following commandline before either the system
> >> destabilized or the OOM killed killed qemu
> >>
> >> -m 2T,maxmem=9T,slots=1 \
> >> -object 
> >> memory-backend-file,id=mem0,size=2T,mem-path=/data/temp/memfile,prealloc=off
> >>  \
> >> -machine memory-backend=mem0 \
> >> -chardev file,path=/tmp/debugcon2.txt,id=debugcon \
> >> -device isa-debugcon,iobase=0x402,chardev=debugcon \
> >>
> >> I have attached the debugcon output from 2 TiB vm.
> >> Is there any other commandline parameters or options I should try?
> >>
> >> thanks
> >> ani
> >
> > $ truncate -s 9T 9tb_sparse_disk.img
> > $ qemu-system-x86_64 -m 9T \
> >   -object 
> > memory-backend-file,id=mem0,size=9T,mem-path=9tb_sparse_disk.img,prealloc=off,share=on
> >  \
> >   -machine memory-backend=mem0
> >
> > works for me till GRUB menu, with sufficient guest kernel
> > persuasion (i.e. CLI limit ram size to something reasonable) you can boot 
> > linux
> > guest on it and inspect SMBIOS tables comfortably.
> >
> >
> > With KVM enabled it bails out with:
> >    qemu-system-x86_64: kvm_set_user_memory_region: 
> > KVM_SET_USER_MEMORY_REGION failed, slot=1, start=0x100000000, 
> > size=0x8ff40000000: Invalid argument
> >

I have seen this in my system but not always. Maybe I should have dug
deeper as to why i do see this all the time.

> > all of that on a host with 32G of RAM/no swap.
> >

My system in 16 Gib of main memory, no swap.

>
> #define KVM_MEM_MAX_NR_PAGES ((1UL << 31) - 1)
>
> ~8 TiB (7,999999)

That's not 8 Tib, thats 2 GiB. But yes, 0x8ff40000000 is certainly greater
than 2 Gib * 4K (assuming 4K size pages).

So in kvm_main.c in kernel, likely we are hitting this:

        new.npages = mem->memory_size >> PAGE_SHIFT;

        if (new.npages > KVM_MEM_MAX_NR_PAGES)
                return -EINVAL;

>
>
>
> In QEMU, we have
>
> static hwaddr kvm_max_slot_size = ~0;
>
> And only s390x sets
>
> kvm_set_max_memslot_size(KVM_SLOT_MAX_BYTES);
>
> with
>
> #define KVM_SLOT_MAX_BYTES (4UL * TiB)
>


So seems in Igor's system its getting limited by kvm not qemu.