Re: PPC64

[Manoel]

On Tue, 2008-10-21 at 15:52 -0500, Hollis Blanchard wrote:
> Please CC me, since I'm no longer subscribed to grub-devel.
> 
> > From: Manoel <address@hidden>
> > To: The development of GRUB 2 <address@hidden>
> > Subject: Re: PPC64
> > Date: Tue, 21 Oct 2008 14:43:25 -0200
> > 
> > Hi Hollis,
> > 
> > On Mon, 2008-10-20 at 14:32 -0500, Hollis Blanchard wrote: 
> > > On Mon, 2008-10-20 at 17:18 -0200, Manoel wrote:
> > > > 
> > > > I'm working in a project to use grub2 to boot some ppc machines(p6 , p5
> > > > and so on) and we had some difficulties with a grub modules problem.
> > > > Grub fails to load modules.
> > > > 
> > > > When debugging I noted that grub try to find the headerinfo modules
> > > > struc (which is identified by the magic number 0x676d696d) in the
> > > > address 0x2d000 (_end + gap aligned in 4k blocks).
> > > > but this address does not contains the headerinfo.
> > > > 
> > > > So i altered the source code such as the memory is searched to find the
> > > > magic number. It is then found at address 0x38f4c and then grub find
> > > > some modules (but fails after) has showed in attachment grub2.txt.
> > > 
> > > ...
> > > ../kern/dl.c:527: module at 0x3e0dc, size 0xc9c
> > > ../kern/dl.c:556: relocating to 0x28720
> > > ../kern/dl.c:513: flushing 0x4 bytes at 0x28190
> > > ../kern/dl.c:513: flushing 0x14 bytes at 0x281d0
> > > ../kern/dl.c:513: flushing 0x68 bytes at 0x28220
> > > ../kern/dl.c:513: flushing 0x410 bytes at 0x282c0
> > > ../kern/dl.c:570: module name: amiga
> > > ../kern/dl.c:571: init function: 0x282c0
> > > ../kern/dl.c:527: module at 0x3ed84, size 0xe28
> > > ../kern/dl.c:556: relocating to 0x280a0
> > > ../kern/dl.c:513: flushing 0x4 bytes at 0x27a30
> > > ../kern/dl.c:513: flushing 0x14 bytes at 0x27a70
> > > ../kern/dl.c:513: flushing 0xfc bytes at 0x27ac0
> > > ../kern/dl.c:513: flushing 0x458 bytes at 0x27bf0
> > > ../kern/dl.c:570: module name: apple
> > > ../kern/dl.c:571: init function: 0x27bf0
> > > ../kern/dl.c:527: module at 0x3fbb8, size 0xeca4
> > > ../kern/dl.c:556: relocating to 0x27940
> > > 
> > > Notice how much larger that last module is than the ones before it.
> > > That's a bit suspicious... do you have any modules that size?
> > > 
> > 
> > I'd like to address this issue later but their size are really messed
> > up. Grub can find the modules (how you can see by the modules names)
> > though. The modules should have 7k at most but grub identified them has
> > having about 50k.
> 
> That is really strange. I wonder if you have an ABI issue like
> sizeof(long)... how is the grub-mkimage tool compiled? Please make sure
> it's 32-bit. The grub binary that executes at boot should also 32-bit.
They are all 32-bit. I'll look deeper in this issue later.
> 
> > I'm also curious why we must have a GAP between _end and the modules.
> > Why do not put the modules right after the _end address.
> 
> We put the modules into a separate PT_LOAD ELF segment, and these must
> be aligned.
> 
> One other possibility is that your firmware doesn't like the way
> grub-mkimage throws out the section table on the ELF file. You could try
> changing that behavior.
> 
> I suppose you could also try to extend the existing PT_LOAD segment
> instead of creating a new one, but architecturally creating a new
> segment for the modules is much nicer.
> 
I talked with some folks at #pppc64 on freenode an they suggest that the
OF is loading things in the wrong place could a problem with my
load-base:

        real-base                c00000              c00000 
        virt-base                ffffffff            ffffffff 
        real-size                1000000             1000000 
        virt-size                ffffffff            ffffffff 
        load-base                4000                4000
 
they suggested to change load-base from 4000 to 200000 but I hava yet to
try it.  They also said that the note section can override load-base
(and maybe we have some problem there?)

I'm now reading PARP documentation to learn more about OF behavior. I
thought these machine were CHRP but they are actually PARP (is that
right?). Son only today I was able to get the correct documentation.
> > I need to look more into the source code but I noted the modules are
> > allocated in address in a decrementing order. The next module is always
> > loaded in a address below the previous module. I don't know if this
> > memory is allocated by the OF or if Grub forces the address to load the
> > modules this way.
> > How I have said before that I will look at this issue after the modules
> > header info location address issue is resolved.
> >  
> > > > that address calculation led me to believe that I can tell where the
> > > > struct will be on memory based in its place in the binary.
> > > > 
> > > > I also noted that basemod ( indicates where the modules sections begin)
> > > > used by grub_mkelfimage is the same calculated by grub (_end + GAP). but
> > > > it seems to not store it on the necessary address.
> > > > 
> > > > using hexedit I could see that in the address 0xCC98 (in the file
> > > > generated by grub_mkelfimage) is stored the struct header info.
> > > 
> > > Well, hmm. Given the readelf output below, file offset 0xcc98 should be
> > > loaded right at 0x2d000. Since you can see the magic number there
> > > (correct?), I can't explain why the ELF loaded places it at 0x38f4c.
> > 
> > Yes, the magic number is exactly at the address 0xcc98 on the file
> > generated by the grub_mkelfimage. How can you tell the address it should
> > appear in memory based on its address in file? Maybe it's only valid in
> > some old OF version? 
> 
> Look at the segment list again:
> > Program Headers:
> >  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
> >  LOAD           0x0000b4 0x00010000 0x00010000 0x0cbe4 0x14098 RWE 0x10
> >  GNU_STACK      0x00cc98 0x00000000 0x00000000 0x00000 0x00000 RWE 0x4
> >  LOAD           0x00cc98 0x0002d000 0x0002d000 0x5c1c8 0x5c1c8 RWE 0x4
> >  NOTE           0x068e60 0x00000000 0x00000000 0x0002c 0x00000 R   0x4
> 
NOTE 0x068e60 0x00000000 0x00000000 0x0002c 0x00000 R 0x4
this line look somehow wrong. NOTE will be loaded at the address 0?
and will occupy no memory? that is the same as don't having NOTE at all
right?

> Offset tells you where in the file each segment begins. FileSiz is how
> many bytes to read from the file. VirtAddr/PhysAddr is where in memory
> to copy it, and MemSiz is how many bytes the segment will occupy in
> memory. (If MemSiz > FileSiz, the trailing bytes are zero.)
> 
> 0x38f4c, where you found the header, is about 50KB inside the second
> LOAD segment (which was added by grub-mkimage and contains the modules).
> Either the firmware's ELF loader did something bad loading the file, or
> grub-mkimage did something bad constructing the file.
> 
> Since you said you can manually verify that file offset 0xcc98 does in
> fact contain the magic number, and we can all see that it *should* be
> loaded at 0x2d000, that makes it seem like the loader did something
> wrong.
> 
> Can you report the bug to the firmware team, supply the broken binary,
> and see if they'll take a look at it?
> 
I'll do that.
> By the way, what filesystem is GRUB located on, and how big is the
> partition? Historically IBM firmware has had bugs loading from many
> filesystems, but I think FAT12 is OK as long as it's on a small
> partition.
> 
We are doing boot through network so we can test easier, and so far we
are only concerned about the initial parts (to find and load modules).
> > > Can you report what memory firmware is using on this system? IIRC you
> > > can decode it from the "available" properties in the memory nodes.
> > > 
> > I couldn't find any apparently useful information in the memory nodes
> > properties. I have attached it anyway, I have also attached the "/" node
> > properties. 
> 
> I didn't get these.
> 
I got documentation about the CHRP binding and it tells about a
memory-controller node, but it doesn't exist (maybe cause it is actually
PARP). I'll try to find something about it in PARP documentation.

> You'll need to refer to the PowerPC and CHRP bindings to IEEE 1275 to
> interpret the /memory/available properties. I don't remember the field
> widths off the top of my head, but they are basically a list of <base,
> length> pairs that denote regions of memory available to applications.
> 
> I was just wondering if maybe firmware was occupying the memory
> specified in the ELF header for the modules segment, and was doing
> something stupid after that.
> 
look likes it, maybe it cant load the section where it should then it
loads it elsewhere, but even so, grub should be able to load all modules
once I find where it starts.
Grub is able to successfully load some modules (first 6) and then
crashes when getting the module name of the 7th one.
Forcing grub to stop in the 6th module they appear listed in the lsmod
command (but I can't tell about it's functionality).

Looking better at the booting the size problem only occurs in the last
module (which crashes grub). So either mkelfimage is doing something
wrong or OF is corrupting the address where its size is stored.
I'll look more into int after the header address issue is done.


-- 
Best Regards,

Manoel Abranches <address@hidden>
IBM Linux Technology Center Brasil