Re: Qemu-PPC problems (was [Qemu-devel] Just to add one single point)

[Rob Landley]

I'm fighting with cvs and haven't been able to update my build directory yet.  
(I've been spoiled by mercurial, where doing an update on a directory with 
modifications in it doesn't _eat_ the directory.  And my attempt to create a 
mercurial copy of the cvs history with tailor so I could tracck development 
more easily revealed an interesting bug tailor that ate an afternoon without 
fixing anything.  Sigh, buried in tangents...)

You've given me some more avenues to pursue and I hope to get to it this 
weekend, but in the meantime lemme catch up with the thread...

On Thursday 12 April 2007 3:56 am, J. Mayer wrote:
> > Oh, and did you ever get the bug report on qemu-ppc not working with 
uClibc 
> > because Linux always zeroes r3 (return value from previous syscall, in 
this 
> > case "exec") and qemu application emulation apparently doesn't?
> 
> This is documented in linux-user/elfload.c
> The fact is r3 is not zeroed to follow the PowerPC ABI and then be able
> to launch BSD program that relies on the kernel following the ABI. If
> Linux does not follow the ABI requirements...

If qemu is built for Linux, why would it be launching BSD programs?  Possibly 
an #ifdef is called for here?

> Once again, it's been a long time I did not check, but Qemu-PPC used to
> be able to launch some BSD programs too.

On BSD, that may be.  (I don't use it.)  But when running Linux programs, it 
doesn't match the real, documented, and consistent behavior of the Linux 
kernel that other software (like uClibc) has come to rely on.  On Linux, it 
may be compliant with a theoretical standard, but it simply doesn't match 
reality.

Again, this sounds like a job for an #ifdef.

To quote from http://wall.riscom.net/books/proc/ppc/cwg/a_abi.html

> IBM has defined three ABIs for the PowerPC architecture: the AIX ABI for
> big-endian 32-bit PowerPC processors and the Windows NT and Workplace ABIs
> for little-endian 32-bit PowerPC processors. Other PowerPC users have
> defined other ABIs. As a practical matter, ABIs tend to be associated with
> a particular operating system or family of operating systems. Programs
> compiled for one ABI are frequently incompatible with programs compiled for
> another ABI because of the low-level strategic decisions required by an
> ABI.

What the kernel does is set r3 to the return value of the exec call (and argc 
in r1), then the dynamic linker shuffles things around so r3 points to argc.  
When a program is statically linked, it gets r3 zeroed, as explained here:

  http://sources.redhat.com/ml/libc-alpha/2003-03/msg00272.html

It's quite possible that command line arguments don't work when the program 
_is_ dynamically linked because the shuffling the dynamic linker does is 
going to overwrite r3 with r1.  I don't know, "hello world" doesn't look at 
argc and my script only uses application emulation as a smoke test for my new 
cross compiler toolchain.

Google brings up this as the abi document:

  http://refspecs.freestandards.org/elf/elfspec_ppc.pdf

Which describes the registers on page 30-32.  Page 38 says the return value is 
in r3 (and Linux is putting the return value of "exec" into r3, which is why 
they expect it to be 0 when exec worked).  Page 44 does talk about the 
register layout you mention later, but on Linux the dynamic linker is what 
sets this up, not the kernel.  (Whether or not the kernel should leave this 
to the dynamic linker is immaterial: reality is that it does.  And it's been 
doing it this way long enough that it's not likely to change.)

> > I sent it to you directly, but your mailserver bounces messages from mine 
as 
> > spam.  (I apparently can't even cc you or you won't get a copy through the 
> > mailing list.)
> 
> Cannot do anything about it, the mailserver is not mine.

Neither is mine, alas.

> > > I'm sorry but I _never_ use custom kernels for tests, apart if I want to
> > > add traces to track a really well hiden bug. I always use stock kernels
> > > delivered with distributions or kernels I recompile under Qemu from the
> > > vanilla sources located at kernel.org, with absolutely no patch. Not all
> > > run, that's true. Some may even say that only a few run.
> > 
> > If a stock kernel boots then it should be possible to build a kernel from 
> > source that will also boot.  I'm happy to debug and document how to do so, 
> > but I'm not good at debugging firmware or bootloaders.
> 
> That's what I do, but using a partitioned file system: I try to act the
> same as what I do when using a real PowerPC based machine. So I compile
> my kernel, install it and modify the yaboot configuration file to use
> it.

My problem is that none of the other platforms I've gotten the kernel booting 
on (so far arm, mips, x86, x86_64, and sparc) work like this.  I got them all 
booting kernel + unpartitioned ext2 filesystem image.  I'm reluctant to go 
back and change all the other platforms because PowerPC currently has unique 
requirements due to options like -kernel not working.  So far I've just 
deferred adding PowerPC support.

I can make a partitioned image for manual testing, but I haven't yet because I 
can't check that into my build system without a major rewrite of unrelated 
things that work as they are.

> > > I also know that some binary blurbs (Linux and real-time OSes based) for
> > > embedded PowerPC targets boot and run well under Qemu PPC. Some I
> > > unfortunately cannot release, some I even don't have, just been reported
> > > they run by their owners. Hope I will have some freely available one of
> > > these days.
> > 
> > Do any of these boot and run without a partitioned filesystem image?  In 
> > theory, I should be able to build an initramfs into the kernel and boot 
with 
> > no hard drive:
> 
> The binary blurbs I can use for test are flash images. That means that
> they completely replace the firmware with proprietary ones. Then, the
> -kernel option is not relevant.
> The -kernel option is a qemu hack. You cannot do that on real hardware,
> so it will never work when using proprietary firmwares.

I don't use proprietary firmware.  I haven't got any, and it would only 
interest me if I was being paid to be interested.  I have pondered using 
u-boot, but it's way down on the todo list...

> > qemu-system-ppc -M prep -nographic -kernel zImage-powerpc -append \
> >   "console=/dev/ttyS0"
> 
> You cannot append anything to the command line this way, with the PPC
> firmware...
> You can append options when using yaboot, not with the -kernel option.
> Then, you should use the CONFIG_CMDLINE kernel option to add the option
> you absolutely need to boot.

I can do that, but with -kernel it wasn't handing over control to the kernel 
at all so I hadn't noticed that part.

> I'm not sure this is a great idea. It would break other things to zero r3 at
> program start. I just checked and I've been able to launch very simple BSD
> programs.  

Did you launch them under BSD, or under Linux?

If QEMU application emulation can launch BSD programs under Linux, but can't 
launch Linux programs, something is wrong.

> If you change this, it would never have a chance to work.
> Linux seem not to follow the ABI but does not zeroes r3 as a result of the
> exec syscall. 
> The r3 register is zeroed after the fork (which is correct) then the exec 
syscall does not seem to set up its value, which is incorrect.

As far as I can tell, the exec syscall returns what other syscalls return, and 
the dynamic linker shuffles stuff around to where the ABI says main() should 
expect them.

> [...]
> > > But
> > > there is still a sufficient support to test at least Linux running,
> > > installing, compiling, with X11 and most application running well, with
> > > one machine and different CPU models available, which is far from beeing
> > > a "nothing works" statement, imho.
> > 
> > I've never gotten it to work, and the problem seems to be that open 
firmware 
> > wants a partitioned image.  Is a partitioned hard drive image a 
requirement 
> > to get it to work?
> 
> you may try to boot kernels in PREP format as they look like regular boot 
partitions...
> It may help.

I'll try it.  My attempts to make my own prep kernel with menuconfig didn't 
produce a zImage (still dunno why), but the kernel has a default prep config 
I can start from...

> > > It would be great to have a lot of more machines, CPU, OS, ...
> > > supported. Some things will come, some are the way, but it will take
> > > time. Feel free to suggest things that you feel that should be a
> > > priority, it may give great ideas...
> > 
> > I have 8 platform variants booting so far with -kernel.
> 
> I was thinking about more PowerPC based CPUs, platforms, OSes, ....

I'm trying to test Linux on as many different platforms as I can, and create a 
unified cross-platform Linux build environment.  (Which is why I do as little 
cross compiling as possible before getting a native build environment running 
under qemu, and then build natively under emulation the rest of the way.  
Much much much less to debug that way.  It's slow, but I can teach the thing 
to call out to the cross compiler via distcc to do the heavy lifting, in 
which case it's still doing ./configure and make and #include resolution and 
library linking inside the emulator so 95% of the suckage of cross compiling 
goes away...)

Now that I've gotten a few platforms building "hello world" I've paused on the 
depth-first bit of building a whole gentoo system in there (and getting the 
distcc thing working), and I'm going breadth-first a bit to try to add lots 
of platforms and get them to build "hello world" natively inside qemu.

It's a bit frustrating in places, but most of it's been off in 
gcc/binutils/uClibc/linux land.  (All of which I've broken more than once.)  
My general approach to qemu has generally been "ask a couple of dumb 
questions and wait for the next release", because development here is a bit 
rapid for me to keep up with in the amount of time I have to spend on it.

This is why I've been approaching it from "ok, all these these other platforms 
work this way, but powerpc doesn't".

Thanks,

Rob
-- 
Penguicon 5.0 Apr 20-22, Linux Expo/SF Convention.  Bruce Schneier, Christine 
Peterson, Steve Jackson, Randy Milholland, Elizabeth Bear, Charlie Stross...