Re: [PATCH 00/18] hw/riscv: Add Microchip PolarFire SoC Icicle Kit board support

[Alistair Francis]

On Wed, Aug 19, 2020 at 3:13 AM <Cyril.Jean@microchip.com> wrote:
>
> On 8/19/20 2:34 AM, Bin Meng wrote:
> > EXTERNAL EMAIL: Do not click links or open attachments unless you know the 
> > content is safe
> >
> > On Tue, Aug 18, 2020 at 9:55 PM Anup Patel <anup@brainfault.org> wrote:
> >> On Tue, Aug 18, 2020 at 6:39 PM <Cyril.Jean@microchip.com> wrote:
> >>> On 8/18/20 7:17 AM, Anup Patel wrote:
> >>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know 
> >>>> the content is safe
> >>>>
> >>>> On Tue, Aug 18, 2020 at 1:23 AM <Cyril.Jean@microchip.com> wrote:
> >>>>> On 8/17/20 8:28 PM, Alistair Francis wrote:
> >>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know 
> >>>>>> the content is safe
> >>>>>>
> >>>>>> On Mon, Aug 17, 2020 at 11:12 AM via <qemu-devel@nongnu.org> wrote:
> >>>>>>> Hi Anup,
> >>>>>>>
> >>>>>>> On 8/17/20 11:30 AM, Bin Meng wrote:
> >>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you 
> >>>>>>>> know the content is safe
> >>>>>>>>
> >>>>>>>> Hi Anup,
> >>>>>>>>
> >>>>>>>> On Sat, Aug 15, 2020 at 1:44 AM Anup Patel <anup@brainfault.org> 
> >>>>>>>> wrote:
> >>>>>>>>> On Fri, Aug 14, 2020 at 10:12 PM Bin Meng <bmeng.cn@gmail.com> 
> >>>>>>>>> wrote:
> >>>>>>>>>> From: Bin Meng <bin.meng@windriver.com>
> >>>>>>>>>>
> >>>>>>>>>> This adds support for Microchip PolarFire SoC Icicle Kit board.
> >>>>>>>>>> The Icicle Kit board integrates a PolarFire SoC, with one SiFive's
> >>>>>>>>>> E51 plus four U54 cores and many on-chip peripherals and an FPGA.
> >>>>>>>>> Nice Work !!! This is very helpful.
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>>> The Microchip HSS is quite convoluted. It has:
> >>>>>>>>> 1. DDR Init
> >>>>>>>>> 2. Boot device support
> >>>>>>>>> 3. SBI support using OpenSBI as library
> >>>>>>>>> 4. Simple TEE support
> >>>>>>>>>
> >>>>>>>>> I think point 1) and 2) above should be part of U-Boot SPL.
> >>>>>>>>> The point 3) can be OpenSBI FW_DYNAMIC.
> >>>>>>>>>
> >>>>>>>>> Lastly,for point 4), we are working on a new OpenSBI feature using
> >>>>>>>>> which we can run independent Secure OS and Non-Secure OS using
> >>>>>>>>> U-Boot_SPL+OpenSBI (for both SiFive Unleashed and Microchip
> >>>>>>>>> PolarFire).
> >>>>>>>>>
> >>>>>>>>> Do you have plans for adding U-Boot SPL support for this board ??
> >>>>>>>> + Cyril Jean from Microchip
> >>>>>>>>
> >>>>>>>> I will have to leave this question to Cyril to comment.
> >>>>>>>>
> >>>>>>> I currently do not have a plan to support U-Boot SPL. The idea of the
> >>>>>>> HSS is to contain all the silicon specific initialization and
> >>>>>>> configuration code within the HSS before jumping to U-Boot S-mode. I
> >>>>>>> would rather keep all this within the HSS for the time being. I would
> >>>>>>> wait until we reach production silicon before attempting to move this 
> >>>>>>> to
> >>>>>>> U-Boot SPL as the HSS is likely to contain some opaque silicon related
> >>>>>>> changes for another while.
> >>>>>> That is unfortunate, a lot of work has gone into making the boot flow
> >>>>>> simple and easy to use.
> >>>>>>
> >>>>>> QEMU now includes OpenSBI by default to make it easy for users to boot
> >>>>>> Linux. The Icicle Kit board is now the most difficult QEMU board to
> >>>>>> boot Linux on. Not to mention it makes it hard to impossible to
> >>>>>> support it in standard tool flows such as meta-riscv.
> >>>>>>
> >>>>>> Alistair
> >>>>> If it is such a problem we can add a U-Boot SPL stage and the HSS can be
> >>>>> treated as standard SoC ROM code.
> >>>> It's not mandatory for U-Boot SPL to have stable DRAM calibration 
> >>>> settings
> >>>> from the start itself. The initial U-Boot SPL support for most
> >>>> platforms (accross
> >>>> architectures) usually include basic working DRAM calibration settings 
> >>>> which
> >>>> is later on updated with separate patches. Also, we don't need all U-Boot
> >>>> drivers to be upstreamed in one-go as this can be done in phases.
> >>>>
> >>>> The advantage we have with PolarFire SoC Icicle board is that we already
> >>>> have a U-Boot S-mode. and we believe the OpenSBI generic platform will
> >>>> work fine for PolarFire SoC Icicle board so the only thing missing right 
> >>>> now
> >>>> is the U-Boot SPL support for OpenSource boot-flow.
> >>>>
> >>>> It will certainly accelerate open-source development if we have boot-flow
> >>>> U-Boot_SPL => OpenSBI (FW_DYNAMIC) => U-Boot_S-mode working
> >>>> on PolarFire SoC Icicle board. Initially, we just need DRAM, SD/eMMC,
> >>>> and Serial port support for U-Boot SPL and U-Boot S-mode. Later on,
> >>>> more patches can add ethernet and other booting device drivers to U-Boot.
> >>>>
> >>>> Regarding security services of HSS, we are working on a OpenSBI
> >>>> feature which will allow HSS security services to run as independent
> >>>> binary in M-mode (not linked to OpenSBI) and OpenSBI FW_DYNAMIC
> >>>> will be a separate binary acting as a secure monitor.
> >>>>
> >>>> Regards,
> >>>> Anup
> >>> What I have in mind is that the external memory will be up and running
> >>> by the time we get to U-Boot SPL. In the case of PolarFire SoC the ROM
> >>> code equivalent brings up the DDR memory interface so we do not need to
> >>> worry about this as part of U-Boot.
> >> Keeping DRAM configuration as part of a separate ROM booting stage prior
> >> to the U-Boot SPL sounds good to me. This will lead to following boot-flow:
> >>
> >> ROM/HSS (M-mode) => U-Boot SPL (M-mode) => OpenSBI (M-mode) => U-Boot 
> >> S-mode
> > If we want to keep the HSS, meaning that DDR memory is already
> > initialized, then there is no need to create an additional step of
> > U-Boot SPL phase, because there is no size limitation any more.
> >
> > Right now, the boot workflow on PolarFire is:
> >
> > HSS + OpenSBI (as a library) (M-mode) => U-Boot S-mode
> >
> > It's just not the canonical way because of the HSS and OpenSBI as a library.
> >
> > A canonical way should be:
> >
> > U-Boot SPL (M-mode) => OpenSBI dynamic (M-mode) => U-Boot S-mode
> >
> > So we should be aiming to replace HSS with U-Boot SPL.
> >
> >>> Sounds to me the component that needs to be upstreamed next is the eMMC
> >>> support so that it can be used as part of U-Boot SPL. Correct?
> >> Maybe as a PHASE1 patch series for PolarFire U-Boot support can
> >> target the following:
> >> 1. Minimal U-Boot board support for PolarFire IcIcle Board with
> >> single defconfig to build both U-Boot SPL and U-Boot S-mode
> >> 2. Serial port driver (probably re-use existing driver)
> >> 3. SD and eMMC driver
> >>
> >> Supporting SD booting is highly desirable for PHASE1. At least,
> >> U-Boot SPL, OpenSBI, U-Boot S-mode, and Linux should come
> >> as separate images from SD card. The ROM/HSS can reside and
> >> boot from on-board flash/eMMC.
> >>
> >> Above is my suggestion based on experience with SiFive Unleashed.
> >>
> >> The general idea behind OpenSource boot-flow is to have each
> >> booting stage as a separate binary so that users can selectively
> >> upgrade a particular booting stage without re-compiling/re-flashing
> >> other booting stages.
> >>
> >> Various distros and yocto already support building most of the above
> >> booting stages (U-Boot, OpenSBI, Linux, etc) so we can reuse a lot
> >> of existing work. Eventually, I am confident we will endup moving
> >> more stuff from ROM/HSS to U-Boot SPL for ease in maintenance.
> > Regards,
> > Bin
>
> I am OK with supporting U-Boot SPL as a community boot flow but the HSS
> is not going to disappear completely for many PolarFire SoC deployments.
> We need to have the HSS running on the monitor core after the system has
> booted for various reasons. Some of the current HSS functionalities can
> be migrated at a later time.
>
> The HSS was designed to easily enable and disable features through
> Kconfig which should allow to assign features to different boot stages
> depending on the overall system deployment. Keeping the HSS as ROM code
> capable of starting various early boot stages in either M mode or S mode
> allows us (Microchip engineering and non pure Linux users) to mix
> different operating systems on the same platform and more easily debug
> hardware bring-up.

Thanks for explaining this.

I have changed the CC list a bit to be a little more targeted.

>
> What I am trying to get at is that we need to balance the QEMU ease of
> use with the ability to bring-up and debug actual hardware. My preferred

Agreed. I don't think anyone has been saying that these changes should
be for QEMU only. All of the changes talked about apply to both the
QEMU model and the hardware.

> approach is to keep all the hardware bring up in one place, the HSS, so
> that you do not have to worry about obscure silicon idiosyncrasies once
> you reach the open source boot flow. Silicon/board bring up

That is a noble goal, but doesn't HSS currently pull in the OpenSBI
library, so it's doing more than just dealing with silicon bugs. As
well as that HSS is not supported by distros, so distros can't build
or ship it. For example OE won't build it in the meta-riscv layer,
meaning that the Icle board just isn't supported. This means the board
misses out on core work, for example it won't be able to utilise the
RISC-V AGL work that has started.

Also, how are users expected to update HSS as bugs and vulnerabilities
are found if it isn't shipped by distros?

> steps/features can be brought out of the HSS into the open source boot
> stages once these steps are mature.
>
> PolarFire SoC is the third family of SoC FPGA I have worked on. So I
> have a fair idea of the amount of pain in front of me to bring the
> entire device family to market :-) We decided early on in the
> architecture of PolarFire SoC that we would make all code executing on
> the RISC-V processors open source and user modifiable. This includes all

Awesome!!! That is so great to hear.

> code typically hidden away in your typical SoC ROM code. This is why

Again, this is just awesome!

> PolarFire SoC has on-chip flash memory used to store the HSS and why the
> HSS should be thought as including ROM code. The current Icicle Kit uses

If we could change it to be more like a traditional ROM in terms of
features then that would be really helpful. In that case we wouldn't
need it for QEMU and only use it for hardware. It would only handle
basic hardware boot up and then hand it over to U-Boot SPL/OpenSBI.

> engineering samples from the first device in the PolarFire SoC family.
> So we are really at the beginning of the process of validating very
> early boot steps on a significant number of devices. I do not want to
> expose the open source community to the pain of bringing up a family of
> devices because of the decision to open source all code. So I am asking

I understand what you mean. In saying that it is usually easiest to
start with the goal of upstreaming and then work towards that. That
way all of the architectures decisions are made in that context. Maybe
at first you have a fork while everything is in flux, but then it's
really helpful to have the work upstreamed.

I think we have all seen too many "open source" projects that just
fork everything, dump their code and never contribute back.
Unfortunately when that happens we end up with duplicate work and
frustrated engineers. As well as that the original project misses out
on all the advantages of open source as they aren't benefiting from
others work as well.

> you to not rush in trying to migrate components from the HSS to a later
> boot stage just yet. It can be done but based on experience I think that
> now is not the right time yet.

The open source focus is great to hear. I think we are all really glad
that MicroChip is doing this. It's a great first step, but it doesn't
end there.

Alistair

>
> Regards,
>
> Cyril.
>