Re: Upstreaming my Android channel

[Maxim Cournoyer]

Hi Julien,

Julien Lepiller <julien@lepiller.eu> writes:

> Hi Guix!
>
> I have been working on updating Android packages for some time now. I
> think it's at a point I'm confident I can share and move them upstream.
> My work is currently in a separate channel at
> https://framagit.org/tyreunom/guix-android. I have been working on the
> SDK and tools, which we already have. We currently are limited to
> Android 7 versions, because later Android versions do not use the same
> build system anymore.
>
> Since then, Android uses the Soong build system, based on blueprint.
> It's pretty complex, and specific. I tried to build it, but it assumes
> too many things to be useful. Instead, I "reimplemented" it by
> following what a Nix contributor did in soongnix:
> https://github.com/danielfullmer/soongnix
>
> I created a soong-build-system that has a few modules for cc, art and
> java types of packages. It supports building most packages from the SDK
> and other Android tools: aapt, appt2, adb, aidl, apksigner, dexdump,
> dmtracedump, dx, etc1tool, fastboot, hprof-conv, libaapt2_jni,
> split-select, zipalign.

This sounds good!

> I have an importer that is a bit different from the existing importers.
> First, it needs to checkout ~30GB of git repositories (and that's with
> --depth=1 for all of them!), explore the set of Android.bp files (used
> by soong) and create one package for each package in these Android.bp
> files. That creates packages that can be a bit smaller than expected.
> Instead of printing, the importer creates two files: one for the source
> definitions (many packages share the same source) and one for the
> packages and their dependencies.

30 GiB!  Wow.  Is there no lighter means to get at the needed data?
Perhaps some tarball snapshot with test data filtered or similar?  I
hope this data is at least gets reused (cached) between invocations?
But I'd expect even just updating such a large set of git repository
would be slow... potentially slowing each invocation of the importer?
Or do you handle this?

> Since I have all of that in a channel, the importer doesn't work from
> the command-line, but I have a relatively simple API that you can use
> to import packages:
>
> ```
> ,use (android import repo)
> (define manifest (get-manifest))
>
> ;; Fetch repositories that form the android distribution.
> ;; This is very big (> 28GB) and will take some time.  If it fails
> ;; because of a network issue, restart that command again until
> ;; everything is downloaded.
> ;;
> ;; You can also add an optional destination directory where the
> ;; sources are to be downloaded, e.g. on a different file-system if
> ;; you lack space on the file system your home directory is on.
> (define root (fetch-manifest-repositories manifest))
>
> ;; Finds all Android.bp files from which to import, takes ~10s on warm
> ;; cache, significantly longer on cold cache and spiny disks.
> (define bp-files (get-all-bp-files root))
>
> ;; Creates the shared structure: a map of modules and variables to the
> ;; files that define them, so we can find them, without actually parsing
> ;; everything.  Will take ~5mins on an HDD, could be faster on SSD.
> (define bp-maps (get-bp-maps bp-files))
>
> ;; Run the import.  The last two arguments are the names of files that
> ;; will be generated.
> (import-recursively manifest root bp-maps '("adb" "fastboot")
>                     "sources.scm" "packages.scm")
> ```
>
> That would import adb and fastboot

Cool!

> You can see the result of importing a bit more than that at
> https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/android-tools.scm
> and
> https://framagit.org/tyreunom/guix-android/-/blob/master/android/packages/android-sources.scm
>
> It's pretty great but missing a few things. These packages don't have a
> home page, synopsis, description, ... and I don't see a way to easily
> import that. Although I could manually add them, the work will have to
> be done again at each update. Every month, Google releases a new
> Android revision (not necessarily a major version) that modifies some
> sources, and can impact the set of dependencies we need to build.
>
> I was aiming at having an importer that was able to recreate all the
> files with no human intervention. I think it's the only sane way to
> take care of all those packages.

Perhaps there's another web site (like pkg.go.dev for Go packages)
holding such metadata that we could query to fill the missing blanks?

> Currently I think all of android/build, android/build-system,
> android/import, android/packages/android-{headers,tools,sources}.scm
> and android/packages/{bison,clang}.scm can be upstreamed. The rest is
> wip or broken packages.
>
> Do you have any ideas how we could plan upstreaming all this?
> Especially, how could the importer fit in the current Guix
> infrastructure?

If the quirks (30 GiB checkout for one) are well documented, also
perhaps directly interactively (warning: the importer needs to fetch
about 30 GiB of sources; hit enter to proceed), I don't see why the
importer couldn't sit right next to the others!

Thank you!

Maxim