Nightly Build status

[nfriedly]

See #336 (comment) for a status update

---

Now that I can finally build everything from source and end up with a working image (edit: not really, see below), I'd like to set up a nightly build that will automatically pull in new changes from the various parts of this project (and, potentially, external things such as emulators) and produce a usable image. Here's what I'm thinking:

1. Choose some main repo, probably this one (although I think it would make more sense if we moved it to the MiyooCFW org)
   • The mainboot repo is another possibility
   • DONE: we're going to use TriForceX/MiyooCFW (this repo)
2. Add a github action that updates all of the submodules to their latest respective releases, then commits to master and pushes the changes
   • Schedule the update action to run nightly
   • DONE: Add github action to update all submodules each night #338 was merged in
3. Add a build github action that builds all of the sources, similar to Acnor's WIP one except at the end, this one runs generate_image_file.sh to build an image with all of the binaries it just compiled
   • DONE-ish: Build all the things! #341
     • But.. only for the V90/Q90
     • ... and the resulting image doesn't boot past the bootloader
     • so, I added a -mixed image that uses an older precompiled uboot and rootfs from the sdcard repo
       • and dameon, because it currently breaks the controls when we build it from source
4. Over time, we add additional submodules for various emulators, games, and apps that we want to include with the CFW.
   • Third-party stuff like RetroArch can be included as binaries if we don't need to make any changes and the source is readily available.
   • If binaries aren't available but we can build it from source, then add a submodule pointing to the third-party source.
     • Maybe don't auto-update these for nightlies?
   • If we have to make changes to the source that can't be up-streamed, then we fork it and submodule the fork..
5. When we think we're close to a release, we can optionally create a git branch for the version, e.g. 2.0.x. That would essentially be a "code freeze" because the nightly script won't update the branch. Then we can pull in changes manually, without worrying about anything else being changed automatically.
   • Manual changes will still trigger a new image to be generated.
6. When we're finally ready to ship a release, we make a git tag for the version, e.g. 2.0.0. This will trigger a new image and also update the CFW version that gets displayed in the menu.

Eventually I'd like to switch from the custom script to using buildroot to generate the image and potentially manage compilation of things. That would move us from "building an image depends on doing a bunch of things and assembling the result" to "building an image depends on a single make command", which would make it easier for someone to do on their own machine instead of depending on github's infrastructure.

Oh, and I'm going to start with the V90 because that's what I can actually test, but we can also build for the other devices.

[jSTE0]

+1 for building emulators, games, and apps from source though that could be a significant undertaking (cue the bikeshedding as to which ones ought to be included). For something like RetroArch, which already has its own CI build pipeline, IMHO there's little benefit to building it again.
There are also some emulators like snes9x4d which use PGO, and have a significant performance benefit from doing so, and automating that would be tricky so it may make sense to ship the upstream binary.

[nfriedly]

Mum, that's some fair points. Maybe the rule should be "If we have to modify the source to include it, then we build from source for each release. But pre-compiled binaries are OK if we can include them without modification and the original source code is easy to find."

As for which ones to include, I was thinking about a tweak to GmenuNX where to provide it a way to hide emulators that don't have any ROMs and Ports that need additional content. I might start a new thread with that and some other quality-of-life improvements that I'd like to add. But I just want to get a release shipped before we go too deep into that.

[jSTE0]

Sounds like a good rule.

That tweak doesn't sound like part of the "MVP" in #231 so should be punted to post-release 😄

[nfriedly]

Here's my post-MVP thread: #337

[nfriedly]

I've been editing the original post, but it's getting a bit hairy, so here is a quick summary of the current status:

• All of the individual pieces now have working CI builds - click the Actions tab for any repo on https://github.com/MiyooCFW
• The main repo does a nightly update to pull in any changes and then if there were any, it does a "mega build" where it compiles all of the submodules and assembles everything together into a final image
  • It also builds a secondary "-mixed" image that uses a pre-built uboot and rootfs - more on that below
• Buildroot has been split into two parts: one to build the rootfs, and a second one to build the toolchain and docker image
  • I also created a toolchain release where you can download it for local usage without docker (linux only).

Current issues:

• When we use a uboot bootloader compiled from source, uboot can display a splash logo, but linux cannot initialize and use the display
  • This happens even if we disable the initial splash logo
  • I do get sound, though, so I know it's booted into linux.
• When we use the rootfs that buildroot builds, I don't get anything past the initial boot logo from uboot, not even sound. I'm not sure if linux even boots
  • I added our /etc/inittab and /etc/main to the sdcard repo, but even with those I still don't get anything.
• For the -mixed image, it boots and loads gmenunx, but the controls are all wonky
  • I managed to launch circut dude once, and the controls are all wonky there too, so I think it's a keyboard driver issue, not a GmenuNX issue. It was the daemon - fixed in More build tweaks #348
• Occasionally the mega build seems to hang on submodule update --init --recursive -- buildroot for hours (example). It seems to always complete eventually, but it should only take like a minute for the submodule step and 20-30 minutes for the entire checkout-and-build job.
  • Fixed in MiyooCFW/buildroot@f39b331 by checking in the dependencies that it was timing out while trying to download.
  • Further improved with the combination of MiyooCFW/buildroot@ee6d894 and buildroot reusable workflow #354

[TriForceX]

On an unrelated note, we should add miyooctl2 to the main repo and include it in the mega build.

well we should rename it back to "miyooctl" only, without the "2", the 2 is just an update of the 1, but we should keep the 1 as a release and then make the update release properly.

not store old code like this!

[nfriedly]

Alright, I figured out the buildroot taking forever issue - the main problem was dependencies that it would download at build time, one of the websites that hosts them would time out. It would retry that site 3 times before moving to the next mirror. I think the site is probably blocking github actions. I fixed it by just checking in the dependencies to enable offline builds. That shaves like a minute off of the normal 20-30 minute build time, but it completely prevents the timeout issue that led to frequent 3+ hour build times.

Before that I also did one other improvement that I decided to keep: for submodule builds in the main repo, it now caches the built rootfs.tar.xz file - so the job completes in like 35 seconds if it finds a cache hit.

I'm going to try to do the same caching trick for the kernel, which takes 5-10 minutes to build. After that, most image builds from the main repo should take ~2 minutes.

[nfriedly]

Oh, and I've made some progress on the video driver. Auto detection still fails, but when I force it to the correct panel, I can get a very broken picture - which is better than no picture at all!

[TriForceX]

what you mean with "auto-detection"? i think ive asked you before, but i also said theres no way to auto-detect anything due the system always send different hardware info...

[nfriedly]

This is the auto-detection I'm referring to: https://github.com/MiyooCFW/kernel/blob/c0b72c9f3215c2620d4400830a38c139550a43f6/drivers/video/fbdev/r61520fb.c#L406-L434 - the video driver can normally read a few settings from the panel, but it gets mostly 0's instead of the expected values when I use uboot compiled from source vs https://github.com/MiyooCFW/sdcard/blob/master/boot/misc/u-boot-bins/u-boot-v90_q90_pocketgo.bin

[nfriedly]

I think it's time for a new update. I haven't made as much progress as I'd like, but I have made some:

• The code pushes from the nightly update job now triggers the "mega build" the way I originally intended. This enabled the next point:
• There's less duplication in the build process: the "mega build" workflow in the main repo can now reuse the build workflows from submodules instead of copying-pasting. This is currently done for the kernel, buildroot, and miyooctl submodules.
  • Now when there are build changes, the only need to be updated in one place.
  • The down side is that the submodule build job is a little more complex
  • Also, it always uses the build job from the submodule's master branch, even when building code from some other branch
  • Still, I think it's a win overall
• Most megabuilds are now a lot faster, typically 2-3 minutes, thanks to two changes:
  • I checked in the buildroot dependencies. There was some intermittent network issue causing rootfs builds to occasionally take 4+ hours.
  • Caching of kernel and rootfs builds - these builds take 20-30 and 5-7 minutes respectively, but we now cache the outputs and can skip re-compiling them when the didn't change.
• I narrowed the input issues down to the daemon and, I think, this line in particular. I need to do a bit more testing, but it is definitely less broken with that line commented out.
• I added some documentation to our kernel video driver, including identifying 3/4 of the panel controller ICs that it supports: document a few things MiyooCFW/kernel#12

Outstanding issues:

• The kernel video driver is still broken in the v2 panel (ST7789S IC). My best theory is that it depends on some initialization that the uboot binary does, but our uboot source code is missing. I've tried a number of different avenues looking for a solution, including:
  • Reading the source and documentation that we do have, hoping that something would jump out at me. There's still more to read.
  • Just logging a bunch of stuff to see what's different when we boot from the two different u-boots. So far I've identified that:
    • the result of readl(iomm.lcdc + TCON_CLK_CTRL_REG) is
      • with pre-compiled u-boot: 0xF0000006
      • with u-boot from source: 0
    • after sending command 0x04 to identify the panel IC, the response (4 calls to lcdc_rd_dat()) is
      • with pre-compiled u-boot: 0, 0, 0x85, 0x85 (usually)
      • with u-boot from source: 0, 0, 0, 0
      • sometimes the pre-compiled u-boot also returns all 0's, so it may just be a timing issue?
    • I want to try doing something similar to this and see if anything jumps out: https://github.com/prenticedavid/MCUFRIEND_kbv/blob/master/examples/LCD_ID_readreg/LCD_ID_readreg.ino
  • Using Ghidra's Version Tracking tool to decompile the two different u-boot binaries and find differences between them
    • I wasn't making much progress, but I read something recently suggesting that I strip out the u-boot header and only give ghidra the remaining bits. So, I may try that sometime soon.
  • I want to go back and make sure there isn't something in the boot.cmd/boot.scr - I think there's a way to use that to pass an initialized video device from u-boot to linux.
  • I tried compiling the 1.2 kernel but it didn't seem to work for me
    • Our u-boot source appears to be identical to the 1.2 u-boot source code there
    • I haven't tried building everything from 1.2, but I may yet.
  • I reached out to a couple of people for help, and steward-fu suggested I look at the ST7789V driver in https://github.com/steward-fu/fc3000/releases/tag/source but noted that the v90 lacks the TE pin. I haven't dug into this yet.
• U-Boot video driver: I haven't spent much time trying to turn our u-boot video driver into a "proper" driver that can display content from u-boot instead of only a hard-coded image, but I think that is the next step towards adding the version to the u-boot screen, and may also help with the linux video issue.
• Buildroot root FS: I haven't touched the rootfs, I'd like to get a video driver working first.

I think there's more room to go down any of these paths for the kernel video driver, but I'm not sure which one I want to put more time into first.

I also ordered a pocketgo so that hopefully I can get a UART connection and get some useful insights from that.

[TriForceX]

The kernel video driver is still broken in the v2 panel (ST7789S IC).

we should have an "status table" somewhere to have a test log, something like:

console	status	note
bb2	failed	something ...
bb3	working	something ...
pgo	failed	something ...
v90	working	something ...

but our uboot source code is missing.

I dont remember if we tried/checked the old one https://github.com/NxHope/f1c500s_uboot

I think there's more room to go down any of these paths for the kernel video driver, but I'm not sure which one I want to put more time into first.

Yea there are many things to "fix/improve/optimize" but i would go for the "fix" path, getting the stuff working for the 2.0 release, after that we should continue improving stuff like the video driver or buildroot thing.

I also ordered a pocketgo so that hopefully I can get a UART connection and get some useful insights from that.

Do you get UART done on v90? due i have one i would like to do some tests

[nfriedly]

we should have an "status table"

I think it's going to be broken on all of them due to the rootfs issues that I haven't gotten very far on. (Although, I suppose we could add a column for the mixed build with some things working there.)

I dont remember if we tried/checked the old one https://github.com/NxHope/f1c500s_uboot

I don't think so, but I'll add it to my list.

Yea there are many things to "fix/improve/optimize" but i would go for the "fix" path, getting the stuff working for the 2.0 release, after that we should continue improving stuff like the video driver or buildroot thing.

Both the video driver fix and the buildroot rootfs fix are necessary to be able to build a 2.0 image from source. Otherwise we're back to assembling an image out of random binaries.

Do you get UART done on v90? due i have one i would like to do some tests

No. It might be possible, but I haven't figured it out yet. Whereas the PocketGo looks to be more straightforward.

I also messaged someone on Reddit who is selling an "OG Bittboy" which, based on the picture, I think is actually a New Bittboy. So that might give me a third device to test on.

[nfriedly]

I just ordered a LicheePi nano also - Seeed Studio is sold out, but digikey has it in the more expensive bundle that comes with a wifi module. (Mouser does too, but their shipping was higher.) I can definitely get a UART on that device, and I'm hoping that I can connect it directly to the v90's display. Regardless of the display, I think it will help with the buildroot / rootfs issues. I'll update more after it arrives.

[nfriedly]

Quick update: Thanks to work from @tiopex (MiyooCFW/uboot#5, MiyooCFW/uboot#6), we're now using uboot from source! (And thank you to @Apaczer also!)

There are a couple of issues, but I think they'll be easier to address now that we have a working foundation in place.