In-tree projects

[puddly]

This PR will rewrite our firmware builder:

1. Every radio is maintained as a complete project.
2. Local development is done by just git cloneing the repository and opening a specific project file within Simplicity Studio (or your editor of choice).
3. Local development uses the same post-build hooks as CI to generate output GBL files with automatic metadata:
   

The tools/create_gbl.py script has full access to the source tree and is able to automatically inject OpenThread, EmberZNet, and Z-Wave metadata into the final GBL. The goal with this script is to make CI and local development produce identical artifacts.

CI needs just a small tweak for the post-build hook:

# Generate makefiles
/opt/slc_cli/slc generate --project-file=SomeProject.slcp --export-destination=build/ --sdk /opt/gecko_sdk

# Overwrite config
cp -Rfv config build

# Inject the post-build script into the makefiles (TODO write a sentinel `.slpb` to search/replace)
sed -i 's|# No post-build defined|-../../tools/create_gbl.py $(OUTPUT_DIR)/$(PROJECTNAME).out|' build/SomeProject.project.mak
make -C build -f SomeProject.Makefile

The main downside of this approach is that bumping the Gecko SDK version will require manually opening the projects in Simplicity Studio and re-linking the SDK. We also would need to keep track of changes to defaults, constants, etc. from the SiLabs example projects.

Thoughts?

[darkxst]

It certainly is a pain to get a local development environment currently! However I have a lot more targets in my builder and manually loading each project to bump the SDK would also be a pain!

I have been pondering this for a while but never tried to implement yet, but now that act is working it maybe possible. The idea somewhat inspired by how debian packaging works with git build package, where the source package is imported and patches applied on top of that so you can edit/rebase whatever you want with the patch series in git.

So essentially to have a workflow target (based around existing build flow) that setups the container, applies config, git am's patches but stops short of actually building the firmware. It could probably go as far as replicating the required project for opening in silabs studio, but I would probably envision just loading it up in Vscode devcontainer.

[agners]

I see that to work on a particular firmware, committing the actual firmware would be much nicer. But yeah, trade-offs... 😅

Maybe we can also consider supporting both approaches with the CI system? This way we could maintain source trees for firmwares we are actively working on/have many changes on (multi-pan currently), and have a pretty much fully automated update system for the rest (ot-rcp/zigbee).

I have been pondering this for a while but never tried to implement yet, but now that act is working it maybe possible. The idea somewhat inspired by how debian packaging works with git build package, where the source package is imported and patches applied on top of that so you can edit/rebase whatever you want with the patch series in git.

So essentially to have a workflow target (based around existing build flow) that setups the container, applies config, git am's patches but stops short of actually building the firmware. It could probably go as far as replicating the required project for opening in silabs studio, but I would probably envision just loading it up in Vscode devcontainer.

I like that idea! Maybe the devcontainer could be the one we have today essentially, and setting up a particular firmware/applying patches is just a couple of VS Code tasks (calling scripts we share between CI/devcontainer)?

I think Silabs is also moving towards VSCode. There is already an extension: https://docs.silabs.com/simplicity-studio-5-users-guide/latest/ss-5-users-guide-vscode-ide/.

[puddly]

I like mixing the approaches too:

1. We have three canonical projects: Zigbee, Thread, and multi-PAN. For simplicity, let's use the SkyConnect for all three, as the Yellow differs only by having LEDs.
2. All other firmwares are derived from these projects either via YAML modification scripts or with .patches. The only differences are chip model IDs and pins.

That way, we can still work on firmwares while being able to automate builds for the many other individual devices.

[darkxst]

I think Silabs is also moving towards VSCode. There is already an extension: https://docs.silabs.com/simplicity-studio-5-users-guide/latest/ss-5-users-guide-vscode-ide/.

Yes, they are certainly moving towards providing the option of using VSCode. SS is still required for some configuration such as pintool, however slc already includes support for generating vscode projects.

Vscode projects use cmake/ninja as build system rather than Makefiles. I suppose for consistency we should switch the CI to also use cmake, then its just a case of having a shared script that run slc generate , applies patches and generates metadata. I think we could also make use of postbuild .slpb hooks to generate the .gbl files.

[puddly]

I think we could also make use of postbuild .slpb hooks to generate the .gbl files.

Unfortunately, post-build hooks can only chain together one of a few pre-made actions, none of which can run external code. While it can generate a GBL, it won't be able to generate the metadata binary that we inject into the GBL, which still necessitates automatically running some tool. These hooks all get translated into a commander command line in the end in all three build systems so as long as we have a way to run some hook, we can run it via Python and replace the functionality. This is what create_gbl.py does.

My idea for this was to create a dummy .slpb file just so that the commander call is present in all three build systems, which will make it easy to replace with something more usable.

[puddly]

Inspired by the JSON metadata you use for your builds @darkxst, how about something like this? One file would be created for every firmware:

name: SkyConnect
device: EFR32MG21A020F512IM32
base_project: ncp-uart-hw

# Only used by Z-Wave firmwares
gbl:
  compression: lz4
  sign_key: /path/to/sign.key
  encrypt_key: /path/to/encrypt.key
  dynamic: ["ezsp_version"]

# Extra components to include in the generated SLCP file
add_components:
- instance: [board_activity]
  id: simple_led

# Components to remove from the generated SLCP file
remove_components:
- instance: [some_component]

# Extra configuration to inject into the generated SLCP file
configuration:
  SL_RAIL_UTIL_PTI_MODE: RAIL_PTI_MODE_DISABLED
  SL_IOSTREAM_USART_VCOM_BAUDRATE: 115200
  SL_SIMPLE_LED_BOARD_ACTIVITY_POLARITY: SL_SIMPLE_LED_POLARITY_ACTIVE_LOW

# Same with C defines
c_defines:
  EMBER_ADDRESS_TABLE_SIZE: 16
  EMBER_MULTICAST_TABLE_SIZE: 16

  # Pin config is always specified as a C define
  SL_SIMPLE_LED_BOARD_ACTIVITY_PORT: gpioPortB
  SL_SIMPLE_LED_BOARD_ACTIVITY_PIN: 1

  SL_UARTDRV_USART_VCOM_PERIPHERAL: USART0
  SL_UARTDRV_USART_VCOM_PERIPHERAL_NO: 0

  SL_UARTDRV_USART_VCOM_TX_PORT: gpioPortA
  SL_UARTDRV_USART_VCOM_TX_PIN: 5

  SL_UARTDRV_USART_VCOM_RX_PORT: gpioPortA
  SL_UARTDRV_USART_VCOM_RX_PIN: 6

  SL_UARTDRV_USART_VCOM_CTS_PORT: gpioPortD
  SL_UARTDRV_USART_VCOM_CTS_PIN: 2

  SL_UARTDRV_USART_VCOM_RTS_PORT: gpioPortC
  SL_UARTDRV_USART_VCOM_RTS_PIN: 1

I believe this format encompasses all of the stick- and firmware-specific configuration necessary.

We would have three base projects maintained in-tree: one for each firmware type. For simplicity, I'll use the SkyConnect as the base target. Its project folder would then be transformed with a project generator (generator) to create projects for all other sticks, which we can then compile.

[darkxst]

yes true the postbuild hooks can't generate the metadata.

in all three build systems so as long as we have a way to run some hook,

There is really only the two build systems (unless you include IAR) Makefiles (for SS and current CI) or cmake (vscode), the IDE targets just add some glue for the IDE's from what I can tell.

We can likewise also inject the post build into cmake with something like

set(post_build_command ../../tools/create_gbl.py)

Inspired by the JSON metadata you use for your builds @darkxst, how about something like this?

Yes this would work, so the base skyconnect project would use commited slcp and others patch this?

  # Pin config is always specified as a C define

Getting rid of the pin patches would be most welcome. I had considered using the .pintool files to generate these defines, as they are last device specific patches in my builders. Doing it like this would work as well. Maybe one day Silabs will actually provide CLI support for the pintool files!

I must say on my system SS is near unusable, partly due to messed up rendering on Linux HiDPI displays, but also it crashes alot! So vscode support is very appealing!

[puddly]

I've made quite a bit of progress with making project templates work and have both the SkyConnect and Yellow firmwares NCP/RCP firmwares functional.

I primarily develop on a dev kit so the default projects are based on those. Project generation still needs a bit of work to handle all of the three ways pins and config is defined but I am able to take an unmodified Simplicity Studio project, retarget it to another chip, generate a CMake build environment, build it with Ninja, and produce a GBL:

mkdir build
python ../tools/generate_project.py ../manifests/yellow_ncp-uart-hw.yaml .

This takes about 20s, most of which is spent waiting for slc-cli to start up and run. The scripts currently have some hard-coded SDK and toolchain paths for macOS but once everything is functional I'll fix it up with argparse.

[darkxst]

I primarily develop on a dev kit so the default projects are based on those.

I recently discovered the Sparkfun MGM240P dev kits, which provide a very cost effective way to get dev kit debug goodness! albeit its based on EFR32MG24, which probably has its own set of bugs compared to mg21..

The scripts currently have some hard-coded SDK and toolchain paths for macOS but once everything is functional I'll fix it up with argparse.

When building in the CI container, these are not required to be specified as they have already been set by slc configuration and get picked up automatically with slc generate and in toolchain.cmake.

Also on linux slc-cli binary is just slc.

[puddly]

When building in the CI container, these are not required to be specified as they have already been set by slc configuration and get picked up automatically with slc generate and in toolchain.cmake.

I'm hoping to avoid as much external state as possible with slc; it's bad enough to have to slc signature trust every SDK, extension, and toolchain.

Also on linux slc-cli binary is just slc.

Both are now supported by the main project generator.

I've added support for bootloader building as well and have verified compatibility with the Sonoff stick as a demo. This should be ready for review once I finish the OpenThread source tree and fix up CI.

On the topic of CI: how are we going to go through with this? Personal access tokens?

[puddly]

I've tested the rewritten CI locally with Act and have successfully built all of the referenced firmwares so I think this PR is done! I'll move the firmwares themselves into src/, as otherwise the root folder is a little cluttered.

I've also switched back to Makefiles but left around the CMake + Ninja code: CMake is just not usable from SLC-CLI yet. Some problems can be patched, others I don't have time to investigate.

You can generate firmware locally as follows:

python tools/generate_project.py \
    --manifest manifests/skyconnect_ot-rcp.yaml \
    --build-dir build \
    --build-system makefile \
    # Multiple SDKs can be specified, the correct one will be used
    --sdk /path/to/gecko_sdk \
    --sdk /path/to/gecko_sdk_4.3.1 \
    # Similarly for toolchains
    --toolchain /path/to/gnu_arm/10.3_2021.10 \
    --toolchain /path/to/gnu_arm/12.2.rel1_2023.7 \
    # Output files are prefixed by the extension so you can export s37, out, map, gbl, etc.
    --output gbl:test.gbl \
    --output out:test.out

CI has been rewritten to bundle multiple SDKs and toolchains in the builder image. This allows us to have firmware-specific Gecko SDKs.

The next step will be organizing build artifacts into proper releases and actually getting CI working for pull requests.

[agners]

It moves quite some logic into the generate_project.py. We'll see how much maintenance this is, but I guess it would have been the same if we continue to add logic into the GH action steps. The Python script will definitely be easier to execute locally and iterate. And of course the manifest support is nice 👍 🤩

[github-actions]

There hasn't been any activity on this pull request recently. This pull request has been automatically marked as stale because of that and will be closed if no further activity occurs within 7 days.
Thank you for your contributions.