Release 2022.07 - RC3 #257

chrysn · 2022-08-09T13:47:32Z

This issue lists the status of all tests for the Release Candidate 3 of the 2022.07 release.

Specs tested:

chrysn · 2022-08-09T13:51:55Z

Ticking off "Task #01 - Compile test" as the last commit on the release branch passed a murdock run that did not seem to skip any tests.

riot-ci · 2022-08-09T13:53:07Z

Test Report

	Task	Outcome
✔	01. Task 02 - Unittests on native	PASSED
✔	01. Task 03 - Unittests on native separated [tests-analog_util]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-base64]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bcd]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bitfield]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bloom]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-bluetil]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-checksum]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-chunked_ringbuffer]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-clif]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-color]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-core]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-credman]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-div]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-dns_cache]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ecc]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-fib]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-fib_sr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-fmt]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-frac]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gcoap]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_ipv6]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_ipv6_hdr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_ipv6_nib]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_netif_pktq]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_sixlowpan_frag_vrb]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-gnrc_udp]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-hashes]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ieee802154]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-inet_csum]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ipv4_addr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ipv6_addr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ipv6_hdr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-kernel_defines]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-luid]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-matstat]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-mtd]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-nanocoap]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-nanocoap_cache]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-netopt]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-phydat]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-pkt]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-pktbuf]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-pktqueue]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-printf_float]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-priority_pktqueue]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-rtc]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-rtt_rtc]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-saul_reg]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-scanf_float]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-seq]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sht1x]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sixlowpan]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sixlowpan_ctx]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sixlowpan_sfr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-sock_util]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-timex]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-tsrb]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-turo]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-uri_parser]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ut_process]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-uuid]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-vfs]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-zptr]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ztimer64]	PASSED
✔	01. Task 03 - Unittests on native separated [tests-ztimer]	PASSED
✔	03. Task 01 - ICMPv6 multicast echo on native	PASSED
✔	03. Task 02 - ICMPv6 link-local echo on native	PASSED
✔	03. Task 03 - ICMPv6 link-local echo on native (1 hour)	PASSED
✔	03. Task 04 - ICMPv6 stress test on native (1 hour)	PASSED
🟡	03. Task 05 - ICMPv6 stress test on native (neighbor cache stress)	SKIPPED
✔	03. Task 06 - ICMPv6 link-local echo on native (IPv6 fragmentation)	PASSED
✔	04. Task 01 - ICMPv6 link-local echo with iotlab-m3	PASSED
✔	04. Task 02 - ICMPv6 multicast echo with iotlab-m3/samr21-xpro	PASSED
✔	04. Task 03 - ICMPv6 echo with large payload	PASSED
✔	04. Task 04 - ICMPv6 echo with iotlab-m3/samr21-xpro 15 minutes	PASSED
🟡	04. Task 05 (Experimental) - ICMPv6 multicast echo with samr21-xpro/cc2538 [lille]	SKIPPED
🟡	04. Task 06 (Experimental)- ICMPv6 link-local echo with samr21-xpro/cc2538 [lille]	SKIPPED
🟡	04. Task 07 (Experimental) - ICMPv6 multicast echo with samr21-xpro/zero + xbee	SKIPPED
🟡	04. Task 08 (Experimental) - ICMPv6 echo with samr21-xpro/zero + xbee	SKIPPED
✔	04. Task 09 - ICMPv6 stress test on iotlab-m3	PASSED
✔	04. Task 10 (Experimental) - ICMPv6 echo with large payload (IPv6 fragmentation)	PASSED
✔	04. Task 11 (Experimental) - ICMPv6 stress test on nrf802154	PASSED
✔	04. Task 12 (Experimental) - ICMPv6 multicast echo with iotlab-m3/nrf802154	PASSED
✔	04. Task 13 (Experimental) - ICMPv6 link-local echo with iotlab-m3/nrf802154	PASSED
✔	05. Task 01 - ICMPv6 echo unicast addresess on native (default route)	PASSED
✔	05. Task 02 - ICMPv6 echo unicast addresess on iotlab-m3 (default route)	PASSED
✔	05. Task 03 - ICMPv6 echo unicast addresess on native (specific route)	PASSED
✔	05. Task 04 - ICMPv6 echo unicast addresess on iotlab-m3 (static route)	PASSED
✔	06. Task 01 - UDP on iotlab-m3	PASSED
✔	06. Task 02 - UDP on iotlab-m3 (UDP port compression)	PASSED
✔	06. Task 03 - UDP on native (non-existent neighbor)	PASSED
✔	06. Task 04 - UDP on iotlab-m3 (non-existent neighbor)	PASSED
✔	06. Task 05 - Empty UDP on native	PASSED
✔	06. Task 06 - Empty UDP on iotlab-m3	PASSED
✔	07. Task 01 - ICMPv6 echo on iotlab-m3 with three hops (static route)	PASSED
✔	07. Task 02 - UDP on iotlab-m3 with three hops (static route)	PASSED
✔	07. Task 03 - ICMPv6 echo on iotlab-m3 with three hops (RPL route)	PASSED
✔	07. Task 04 - UDP on iotlab-m3 with three hops (RPL route)	PASSED
🟡	07. Task 05 (Experimental) - UDP with large payload on iotlab-m3 with three hops (RPL route)	SKIPPED
✔	08. Task 01 - ICMPv6 echo between native and Linux	PASSED
✔	08. Task 08 - UDP between GNRC and lwIP on iotlab-m3	PASSED
✔	09. Task 01 - CORD Endpoint	PASSED
✔	09. Task 02 - Confirmable retries [False-0]	PASSED
✔	09. Task 02 - Confirmable retries [True-1]	PASSED
✔	09. Task 03 - Block1	PASSED
✔	09. Task 04 - Block2	PASSED
✔	09. Task 05 - Observe registration and notification	PASSED
✔	10. Task 01 - Destination unreachable - no route to destination	PASSED
✔	10. Task 02 - Destination unreachable - Beyond scope of source address	PASSED
✔	10. Task 03 - Destination unreachable - address unreachable (target node address)	PASSED
✔	10. Task 04 - Destination unreachable - address unreachable (neighbor cache miss)	PASSED
✔	10. Task 05 - Destination unreachable - port unreachable	PASSED
✔	10. Task 06 - Destination unreachable - port unreachable (large payload)	PASSED
✔	10. Task 07 - Packet too big	PASSED
✔	10. Task 08 - Time exceeded - hop limit exceeded in transit	PASSED
✔	10. Task 09 - Parameter problem - erroneous header field encountered	PASSED
✔	10. Task 10 - IPv6-in-IPv6 encapsulation	PASSED
✔	11. Task 01 - LoRaWAN example	PASSED
✔	11. Task 02 - OTAA join procedure	PASSED
✔	11. Task 03 - ABP join procedure	PASSED
✔	11. Task 04 - LoRaWAN device parameters persistence	PASSED
✔	11. Task 05 - GNRC LoRaWAN (Over The Air Activation) [otaa]	PASSED
🟡	11. Task 06 - GNRC LoRaWAN (ABP) [abp]	SKIPPED

chrysn · 2022-08-20T07:58:12Z

Tests ran manually on iotlab-m3:

$ BOARD=iotlab-m3 IOTLAB_NODES=1 IOTLAB_DURATION=10 make iotlab-exp -C tests/tests_unittests
$ iotlab-experiment get -n
[...] m3-101.grenoble.iot-lab.info
$ BOARD=iotlab-m3 IOTLAB_NODE=m3-101.grenoble.iot-lab.info make all flash test -C tests/tests_unittests
$ BOARD=iotlab-m3 IOTLAB_NODE=(actually a different experiment) ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . iotlab-m3

This is ongoing, and I'll tick off 02-tests / "Task #2 - Subset of tests on iotlab-m3" when that's done.

chrysn · 2022-08-20T11:16:38Z

I've run through the native tests (compile-and-test), with some failures -- but AFAICT they were all related to odd versions of compilers used here:

Rust applications break; this is known as #18389 and generally we only support the nightly in CI -- but given it showed up here I better file for a backport.
examples/ccn-lite-relay build failed with Use-after-free in ccnl-core RIOT#18480
Some just didn't work because I lacked depenencies locally
- fuzzing was not tested for lack of an AFL compiler
- For C++ tests I didn't have the right headers around
- For several pkgs, C++ headers, Python libraries or programs (protoc) were missing.

Except for the 18480 issue, I consider these good enough. If that turns out to be a false positive, I'll tick the "build on native" box.

chrysn · 2022-08-23T10:39:04Z

Going through the native tests, they worked except

micropython: Example test fails RIOT#18487 (which I consider not critical)
ztimer_periodic: Unrealistic expectations on native RIOT#18490 (which is about unrealistic expectations on the microsecond timer on native; not ciritcal)
tests/thread_float fails on native RIOT#17170 (tests/thread_float which is a known issue and highly dependent on the local setup on native)

I consider these tests good enough to tick the "native" box.

chrysn · 2022-08-23T12:02:34Z

At iotlab-m3, local building and testing-at-IoT-LAB worked except:

where building lacked the C++ headers or other local dependencies (like on native)
if (deviating from the documentation) nightly is not installed as the default Rust version, tests/rust_libs fails. That's OK for this release, but I'm taking the opportunity for Modules implemented in Rust should build on stable Rust RIOT#18500

A few tests failed (but hey, this is already running all runnable ones rather than "a subset of"; still):

Several tests just failed once but worked in a rerun (eg. bench_msg_pingpong, bench_mutex_pingpong up to tests/deferred_yield_higher) ... looks a bit like I just had network trouble during that time in the first round
tests/turo passed on second attempt, maybe in the same category as bench_msg_pingpong..deferred_yield_higher
tests/buttons is not runnable on that board ("no buttons available"), indicating that the test should not be run there (TBD)
tests/riotboot failed but AIU that's not applicable w/o a bootloader (which we don't have there -- TBD)
examples/micropython fails, but we've had that on native already.

Up to be investigated are still:

gnrc_rpl -- Makefile already says that automated tests only work on nightly -- TBD as with the above: check why compile_and_test_for_board disregards that)
periph_timer_short_relative_set -- similarly, Makefile says it's OK, TBD

both of which fail in repeated tests on m3-100.grenoble.iot-lab.info.

(all done in manual runs like BOARD=iotlab-m3 IOTLAB_NODE=m3-100.grenoble.iot-lab.info make flash reset test -C tests/...)

With the above boxes ticked, some things are left to do in tests cleanup, but none of that is release critical; ticking the iotlab-m3 box.

chrysn · 2022-08-24T08:50:11Z

Closing in favor of #258

chrysn added the help wanted label Aug 9, 2022

chrysn mentioned this issue Aug 9, 2022

Release 2022.07 - RC2 #254

Closed

88 tasks

chrysn mentioned this issue Aug 20, 2022

pkg/ccn-lite: patch to fix use-after-free RIOT-OS/RIOT#18483

Merged

maribu mentioned this issue Aug 21, 2022

pkg/ccn-lite: patch to fix use-after-free [backport 2022.07] RIOT-OS/RIOT#18488

Merged

chrysn closed this as completed Aug 24, 2022

chrysn mentioned this issue Aug 24, 2022

Release 2022.07 - RC4 #258

Closed

88 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release 2022.07 - RC3 #257

Release 2022.07 - RC3 #257

chrysn commented Aug 9, 2022 •

edited

Loading

chrysn commented Aug 9, 2022

riot-ci commented Aug 9, 2022 •

edited

Loading

chrysn commented Aug 20, 2022

chrysn commented Aug 20, 2022

chrysn commented Aug 23, 2022

chrysn commented Aug 23, 2022 •

edited

Loading

chrysn commented Aug 24, 2022

Release 2022.07 - RC3 #257

Release 2022.07 - RC3 #257

Comments

chrysn commented Aug 9, 2022 • edited Loading

chrysn commented Aug 9, 2022

riot-ci commented Aug 9, 2022 • edited Loading

Test Report

chrysn commented Aug 20, 2022

chrysn commented Aug 20, 2022

chrysn commented Aug 23, 2022

chrysn commented Aug 23, 2022 • edited Loading

chrysn commented Aug 24, 2022

chrysn commented Aug 9, 2022 •

edited

Loading

riot-ci commented Aug 9, 2022 •

edited

Loading

chrysn commented Aug 23, 2022 •

edited

Loading