04-single-hop-6lowpan-icmp: initial import of automated tests #161

miri64 · 2020-07-13T15:39:12Z

Taken out of the original #155, this provides automated tests (where possible with the current framework) for spec 4 to the new, pytest-based framework.

Requires #155

miri64 · 2020-07-15T13:29:15Z

Rebased to current master.

04-single-hop-6lowpan-icmp/test_spec04.py

miri64 · 2020-07-16T10:36:49Z

And rebased to current master

fjmolinas

Looks good, I think in the specs the wait time for the pktbuf to be empty should match the wait time we use in the scripts. Otherwise this gives a good first base to have, I'll run the all the tests once to verify that everything is OK, before ACKing. Tests 6-8 I'll run locally.

fjmolinas · 2020-07-16T14:02:27Z

One question, why not simply test for the script? Seem redundant since the folder is already 04-single...

fjmolinas · 2020-07-16T14:13:57Z

04-single-hop-6lowpan-icmp/test_spec04.py

+    assert pktbuf(pinger).is_empty()
+
+
+@pytest.mark.xfail(reason="Experimental task")


Realizing this while running the tests, but I'm not sure these should be marked as xfail, if they are then we get no information about the failure, I think we should not mark them as such.

RIOT_TERMINAL=socat RIOTBASE=/home/francisco/workspace/RIOT TRIBE_CI=1 pytest 04-single-hop-6lowpan-icmp/test_spec04.py -k="test_task05 or test_task06 or test_task07 or test_task08" --local ========================================================== test session starts ========================================================== platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /home/francisco/.virtualenvs/release/bin/python3 cachedir: .pytest_cache rootdir: /home/francisco/workspace/Release-Specs, inifile: setup.cfg collected 10 items / 6 deselected / 4 selected 04-single-hop-6lowpan-icmp/test_spec04.py::test_task05[nodes0] XFAIL [ 25%] 04-single-hop-6lowpan-icmp/test_spec04.py::test_task06[nodes0] XFAIL [ 50%] 04-single-hop-6lowpan-icmp/test_spec04.py::test_task07[nodes0] XPASS [ 75%] 04-single-hop-6lowpan-icmp/test_spec04.py::test_task08[nodes0] XFAIL [100%] ------------------------------ generated xml file: /home/francisco/workspace/Release-Specs/test-report.xml ------------------------------ ======================================== 6 deselected, 3 xfailed, 1 xpassed in 292.06s (0:04:52) ========================================

For now what I would rather do is mark 05-08 as local only since arduino-zero is currently broken in iotlab, that way they will be skipped anyway

(Don't be alarmed by failing tests, I don't actually have all the boards on the test, so its normal for them to fail (and xbee initialization seem to still be flimsy).

Will do. There are however also other tests in other specs that are marked as xfail due to their experimental nature. Shall I remove them there as well?

I think we should, can you point me to the others that were marked?

Appearently the only one remaining is in #160:

03-single-hop-ipv6-icmp/test_spec03.py:@pytest.mark.xfail(reason="See https://github.com/RIOT-OS/RIOT/issues/12565")

miri64 · 2020-07-16T14:19:42Z

One question, why not simply test for the script? Seem redundant since the folder is already 04-single...

pytest did not collect that for me.

miri64 · 2020-07-16T14:22:32Z

Looks good, I think in the specs the wait time for the pktbuf to be empty should match the wait time we use in the scripts.

I think it is not breaking the specs if we wait longer. The times in the specs are there so people don't try it immediately after. By waiting a bit longer in the automated tests we ensure we do not hit the cleaning of the packet buffer just at that moment accidentally.

miri64 · 2020-07-16T14:24:52Z

One question, why not simply test for the script? Seem redundant since the folder is already 04-single...

pytest did not collect that for me.

(sure, could have been just test_spec.py as well, but I won't change this for all the specs now ^^")

miri64 · 2020-07-16T14:32:53Z

Applied comments and ~~squashed~~ rebased.

fjmolinas · 2020-07-16T14:32:57Z

Test results:

========================================================== test session starts ==========================================================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /home/francisco/.virtualenvs/release/bin/python3
cachedir: .pytest_cache
rootdir: /home/francisco/workspace/Release-Specs, inifile: setup.cfg
collected 10 items                                                                                                                      

04-single-hop-6lowpan-icmp/test_spec04.py::test_task01[nodes0] PASSED                                                             [ 10%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task02[nodes0] PASSED                                                             [ 20%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task03[nodes0] PASSED                                                             [ 30%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task04[nodes0] PASSED                                                             [ 40%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task05[nodes0] SKIPPED                                                            [ 50%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task06[nodes0] SKIPPED                                                            [ 60%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task07[nodes0] XFAIL                                                              [ 70%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task08[nodes0] XFAIL                                                              [ 80%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task09[nodes0] PASSED                                                             [ 90%]
04-single-hop-6lowpan-icmp/test_spec04.py::test_task10[nodes0] XPASS                                                              [100%]

------------------------------ generated xml file: /home/francisco/workspace/Release-Specs/test-report.xml ------------------------------
======================================================== short test summary info ========================================================
SKIPPED [1] 04-single-hop-6lowpan-icmp/test_spec04.py:120: Test can't run on IoT-LAB
SKIPPED [1] 04-single-hop-6lowpan-icmp/test_spec04.py:144: Test can't run on IoT-LAB

miri64 · 2020-07-16T14:33:48Z

04-single-hop-6lowpan-icmp/test_spec04.py::test_task10[nodes0] XPASS

Interesting oO

fjmolinas · 2020-07-16T14:34:05Z

Please squash @miri64!

Co-Authored-By: Alexandre Abadie <alexandre.abadie@inria.fr> Co-Authored-By: Martine S. Lenders <m.lenders@fu-berlin.de>

miri64 · 2020-07-16T14:37:29Z

Squashed

fjmolinas

ACK!

miri64 · 2020-07-16T15:05:12Z

Thanks for the review!

miri64 added the enhancement label Jul 13, 2020

miri64 requested review from aabadie and fjmolinas July 13, 2020 15:39

This was referenced Jul 13, 2020

testutils: Automated testing environment for release tests using pytest #155

Merged

pytest: tracking porting efforts of release tests #164

Closed

miri64 force-pushed the pytest/env/spec04 branch from e224e41 to 964db19 Compare July 15, 2020 13:29

fjmolinas reviewed Jul 16, 2020

View reviewed changes

04-single-hop-6lowpan-icmp/test_spec04.py Outdated Show resolved Hide resolved

miri64 force-pushed the pytest/env/spec04 branch 2 times, most recently from b8fff84 to b67f945 Compare July 16, 2020 10:36

fjmolinas reviewed Jul 16, 2020

View reviewed changes

miri64 force-pushed the pytest/env/spec04 branch from 0ce8bdf to c673ec8 Compare July 16, 2020 14:32

04-single-hop-6lowpan-icmp: initial import of automated tests

666c551

Co-Authored-By: Alexandre Abadie <alexandre.abadie@inria.fr> Co-Authored-By: Martine S. Lenders <m.lenders@fu-berlin.de>

miri64 force-pushed the pytest/env/spec04 branch from c673ec8 to 666c551 Compare July 16, 2020 14:35

fjmolinas approved these changes Jul 16, 2020

View reviewed changes

fjmolinas merged commit 09a0af9 into RIOT-OS:master Jul 16, 2020

miri64 deleted the pytest/env/spec04 branch July 16, 2020 15:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

04-single-hop-6lowpan-icmp: initial import of automated tests #161

04-single-hop-6lowpan-icmp: initial import of automated tests #161

miri64 commented Jul 13, 2020

miri64 commented Jul 15, 2020

miri64 commented Jul 16, 2020

fjmolinas left a comment

fjmolinas commented Jul 16, 2020

fjmolinas Jul 16, 2020

fjmolinas Jul 16, 2020

miri64 Jul 16, 2020

fjmolinas Jul 16, 2020

miri64 Jul 16, 2020

miri64 commented Jul 16, 2020

miri64 commented Jul 16, 2020

miri64 commented Jul 16, 2020 •

edited

Loading

miri64 commented Jul 16, 2020 •

edited

Loading

fjmolinas commented Jul 16, 2020

miri64 commented Jul 16, 2020

fjmolinas commented Jul 16, 2020

miri64 commented Jul 16, 2020

fjmolinas left a comment

miri64 commented Jul 16, 2020

		assert pktbuf(pinger).is_empty()


		@pytest.mark.xfail(reason="Experimental task")

04-single-hop-6lowpan-icmp: initial import of automated tests #161

04-single-hop-6lowpan-icmp: initial import of automated tests #161

Conversation

miri64 commented Jul 13, 2020

miri64 commented Jul 15, 2020

miri64 commented Jul 16, 2020

fjmolinas left a comment

Choose a reason for hiding this comment

fjmolinas commented Jul 16, 2020

fjmolinas Jul 16, 2020

Choose a reason for hiding this comment

fjmolinas Jul 16, 2020

Choose a reason for hiding this comment

miri64 Jul 16, 2020

Choose a reason for hiding this comment

fjmolinas Jul 16, 2020

Choose a reason for hiding this comment

miri64 Jul 16, 2020

Choose a reason for hiding this comment

miri64 commented Jul 16, 2020

miri64 commented Jul 16, 2020

miri64 commented Jul 16, 2020 • edited Loading

miri64 commented Jul 16, 2020 • edited Loading

fjmolinas commented Jul 16, 2020

miri64 commented Jul 16, 2020

fjmolinas commented Jul 16, 2020

miri64 commented Jul 16, 2020

fjmolinas left a comment

Choose a reason for hiding this comment

miri64 commented Jul 16, 2020

miri64 commented Jul 16, 2020 •

edited

Loading

miri64 commented Jul 16, 2020 •

edited

Loading