Tests failed when using Python 3.12.1

[tklam]

Using Python 3.12.1 leads to many failed test cases:

poetry shell
poetry env use python3.12
poetry install --with=dev
poetry run pytest 

Then the test results (attached)
magia.test.19jan2024.txt
:

============================= test session starts ==============================
platform linux -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/tklam/development/magia-hdl/magia
configfile: pyproject.toml
plugins: cocotb-test-0.2.4, xdist-3.5.0, github-report-0.0.1
collected 186 items
 
tests/test_external_mod.py .................                             [  9%] 
tests/test_memory.py FFFF                                                [ 11%] 
tests/test_module_instance.py .....                                      [ 13%]
tests/test_smoke.py ...                                                  [ 15%]
tests/bundle/test_bundle.py FF                                           [ 16%]
tests/core/test_const.py F........                                       [ 21%]
tests/core/test_op.py FF........FFFFFFFFFFFFF                            [ 33%]
tests/core/test_reg.py FFFFFFFFF                                         [ 38%]
tests/core/test_signal.py .F.                                            [ 40%]
tests/std/test_fixed_point.py .......................................... [ 62%] 
....................................................                     [ 90%] 
tests/std/test_stdio.py F                                                [ 91%] 
tests/std/encoding/test_onehot.py FFFFF.FFFFFFFFFF                       [100%] 

[khwong-c]

TL;DR

It's a problem of Verilator. It used the System Default /usr/bin/python3 as it's python interpreter internally.

Investigation

We can look for the Makefile of the simulator, from the official release of Verilator (no matter by apt get or in the docker image)
In verilator/include/verilated.mk, it states the Python interpreter is fixed to /usr/bin/python3

This matched the observation from the full log. When we run the cocotb simulator, the following log comes out:

INFO     cocotb:simulator.py:302 /usr/bin/python3 /usr/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include Vtop.cpp Vtop___024root__DepSet_h84412442__0.cpp Vtop___024root__DepSet_heccd7ead__0.cpp Vtop__Dpi.cpp Vtop___024root__Slow.cpp Vtop___024root__DepSet_h84412442__0__Slow.cpp Vtop___024root__DepSet_heccd7ead__0__Slow.cpp Vtop__Syms.cpp > Vtop__ALL.cpp
ERROR    cocotb:simulator.py:302 Traceback (most recent call last):
ERROR    cocotb:simulator.py:302   File "/usr/share/verilator/bin/verilator_includer", line 12, in <module>
ERROR    cocotb:simulator.py:302     import re
ERROR    cocotb:simulator.py:302   File "/usr/lib/python3.12/re/__init__.py", line 125, in <module>
ERROR    cocotb:simulator.py:302     from . import _compiler, _parser
ERROR    cocotb:simulator.py:302   File "/usr/lib/python3.12/re/_compiler.py", line 18, in <module>
ERROR    cocotb:simulator.py:302     assert _sre.MAGIC == MAGIC, "SRE module mismatch"
ERROR    cocotb:simulator.py:302            ^^^^^^^^^^^^^^^^^^^
ERROR    cocotb:simulator.py:302 AssertionError: SRE module mismatch
ERROR    cocotb:simulator.py:302 make: *** [/usr/share/verilator/include/verilated.mk:212: Vtop__ALL.cpp] Error 1
INFO     cocotb:simulator.py:302 make: Leaving directory '/tmp/tmpo76rs7c4'

The error is caused by mismatched between library between the one imported from the VirtualEnv and the Systemwide one.

It is quite hard to fix. Need more effort on this one.

[khwong-c]

It is mentioned an override on PYTHON3 In Verilator's Document.
However, it is not respected in runtime, on 5.020 the latest version.

It can be changed only when compiling verilator itself.
This will add an extra 5min to our CI, and this is expensive.
Furthermore, this doesn't fix our situation when individual developer try to integrate their dev environment with verilator.

I couldn't think of a decent way to fix that.
For the CI, the quick and dirty way is "Relink /usr/bin/python3 with the python in the virtual env"

Yup. It is really dirty.

https://verilator.org/guide/latest/environment.html

[khwong-c]

Fixed the Issue on CI with #16
But we need to handle this later on when we integrate magia-hdl with the simulation workflow.

@tklam proposed to raise an issue/PR on verilator's repo.