Add option to launch each test in separate process

[virtuald]

Either specify directly on command line, or add this to pyproject.toml:

[tool.robotpy.pyfrc]
isolated = true

If it works out, expect to make this default in 2026. Replaces #236

Currently, this almost works, but I haven't tried it on Windows yet.

• The examples/Timed project works
• The examples/Physics project does not work yet (any project that has a embedded 'tests' directory). Need to get rid of directory changing as auscompgeek suggested.
• This will work if the project has no test directory and the user runs robotpy tests --builtin

[virtuald]

This works in all the cases I tried on Linux.

[virtuald]

Potentially blocked on robotpy/mostrobotpy#146

[auscompgeek]

I tried this on my team's robot code today. Unit tests run fine, but the pyfrc tests mysteriously fail with no further information.

❱ .venv/bin/robotpy test --isolated -- --maxfail=0
12:22:55:462 INFO    : faulthandler        : registered SIGUSR2 for PID 153549
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
platform linux -- Python 3.13.1, pytest-8.3.3, pluggy-1.5.0 -- /home/davo/dev/frc/thedropbears/pyreefscape/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/davo/dev/frc/thedropbears/pyreefscape/.hypothesis/examples'))
rootdir: /home/davo/dev/frc/thedropbears/pyreefscape
configfile: pyproject.toml
testpaths: tests
plugins: integration-0.2.3, reraise-2.1.2, hypothesis-6.119.3, typeguard-4.3.0
collected 25 items                                                             

tests/pyfrc_test.py::test_disabled FAILED                                [  4%]
tests/pyfrc_test.py::test_operator_control FAILED                        [  8%]
tests/test_caching.py::test_cache_per_loop PASSED                        [ 12%]
tests/test_constrain_angle.py::test_happy PASSED                         [ 16%]
tests/test_constrain_angle.py::test_all PASSED                           [ 20%]
tests/test_constrain_angle.py::test_zero PASSED                          [ 24%]
tests/test_constrain_angle.py::test_edge_pos PASSED                      [ 28%]
tests/test_constrain_angle.py::test_edge_neg PASSED                      [ 32%]
tests/test_constrain_angle.py::test_revolution_pos PASSED                [ 36%]
tests/test_constrain_angle.py::test_revolution_neg PASSED                [ 40%]
tests/test_constrain_angle.py::test_one_wrap_positive_half PASSED        [ 44%]
tests/test_constrain_angle.py::test_one_wrap_negative_half PASSED        [ 48%]
tests/test_functions.py::test_rate_limit2d PASSED                        [ 52%]
tests/test_functions.py::test_rate_limit_2d_limit PASSED                 [ 56%]
tests/test_functions.py::test_clamp2d_noconstrain PASSED                 [ 60%]
tests/test_functions.py::test_clamp2d_constrain PASSED                   [ 64%]
tests/test_scalers.py::test_deadzone PASSED                              [ 68%]
tests/test_scalers.py::test_deadzone_zero_threshold PASSED               [ 72%]
tests/test_scalers.py::test_exponential PASSED                           [ 76%]
tests/test_scalers.py::test_scale_value PASSED                           [ 80%]
tests/fuzz_test.py::test_fuzz[Blue2] SKIPPED (Integration tests skipped) [ 84%]
tests/fuzz_test.py::test_fuzz[Red1] SKIPPED (Integration tests skipped)  [ 88%]
tests/fuzz_test.py::test_fuzz_test SKIPPED (Integration tests skipped)   [ 92%]
tests/autonomous_test.py::test_all_autonomous[Red] SKIPPED (Slow int...) [ 96%]
tests/autonomous_test.py::test_all_autonomous[Blue] SKIPPED (Slow in...) [100%]

=================================== FAILURES ===================================
________________________________ test_disabled _________________________________
Test failed in subprocess: tests/pyfrc_test.py::test_disabled
----------------------------- Captured stdout call -----------------------------
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
collecting ... collected 1 item

tests/pyfrc_test.py::test_disabled FAILED                                [100%]

============================== 1 failed in 0.12s ===============================
----------------------------- Captured stderr call -----------------------------
12:22:56:843 INFO    : pyfrc.physics       : Physics support successfully enabled
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)
	
[phoenix] Library shutdown cleanly
	
____________________________ test_operator_control _____________________________
Test failed in subprocess: tests/pyfrc_test.py::test_operator_control
----------------------------- Captured stdout call -----------------------------
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
collecting ... collected 1 item

tests/pyfrc_test.py::test_operator_control FAILED                        [100%]

============================== 1 failed in 0.14s ===============================
----------------------------- Captured stderr call -----------------------------
12:22:58:254 INFO    : pyfrc.physics       : Physics support successfully enabled
[phoenix] Library initialization is complete.
	
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)
	
[phoenix] Library shutdown cleanly
	
=========================== short test summary info ============================
FAILED tests/pyfrc_test.py::test_disabled - Failed: Test failed in subprocess: tests/pyfrc_test.py::test_disabled
FAILED tests/pyfrc_test.py::test_operator_control - Failed: Test failed in subprocess: tests/pyfrc_test.py::test_operator_control
=================== 2 failed, 18 passed, 5 skipped in 29.43s ===================
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)
	
[phoenix] Library shutdown cleanly
	

[virtuald]

The --no-summary flag caused the error information to not be displayed. I removed it, now it shows an actionable error. @auscompgeek can you take a look and see if this is good for you now?

[auscompgeek]

Got the failure details now:

❱ .venv/bin/robotpy test --isolated -- -x
00:07:58:188 INFO    : faulthandler        : registered SIGUSR2 for PID 38788
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
======================================= test session starts =======================================
platform darwin -- Python 3.13.1, pytest-8.3.3, pluggy-1.5.0 -- /Users/davidv/dev/frc/thedropbears/pyreefscape/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/Users/davidv/dev/frc/thedropbears/pyreefscape/.hypothesis/examples'))
rootdir: /Users/davidv/dev/frc/thedropbears/pyreefscape
configfile: pyproject.toml
testpaths: tests
plugins: integration-0.2.3, hypothesis-6.119.3, reraise-2.1.2, typeguard-4.3.0
collected 26 items

tests/pyfrc_test.py::test_disabled FAILED                                                   [  3%]

============================================ FAILURES =============================================
__________________________________________ test_disabled __________________________________________
Test failed in subprocess: tests/pyfrc_test.py::test_disabled (exit code 1)
-------------------------------------- Captured stdout call ---------------------------------------
[phoenix] CANbus Connected: sim
[phoenix] CANbus Network Up: sim
============================= test session starts ==============================
collecting ... collected 1 item

tests/pyfrc_test.py::test_disabled FAILED                                [100%]

=================================== FAILURES ===================================
________________________________ test_disabled _________________________________

self = <pyfrc.test_support.controller.TestController object at 0x11aac30e0>
robot = <weakproxy at 0x11ab1d260; to 'robot.MyRobot' at 0x11aace270>

    def _robot_thread(self, robot):
        with self._cond:
            self._robot_started = True
            self._cond.notify_all()

        with self._reraise(catch=True):
            assert robot is not None  # shouldn't happen...

            robot._TestRobot__robotInitialized = self._on_robot_initialized

            try:
>               robot.startCompetition()

../../robotpy/pyfrc/pyfrc/test_support/controller.py:42:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.13/site-packages/magicbot/magicrobot.py:365: in startCompetition
    self.robotInit()
../../robotpy/pyfrc/pyfrc/test_support/pytest_plugin.py:53: in robotInit
    super().robotInit()
.venv/lib/python3.13/site-packages/magicbot/magicrobot.py:111: in robotInit
    self._automodes = AutonomousModeSelector("autonomous")
.venv/lib/python3.13/site-packages/robotpy_ext/autonomous/selector.py:145: in __init__
    instance = obj(*args, **kwargs)
autonomous/coral_auto.py:8: in __init__
    super().__init__("Coral")
autonomous/coral_auto_base.py:28: in __init__
    self.trajectory = choreo.load_swerve_trajectory(trajectory_name)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

trajectory_name = 'Coral'

    def load_swerve_trajectory(trajectory_name: str) -> SwerveTrajectory:
        """Load a swerve trajectory from a file.

        Parameter ``trajectory_name``:
            The path name in Choreo, which matches the file name in the deploy
            directory. Do not include ".traj" here.
        """
>       with open(
            os.path.join(getDeployDirectory(), "choreo", trajectory_name + ".traj"),
            "r",
            encoding="utf-8",
        ) as trajectory_file:
E       FileNotFoundError: [Errno 2] No such file or directory: '/Users/davidv/dev/frc/thedropbears/pyreefscape/.venv/bin/deploy/choreo/Coral.traj'

.venv/lib/python3.13/site-packages/choreo/__init__.py:139: FileNotFoundError

During handling of the above exception, another exception occurred:

item = <Function test_disabled>

    @pytest.hookimpl(hookwrapper=True, tryfirst=True)
    def pytest_runtest_call(item: Item):
        result = yield
        if hasattr(item, "funcargs") and "reraise" in item.funcargs:
            reraise = item.funcargs["reraise"]

            # Override any non-re-raised exception in the main thread by calling `reraise()`
            if result.excinfo is None or not hasattr(
                result.excinfo[1], "_is_from_pytest_reraise"
            ):
>               reraise()

.venv/lib/python3.13/site-packages/pytest_reraise/reraise.py:136:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.13/site-packages/pytest_reraise/reraise.py:72: in __call__
    raise e
../../robotpy/pyfrc/pyfrc/test_support/controller.py:46: in _robot_thread
    robot.endCompetition()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <robot.MyRobot object at 0x11aace270>

    def endCompetition(self) -> None:
        self.__done = True
>       self._automodes.endCompetition()
E       AttributeError: 'MyRobot' object has no attribute '_automodes'

.venv/lib/python3.13/site-packages/magicbot/magicrobot.py:384: AttributeError
---------------------------- Captured stdout setup -----------------------------
Not loading CameraServerShared
----------------------------- Captured stdout call -----------------------------
DataLog: Logging to '/Users/davidv/dev/frc/thedropbears/pyreefscape/logs/FRC_TBD_fd0c1ee96732ae68.wpilog' (12.9 GiB free space)
----------------------------- Captured stderr call -----------------------------
00:07:58:862 INFO    : autonomous          : Begin initializing autonomous mode switcher
------------------------------ Captured log call -------------------------------
INFO     autonomous:selector.py:86 Begin initializing autonomous mode switcher
=========================== short test summary info ============================
FAILED tests/pyfrc_test.py::test_disabled - AttributeError: 'MyRobot' object ...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================== 1 failed in 0.11s ===============================
-------------------------------------- Captured stderr call ---------------------------------------
00:07:58:795 INFO    : pyfrc.physics       : Physics support successfully enabled
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)

[phoenix] Library shutdown cleanly

===================================== short test summary info =====================================
FAILED tests/pyfrc_test.py::test_disabled - Failed: Test failed in subprocess: tests/pyfrc_test.py::test_disabled (exit code 1)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================== 1 failed in 0.77s ========================================
[phoenix-diagnostics] Server shutdown cleanly. (dur:0)

[phoenix] Library shutdown cleanly