[Bug]: Testing.get_tap_test_class fails with "fixture 'stream' not found"

[qbatten]

Singer SDK Version

0.19.0

Python Version

3.10

Bug scope

Taps (catalog, state, stream maps, etc.)

Operating System

Mac OS

Description

I'm trying to use the new testing fixtures in 0.19.0. Using a minimal example that matches this section of the SDK testing docs. When I try to run it, I get several errors about missing a stream fixture.

---

To reproduce, you can clone MeltanoLabs/tap-postgres and add a new test file that contains the following.

Code to reproduce

import pendulum

from singer_sdk.testing import get_tap_test_class

from tap_postgres.tap import TapPostgres

SAMPLE_CONFIG = {
    "start_date": pendulum.datetime(2022,11,1).to_iso8601_string(),
    "sqlalchemy_url": "postgresql://postgres:postgres@localhost:5432/postgres",
}

TestTapPostgres = get_tap_test_class(
    tap_class=TapPostgres,
    config=SAMPLE_CONFIG,
)

---

Here's a full printout of the errors I'm getting.

Error

❯ poetry run pytest 
=============================================================== test session starts ================================================================
platform darwin -- Python 3.10.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/quinn/Documents/personal/code/meltano-all/tap-postgres
plugins: Faker-15.3.4
collected 10 items                                                                                                                                 

tap_postgres/tests/test_sdk_new.py ...EEEEEEE                                                                                                [100%]

====================================================================== ERRORS ======================================================================
_________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_catalog_schema_matches_record __________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 165
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:165
_________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_record_schema_matches_catalog __________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 165
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:165
_________________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_returns_record _________________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 165
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:165
__________________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_primary_keys __________________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 165
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:165
_____________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_attribute_is_datetime ______________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 199
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:199
______________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_attribute_is_integer ______________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 199
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:199
_______________________________________ ERROR at setup of TestTapPostgres.test_tap_stream_attribute_not_null _______________________________________
file /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py, line 199
      def run(  # type: ignore[override]
E       fixture 'stream' not found
>       available fixtures: _session_faker, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, config, doctest_namespace, faker, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, resource, runner, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/templates.py:199
================================================================= warnings summary =================================================================
../../../../../Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/suites.py:44
  /Users/quinn/Library/Caches/pypoetry/virtualenvs/tap-postgres-iDOMORjl-py3.10/lib/python3.10/site-packages/singer_sdk/testing/suites.py:44: PytestCollectionWarning: cannot collect test class 'TestSuite' because it has a __init__ constructor (from: tap_postgres/tests/test_sdk_new.py)
    @dataclass

tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_connections
  /Users/quinn/Documents/personal/code/meltano-all/tap-postgres/tap_postgres/client.py:138: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
    yield dict(row)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================= short test summary info ==============================================================
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_catalog_schema_matches_record
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_record_schema_matches_catalog
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_returns_record
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_primary_keys
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_attribute_is_datetime
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_attribute_is_integer
ERROR tap_postgres/tests/test_sdk_new.py::TestTapPostgres::test_tap_stream_attribute_not_null
===================================================== 3 passed, 2 warnings, 7 errors in 11.56s =====================================================
(base) 

---

Excluding stream and stream_attribute tests results in no errors. (Adding these kwargs to get_tap_test_class in the reproducible example above.)

    include_stream_tests=False,
    include_stream_attribute_tests=False

I think this is related to StreamTestTemplate, either instantiation or its run method being called.

Code

No response

[tayloramurphy]

@qbatten are you using Python 3.6? I don't believe we're officially supporting that version.

@edgarrmondragon I see you thumbsupped this - is this related to the python version or is there another potential solution?

[qbatten]

@tayloramurphy Ah! Sorry! I set the version wrong in the issue. I am on 3.10. Just fixed it in the description.

Also pushed up a commit that has the full repro https://github.com/qbatten/tap-postgres/tree/sdk-issue-1370 (It's as-described in the ticket but figured this way is more convenient for anyone investigating)

[tayloramurphy]

@qbatten thanks for updating!

cc @visch since you've been working on some of the testing stuff as well.

[edgarrmondragon]

@qbatten I think you need to add the following to a tests/conftest.py:

"""Test Configuration."""

pytest_plugins = ("singer_sdk.testing.pytest_plugin",)

see #1171 (review) and https://github.com/MeltanoLabs/tap-stackexchange/blob/a188ae690f7c1f1059c778928dd3c7ec7ae07285/tests/conftest.py

cc @tayloramurphy

[qbatten]

Aha! Yes, that solved this. I also had to update pytest and, along with that, move the tests folder out into the rootdir of the repo, from tap-postgres/tap-postgres/tests/ -> tap-postgres/tests/. Thank you!!