Merge pull request #48 from automaticp/parser-tests-with-pytest

Parser tests: replace shell scripts with pytest scripts

justfile

@@ -19,7 +19,7 @@ watch *ARGS:
 test:
     zig build test --summary failures {{flags}}
     just install
-    tests/run-all-tests.sh
+    pytest tests/
 
 test-file *ARGS:
     zig test {{flags}} "$@"

tests/README.md

@@ -1,7 +1,7 @@
 # `glsl_analyzer/tests`
 
+- [How do I run the tests?](#how-do-i-run-the-tests)
 - [LSP tests](#lsp-tests)
-    - [How do I run the tests?](#how-do-i-run-the-tests)
     - [How do I debug failing tests?](#how-do-i-debug-failing-tests)
     - [How do I add more files for testing / write new test cases?](#how-do-i-add-more-files-for-testing--write-new-test-cases)
         - [How tests and test input are structured](#how-tests-and-test-input-are-structured)
@@ -13,10 +13,8 @@
     - [Expecting failure](#expecting-failure)
 - [Parser tests](#parser-tests)
 
-## LSP tests
-
 
-### How do I run the tests?
+## How do I run the tests?
 
 **Requirements:** `python 3.10` and `requirements.txt`.
 
@@ -34,9 +32,18 @@ $ cd tests/ && pytest
 
 `pytest` will automatically pick up tests from modules that have names like `test_*.py` and contain functions `test_*` and classes `Test*`. More patterns are available, see the `pytest` docs.
 
+If you want to run individual test files, just specify them as arguments:
+
+```sh
+$ pytest tests/test_lsp.py
+```
+
 *Caution: do not run `pytest` with the `--runxfail` flag on these tests. We do not use `xfail` in its conventional form in our test suite; using this flag will make the tests fail in unpredictable ways.*
 
 
+## LSP tests
+
+
 ### How do I debug failing tests?
 
 The current test routines are set up to output the glsl file location that was targeted in the test:
@@ -259,17 +266,13 @@ We always expect failure in *strict mode*: if the test was expected to fail, but
 
 ## Parser tests
 
-Parser tests are simple bash scripts that run the parser with `glsl_analyzer --parse-file` on a set of files. Currently we only test successful parsing on well-formed code.
+Parser tests are simple pytest scripts that run the parser with `glsl_analyzer --parse-file` on a set of files. Currently we only test successful parsing on well-formed code and check that the `glsl_analyzer` did not output anything into `stderr`.
 
-*The `glsl_analyzer` version you want to test must be available in your PATH.*
-
-You can run the tests by executing
+Run only the parser tests by executing
 
 ```sh
-$ ./run-all-tests.sh
+$ pytest test_parser.py
 ```
 
-You can add more directories for testing by adding them to the `well_formed_dirs` in `run-parser-tests.sh`.
-
-*This setup will most likely be rewritten in the near future.*
+You can add more directories for testing by adding them to the  `dir_with_test_files` fixture `params` in `test_parser.py`.
 

tests/test_parser.py

@@ -0,0 +1,42 @@
+import subprocess
+from pathlib import Path
+
+import pytest
+import pytest_subtests
+
+
+base_directory = Path(__file__).parent.resolve()
+
+glsl_extensions = (
+    ".glsl",  ".vert",  ".frag",  ".geom",
+    ".comp",  ".tesc",  ".tese",  ".rgen",
+    ".rint",  ".rahit", ".rchit", ".rmiss",
+    ".rcall", ".mesh",  ".task"
+)
+
+
+@pytest.fixture(
+    params=("glsl-samples/well-formed/glslang/",)
+)
+def dir_with_test_files(request) -> Path:
+    return base_directory / Path(request.param)
+
+
+def test_parser_in_directory(
+    subtests: pytest_subtests.SubTests,
+    dir_with_test_files: Path
+):
+
+    for file in dir_with_test_files.iterdir():
+        if file.suffix not in glsl_extensions:
+            continue
+
+        with subtests.test(msg=str(file)):
+            output = subprocess.run(
+                args=("glsl_analyzer", "--parse-file", str(file)),
+                capture_output=True,
+                text=True
+            )
+
+            assert output.returncode == 0 and len(output.stderr) == 0, \
+                output.stderr