Implement end-to-end testing (#92)

* Implements end-to-end test suite using pytest, git-dummy, and adds testing docs

Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>

---------

Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
Co-authored-by: Eric Matthes <eric@Erics-Mac-Studio.local>

Author: ehmatthes

.gitignore

@@ -6,3 +6,5 @@ git-sim_media/
 build/
 dist/
 git_sim.egg-info/
+
+.venv/

tests/README.md

@@ -0,0 +1,73 @@
+# Testing
+---
+
+Testing is done with pytest. The focus for now is on end-to-end tests, which show that the overall project is working as it should.
+
+## Running tests
+
+The following instructions will let you run tests as soon as you clone the repository:
+
+```sh
+$ git clone https://github.com/initialcommit-com/git-sim.git
+$ cd git-sim
+$ python3 -m venv .venv
+$ source venv/bin/activate
+(.venv)$ pip install -e .
+(.venv)$ pip install pytest
+(.venv)$ pytest -s
+```
+
+Including the `-s` flag tells pytest to include diagnostic information in the test output. This will show you where the test data is being written:
+
+```sh
+(.venv)$ pytest -s
+===== test session starts ==========================================
+platform darwin -- Python 3.11.2, pytest-7.3.2, pluggy-1.0.0
+rootdir: /Users/.../git-sim
+collected 3 items
+
+tests/e2e_tests/test_core_commands.py 
+
+Temp repo directory:
+  /private/var/folders/.../pytest-108/sample_repo0
+
+...
+
+===== 3 passed in 6.58s ============================================
+```
+
+## Helpful pytest notes
+
+- `pytest -x`: Stop after the first test fails
+
+## Adding more tests
+
+To add another test:
+
+- Work in `tests/e2e_tests/test_core_commands.py`.
+- Duplicate one of the existing test functions.
+- Replace the value of `raw_cmd` with the command you want to test.
+- Run the test suite once with `pytest -sx`. The test should fail, but it will generate the output you need to finish the process.
+- Look in the "Temp repo directory" specified at the start of the test output.
+    - Find the `git-sim_media/` directory there, and find the output file that was generated for the test you just wrote.
+    - Open that file, and make sure it's correct.
+    - If it is, copy that file into `tests/e2e_tests/reference_files/`, with an appropriate name.
+    - Update your new test function so that `fp_reference` points to this new reference file.
+- Run the test suite again, and your test should pass.
+- You will need to repeat this process once on macOS or Linux, and once on Windows.
+
+## Cross-platform issues
+
+There are two cross-platform issues to be aware of.
+
+### Inconsistent png and jpg output
+
+When git-sim generates a jpg or png file, that file can be slightly different on different systems. Files can be slightly different depending on the architecture, and which system libraries are installed. Even Intel and Apple-silicon Macs can end up generating non-identical image files.
+
+These issues are mostly addressed by checking that image files are similar within a given threshold, rather than identical.
+
+### Inconsistent Windows and macOS output
+
+The differences across OSes is even greater. I believe this may have something to do with which fonts are available on each system.
+
+This is dealt with by having Windows-specific reference files.

tests/e2e_tests/conftest.py

@@ -0,0 +1,31 @@
+import subprocess, os
+from pathlib import Path
+from shlex import split
+
+import pytest
+
+import utils
+
+
+@pytest.fixture(scope="session")
+def tmp_repo(tmp_path_factory):
+    """Create a copy of the sample repo, which we can run all tests against.
+
+    Returns: path to tmp dir containing sample test repository.
+    """
+
+    tmp_repo_dir = tmp_path_factory.mktemp("sample_repo")
+
+    # To see where tmp_repo_dir is located, run pytest with the `-s` flag.
+    print(f"\n\nTemp repo directory:\n  {tmp_repo_dir}\n")
+
+    # Create the sample repo for testing.
+    os.chdir(tmp_repo_dir)
+
+    # When defining cmd, as_posix() is required for Windows compatibility.
+    git_dummy_path = utils.get_venv_path() / "git-dummy"
+    cmd = f"{git_dummy_path.as_posix()} --commits=10 --branches=4 --merge=1 --constant-sha --name=sample_repo --diverge-at=2"
+    cmd_parts = split(cmd)
+    subprocess.run(cmd_parts)
+
+    return tmp_repo_dir / "sample_repo"

tests/e2e_tests/reference_files/git-sim-log.png

@@ -0,0 +1,54 @@
+"""Tests for the core commands implemented in git-sim.
+
+All test runs use the -d flag to prevent images from opening automatically.
+
+To induce failure, include a call to `run_git_reset()` in one of the
+  test functions.
+"""
+
+import os, subprocess
+from pathlib import Path
+
+from utils import get_cmd_parts, compare_images, run_git_reset
+
+
+def test_log(tmp_repo):
+    """Test a simple `git-sim log` command."""
+    raw_cmd = "git-sim log"
+    cmd_parts = get_cmd_parts(raw_cmd)
+
+    os.chdir(tmp_repo)
+    output = subprocess.run(cmd_parts, capture_output=True)
+
+    fp_generated = Path(output.stdout.decode().strip())
+    fp_reference = Path(__file__).parent / "reference_files/git-sim-log.png"
+
+    assert compare_images(fp_generated, fp_reference)
+
+
+def test_status(tmp_repo):
+    """Test a simple `git-sim status` command."""
+    raw_cmd = "git-sim status"
+    cmd_parts = get_cmd_parts(raw_cmd)
+
+    os.chdir(tmp_repo)
+    output = subprocess.run(cmd_parts, capture_output=True)
+
+    fp_generated = Path(output.stdout.decode().strip())
+    fp_reference = Path(__file__).parent / "reference_files/git-sim-status.png"
+
+    assert compare_images(fp_generated, fp_reference)
+
+
+def test_merge(tmp_repo):
+    """Test a simple `git-sim merge` command."""
+    raw_cmd = "git-sim merge branch2"
+    cmd_parts = get_cmd_parts(raw_cmd)
+
+    os.chdir(tmp_repo)
+    output = subprocess.run(cmd_parts, capture_output=True)
+
+    fp_generated = Path(output.stdout.decode().strip())
+    fp_reference = Path(__file__).parent / "reference_files/git-sim-merge.png"
+
+    assert compare_images(fp_generated, fp_reference)

tests/e2e_tests/reference_files/git-sim-log_windows.png

@@ -0,0 +1,107 @@
+import os, subprocess
+from pathlib import Path
+from shlex import split
+
+import numpy as np
+
+from PIL import Image, ImageChops
+
+
+def compare_images(path_gen, path_ref):
+    """Compare a generated image against a reference image.
+
+    This is a simple pixel-by-pixel comparison, with a threshold for
+      an allowable difference.
+
+    Parameters: file path to generated and reference image files
+    Returns: True/ False
+    """
+    if os.name == "nt":
+        # Use Windows-specific reference files.
+        path_ref = path_ref.with_name(path_ref.stem + "_windows" + path_ref.suffix)
+
+    img_gen = Image.open(path_gen)
+    img_ref = Image.open(path_ref)
+
+    img_diff = ImageChops.difference(img_gen, img_ref)
+
+    # We're only concerned with pixels that differ by a total of 20 or more
+    #   over all RGB values.
+    # Convert the image data to a NumPy array for processing.
+    data_diff = np.array(img_diff)
+
+    # Calculate the sum along the color axis (axis 2) and then check
+    # if the sum is greater than or equal to 20. This will return a 2D
+    # boolean array where True represents pixels that differ significantly.
+    pixels_diff = np.sum(data_diff, axis=2) >= 20
+
+    # Calculate the ratio of pixels that differ significantly.
+    ratio_diff = np.mean(pixels_diff)
+
+    # Images are similar if only a small % of pixels differ significantly.
+    #   This value can be increased if tests are failing when they shouldn't.
+    #   It can be decreased if tests are passing when they shouldn't.
+    if ratio_diff < 0.0075:
+        return True
+    else:
+        print("bad pixel ratio:", ratio_diff)
+        return False
+
+
+def get_cmd_parts(raw_command):
+    """
+    Convert a raw git-sim command to the full version we need to use
+      when testing, then split the full command into parts for use in
+      subprocess.run(). This allows test functions to explicitly state
+      the actual command that users would run.
+
+    For example, the command:
+        `git-sim log`
+    becomes:
+        `</path/to/git-sim> -d --output-only-path --img-format=png log`
+
+    This prevents images from auto-opening, simplifies parsing output to
+      identify the images we need to check, and prefers png for test runs.
+
+    Returns: list of command parts, ready to be run with subprocess.run()
+    """
+    # Add the global flags needed for testing.
+    cmd = raw_command.replace(
+        "git-sim", "git-sim -d --output-only-path --img-format=png"
+    )
+
+    # Replace `git-sim` with the full path to the binary.
+    #  as_posix() is needed for Windows compatibility.
+    git_sim_path = get_venv_path() / "git-sim"
+    cmd = cmd.replace("git-sim", git_sim_path.as_posix())
+
+    return split(cmd)
+
+
+def run_git_reset(tmp_repo):
+    """Run `git reset`, in order to induce a failure.
+
+    This is particularly useful when testing the image comparison algorithm.
+    - Running `git reset` makes many of the generated images different.
+    - For example, `git-sim log` then generates a valid image, but it doesn't
+      match the reference image.
+
+    Note: tmp_repo is a required argument, to make sure this command is not
+      accidentally called in a different directory.
+    """
+    cmd = "git reset --hard 60bce95465a890960adcacdcd7fa726d6fad4cf3"
+    cmd_parts = split(cmd)
+
+    os.chdir(tmp_repo)
+    subprocess.run(cmd_parts)
+
+
+def get_venv_path():
+    """Get the path to the active virtual environment.
+
+    We actually need the bin/ or Scripts/ dir, not just the path to venv/.
+    """
+    if os.name == "nt":
+        return Path(os.environ.get("VIRTUAL_ENV")) / "Scripts"
+    else:
+        return Path(os.environ.get("VIRTUAL_ENV")) / "bin"