Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Implement end-to-end testing. #92

Merged
merged 3 commits into from
Jun 16, 2023

Conversation

ehmatthes
Copy link
Contributor

As discussed in #55, implements a small set of end-to-end tests.

  • To run tests, you'll need to run pip install pytest if you don't already have it available.
  • I've run the tests on macOS and Windows. I believe they should run on most Linux systems, but have not checked that yet.
  • The image files that are generated on Windows are significantly different (pixel-wise) than what's generated on macOS. This looks like a font-rendering difference. To address this, there are Windows-specific reference files in the test resources.
  • I don't know if this is a Windows-specific issue, or something particular to my Windows setup. We may end up needing to have a small variety of reference images, and only fail a test if none of the reference images match.
  • Includes brief documentation about running tests, and adding new tests.
  • You can make tests fail by adding a call to run_git_reset(tmp_repo) in any of the test functions. This is helpful for seeing what test failures look like, and to make sure the image comparison function can actually fail appropriately.
  • The test functions are written so that every test function contains the raw command that we would expect users to run, such as git-sim log. A helper function then inserts the global options needed to make the tests work.

Eric Matthes added 3 commits June 13, 2023 20:48
Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
Signed-off-by: Eric Matthes <eric@Erics-Mac-Studio.local>
@ehmatthes
Copy link
Contributor Author

11 files changed is better than 92. :)

I didn't see anything to change in the documentation; we still end up with a temp folder called sample_repo, which pytest renames to sample_repo0.

If you end up merging this, I think you should add a couple tests for commands you'd like to see covered, and ask any questions you have about working with the test suite. I don't have any ambitions to add a bunch of tests; my goal was to help get a test suite off the ground. If you want help with testing I'm happy to do more, but I think the bigger priority is that you're comfortable working with what's started here.

I would note that while using git-dummy is perfectly reasonable and understandable, I don't think it's as fast as copying a static repo. I think it might be reasonable to have a git-dummy call per test module, but I don't think you'd want a separate call per test function. I think the git-dummy call took about 2 seconds on my system, but I didn't benchmark that at all. I just noticed it went from ~6.3s for the test run with the static repo to ~8.3s using git-dummy. That would add up fast. You can start to see the impact by changing the scope of the tmp_repo() fixture from session to function. That runs the fixture once per test function rather than once for the whole session. For me, that's 8.3s -> 10.0s.

It's all good, you'll figure out how much test coverage is helpful to have without slowing down your development process. :)

@initialcommit-io initialcommit-io merged commit 0a9b2d3 into initialcommit-com:dev Jun 16, 2023
@ehmatthes ehmatthes mentioned this pull request Jun 16, 2023
8 tasks
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants