Test overhaul

[akx]

Describe what this pull request is trying to achieve.

This PR is trying to enable a slightly more standard method of running tests, using Py.test.

It also enables coverage reporting - one could enable e.g. codecov.io or another service for tracking coverage metrics. Right now you can look at the output and HTML coverage reports in the Check Artifacts.

Additional notes and description of your changes

• Instead of launch.py working double duty as a test runner, py.test is run separately against a regular webui server (running launch.py with --test-server configures it for tests, as it had done before with --tests ***).
• Since there was really no documentation for the testing, there still is none.

Environment this was tested in

• OS: Windows, macOS, GitHub Actions
• Browser: not really
• Graphics card: sure

[AUTOMATIC1111]

27 files changes

files added

files deleted

aaaaaa

[akx]

aaaaaa

Sorry, it's just renaming files to match the usual py.test pattern of test_*.py 🥲

[AUTOMATIC1111]

Will you update the wiki page with instructions for people to run tests on their own machines? https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Tests

[AUTOMATIC1111]

are you considering the question

[akx]

are you considering the question

Sorry, missed the notif. Sure, I'll update the wiki if this gets merged 👍

[illtellyoulater]

While I was experimenting with command line arguments I used "--server-test" and I got something like this:

Launching Web UI with arguments: --test-server --api --ckpt test/test_files/empty.pt --skip-torch-cuda-test --disable-nan-check
Loading weights [d030ad8db7] from test/test_files/empty.pt
Creating model from config: configs/v1-inference.yaml
Downloading pytorch_model.bin:   9%|████████▍                                                                                   | 157M/1.71G 
Downloading pytorch_model.bin:  30%|███████████████████████████▋                              | 514M/1.71G 
^C
Interrupted with signal 2

As you can see I stopped the execution with ctrl-c as I was not sure what was going on and why it was downloading additional model files, how many it would download - and most importantly - how these operations would affect my current webui install.

Can anyone tell me whether the above operations made permanent changes to my webui install?
If so, is there a way to revert them without having to clone the webui repo from scratch?

The only documentation I could find about --test-sever (https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Tests) says very little and doesn't answer to my questions... I later found the partly downloaded model in "/home/myusername/.cache/huggingface/hub/..." and I deleted it.. but other than that I have no idea what else was changed.

[lllyasviel]

Although it might be too late to against this change, but I still want to bring into attention that this change just break all extensions’ github ci tests. Please consider still having “--tests”.
But things are already too late ...

[akx]

@lllyasviel This happened a third of a year ago...

[catboxanon]

I think this is related: #11441