Solution for subtle rendering differences across browsers and operating systems

[archfz]

Problem

I have been using https://github.com/meinaart/cypress-plugin-snapshots for a good while now. When I started to use it I quickly realized that it had a big flaw: using it on locals images were generated with subtle differences which would always cause failures. These differences were due to the fact that initial images were generated on X OS and on Y browser. Now everyone who was using a different combination of OS or browser than what was used initially would bump into this issue. This was a deal breaker. From a quick look I do not see any solution for it in this plugin either.

Semi solution

I have managed to create a solution that works for us:

1. Do not run visual checks when in GUI mode.
2. Run visual checks only in RUN mode and only in a specific docker container, which is also shared with the CI pipeline. Thus keeping the same OS+browser always when screenshots are generated.
3. For local setup a simple shellscript that would slightly improve development with the visual tests: allow quickly selecting which test to run in RUN mode and to decide whether to accept or not the changes.

But this has certain inconveniences:

• Visual tests don't run in GUI mode. This means differences cannot quickly be seen and accepted in GUI. Devs need to manually run the separate command and to manually open files to check for differences.
• It breaks up the development cycle: you use GUI for general cypress assertion and another command for visual checks.

Theoretical solution

I've been thinking that maybe this whole process with docker could be automated and even connected with the GUI so that previously mentioned issues/inconveniences would be solved. The high level idea is something like this:

1. Optionally with configuration enable taking screenshots through docker, for consistent images in any host. This would atleast imply a docker image configured.
2. On visual assertion command spin up a container and generate the screenshot inside it (using cypress), instead of generating it in the host.

For this to work the main concerns are:

1. How can we efficiently move the current DOM from host cypress into docker cypress? As we don't want to rerun all previous steps from a test given we want it to happen as fast as possible. Here I've been thinking that maybe we could save the current DOM from under the iframe into a simple html file. We would then inject this into the container, and the cypress from within would be specially configured to open this file. The other thing that needs to be supported here is loading of any outside resource of the DOM, for which we could connect the docker container directly to host network.
2. How would we actually take the screenshot? In host cypress we could just register the parameters for the screenshot command and send it to the docker cypress to actually run it.
3. Even if all this works wouldn't it be too slow? Indeed I would expect that spinning up the container, booting cypress and then taking the screenshot could be 3-5 seconds, or more. Infact starting a container is not that slow, that is quite fast. The most time that is spent in addition is on booting cypress. We could maybe have a very bare-bone configuration on this instance (like not using bundlers / typescript). An additional idea is that maybe we can have only one container doing all the screenshots during the whole lifecycle of the host cypress, this way we'd only need to boot once and we could probably do it before any tests start. The thing to figure out here would be how we can run cypress in RUN mode in an infinite loop manner, so it can continually accept new DOMs and make new screenshots.
4. Would we need to share any cypress configs between host and docker instance? I am really unsure about this. Only thing that pops in my mind is maybe the default resolution.

Let me hear what you think, or whether you have different ideas. Also I am open to collaborate on the implementation for this.

[FRSgit]

Hey,
Sorry I needed to take a break from OSS and focus on my full-time job for last 3 months.
Thanks for this nice write up!

From my perspective there are two flows that can be used to achieve good visual testing workflow:

1. Always work with visual tests in cypress headless mode within docker container. That way screenshots should always be matching as you're running them in the same, virtual environment. But of course this method has some issues (mentioned by you earlier):
   1. User won't be able to interact with screenshots using GUI in current state of plugin. Though this one could be overcome by extracting the "diff window" and creating single web app which would allow you to go through the screenshots and update them.
   2. Running cypress in docker is not as efficient as running it in the host OS and makes it harder to work with other types of tests.
2. Have different images stored locally (for you own testing purposes) and in the repo (which is used by CI). This is the way I've went within my project: when I'm testing locally, screenshots are being stored within __local_images__ directories. These directories are also included in .gitignore and never committed to the repository. On the other hand there is a set of images which is a part of repo and which CI is comparing screenshots against. There images live in directory __image_snapshots__ and can be updated only by using [update screenshots] at the end of commit message within PR (this triggers CI workflow that generates new images and commits new to PR branch).
   1. This workflow is quicker to run locally.
   2. It's a bit slower when you need to update images as this happens only on CI. And requires PR owners to review which images has been updated.
   3. Users needs to run the image generation at the beginning of their work to generate the "working version" of baseline images.
3. Try to use cypress open within docker (it's possible):
   1. This probably will be the slowest way for local development.
   2. In cypress interactive mode you cannot run all tests at once (or, you can, but you might encounter some issues when doing so).
   3. There is still issue of having different images on the CI and locally.
4. Doing such "hybrid" as you've mentioned (with all of it's pros & cons that you've described).

I've also seen a variant of point 2 where visual tests were just skipped entirely on local runs and report issues only on the CI.

I think one or more of solutions described above could be implemented as a part of this plugin (separate libs in this repo). But my main concern is - which will be the best developer experience. And I'm wondering if we could get more community feedback on that.