Notes and FAQ

Notes

For accurate results, you should generate those images via CodeceptJS. For example, by using the built-in method I.saveScreenshot(), or the method I.takeScreenshot() provided by this helper.

The reason is, that every browser renders the image slightly different. Only when you generate both images (expected and actual image) using the same browser/settings, there will be no differences.

We've even seen differences between images taken by the same test setup by only changing the browser from default to headless mode.

FAQ

What's a base image, actual image, diff image, variation image, ...?

There are three basic image types when we're talking about image comparison tests:

1. Base Images
   • Also called "baseline image" or "expected image".
   • This is a screenshot of how the website should look like.
   • Usually, you create base images once, from a stable state and commit those base images to your git repository.
2. Actual Images
   • Also called "screenshots".
   • Those are the images of how the website actually looks like.
   • Those actual images are always generated fresh for every test, and are compared against the static base images.
   • Actual images should not be added to your git repository.
3. Diff Image
   • Those images document the difference between a base image and an actual image.
   • They are a means of telling you what is wrong (or unexpected) on your website. Usually, diff images are only generated when a match fails.
   • Diff images should not be added to your git repository.

In other words: Base images and actual images provide the input for the comparison, and the diff image is the result of that comparison.

We also invented the term "variation image":

4. Variation Image
   • Is a special type of base image that is relevant for tests that use multiple base images

What's the difference between tolerance and threshold?

• tolerance defines the number of pixels that are allowed to be different between both images. When the relative count of different pixels is below the tolerance, the images are considered equal. Tolerance simply counts every different pixel, regardless of how different that pixel is.
• threshold is used by the pixelmatch library to determine, which pixels are actually different. By raising the threshold, you will get a lower count of different pixels. Threshold inspects the color-difference between two pixels to determine if they are different.

If tests are too sensitive, you can either increase tolerance (to allow a higher number of different pixels), or increase the threshold (to reduce the number of different pixels).

---

← Multiple base images | Contribute →