Baseline Image is modified during test run

[mbehzad]

Describe the bug
Hi,
I upgraded from 3.1.7 to 3.2.3 and now when I run the tests (with pluginVisualRegressionUpdateImages: false) the baseline images (something-something#0.png) is also modified. not the content but metadata (last-modified).
This doesn't happen when I switch back to 3.1.7

To Reproduce
Steps to reproduce the behavior:

If this can't be reproduced by you, then let me know and i'll try adding a minimal reproducible repo for this.

Expected behavior

Baseline screenshots are not touched unless updateImages flag is set.

Please complete the following information:

• OS and version: Windows 10
• Browser and version: Chrome 107 (headless) | Electron 106 (headless)
• Cypress version 10.11.0 | 11.2.0

[FRSgit]

Hey!

To avoid unexpected behaviour after migration, this plugin always regenerates baseline images whenever its' version has been changes (so, every time you upgrade it in your repository). After single regeneration right after version update everything should work as expected - create a new issue please if that's not the case for you.

And regarding your original issue - do you have any ideas how we could improve this situation?
I'm thinking about writing documentation of this feature, do you think it will be enough? What would your like to see in a description of this feature?
And ofc - PR for it is more than welcome 🙏

[mbehzad]

Hi Jakub, thank you very much for your response.
I can confirm, the baseline image changes happen only once after the dependency update.
Sure I think in the READMEs FAQ section if this behavior is mentioned then it would help people facing this issue. And I'm also curious why this is necessary. I naively would assume that the screenshot is taken by cy/browser and the options passed by the user therefor the baseline image wouldn't change if the plugin is upgraded.
(should i close this issue?)

[wkillerud]

For what it's worth, this surprised me and my colleague as well. We also assumed the baseline images would be kept as-is between upgrades.

Out of curiosity, has there been any unexpected behavior after upgrades before?

Personally, I would be fine with a situation where an upgrade of this plugin fails on CI for some or all snapshots. Deleting failing baselines is already part of our workflow for when visual changes are intended.

We have a few hundred baseline images in our test suite. Depending on the update frequency we could end up with a larger-than-necessary .git folder.

That said, we're also fine with keeping the behavior as-is. But maybe add an entry to the FAQ 😄

[adamdehaven]

I agree -- updating all images on a version change is unexpected, especially when it's just a version patch.

[FRSgit]

Thanks for your feedback. Yes, there were some issues in the past when we did changes to default image comparison configuration.
I think we can make this situation a bit better and provide separate versioning for when images need to be updated. I'll create a PR for the plugin soon

[FRSgit]

I've just created a PR regarding this behaviour, feel free to review it: #221

[github-actions]

🎉 This issue has been resolved in version 3.3.0 🎉

The release is available on:

• GitHub release
• npm package (@latest dist-tag)

Your semantic-release bot 📦🚀