Only run visual regression tests if there are code changes that might cause a visual change

[36degrees]

What

Only run Percy's visual regression tests if there are code changes that might cause a visual change.

As a (non-exhaustive) example, do not run visual regression tests if the only files changed in a PR are tests, docs, internal tooling, or in the dist and package folder.

Why

We have a limited number of screenshots available to us in any given billing period. Only running the visual regression tests when they are useful helps us to burning through our included screenshots too quickly, allowing us to remain on a cheaper plan.

Who needs to work on this

Developers

Who needs to review this

Developers

Done when

• Percy's visual regression tests are only run if there are code changes that might cause a visual change

[36degrees]

We could also only do the split JS/non-JS variants for components that have JavaScript.

[colinrotherham]

Some suggestions from me:

1. Adding Jest --changedSince flag (via commit hash)
   Percy has PERCY_PARTIAL_BUILD and we could skip components that were already tested

2. Adding GitHub workflow path filters
   So only component Nunjucks/JavaScript (or their tests) trigger Percy

[colinrotherham]

We're also running the Tests workflow twice for every push to a pull request

.github/workflows/tests.yml
Triggers on pushes twice if a pull request is also open

on: [push, pull_request]

[36degrees]

I think this closed by #2878.

[colinrotherham]

Going to reopen this one as we've still running Percy for code changes that don't affect component UI

[colinrotherham]

We're now running git diff on our npm package:

• Add PR diff comments for npm package changes #4511

We could use this information (e.g. no diff, nothing has changed) to prevent Percy from running

[colinrotherham]

Some time later, but here's an example of how to use git diff to conditionally skip tests:

• Skip visual regression tests without a diff nhsuk/nhsuk-frontend#1164

      - name: Run backstop
        run: npm run test:visual
+       if: ${{ steps.changes.outputs.changes == 'true' }}