Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce GitHub workflow runs (including Percy screenshot uploads) #2878

Merged
merged 15 commits into from
Sep 30, 2022

Conversation

colinrotherham
Copy link
Contributor

@colinrotherham colinrotherham commented Sep 22, 2022

Just adding some GitHub Action workflow tweaks here

Ideas so far

Cancel in-progress GitHub workflows when pushed again
Forgot a change? Pushing up a second time with cancel the 1st test run

Remove push event from GitHub Actions branch workflows
Prevents duplicate runs as pull_request event already fires for PR pushes

Skip tests for documentation changes
Don't need to run JavaScript tests when only docs/**/*.md files have changed (??)

☝️ Moved to branch main...sass-test-matrix

Move “All components” template tests out of Puppeteer environment
Don't lose time spinning up Chromium (headless) to run HTML validation checks etc

Only take “JavaScript disabled” screenshots for components with JS
Skip "JavaScript disabled" tests for components lacking *.mjs files as screenshots will be identical

Before

  1. Multiple workflow [pull_request, push] events for PRs
  2. Screenshots all components with JavaScript enabled
  3. Screenshots all components with JavaScript disabled

GitHub Workflow: 14 checks total (7 checks per run)
Screenshots (JavaScript enabled): 64 images (32 per run)
Screenshots (JavaScript disabled): 64 images (32 per run)

Time:        89.322 s, estimated 96 s

After

  1. Single workflow pull_request events for PRs
  2. Screenshots all components with JavaScript enabled
  3. Screenshots "JS capable" components with JavaScript disabled

GitHub Workflow: 7 checks total
Percy screenshots (JavaScript enabled): 32 images
Percy screenshots (JavaScript disabled): 11 images

Time:        58.233 s, estimated 96 s

@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 16:09 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 16:15 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 16:40 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 16:46 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 16:54 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 16:55 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 20:41 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 20:50 Inactive
@colinrotherham colinrotherham changed the title For discussion: Strategies to reduce Percy usage [WIP] For discussion: Strategies to reduce GitHub Action workflow runs (including Percy) [WIP] Sep 22, 2022
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 20:55 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 21:01 Inactive
@colinrotherham colinrotherham changed the title For discussion: Strategies to reduce GitHub Action workflow runs (including Percy) [WIP] For discussion: Strategies to reduce GitHub workflow runs (+ Percy) Sep 22, 2022
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 21:06 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 22, 2022 21:14 Inactive
@colinrotherham
Copy link
Contributor Author

@36degrees Morning! This commit for “JavaScript disabled” screenshots follows your idea in #2871

I've added a getFiles() directory listing helper to filter only components with ${component}.mjs, but we can run it before all the tests so once Puppeteer opens the Chromium window it stays rapid

@colinrotherham colinrotherham marked this pull request as ready for review September 23, 2022 07:26
@colinrotherham colinrotherham requested a review from a team as a code owner September 23, 2022 07:26
Prevents duplicate runs as `pull_request` event already fires for PR pushes
By removing `push` events we’ve lost the abilty to run tests without a PR being open, not any more
When tests were run with the Node.js or jsdom test environment, we were including helper code that tried to access the Puppeteer `page` object
Makes sure we’re also testing the default environment for all components
Previously the Sass rendering tests were combined with the screenshot generation `describe()` block
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 29, 2022 20:51 Inactive
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 29, 2022 21:20 Inactive
@colinrotherham
Copy link
Contributor Author

Morning @36degrees

I've gone through and split out all the commits, explaining what's happening and why

If you can take another look please 😊

Reloads the page (without JavaScript) for second capture
…test.js`

These tests don’t need a Chromium browser window (Puppeteer environment)

This change brings Nunjucks HTML render + validation together
These tests don’t need a Chromium browser window (Puppeteer environment)
We’ve configured Jest + Puppeteer with `browserPerWorker: true` to open multiple browsers and speed up our tests: https://github.com/smooth-code/jest-puppeteer#jest-puppeteerconfigjs

This stops Node.js warning us about potential memory leaks
…isting()`

Preparation for tests that need to filter components by certain files, for example “All components with `${component}.mjs`”
This change reduces our Percy screenshots down from 64 to 42 as we don’t need to capture identical screenshots for components that lack JavaScript functionality
@govuk-design-system-ci govuk-design-system-ci temporarily deployed to govuk-frontend-pr-2878 September 30, 2022 09:27 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

Successfully merging this pull request may close these issues.

Only run visual regression tests if there are code changes that might cause a visual change
5 participants