Handle GitHub CI secrets in Nova jobs

Nova Linux, MacOS, and Windows jobs are the basic building blocks for many CI jobs in PyTorch. They provide a common interface to hide the complexity of setting up the infrastructure from the repo owners so that they can focus more on the actual CI logic. Nevertheless, one of the remaining rough edges in Nova jobs is how it handles secrets. It's important to get this done right because of the security implication involved. Simply said, leaking secret is bad. And the remaining of this wiki will explain how to safely handle GitHub CI secrets in Nova jobs to avoid similar incidents in the future.

What is GitHub CI secret?

Short answer: a GitHub CI secret is a secret string that GitHub keeps in the repository settings. Only repo owners and org admins can add or modify its secrets. The secret can then be used in CI jobs to access remote services either in read-only mode, i.e. HF_TOKEN to download model weights, or with write access, i.e. PYPI_TOKEN.

Some important points to keep in mind:

• GitHub secrets have scope and can be:
  • Repository secrets which are available for all workflows in the repo
  • [RECOMMEND] Environment secrets that are limited only to specific environments. The key difference is that the environment has protection rules saying the conditions where the access is allowed, i.e. only from the protected main branch.
  • There is also org-wide secrets, but I have never seen one before nor plan to add one. So, let's exclude this.
• A fork pull request doesn't have access to the secrets of the target repo. In the context of PyTorch where ghstack is widely used, this is a common gotcha because ghstack PRs are non-fork so they have full access to PyTorch repo secrets while regular non-fork PRs don't.
  • For metamates, exporting an internal diff could map to either a fork (the default option) or non-fork PR. Going the default gives the impression that CI jobs with secrets don't work, i.e. #6865
• Using secrets on a non-ephemeral self-hosted runner exposes the possibility of them being stolen. As such, GitHub recommends using ephemeral runners whenever a secret is needed in CI.

How Nova jobs use secrets

When using Nova reusable workflows, it's important to keep in mind that they are located in pytorch/test-infra so that they can be shared with all repositories in the org. This setup requires that any repo using them (the caller) will need to explicitly pass over their secrets to Nova (the callee). This is done using the secrets-env parameter.

Here is a concrete example on how it's done:

1. Add secrets: inherit to the workflow. This grants Nova access to all the secrets from its caller. Note that if the caller workflow doesn't have access to a secret, for example from a workflow triggered by a fork PR, it won't be able to pass that over
2. Set the list of secrets to pass over to Nova among all the secrets available to the caller using secrets-env parameter. This is a list of secret names separated by space.
3. Access the secrets inside Nova script parameter as environment variables. Note that the prefix SECRET_ is added to the secret names from step 2 to distinguish them from regular variables. For example, HF_TOKEN becomes SECRET_HF_TOKEN.

There is, however, one important caveat to keep in mind that GitHub environment secrets are not usable here. The root cause lies in the limitation of GitHub reusable workflow itself, quoting the GitHub wiki:

Environment secrets cannot be passed from the caller workflow as on.workflow_call does not support the environment keyword. If you include environment in the reusable workflow at the job level, the environment secret will be used, and not the secret passed from the caller workflow

This is where the trouble begins because this limitation implies that only repository secrets are available to Nova workflows. And we know from a past incident that using repository secrets is an anti-pattern to be discouraged.