Skip to content

Commit

Permalink
fix(cli): tolerate #2702 stderr noise (#2704)
Browse files Browse the repository at this point in the history
Closes: #2702 
Refs: #XXXX

## Description

As shown by #2702:
When tests are run under the vscode JavaScript Debug Terminal, every
time that test spawns a subshell, extra noise appears on the stderr of
that subshell, such as
```
Debugger attached.␊
  - Waiting for the debugger to disconnect...
```

This seems symptomatic of a widespread assumption that other tools on
the toolchain can inject extra output into the stderr of subshells
without typically causing problems.

The test framework code changed by this PR had defaulted both an absent
`stdout` expectation and an absent `stderr` expectation to expecting the
empty string. This PR changes the `stderr` default expectation to an
regexp that accepts anything. This can still be overridden by an
explicitly provided `stderr` expectation, including one explicitly
expecting an empty string, so there is no loss of generality. Just a
change in the default.

### Security Considerations

This change will cause some tests to pass that might have otherwise
failed. Some of these masked failures may have been symptoms of actual
problems, rather than tool generated noise. This PR will mask such
problems.

### Scaling Considerations

none

### Documentation Considerations

none

### Testing Considerations

Because the #2702 bug this PR fixes is about a behavior seen under the
vscode JavaScript Debug Terminal, I didn't see a way to write an
automated test to check the fix under CI. Rather, I just verified the
fix locally and interactively using vscode.

### Compatibility Considerations

As mentioned in Security Considerations above, this PR may mask genuine
errors that would have been flagged prior to this PR.

### Upgrade Considerations

none
  • Loading branch information
erights authored Feb 6, 2025
1 parent 895ea0a commit 1bf94a1
Showing 1 changed file with 9 additions and 3 deletions.
12 changes: 9 additions & 3 deletions packages/cli/test/_section.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@
* Transforms a testRoutine into an ava test.
* The testCommand function asserts that a given awaitable command produces the expected stdout and stderr.
*
* An absent `expectation.stdout` defaults to the empty string.
* An absent `expectation.stderr` now defaults to an accept-anything regexp
* in order to tolerate extra tool-generated noise that happens, for example,
* when run under a vscode JavaScript Debug Terminal
* @see https://github.com/endojs/endo/issues/2702
*
* @param {Execa} execa - the command execution environment
* @param {TestRoutine} testRoutine - the test logic implementation
* @returns {(t: t) => Promise<void>}
Expand All @@ -15,16 +21,16 @@ export function makeSectionTest(execa, testRoutine) {
const matchExpecation = (expectation, result, errMsg) => {
(expectation instanceof RegExp ? t.regex : t.is)(
result,
expectation ?? '',
expectation,
errMsg,
);
};
const testCommand = async (command, expectation) => {
const result = await command;
if (expectation !== undefined) {
const errMsg = JSON.stringify({ expectation, result }, null, 2);
matchExpecation(expectation.stdout, result.stdout, errMsg);
matchExpecation(expectation.stderr, result.stderr, errMsg);
matchExpecation(expectation.stdout ?? '', result.stdout, errMsg);
matchExpecation(expectation.stderr ?? /.*/, result.stderr, errMsg);
}
};
await testRoutine(execa, testCommand);
Expand Down

0 comments on commit 1bf94a1

Please sign in to comment.