Equivalent of rust's "crater" (test changes against public code to see what visibly breaks)

[SamB]

I was looking at Why a port instead of a rewrite? What's the difference? (#410) and it occurred to me to wonder what tooling you have that is roughly analogous to Rust's crater ("Run experiments across parts of the Rust ecosystem!"), which runs experiments (usually comparing cargo build/cargo run before and after a toolchain change, but apparently custom experiments can be done too) across a wide sample of rust code (chosen from the crates.io package repository and public GitHub repositories).

I realize this is substantially more challenging for TypeScript because

• There's nothing like an 80% build procedure for the JavaScript ecosystem: there are at least three big names (npm, yarn, pnpm) in package management, and there's no real standard about what commands are used to typecheck, build, and test packages - they are more like guidelines
• You can't work around that by checking the uploaded packages, as there's a strong stigma against shipping the TypeScript sources in those
• The typescript compiler tends to be an explicit dependency, but that dependency says very little about how the compiler is used: cli, library api, or server interface?
• Typescript's intentionally unsound typing, while necessary for adding types to idiomatic JavaScript, does make it harder to tell if something has broken, especially when using less-strict settings / only for intellisense

But I'm assuming there's something kicking around that's used to check for breakage of code "in the wild", despite the above challenges. But nobody seems to have used the word crater in issues for either this repository or typescript's, nor in these discussions, prior to now.

So, again: what do you have to check for breakages in a sampling of the ecosystem?

[jakebailey]

There's nothing like an 80% build procedure for the JavaScript ecosystem: there are at least three big names (npm, yarn, pnpm) in package management, and there's no real standard about what commands are used to typecheck, build, and test packages - they are more like guidelines
You can't work around that by checking the uploaded packages, as there's a strong stigma against shipping the TypeScript sources in those

We have an extensive test suite with a large number of baselines. You can see existing PRs and how they affect baselines as we improve compatibility.

https://github.com/microsoft/TypeScript-error-deltas exists to let us query known-important repos but also an arbitrary of "top" TypeScript repos (as given by GitHub); this tool automatically runs and compares the outputs of two different builds of TypeScript. It's effectively an equivalent of crater. We can do this because tsconfig is effectively all one needs to verify TS behavior.

The typescript compiler tends to be an explicit dependency, but that dependency says very little about how the compiler is used: cli, library api, or server interface?

The CLI is easy to test, but we know already that the API will break and we're working to get that going. For API breaks themselves (outside of a major break like this one), we baseline the API surface itself and inspect it when it changes. Look for typescript.d.ts committed to the main TS repo. For the server, I assume you mean tsserver, which we're redoing in LSP.

Typescript's intentionally unsound typing, while necessary for adding types to idiomatic JavaScript, does make it harder to tell if something has broken, especially when using less-strict settings / only for intellisense

TypeScript does not affect runtime behavior outside a set of non-ECMAScript standard features, which are extremely well tested. The type system's behavior cannot affect runtime code.

[SamB]

I suppose I was overthinking things because, in theory, there's no reason some of TypeScript's input couldn't be freshly generated during the build process. And yeah, I understand that the existing API and tsserver protocol won't work for the Go port; as long as the LSP has parity with https://github.com/typescript-language-server/typescript-language-server, this will probably be fine.

TypeScript does not affect runtime behavior outside a set of non-ECMAScript standard features, which are extremely well tested. The type system's behavior cannot affect runtime code.

This is true, for good or ill. Still, if unchecked JavaScript projects lose completion, this won't show up in the tests, will it?

[jakebailey]

if unchecked JavaScript projects lose completion, this won't show up in the tests, will it?

I assume you mean in terms of editor completions? I'd think it would; we test JS code in our tests, and it's the same code paths save for a couple of differences.