Incident 1 - h3 location lexicon

[ngerakines]

This is a report of incident 1. It involves the events and locations lexicon that was merged on February 3rd, 2025¹. That lexicon has a bug that rendered one of the types invalid.

Investigation and Diagnosis

The issue, as initially reported on February 9th, 2025², is that the community.lexicon.location.h3 type string is invalid. The AT Protocol NSID spec³ states that the "name" component has the following criteria:

digits and hyphens are not allowed

The community.lexicon.location.h3 type has the authority community.lexicon.location and name h3. That name element does not conform to the spec.

Resolution and Recovery

On 26 February 2025 a pull request was created (and subsequently approved) to resolve the issue: #43

This renames the community.lexicon.location.h3 type to community.lexicon.location.hthree to conform to the spec.

Although this is not technically a backwards compatible change, I think that because there aren't any records using it at the moment and also because this change is preventing the lexicon from being discovered, it is a safe thing to change in this way.

Alternatively, there could be an appetite to adjust the spec to support this format, but I think that it would take longer to adopt.

Alternative solutions are welcome in the form of pull-requests so they can be compared and discussed.

Fortunately, the current lexicon discovery tool is processing adjacent lexicon types, including those that directly reference the invalid lexicon type, correctly⁴.

Follow-up

Open questions:

• What post-push, post-merge, and pre-release actions can be implemented to lint lexicons more thoroughly?
• What guidance can we give developers building against lexicons for when invalid types are encountered?
• How can lexicons, including dependencies, be resolved in a way that reduces the blast radius of lexicon validation issues?

Footnotes

1. https://github.com/lexicon-community/lexicon/pull/23#event-16162748509 ↩

2. https://github.com/lexicon-community/lexicon/issues/40 ↩

3. https://atproto.com/specs/nsid ↩

4. https://lexidex.bsky.dev/domain/lexicon.community ↩

[ngerakines]

What post-push, post-merge, and pre-release actions can be implemented to lint lexicons more thoroughly?

I think a simple GitHub Action would help prevent this from happening in the future.

• On git push, lint the schema files and fail action if schema validation fails.

[ngerakines]

How can lexicons, including dependencies, be resolved in a way that reduces the blast radius of lexicon validation issues?

@tom-sherman I'm interested in your take here, given your recent tooling work to recursively resolve lexicons.

[tom-sherman]

I had been thinking if it was valuable to have schema tooling that can validate/parse a record partially - if it encountered refs that were invalid or unavailable for whatever reason, just drop those properties from the data (assuming they're optional).

[tom-sherman]

It's unfortunate that the more complex the web of schema references becomes, the more likely this issue is to crop up - all it takes is one person to publish a bad schema.

This is IMO another reason why caching (and lockfiles based on CIDs) is important - I don't want my workflows to be blocked by a temporarily broken schema.

[ebwinters]

I think something like this would suffice without any custom linting

name: Run lex-cli

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  run-lex-cli:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install lex-cli
        run: npm install -g @atproto/lex-cli

      - name: Generate API from lexicon schemas
        run: lex gen-api ./api/src ./community/lexicon/*.json

[ngerakines]

That looks very practical, nice,

[ngerakines]

Following up on this, I've created issue #45 and PR #47.

[bnewbold]

As an option, the goat command has a goat lex parse sub-command, for individual Lexicon JSON files. It doesn't current do recursive lookups and validation, but that would not be too hard to add. I'm definitely interested in making it a robust tool for Lexicon developers of all stripes, including lexicon-community.

[ngerakines]

Yeah, the infra script that I used to create the DNS and PDS records calls that before publication as a safety check. Unfortunately, I don't think it validates this one part of the spec.

$ goat lex parse ../lexicon/community/lexicon/location/h3.json
../lexicon/community/lexicon/location/h3.json: success

If I were to give something an invalid type, with the following diff as an example:

diff --git a/community/lexicon/location/h3.json b/community/lexicon/location/h3.json
index 84929dc..ab47f41 100644
--- a/community/lexicon/location/h3.json
+++ b/community/lexicon/location/h3.json
@@ -3,7 +3,7 @@
     "id": "community.lexicon.location.h3",
     "defs": {
         "main": {
-            "type": "object",
+            "type": "smobject",
             "description": "A physical location in the form of a H3 encoded location.",
             "required": [
                 "value"

Then an error would surface:

error: failed to parse ../lexicon/community/lexicon/location/h3.json: unexpected schema type: smobject

[ngerakines]

Closing this as discussion as trailed off and the resolution implemented.