🧹 Clarification: Define In-Place Applicators' Subschemas and Results

[hackowitz-af]

Specification section

10.2. Keywords for Applying Subschemas in Place
These keywords apply subschemas to the same location in the instance as the parent schema is being applied. They allow combining or modifying the subschema results in various ways.

[10.2.*.*] [...] An instance validates successfully against this keyword if it validates successfully against [some subset of] schema[(s)] defined by this keyword's value.

What is unclear?

• It doesn't define which applicator schema(s) are applied to the parent as subschemas.
• It doesn't define which results are gathered

The spec says the applicator applies subschemas - but not all of the applicator's contained schemas are actually subschemas of the instance. Many of them are only candidate subschemas as in the oneOf applicator, explicitly not subschemas as in the not applicator, or conditional as in the if/then/else.

The spec says the applicators allow combining or modifying "subschema" results, but each applicator only defines how it determines the boolean "validity". Most of the applicators don't state anything about how they combine any other results like annotations or evaluated properties. For example, should the annotations of a valid schema with in a not applicator be gathered? What about an invalid one? How about the properties of every valid schema within a anyOf applicator be considered by unevaluatedProperties, like with annotations?

Proposal

Here, I use "results" to mean all annotations, "evaluated" items/properties per unevaluated* and additional* keywords, and possibly other results like validation errors.

The spec should clarify, either for each applicator individually or for all applicators globally:

• The applicator provides results from all subschemas - even if they are invalid.
• The applicator does not provides results from non-subschemas. It uses their results to determine it's own, not to pass through.
• All valid, evaluated, applicator schemas are subschemas.
  • For example, a valid schema in a not applicator is a subschema and can contribute results, even though it causes the applicator to be invalid.
  • The anyOf docs already clarify that all schemas need to be evaluated when collecting full results, but it might need to be added if this should include "evaluated" items/properties, or annotations only
• Invalid or unevaluated applicator schemas might still be subschemas, depending on the applicator's defined logic.
  • For example, a valid else schema following a valid if schema, is not evaluated and does not produce results.
  • For example, each allOf schema is required to a subschema, even if it it invalid, and must provide results.

This would help quite a bit for unevaluated* keywords, which seem to struggle from how to determine evaluated properties, especially from subschemas, per #1604 .

Do you think this work might require an [Architectural Decision Record (ADR)]? (significant or noteworthy)

No

[hackowitz-af]

I guess the most simple summary is that I think that all subschemas applied by applicators should behave consistently with the current behavior of the $ref applicator.

The Difference

It seems that in 2020-12, results are gathered from a failing #/$ref as if they were from the schema #/ itself. But results from subschemas are not.

Let's compare using a car that fails validation (if it's a car at all - since it fails to validate as one):

{"make": "Mercedes-Benz", "model": "G63 AMG 6x6", "wheels": 6}

⚠ My results are produced in python with jsonschema==4.23.0. If these deviate from the spec, and have caused my misinterpretation of the spec, I will migrate my issue over there.

The $ref Applicator

Let's include the car subschema using $ref. The unevaluatedProperties will shine some light on what annotations (if any) are gathered from the failing car subschema.

{
  "$ref": "#/$defs/car",
  "unevaluatedProperties": false,
  "$defs": {
    "car": {
      "required": ["make", "model", "wheels"],
      "properties": {
        "make": {"type": "string"},
        "model": {"type": "string"},
        "wheels": {"type": "integer", "minimum": 2, "maximum": 4}
      }
    }
  }
}

produces one error:

ValidationError: 6 is greater than the maximum of 4

Failed validating 'maximum' in schema['properties']['wheels']:
    {'type': 'integer', 'minimum': 2, 'maximum': 4}

On instance['wheels']:
    6

Note that the "make" and "model" are not unevaluated, meaning annotation results are gathered from the failing subschema when specified by the $ref applicator.

Also, the reported "path" is #/properties/wheels, not #/$ref/properties/wheels or #/$defs/car/properties/wheels, but I'm not sure what to make of that.

Other Applicators

Let's see the same example, this time using an allOf as the applicator instead of $ref:

{
  "unevaluatedProperties": false,
  "allOf": [
    {
      "required": ["make", "model", "wheels"],
      "properties": {
        "make": {"type": "string"},
        "model": {"type": "string"},
        "wheels": {"type": "integer", "minimum": 2, "maximum": 4}
      }
    }
  ]
}

It now produces two errors:

ValidationError: Unevaluated properties are not allowed ('make', 'model', 'wheels' were unexpected)

Failed validating 'unevaluatedProperties' in schema:
    {'unevaluatedProperties': False,
     'allOf': [{'required': ['make', 'model', 'wheels'],
                'properties': {'make': {'type': 'string'},
                               'model': {'type': 'string'},
                               'wheels': {'type': 'integer',
                                          'minimum': 2,
                                          'maximum': 4}}}]}

On instance:
    {'make': 'Mercedes-Benz', 'model': 'G63 AMG 6x6', 'wheels': 6}

ValidationError: 6 is greater than the maximum of 4

Failed validating 'maximum' in schema['allOf'][0]['properties']['wheels']:
    {'type': 'integer', 'minimum': 2, 'maximum': 4}

On instance['wheels']:
    6

The subschema behavior seems to differ between $ref and the other applicator, in a meaningful way! Now the "make" and "model" are unevaluated, meaning annotation results are not gathered from the failing subschema when specified by the allOf applicator.

---

• Is my validator not following the $ref spec by producing these results? If not...
• Is it defined anywhere that $ref is a special case that behaves differently from other applicators?
• Should $ref behave differently from other applicators? If not, which should change?

[jdesrosiers]

First of all, thank you for providing feedback on the spec! We need it and we appreciate you taking the time.

not all of the applicator's contained schemas are actually subschemas of the instance.

That's not actually how we use the term subschema. A subschema is any schema within a schema resource. The term is independent of the instance that's being validated.

Most of the applicators don't state anything about how they combine any other results like annotations or evaluated properties.

Evaluated properties are defined as annotations. There's no difference in how how they propagate compared to annotations.

Annotations propagate the same way for every keyword. They either get dropped because the schema failed validation or they get aggregated with the annotations from all the passing schemas. This should be covered in the 7.7 Annotations section, but as I'm reading it now, it seems to be lacking.

So,

For example, should the annotations of a valid schema with in a not applicator be gathered?

Yes

What about an invalid one?

No

How about the properties of every valid schema within a anyOf applicator be considered by unevaluatedProperties, like with annotations?

Yes. That part is covered in the last paragraph of 7.7 about short circuiting.

The subschema behavior seems to differ between $ref and the other applicator, in a meaningful way! Now the "make" and "model" are unevaluated, meaning annotation results are not gathered from the failing subschema when specified by the allOf applicator.

This is an issue with the validator you're using. Validators are allowed to present errors however they like, so it's not technically wrong, but I doubt it was the intention for those two examples to produce different results.