🧹 Clarification: "evaluated"/"unevaluated" Definitions

[hackowitz-af]

Specification section

Throughout the Draft 2020-12, but particularly:

7. Keyword Behaviors

Evaluating an instance against a schema involves processing all of the keywords in the schema against the appropriate locations within the instance. Typically, applicator keywords are processed until a schema object with no applicators (and therefore no subschemas) is reached. The appropriate location in the instance is evaluated against the assertion and annotation keywords in the schema object, and their results are gathered into the parent schema according to the rules of the applicator.

11. A Vocabulary for Unevaluated Locations

[Unevaluated] instance items or properties may have been unsuccessfully evaluated against one or more adjacent keyword subschemas, such as when an assertion in a branch of an "anyOf" fails. Such failed evaluations are not considered to contribute to whether or not the item or property has been evaluated. Only successful evaluations are considered.

What is unclear?

The usages of "evaluated" and "unevaluated" are not antonyms - and "evaluated" is open to some interpretation.

Results

Just for context - I use the term "results" as defined in the spec - a boolean validity that may have with additional information attached by annotations. The JSON Schema recommended output schema provides an example.

Evaluated/Evaluation

The current usage of "evaluate" seems to mean "process all keywords and gather the results". This interpretation means "evaluation" is successfully completed once all keywords are processed and results gathered - regardless of the boolean value of the result. This would also mean that if keywords are processed but their results are not gathered, the keyword is not "evaluated".

I think this is a good definition, but it's not given explicitly. See the discussion in python-jsonschema #1365 (⚠ disclosure: my own issue), where the unevaluatedProperties validator does process it's dependent subschemas for it's own internal use then discards the results and does not gather them for the validator - but considered the subschemas "evaluated". It is not clear from the spec itself which interpretation is correct.

Unevaluated

From the above definition of "evaluated" as "processed to produce a result", "unevaluated" should be it's mutually-exclusive antonym - "not evaluated", that is "not processed to produce a result". Note that the value of the result is not part of the definition; an evaluation with a falsy result is still an evaluation.

However, in section 11, "unevaluated" does not mean "not evaluated", it means "not validated". Properties that are "evaluated" can also be "unevaluated" at the same time if the result is falsy! Further, the term "failed evaluations" also seems wrong; an evaluation that successfully produces a falsy result didn't fail, it succeeded!

Proposal

Definitions of Evaluated/Unevaluated

I propose to change the phrasing in section 7 to clearly define "evaluated" as "processed and results gathered". I can do this myself if is a desired clarification.

The phrasing and defined keyword behavior in section 11 is more tricky. If we were starting fresh, I think "unevaluated" should be defined as "not evaluated" in the spec, and unevaluated* keywords would change to follow the definition. Depending on the collection of subschemas, this might be able to produce the same results as before without significant breaking changes (see below).

If these definitions cannot be used, I think the docs need to be painfully explicit that "unevaluated" does not mean "not evaluated".

Specification for how applicators apply subschemas and gather their results

I think the intent of current unevaluated* behavior could largely be preserved if the applicator spec were more clear on how it applies subschemas and gathers their results. It seems like we have to treat falsy evaluations as unevaluated only because the falsy value indicates a subschema that should not have been applied. This may be wrong, or become a bigger topic, so I will open a new issue and cross-link them.

Do you think this work might require an [Architectural Decision Record (ADR)]? (significant or noteworthy)

Yes

[hackowitz-af]

#1605 addresses how applicators apply subschemas and gather their results. With respect to unevaluated* keywords, I'm curious how this might affect the reason(s) for "evaluated to be invalid" to be called "unevaluated". From the spec:

For example if a subschema represents a car, which requires between 2-4 wheels, and the value of "wheels" is 6, the instance object is not "evaluated" to be a car, and the "wheels" property is considered "unevaluated (successfully as a known thing)", and does not retain any annotations.

I don't mean to pick apart the example itself - but this highlights the issue fairly nicely, in the same terms the spec uses to justify it:

{
    "title": "A flying car",
    "allOf": [
        {
            "title": "car",
            "required": ["make", "model", "wheels"],
            "properties": {
                "make": {"type": "string"},
                "model": {"type": "string"},
                "wheels": {"type": "integer", "minimum": 2, "maximum": 4}
            }
        },
        {
            "title": "flying object",
            "required": ["wings"],
            "properties": {
                "wings": {"title": "A prompt for ChatGPT to design the wings", "type": "string"}
            }
        }
    ],
    "unevaluatedProperties": false
}

and the instance:

{"make": "Mercedes-Benz", "model": "G63 AMG 6x6", "wheels": 6, "wings": "great big wings that flap like a bird"}

Awesome! The passengers in the back seat might be unhappy, but the only thing that the instance author did wrong was to give the car too many wheels. Maybe the schema is wrong for claiming cars can't have that many wheels, or maybe the schema is right and it really does have too many wheels to be a car, but the number of wheels is the single issue needing resolution.

But from the docs, we would (and even should) get additional error(s), that "make", "model", and "wheels" were unevaluated! This is unhelpful at best, and might seem misleading or false entirely depending on the user's access/familiarity with the specification. "make" and "model" were, valid, required, and evaluated - but the message claims there is a problem with them. "wheels" is required and evaluated, and produces it's own validation result, but seems to not be evaluated.

This becomes very frustrating for large deeply-nested schema, especially with many $refs in a schema registry. For example, on an aircraft "interface control document", a digital "bus" claims that none of it's properties ("protocol", "name", "description", "terminals", etc.) are evaluated because one of the "encodings" in one of the "fields" in one of the "messages" from one of the "interfaces" was wrongly capitalized and was not in it's schema's enum.

Also, depending on interpretation of "evaluation", and if the sequence of evaluation must be maintained by the sequence of results, a fail-early implementation might only report the unevaluatedProperties and never reach the real issues. For example, an implementation of unevaluatedProperties might internally process the allOf and discard the result (claiming that this satisfies the spec that allOf be evaluated first), then raise the error and terminate before any results of allOf are ever reported to the user.

[hackowitz-af]

Vice versa if the car subschema was unambiguously not a car (vs. a car that's just not up to spec.), the car subschema should still be the one to say so. In 11.1. Keyword Independence and 10.1. Keyword Independence, it seems like it's preferred for keywords and subschemas to operate independently where possible. An error in $.unevaluatedProperties coupled to an error nested withing a sibling property at $.allOf[0].properties.wheels.maximum seems against the design philosophy.

For example, an X-wing shouldn't be admitted to a flying car show for exactly 2 reasons:

{"make": "Incom", "model": "T-65B", "wings": "A Star Wars X-Wing from the original trilogy", "laser cannons": 4}

• If it was a car, it would have wheels!
  • Fails $.allOf[0].required on "wheels"
  • Has no effect on "make" and "model", because of keyword independence! $.unevaluatedProperties takes a coffee break
• We didn't expect anyone to bring laser cannons, and we don't want anything unexpected!
  • $.unevaluatedProperties comes back from coffee and cites the X-Wing for bringing laser cannons to a car show.
  • Has no effect on "make" and "model"; we require these for everything at a flying car show, and everyone's looking at the tractor beam anyway

But if the registration desk told me my Incom T-65B X-Wing didn't have a "make" or "model" because there was landing gear where the wheels should be, I would disagree. I bet if I took of the laser cannons and put skateboards under the landing gear they would realize that it does have a make and model

[gregsdennis]

From the above definition of "evaluated" as "processed to produce a result"

I think this is the misunderstanding. A property/item is considered "evaluated" it if it is defined within a subschema that:

1. passes validation
2. all ancestor subschemas between the "unevaluated*" keyword and the property definition pass validation

More practically, assuming the implementation collects annotations, there must be an items or properties keyword that produces an annotation that reaches the subschema with unevaluated*. Given that annotations produced by a subschema are dropped when that subschema fails validation, it follows that the subschema validations from unevaluated* to the item or property definition must all pass.

I agree that the use of "evaluated" and "unevaluated" could be clearer. When we were putting in the keywords, we went round and round on the fact that the way we are using "evaluated" actually means "validation passed for this property/item in some subschema of a sibling keyword". However, using that as the definition, "unevaluated" does mean the opposite: "validation has not passed for this property/item in some subschema of a sibling keyword". It could be that the property/item was validated and failed, but that doesn't count for this purpose.

In 11.1. Keyword Independence and 10.1. Keyword Independence, it seems like it's preferred for keywords and subschemas to operate independently where possible. An error in $.unevaluatedProperties coupled to an error nested withing a sibling property at $.allOf[0].properties.wheels.maximum seems against the design philosophy.

This idea of keyword independence is preferred, yes, however there are cases like this where a dependency is clearly required. For example, the dependency between properties and additionalProperties has always been there. While additionalProperties doesn't have to be processed such that it requires the results of properties, processing it would still require looking at properties to see what keywords it defines.

If a keyword can be made to operate independently, it should, but sometimes it can't be made to do so.

But from the docs, we would (and even should) get additional error(s), that "make", "model", and "wheels" were unevaluated!

Your instance, in fact, does fail validation of that schema, precisely because of the reason you mention: wheels is 6.

(click to expand) Here's the output from my validator, https://json-everything.net/json-schema:

[!NOTE]
For clarity, this is from the next version of the spec, not 2020-12. 2020-12 is remarkably difficult to read (and produce).

{
  "valid": false,
  "evaluationPath": "",
  "schemaLocation": "https://json-everything.net/7422d56271#",
  "instanceLocation": "",
  "details": [
    {
      "valid": false,
      "evaluationPath": "/allOf/0",
      "schemaLocation": "https://json-everything.net/7422d56271#/allOf/0",
      "instanceLocation": "",
      "details": [
        {
          "valid": true,
          "evaluationPath": "/allOf/0/properties/make",
          "schemaLocation": "https://json-everything.net/7422d56271#/allOf/0/properties/make",
          "instanceLocation": "/make"
        },
        {
          "valid": true,
          "evaluationPath": "/allOf/0/properties/model",
          "schemaLocation": "https://json-everything.net/7422d56271#/allOf/0/properties/model",
          "instanceLocation": "/model"
        },
        {
          "valid": false,
          "evaluationPath": "/allOf/0/properties/wheels",
          "schemaLocation": "https://json-everything.net/7422d56271#/allOf/0/properties/wheels",
          "instanceLocation": "/wheels",
          "errors": {
            "maximum": "6 should be at most 4"
          }
        }
      ]
    },
    {
      "valid": true,
      "evaluationPath": "/allOf/1",
      "schemaLocation": "https://json-everything.net/7422d56271#/allOf/1",
      "instanceLocation": "",
      "annotations": {
        "title": "flying object",
        "properties": [
          "wings"
        ]
      },
      "details": [
        {
          "valid": true,
          "evaluationPath": "/allOf/1/properties/wings",
          "schemaLocation": "https://json-everything.net/7422d56271#/allOf/1/properties/wings",
          "instanceLocation": "/wings",
          "annotations": {
            "title": "A prompt for ChatGPT to design the wings"
          }
        }
      ]
    },
    {
      "valid": false,
      "evaluationPath": "/unevaluatedProperties",
      "schemaLocation": "https://json-everything.net/7422d56271#/unevaluatedProperties",
      "instanceLocation": "/make",
      "errors": {
        "": "All values fail against the false schema"
      }
    },
    {
      "valid": false,
      "evaluationPath": "/unevaluatedProperties",
      "schemaLocation": "https://json-everything.net/7422d56271#/unevaluatedProperties",
      "instanceLocation": "/model",
      "errors": {
        "": "All values fail against the false schema"
      }
    },
    {
      "valid": false,
      "evaluationPath": "/unevaluatedProperties",
      "schemaLocation": "https://json-everything.net/7422d56271#/unevaluatedProperties",
      "instanceLocation": "/wheels",
      "errors": {
        "": "All values fail against the false schema"
      }
    }
  ]
}

You can see the error "maximum": "6 should be at most 4" where the subschema is evaluated. Since this subschema fails, all of those properties are considered unevaluated, so they're picked up by unevaluatedProperties, each producing an error.

This becomes very frustrating for large deeply-nested schema

I don't disagree. I have been working on the output for some time, and, for the next version, we've decided to extract it to its own spec so that we can iterate on it separately.

I think the root of your frustration here is the output, not the schema evaluation or even the definition of "evaluated". I've written several blog posts about it, but I think this one will be more helpful to you here. It focuses on output for schemas that branch, which this one does, but it's pretty easy to see how the unevaluated* output can fit into this.

Has no effect on "make" and "model", because of keyword independence!

I think this is another misunderstanding. properties may pass in this subschema, but required fails, and we only consider validation pass/fail at the subschema level. Since the subschema fails, all properties defined within it are still considered unevaluated.

[hackowitz-af]

Thanks, these definitions help a ton for understanding current (2020-12) behavior!

I think the root of your frustration here is the output [...]

Sort of, but not quite - I think what I have trouble with (and would like to help change, if I can) is actually that:

annotations produced by a subschema are dropped when that subschema fails validation

I think that this should hold true for assertion keywords but not for applicators. An applicator can require a subschema, regardless of it's validity, as in allOf. In these cases, the subschemas and parent schema are peers in the sense that they are all required schema of the same json instance. The total result for the instance then would be the combined result of each required schema. It would follow that annotations of evaluated keywords for subschemas are gathered, even if other keywords fail evaluation and cause the subschema as a whole to fail validation.

Are annotations for the root schema dropped when it fails validation? I guess I've assumed they aren't - though I might be wrong.

This would greatly clarify the reported errors for both of my examples by removing the seemingly "false negative" unevaluatedProperties and adding annotations for the newly evaluated "make" and "model". The 6 wheels would still be both "unevaluated" and above the "maximum" - which is unfortunately not solved without a definition change.

[gregsdennis]

I think that this should hold true for assertion keywords but not for applicators.

But it's at the subschema that the annotations are dropped on validation failure, not the keyword. This means that by the time an applicator keyword receives information that the subschema has failed, there aren't any annotations to pass on to its siblings. I do see your point about gathering all annotations for a given instance location, but that's not how the system was designed.

The point of the unevaluated* keywords was that we wanted all properties/items to be successfully validated. I think the catch for you is that we don't consider successfully validated properties within subschemas that otherwise fail. If the subschema fails, we invalidate everything about it. If "make" and "model" pass, but "wheels" fails, then that entire subschema is invalid, and we can't use any of its internal results.

I think I'd be open to a proposal for a new keyword that behaves more like you're expecting, but unevaluated* and the annotation behavior (i.e. dropping when their subschema validation fails) have been around long enough that changing it for the new spec will cause more problems than it solves.

Are annotations for the root schema dropped when it fails validation?

Yes they are.

Have a play on https://json-everything.net/json-schema. It'll include all of the annotations with the specified behavior, and there's even an option to include dropped annotations so you can see what's left out and where it got lost.

[hackowitz-af]

But it's at the subschema that the annotations are dropped on validation failure, not the keyword.

Ah - given that, I agree there's not really anything to do about it. I'll make a proposal for a new keyword in a separate issue and close this one.

[jdesrosiers]

Here's a relevant discussion you might be interested in reading up on, https://github.com/orgs/json-schema-org/discussions/67.

I'd love to see a replacement for the unevaluated keywords. I find them to be highly problematic. But I wasn't clear on what the proposed change is exactly, so I'm looking forward to reading the proposal.