Check all $refs are resolvable

[rhiaro]

I've added 5 json schemas which cross-reference each other and refer to $defs in the same file using $ref into a Registry. Is there a straightforward way to check if all of the $refs resolve successfully? I can of course use resolver.lookup() on individual paths - I'd like that, but automatically, for all $refs. I've spent some time looking at the docs and code for this library and haven't figured it out, but apologies if I've missed something obvious. (If not, I guess I need to use jsonref for this?)

[Julian]

Hi! No worries.

For an arbitrary JSON Schema there actually isn't a feasible way to do this in general -- specifically for $dynamicRef (as opposed to $ref) it's potentially impossible to do, as it may depend on the schema that a subschema ends up embedded into, or on the data, etc.

For some schemas it's possible, particularly if you limit to $ref and no dynamicism -- #27 is somewhat related as really having easy access to a bundler is part of the "recommended solution" upstream from the spec team.

Ultimately my real recommendation actually has nothing to do with $ref in particular -- anytime you write a schema you should be writing unit tests which exercise the branches of the schema, because a schema isn't a replacement for those, as there's no way of ensuring your schema actually enforces what you expected it to enforce without having tests checking it does what you meant on your data. Those unit tests should uncover if you have any broken refs, as you'll see them in your test suite. That's really what anyone worried about this should be doing -- but it's true that for some subset of possibilities, jsonref will do something!

[Julian]

All the above aside, if there are tools that this library should offer to make this easier, I definitely won't claim it's a fully solved problem, so I'm open to ideas, I just don't think something like jsonref's way is feasible in "Modern JSON Schema" or OpenAPI (which this library also means to tackle at some point -- but modern versions are pretty complicated when it comes to referencing behavior.

[rhiaro]

Thanks for the fast reply! The reason I want this is actually because I'm writing tests for the schema. So your recommendation would be to test the $refs by running test data against every single one of them? Which does make sense, I was just hoping for a shortcut, since they're all already just right there in the Registry.. (my schema only uses $ref not $dynamicRef so at least it's not in the realms of impossible).

[Julian]

Maybe I'm assuming the wrong scenario then --

So your recommendation would be to test the $refs by running test data against every single one of them?

If you have refs "urn:example:foo" and "urn:example:bar", I'm not saying to have tests that do literally assert Registry.contents("urn:example:foo") == whatever, even though you indeed could get urn:example:foo and urn:example:bar spit back out at you by iterating over the URLs the registry knows about.

The $refs are applying some behavior presumably, are they not? E.g. you have some schema:

{
"title": "my cool schema",
"$ref": "urn:example:foo",
"$defs": {
  "positiveInteger": {"$id": "urn:example:foo", "type": "integer", "minimum": 1},
  }
}

(or the same applies if you have urn:example:foo in some other resource you loaded into the registry).

So you should have some unit test that instead looks like assert is_valid(37) and assert not is_valid("foo") -- these unit tests a fortiori imply the refs work, but you never test them directly -- and any time you would have, it means just test whatever data actually depends on the $ref.

Does that help, or am I misunderstanding what you have or are doing?

[rhiaro]

so you should have some unit test that instead looks like

Yeah, that's exactly what I meant.

If someone else modifies the schema to add a potentially broken $ref but doesn't add a test, there would be nothing to flag it in the test suite, but that is not a problem for here! Thanks for your help, much appreciated.

[Julian]

Yeah I think for that you have to simply enforce that schema changes come with test changes same as code changes need to. Glad to help.