LinkML-Scala – how and why?

[Ostrzyciel]

Hi all! 👋

This is a copy of the Q&A we posted here in the announcement of LinkML-Scala: https://github.com/orgs/linkml/discussions/3787

We hope this clarifies a bit the context of this project :)

We'll happily answer any other questions you may have! :) Just drop them here in a comment, or start a new discussion.

Q&A

• Why not contribute to linkml instead?
  • There are some things that Python fundamentally cannot do, like performant code or portability. Python is also not a very strict language, which makes maintaining a complex codebase challenging, at least in our opinion.
  • We also believe that a second full implementation is the thing the LinkML community needs to finally give the LinkML specification the attention it deserves. More on that in the next point...
  • We did contribute to the Python implementation in the past (e.g., fixes in the JSON-LD generator, the YARRRML generator), but we likely won't continue this work.
• How do you plan to work with the existing implementation?
  • We want to maintain a second, independent implementation, side-by-side with linkml. We want the LinkML specification to be the ground for deciding how all implementations should behave. To help with that, in the coming weeks and months we will be contributing fixes and improvements to the spec.
• Why Scala?
  • Scala is an amazingly portable language. Many people associate it with the Java Virtual Machine only, but modern Scala also compiles to JavaScript and native binaries. We use GraalVM for native compilation – the other option is Scala Native, but it wasn't as stable as Graal, so we discarded it.
  • It is possible to also make native libraries for languages like Rust, C++, or C. If you need that, please open an issue and let us know!
  • Scala also offers features like metaprogramming and an advanced type system that allowed LinkML-Scala to be both fast and safe – most errors are caught at compilation time.
• What's the feature coverage?
  • Generators: SHACL, JSON Schema, Frictionless Table Schema, RDFS, Scala, LinkML (class derivation). We are working on more – let us know in issues what you need. :)
  • Metamodel features: most of the "common stuff" like imports, ranges, enums, slot derivation, inlining. A notable (yet) unsupported feature are default values. We are constantly implementing more, but our main bottleneck is the spec, it's very incomplete. Please feel free to suggest in issues what should we work on next. :)
• How do you make it more consistent and robust?
  • We moved most of the logic from generators to SchemaView.
  • We also use a strongly-typed language where many wonky errors (e.g., accessing undefined fields in a class) are simply impossible.
  • Finally, we have a model catalog (dubbed: "model zoo") for testing the various LinkML edge cases. We employ it systematically across all generators and also cross-check with format-specific validators (e.g., JSON Schema, SHACL). This catches an unbelievable amount of bugs.
  • This model catalog is language-agnostic, so you can adopt it in the Python implementation as well. Maybe we could make it into a conformance test kit for the spec?
• What about JSON <> RDF and other conversions?
  • We have an internal solution for lifting JSONs and CSVs to RDF, and lowering RDF to JSON/CSV. It works great. We unfortunately cannot open source it at this moment. We plan to open source it later (can't name a date yet, sorry).
  • If you need this in your work, please get in touch with me, we may be able to help.
• How do you use it?
  • Mainly in a big ol' JVM server application. Also as a CLI tool, in GitHub Actions CI and (sometimes) in the browser.
  • We have a lot of closed-source generators that are based on LinkML-Scala, it's very easy to write them.
• Do you use the Python implementation?
  • We wrote the initial bootstrap generator of Scala classes in Python, but we stopped using it the moment we reached self-hosting. We do not plan to release it, it's very much out of date now.
  • In general we are not using the Python implementation internally at all (other than for comparison testing) and we are very unlikely to make any contributions to it. LinkML-Scala is designed to be a full replacement.
• What about the Java and Rust implementations?
  • Don't fit our portability requirements. Java doesn't have a clean & stable compilation path to JS, and Rust doesn't work in the JVM.
  • We also knew from the start that we want to reimplement generators, and to have very good infrastructure for implementing more generators (we need a lot of them!). This is a different scope than these project have.