SDPX 3.0 Support

[mlieberman85]

The SPDX 3.0 spec has been officially released. This library should add support for SPDX 3.0.

What would this look like?

• Add 3.0 models (in spdx/v3/v3_0)
• Update the referenced core models in spdx to be the latest 3.0 version
• Add any necessary conversion functions from 2.3 -> 3.0
  • see: go-struct-converter and a potential change to support backwards conversion if we also want to support 3.0 -> 2.x conversion
  • Add at least one fairly complete 2.3 -> 3.0 conversion test
• Add support in deserialization calls for 3.0:
  • JSON (SPDX 3 only supports JSON-LD presently)
  • YAML
  • Tag Value (also: writer)
• Add tests against some reference 3.0 documents (to be identified)

[kzantow]

@mlieberman85 there are plans to add SPDX 3 support at some point, but there isn't currently a timeframe for it. I hope you don't mind but I've updated the text of this issue to use it a bit more placeholder issue for adding support.

[kzantow]

There have been some people looking in to how we can add support for SPDX 3 to this library. There are some distinct challenges -- namely the SPDX 3 model is based on single-inheritance, which isn't really directly supported in Golang. To share some potential solutions, I created this PR for discussion so anyone interested in what the future of this library looks like can comment on the ideas, suggest others, etc.: #240

[kzantow]

A prototype is available for feedback here: #247

[janjaeschke]

I would like to push this topic or ask for a current state. :)
The linked draft is 3 month old and for me it looks like the work sadly stopped. :(

[kzantow]

Hi @janjaeschke 👋 I have paused working on this until there is some resolution to a serious problem between the current SPDX 3 specification and the implementations.

I believe the draft PR was left in a fully functional state serializing to and deserializing from JSON-LD 1.1. However, documents created from the draft PR do not validate with the online tools and are not compatible with the other SPDX 3 implementations because the other implementations do not accept JSON-LD but rather only JSON conformant to the SPDX 3 JSON schema.

The serializations section of the SPDX 3 spec, however, indicates that JSON-LD is a valid serialization format, but the current implementations do not actually support JSON-LD, but rather a subset that no JSON-LD libraries are able to produce and as such requires a custom serializer, which I am not planning on writing without the spec indicating I should. Here is the text copied verbatim from that page:

RDF serialization
Since the data model is based on RDF, any SPDX data can be serialized in any of the multiple RDF serialization formats, including but not limited to:

• JSON-LD format as defined in JSON-LD 1.1;
• Turtle (Terse RDF Triple Language) format as defined in RDF 1.1 Turtle;
• N-Triples format as defined in RDF 1.1 N-Triples; and
• RDF/XML format as defined in RDF 1.1 XML Syntax.

As it stands I believe the spec is misleading and should have all references to data formats removed that are not going to be compatible and should only indicate JSON compliant to the schema is allowed OR the implementations should support full JSON-LD, in which case the draft PR is fairly close to being done, albeit requiring lots more testing to be performed and more unit tests to be added.

In the meantime, there is another library that only supports SPDX 3 using the custom serialization format you are able to use, but last I saw does not support conversions from 2.x and has an API that is not what I hope to provide in tools-golang here. Since golang was not really considered when designing SPDX 3, the lack of inheritance in the language ends up making the SPDX object model fairly cumbersome to deal with in practice, but I think I came up with a reasonable set of utilities in the draft PR to make people's lives easier, such as listing all items by type within each collection, pervasively through the model. I don't believe the same is true for the library noted above, so you will need to type assert almost everything to be able to use it.

Of course, I'd love to have any help testing the implementation in the draft PR, too, if someone was interested in moving it forward.

[goneall]

@kzantow - Shortly after you raised the issue of not specifying the specific serialization for SPDX 3 (which many of us agreed was an issue), we created this PR: spdx/spdx-spec#1197

Here's the current description of the serialization spec: https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Vocabularies/ExternalIdentifierType/

If this isn't sufficient, please provide some details or perhaps a clarifying PR to address so we can move forward on SPDX 3 support.

I'll be on the implementers call this morning if you'd like to discuss. There are quite a few people who would really like to update this library for SPDX 3 and your contributions to the library are very much appreciated by the community.

[JPEWdev]

In the meantime, there is another library that only supports SPDX 3 using the custom serialization format you are able to use, but last I saw does not support conversions from 2.x and has an API that is not what I hope to provide in tools-golang here. Since golang was not really considered when designing SPDX 3, the lack of inheritance in the language ends up making the SPDX object model fairly cumbersome to deal with in practice, but I think I came up with a reasonable set of utilities in the draft PR to make people's lives easier, such as listing all items by type within each collection, pervasively through the model. I don't believe the same is true for the library noted above, so you will need to type assert almost everything to be able to use it.

The spdx-go-model bindings are low level generated code bindings; the intention is that these are easily generated based on the SPDX spec, so they can be kept up to date. Projects are free to use these bindings if they desire and they should work just fine; they do take steps to guide users into creating correct documents, providing (incomplete) compile time and runtime validation. However, they are missing some QoL features that tools-golang is better suited to do. The intention is that tools-golang take the generated bindings and add these QoL features on top (for example, conversion from SPDX 2 -> SPDX 3, etc.)

Also, I'm not a go programmer, but I really didn't get a lot of help writing the code generation for go bindings (the automatic generation of the code bindings is IMHO paramount for our maintenance of the SPDX spec), so I do sincerely apologize for how non-go-like they are. If a fluent go programmer would like to make more go-like bindings that are more comfortable, please feel free to make a new generator for shacl2code; my only request is that you use the existing test benches to validate the new bindings (the existing go bindings should be helpful to show how to do that).

[janjaeschke]

@kzantow thank you for the brief explanation, that totally explains why you got stuck here.
I hope the mentioned changes described by @goneall will solve at least a part so you might continue or even solve the entire issue.

Sadly I have way too little experience to support here as any kind of developer :(

[henning-krause]

Spdx 3.0 would be awesome. I saw some progress on #247. Any ideas when this will be available? Sadly, I'm not a go dev, so I probably can't help much.

[kzantow]

Hi everyone, here's an update on my SPDX 3 plan for this library: I have merged the code to this repo in a separate branch (spdx3), which I will keep up to date with main (it is currently up to date and there hasn't been a lot of activity in main, but this allows easily making patch versions of v0.5 if needed). I have created a v0.6.0-rc2 release candidate that seems to be relatively bug free, and validates against the Java tools, so you are able to use this today.

If you want to use the SPDX 3 model, I have tried to make it as similar to SPDX 2 as possible, but there are obviously lots of changes. You can use the SPDX 3 model by importing the package github.com/spdx/tools-golang/spdx/v3/v3_0. You should use NewDocument to create the document, as it handles the circular reference of CreationInfo and such.

This is the last call for feedback, I've asked many times in different venues, and without any substantial feedback to the contrary, I'm planning on moving forward with this implementation unless there are strong reasons not to.

Notes:

For the v0.6.0 release, I don't intend to implement backwards conversion from 3 to 2, but there is forwards conversion, so if you use the SPDX 3 data model, you are able to read any prior version that was already supported (2.1 tag-value, 2.2 json or later), but only able to export SPDX 3. Because of this and not yet seeing widespread adoption of SPDX 3, I plan on keeping the v0.6 main data model pinned to 2.3 (the github.com/spdx/tools-golang/spdx import many are using). This means your existing code will continue to work and you are able to export your 2.3 model to 3 relatively easily (it works like previous conversions do; here's an example in Syft). If someone really wants this, PRs are welcome! It would only require a very small test of <full spdx 2 document> -> <spdx 3> -> <spdx 2>, diff them.

After it's been in use for a while in the wild and the consensus is that there are no significant issues, the next minor release, v0.7, I will update the main model to SPDX 3 (where you can continue to use 2.3 directly by changing your import to github.com/spdx/tools-golang/spdx/v2/v2_3), and hopefully add backwards conversion from 3.0 to 2.3, which will allow reading and writing any supported version. Backwards conversion will of course potentially be lossy and the SPDX 3 data model is fairly "loose" to me, so it's a little unclear exactly what to do for the backwards conversion. Hopefully that will be more clear in the future, when we see more SPDX 3 documents.

I have integrated this in Syft and plan to release this soon using the RC; you can see some examples reading there. I owe an update to the README before v0.6; there's an example creating a document here.

Under the hood, document reading is done using JSON-LD and supports extensions, if there are appropriate data types registered -- Cdx properties are already registered, of course, since they're part of the spec; you probably don't need this, but if you do I'm happy to explain it... it's actually pretty simple, just make a struct with appropriate field tags and register it with the LD context.

Trying not to get too far in the weeds, but you should know there are a couple things this library is automatically doing for you:

• every element in SPDX 3 needs a CreationInfo, so when saving SPDX 3, the library traverses the object graph and sets CreationInfo based on what you passed to NewDocument for any objects that don't have it... so if you read a document it will retain the previous CreationInfo on those elements
• the root SpdxDocument "Element" field is supposed to have a reference to all elements in the document, so it collects those for you and sets them

Ok, I think that's about it. Please report any issues, of course!

[jijo-OO7]

I’ve been exploring the v3 → v2.3 conversion approach and wanted to confirm a detail regarding handling fields that don’t have a direct v2.3 equivalent.

Since SPDX 3 introduces richer constructs (e.g., multi-valued fields, additional profiles like AI/Dataset, and extended relationship metadata), a strict one-to-one mapping to v2.3 isn’t always possible.

My current approach is to treat these as lossy conversions, but instead of silently dropping them, I plan to:

Track all such cases during conversion
Return them as a []string warnings alongside the converted document
Clearly indicate which fields or elements were omitted or reduced

This keeps the conversion transparent while still producing a valid v2.3 document.
Also curious if there’s any preference for handling cases where multiple v3 values must be reduced to a single v2 field.

/cc @goneall @kzantow @mlieberman85

[mlieberman85]

I think that approach makes sense, about returning as warnings.