Enforce data structure and data type consistency for JSON metadata #1421
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,10 +14,7 @@ | |
| ] | ||
| }, | ||
| "lang": { | ||
| "$ref": "../doi/language.json" | ||
svogt0511 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }, | ||
| "subject": { "type": "string" }, | ||
| "subjectScheme": { "type": "string" } | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
| "title": "Affiliation", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "object", | ||
| "properties": { | ||
| "affiliationIdentifier": { | ||
| "type": ["string", "null"] | ||
| }, | ||
| "affiliationIdentifierScheme": { | ||
| "type": ["string", "null"] | ||
| }, | ||
| "name": { | ||
| "type": ["string", "null"] | ||
| }, | ||
| "schemeUri": { | ||
| "type": ["string", "null"] | ||
| } | ||
| }, | ||
| "dependentRequired": { | ||
svogt0511 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| "affiliationIdentifier": ["affiliationIdentifierScheme"] | ||
| }, | ||
| "additionalProperties": false | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "title": "Affiliations", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "array", | ||
| "minItems": 0, | ||
| "items": { | ||
| "$ref": "affiliation.json" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "title": "AlternateIdentifier", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "object", | ||
| "properties": { | ||
| "alternateIdentifier": { | ||
| "type": "string" | ||
| }, | ||
| "alternateIdentifierType": { | ||
| "type": "string" | ||
| } | ||
| }, | ||
| "additionalProperties": false, | ||
| "dependentRequired": { | ||
| "alternateIdentifier": ["alternateIdentifierType"] | ||
|
||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "title": "AlternateIdentifiers", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "array", | ||
| "minItems": 0, | ||
svogt0511 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "items": { | ||
| "$ref": "alternate_identifier.json" | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| { | ||
| "title": "Contributor", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "object", | ||
| "properties": { | ||
| "name": { | ||
|
Collaborator
There was a problem hiding this comment. This is an interesting one. I was testing with metadata pulled from some recently created/updated DOIs, to see if this validation would have impacted them. I found this DOI which in the JSON has a contributor "contributors": [
{
"nameType": "Personal",
"givenName": "Jacopo",
"familyName": "Torrisi",
"affiliation": [],
"contributorType": "DataManager",
"nameIdentifiers": [
{
"nameIdentifier": "",
"nameIdentifierScheme": "ORCID"
}
]
}
],Using this metadata to create a DOI on staging failed with this error: {
"errors": [
{
"source": "contributors",
"title": "Object at `/0` is missing required properties: name",
"uid": "10.1111/742r-wc63"
}
]
}From my understanding of the XSD, <contributors>
<contributor contributorType="DataManager">
<contributorName nameType="Personal">Torrisi, Jacopo</contributorName>
<givenName>Jacopo</givenName>
<familyName>Torrisi</familyName>
<nameIdentifier nameIdentifierScheme="ORCID" schemeURI=""/>
<affiliation affiliationIdentifierScheme="ROR"/>
</contributor>
</contributors>How is that contributorName being generated for the XML? I am just thinking through the potential impact on user who are currently providing not providing
Collaborator
There was a problem hiding this comment. Tagging @codycooperross for input into this as well :)
Contributor
There was a problem hiding this comment. With JSON -> XML, we currently generate a contributorName and creatorName based on available familyName and givenName metadata if available. See the description here: https://docs.google.com/spreadsheets/d/1Hy0KXWPxqNx-Pfh-nNFxbsUFXXVYsO8O2sDIytXQv7U/edit?gid=1806954511#gid=1806954511&range=2:2 For the sake of backwards compatibility with existing request patterns and scoping this PR, let's remove the
Contributor
Author
There was a problem hiding this comment. This was always very confusing to me. As I understand it, contributorName is always required. If nameType = Unknown then Contributor can have a given name, family name and a completely independent name. If nameType = Organization then Contributor has any name you want to give it. (**Although, if you are converting the type to organization to one of the other types, givenName and familyName are preserved in the DB (and probably in elasticSearch. A topic for another time.) If nameType = Person, the Name field is generated from the givenName and familyName. I would not remove the name requirement. I would not back out our requirements to fit incorrect data. I thought that if there are incorrect dois in the DB, that the user would be prodded to correct them if/when the doi is updated, otherwise, there is not much we could do, except perhaps, to flag them as being incorrect at some point. Does this make sense? |
||
| "type": "string" | ||
| }, | ||
| "nameType": { | ||
| "$ref": "controlled_vocabularies/name_type.json" | ||
| }, | ||
| "givenName": { | ||
| "type": ["string", "null"] | ||
| }, | ||
| "familyName": { | ||
| "type": ["string", "null"] | ||
| }, | ||
| "contributorType": { | ||
| "oneOf": [ | ||
| { | ||
| "$ref": "controlled_vocabularies/contributor_type.json" | ||
| }, | ||
| { | ||
| "type": "null" | ||
| } | ||
| ] | ||
| }, | ||
| "lang": { | ||
|
Contributor
There was a problem hiding this comment. I don't think It doesn't look like it's populated in the API: https://api.datacite.org/dois?query=contributors.lang:*
Collaborator
There was a problem hiding this comment. That's interesting, it should be allowed. In the XML, it is attached to contributorName: https://datacite-metadata-schema.readthedocs.io/en/4.6/properties/contributor/#id2 There is
Contributor
There was a problem hiding this comment. It looks like we do accept this currently in ParamsSanitizer: lupo/app/lib/params_sanitizer.rb Line 161 in e2556e3 Not sure if it's mapped in OpenSearch or to XML. But I'll close this then. |
||
| "$ref": "language.json" | ||
| }, | ||
| "affiliation": { | ||
| "$ref": "affiliations.json" | ||
| }, | ||
| "nameIdentifiers": { | ||
| "$ref": "name_identifiers.json" | ||
| } | ||
| }, | ||
| "additionalProperties": false, | ||
| "required": [ | ||
| "name" | ||
|
Contributor
There was a problem hiding this comment.
|
||
| ] | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| { | ||
| "title": "Contributors", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "array", | ||
| "minItems": 0, | ||
| "items": { | ||
| "$ref": "contributor.json" | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| { | ||
| "title": "ContributorType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": ["string", "null"], | ||
| "anyOf": [ | ||
| { | ||
| "type": "string", | ||
| "enum": [ | ||
| "ContactPerson", | ||
| "DataCollector", | ||
| "DataCurator", | ||
| "DataManager", | ||
| "Distributor", | ||
| "Editor", | ||
| "HostingInstitution", | ||
| "Producer", | ||
| "ProjectLeader", | ||
| "ProjectManager", | ||
| "ProjectMember", | ||
| "RegistrationAgency", | ||
| "RegistrationAuthority", | ||
| "RelatedPerson", | ||
| "Researcher", | ||
| "ResearchGroup", | ||
| "RightsHolder", | ||
| "Sponsor", | ||
| "Supervisor", | ||
| "Translator", | ||
| "WorkPackageLeader", | ||
| "Other" | ||
| ] | ||
| }, | ||
| { | ||
| "type": "null" | ||
| } | ||
| ] | ||
coderabbitai[bot] marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| { | ||
| "title": "DateType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": ["string", "null"], | ||
| "anyOf": [ | ||
| { | ||
| "type": "string", | ||
| "enum": [ | ||
| "Accepted", | ||
| "Available", | ||
| "Copyrighted", | ||
| "Collected", | ||
| "Coverage", | ||
| "Created", | ||
| "Issued", | ||
| "Submitted", | ||
| "Updated", | ||
| "Valid", | ||
| "Withdrawn", | ||
| "Other" | ||
| ] | ||
| }, | ||
| { | ||
| "type": "null" | ||
| } | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| { | ||
| "title": "DescriptionType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "string", | ||
| "enum": [ | ||
| "Abstract", | ||
| "Methods", | ||
| "SeriesInformation", | ||
| "TableOfContents", | ||
| "TechnicalInfo", | ||
| "Other" | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "title": "FunderIdentifierType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": "string", | ||
| "enum": [ | ||
| "Crossref Funder ID", | ||
| "GRID", | ||
| "ISNI", | ||
| "ROR", | ||
| "Other" | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| { | ||
| "title": "NameType", | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "type": ["string", "null"], | ||
| "anyOf": [ | ||
| { | ||
| "type": "string", | ||
| "enum": ["Organizational", "Personal"] | ||
| }, | ||
| { | ||
| "type": "null" | ||
| } | ||
| ] | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: datacite/lupo
Length of output: 1609
Remove
byebugrequire from model code.Requiring debugger gems in production model code causes boot failures in environments where the gem is not installed and is a release blocker. The
require "pp"should also be removed—standard library utilities should not be required at class definition time.Fix
📝 Committable suggestion
🤖 Prompt for AI Agents