Make Sip is not updating the narthex_facts.txt and mapping.xml

[kiivihal]

The values from the GUI are only used during the initial creation of the SipZip. On each subsequent 'make sip' only the schema version is updated.

So the facts need to be updated in:

• narthex_facts.txt
• mapping_{recdef}.xml

The sip-core and narthex already has the hooks to start updating these. They just need to be added to the switch statement in 'make sip'.

[kiivihal]

The meta-information stored in the TripleStore by Narthex is not being updated in the SipZip after its initial creation. This document contains the background information need to address this issue.

The SipZips are stored in 'NarthexFiles/{orgId}/sips/'. They are created by the 'MakeSip' process.

A SipZips contains the following files:

• record-definition, e.g. => edm_5.2.6_record-definition.xml
• validation XSD, e.g. => edm_5.2.6_validation.xsd
• a hints file, e.g. => hints.txt
• the source data wrapped in delving pockets, e.g. => source.xml.gz
• the actual mapping => mapping_edm.xml
• facts used by mapping, e.g. => narthex_facts.txt

The first four files - record-definition, validation, source and hints - are correct.

The confusing aspect of the facts and mapping is that data is being duplicated - probably for historic reasons. However, during processing only the facts from within the mapping file is used.

So below the fields that should be updated in the narthex_facts.txt:

provider=Rijksdienst voor het Cultureel Erfgoed
name=Collectie Schraven
schemaVersions=edm_5.2.6
orgId=devorg
rights=http://creativecommons.org/licenses/by-nc/3.0/nl/
baseUrl=http://devops.localhost:
* type=IMAGE
spec=coll-schraven
* country=Netherlands
dataProvider=Collectie Schraven
language=nl

The fields marked with the asterisk are legacy and either cannot or should not be updated.

The remainder of the fields either come from the narthex.conf or the dataset metadata (in Fuseki). There is code that generates this on initial creation of the dataset/ initial MakeSip. This now needs to be run each time MakeSip is run.

The mapping is a bit more tricky. Here the facts are part of the XML, see for example:

  <facts>
    <entry>
      <string>country</string>
      <string>Netherlands</string>
    </entry>
    <entry>
      <string>schemaVersions</string>
      <string>icn_1.0.4, ese_3.4.0, lido_1.0.2</string>
    </entry>
    <entry>
      <string>baseUrl</string>
      <string>http://devorg.localhost</string>
    </entry>
    <entry>
      <string>provider</string>
      <string>Rijksdienst voor het Cultureel Erfgoed</string>
    </entry>
    <entry>
      <string>rights</string>
      <string>http://creativecommons.org/licenses/by-nc/3.0/nl/</string>
    </entry>
    <entry>
      <string>name</string>
      <string>Collectie Schraven</string>
    </entry>
    <entry>
      <string>language</string>
      <string>nl</string>
    </entry>
    <entry>
      <string>dataProvider</string>
      <string>Collectie Schraven</string>
    </entry>
    <entry>
      <string>type</string>
      <string>IMAGE</string>
    </entry>
    <entry>
      <string>spec</string>
      <string>coll-schraven</string>
    </entry>
    <entry>
      <string>orgId</string>
      <string>devorg</string>
    </entry>
  </facts>

In the narthex code currently only the 'schemaVersions' is updated. The good news is here that sip-core contains a function to updated the facts, so no custom parsing and updating of the XML should be required.

The same fields should be updated that are updated in the 'narthex_facts.txt'

So to recap:

• hook creation/updating of 'narthex_facts.txt' and 'mapping_edm.xml' to each time MakeSip is run and not only the first time.
• add notice to the interface in the 'metadata' tab that this information is only available in the mapping after 'MakeSip' is run
• write narthex_facts.txt each time and get all the information
• update facts section in mapping_edm.xml

[kiivihal]

The easiest way to reproduce this issue is:

• create a dataset for the myfirst.sip.zip test data
• drop the SipZip
• generate the new SipZip
• download it in the SipCreator
• review the facts info in the 'mapping_edm.xml' and the 'narthex_facts.txt'
• change the metadata in the narthex metadata form
• save this data
• generate a new SipZip
• download it in the SipCreator
• verify that the new facts information is not updated in the 'mapping_edm.xml' and the 'narthex_facts.txt'

There is also some information in facts that comes from the 'narthex.conf' such as the 'orgId' and the 'rdfBaseUrl'. These should also be updated when a new SipZip is created. This should be easier to test:

1. start narthex
2. create SipZip
3. download in the SipCreator and verify in the mapping and facts file
4. stop narthex
5. update the 'rdfBaseUrl' and the 'orgId'
6. repeat steps 1-3 to see if the correct information is available.