Yale - Import data from external source to arches via UI

[azaroth42]

A tracker issue to help Rob / Yale better import data from one system into Arches via a UI.

In particular:

• A page should allow an authenticated user to enter a URL and LUX would then retrieve that record as JSON-LD and import it, with the existing UUID if possible
• The import process should also import the records that the initial one referenced by retrieving them individually
• It should allow the user to request a zip file of records from LUX to avoid the above repeated calls
• It should report errors to the user as to which records did not import, hopefully with an instructive error message as to why they didn't import
• The multi-import steps could be a workflow if that would be easier/better?

Bad example code:
example_code.zip

[whatisgalen]

@azaroth42 are you spec'ing this out as an etl import module?

[apeters]

• It should allow the user to request a zip file of records from LUX to avoid the above repeated calls

@azaroth42 At the moment we'll be focusing on just allowing users to upload a zip file of resources formatted with the same directory structure as the current json-ld loader uses from the command line.

Directory path pattern

outer dir
  - slug for graph
     - first 2 digits of uuid of resource instance
        - json ld files named with {uuid.json}

ie: files/person/04/04020230-0200-0000-020323-0023232923484.json

[apeters]

@azaroth42 Also, we'll assume that this operation either overwrites, or adds a new resource, like the current json-ld loader command does.

[azaroth42]

Yes, agreed re overwrite/add. Building the zip file we can work on. Doesn't seem too hard for the small number of records per interaction to build it only in memory.