Add JSON-LD bulk import module #10798 #10885
Conversation
The tiles triggers can't be disabled during a transaction.
There was a problem hiding this comment.
I'm not seeing any error details when the load fails even though I can see more info in the db.
In the UI:
From the load_errors table:
It would be nice if the error message showed the actual name of the .json file that triggered the error instead of just the "block". Additionally I notice that the errors shown in the UI specify Validation Errors, but the errors I'm seeing are really load errors. Maybe we need both, but for load errors it may be enough to show the json file that errored and it's corresponding error message.
Also, it would be nice if users could opt to pass the --ignore-errors flag so that all resources that pass are loaded to the db and the ones that don't pass log errors in the ui.
|
Thanks, good notes. The current error UX looks like this... 
... and clicking "Full error report" gives: 
I understood load errors and validation errors to be synonyms in this context. What was the distinction you had in mind?
This is an interesting idea, but I think the whole interface of the bulk data manager assumes that you're not going to perform a load unless everything passes. In the scenario you describe, you'd have a partial success / partial failure state. I don't think the bulk data manager will support that. |
|
PS -- I could copy over the |
arches/app/templates/views/components/etl_modules/base-import.htm
Outdated
Show resolved
Hide resolved
| def stage_files(self, files, summary, cursor): | ||
| for file in files: | ||
| self.stage_excel_file(file, summary, cursor) |
There was a problem hiding this comment.
This seems like such a small (unless I'm missing something) but nice refactor that I think we should just do this refactor now.
Allows logging to the correct place in a request-response cycle when called in a view.
|
Solved the magic string and the column names by just overwriting the template block 👍
|
|
The validation error table looks like this now (with the first "Details" link expanded): 
|
serialized_rollback is the doc'd way to ensure the data from the initial migration is present in a TransactionTestCase. The little gotchas: - setUpClass() runs before the serialized data is restored, so it shouldn't do db stuff - signals shouldn't create other related objects if raw=True
There was a problem hiding this comment.
This looks great @jacobtylerwalls . Thanks for all those last minute UI tweaks.
Types of changes
Description of Change
Add a bulk import module that calls the CLI loader for JSON-LD.
Add some hooks to BaseImportModule to make it more extensible for things like:
Issues Solved
Closes #10798
Checklist
Ticket Background
Further comments
Follow-up tickets I found as part of this work:
__get_nodegroup_tree#10889Testing instructions
I tested with the sample models in the unit tests. I opened a django shell, ran most of
arches.tests.importer.jsonld_import_tests.JsonLDImportTests.setUpClass(), hacking bits in/out as needed, and then manually created some resources of the various models, viewed the JSON-LD at the /resources route, and zipped that up according to the instructions in #10798.