records import: add missing nodes#4120
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the Arena Mobile records import job to “fix” imported records using RecordFixer, with the goal of inserting missing nodes (and generally normalizing imported record structures) during import.
Changes:
- Use
RecordFixer.fixRecordduringcleanupCurrentRecordinstead of custom node validation/cleanup logic. - Refactor
trackFileUuidsiteration fromforEachtofor...of.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| super(ArenaMobileDataImportJob.type, params, [new RecordsImportJob(), new FilesImportJob(), new RecordCheckJob()]) | ||
| } |
There was a problem hiding this comment.
RecordCheckJob is now added to the mobile data import pipeline, but (unlike the Collect import / publish flows) this pipeline does not run SurveyRdbCreationJob / RecordsUniquenessValidationJob afterwards. Since RecordCheckJob clears record-keys validation, this can leave imported/updated records without correct duplicate-keys validation. Either add a uniqueness validation step after the check (full or limited to recordUuidsIncluded), or adjust RecordCheckJob so it doesn’t clear record-keys validation when no follow-up uniqueness validation will run.
| if (updateRdb) { | ||
| await RecordManager.assocRefDataToNodes({ survey, nodes: allUpdatedNodesArray }, tx) | ||
| const { rdbUpdates } = RecordManager.generateRdbUpates({ survey, record, nodesArray: allUpdatedNodesArray }) | ||
| await this.rdbUpdatesBatchPersister.addItem(rdbUpdates) | ||
| } |
There was a problem hiding this comment.
updateRdb triggers RDB updates, but this job always calls _clearRecordKeysValidation(record) and validateNodesAndPersistValidation does not validate record key/unique-node uniqueness by default. When RecordCheckJob runs without a subsequent RecordsUniquenessValidationJob/SurveyRdbCreationJob (e.g. the mobile import flow), this can permanently clear existing duplicate-keys validation and never recompute it. Consider either (1) not clearing record-keys validation in this flow, or (2) explicitly re-running record uniqueness validation for the affected records after RDB updates are applied (or flush/apply RDB updates before validating), so record-keys validation remains correct.
|
No description provided.