5758 - Navigation: Data errors

[yaguzmang]

Table validations

• Server-side table node validation

  • Validate table nodes per assessment/cycle/country and cache results in Redis
  • Revalidate edited nodes together with calculated dependents
  • Emit table validation updates to the frontend after node/dependency updates

• Frontend table validation flow

  • Fetch table validations when initializing table data
  • Remove client-side table validation
  • Remove frontend fetching of validation dependency table data
  • Show table validation errors in cells

• Validation summary and submission warning

  • Build validation summary for sections/subsections/tables
  • Show validation error indicators in the nav
  • Warn users before submitting to review

• Test table validations

  • Test cross-cycle validation when editing forest area
  • Test clearing validation errors after fixing data
  • Test updating dependent table validation errors after editing source data
  • Test showing validation errors on initial load
  • Test hiding validation UI on the print route
  • Test revalidating dependent max-area nodes after editing source data

Description validations

• Validate description text

  • Queue link validation after a user edits description text with links
  • Process edited description links in the verify-links worker
  • Store visited description links and locations in the links table
  • Update Redis description-link validations after single-description validation
  • Update Redis description-link validations after full-country/full-assessment link checks
  • Emit description text validation update events
  • Listen for description text validation events in the frontend
  • Rebuild the validation summary in the client when description text validations change
  • Include description text errors in validation summary
  • Show description text link validation errors in the description UI

• Validate data sources

  • Required fields

    • Validate data source fields that are required
    • Store fields validation results in Redis
    • Return/update required-field validation results in the frontend after save
    • Include data source errors in validation summary (backend and frontend)
    • Replace client side errors with server-side validation results
    • Show field validation errors in the nav indicators
    • Include field errors in the submit-to-review warning

  • Reference links

    • Queue reference-link validation after a user saves data sources with reference links
    • Validate reference links through the async links worker
    • Store visited reference links and locations in the links table
    • Clear stale reference-link locations/errors when a reference link is removed
    • Store reference-link validation results in Redis after link validation completes
    • Update Redis reference-link validations after single-description/data-source validation
    • Update Redis reference-link validations after full-country/full-assessment link checks
    • Emit reference-link validation update events
    • Fetch data source validations when initializing section descriptions
    • Listen for reference-link validation events in the frontend
    • Rebuild the validation summary when reference-link validations change
    • Show reference-link validation errors in the data sources UI

• Trigger links validation after approving/disapproving a link that is in a description

• In the links worker, use section name + description name instead of description ids

• In validateAll, validate also descriptions

• To see if we do:

  • Split validation reducer

• Test description validations

  • Add tests for description text link validation flow
  • Add tests for data source validation flow

ODP validations

• BE: Validate all ODPs and store them in redis
• BE: Include ODP validation in validation summary
• BE: Validate Year on change
• BE: Validate national class on change
• BE/FE: Send validation to FE
• FE: Show validation errors
• BE: update validation summary on ODP validation update
• BE: send validation summary update to FE
• BE: Delete NDP validation on NDP deletion
• BE/FE: Validate Comments and Reference to data source
• Update visitCycleLinks so it updates validation cache like the new description and ndp link jobs
• Fix unreachable columns and variables when validating after toggling "Use national data points"
• Divide 1a/1b NDP errors, so they affect the relevant section only
• Make the summary ignore 1b NDP errors if 1b NDP is disabled
• Make the summary ignore 1a datasources if 1a NDP is enabled and visible
• Make the summary ignore 1b datasources if 1b NDP is enabled and visible
• Add a tool to validate links

From Miró e Cosimo

• ValidateCountry[XXX] (e.g. Descriptions) to be moved under tools (purpose: keep controller clean).

• Split Validation reducer -> tableValidationReducer, descriptionValidationReducer ...

• DescriptionValidation:

  • Only call the related method to validate correct description.
  • E.g. when editing description, dont call datasourcevalidation (because we are not editing data sources)
  • src/server/controller/cycleData/validations/descriptions/updateDescriptionValidations.ts:22

• src/server/controller/cycleData/validations/descriptions/updateDescriptionValidations.ts:

  • Move to DataValidationService.validateDescriptions.
  • Remove redundant updateDescriptionLinkValidations.ts line 21 (inline the code).
  • Call updateDataSourceFieldValidations only if needed, which is renamed to validateDataSources.

• visitDescriptionLinks:

  • Move it to LinkService.validateLinks.
  • Why do we call visitDescriptionLinks, and what is the difference with triggerVerifyLinksWorker?

• Revisit DescriptionIdentifier type — maybe use Pick and revisit name.

• context factory for validations:

  • why do we run two iterations in #initQueue

• src/server/controller/cycleData/validations/tables/updateTableValidations.ts

  • Move to DataValidationService.validateNodes()

• _removeValidation → TableValidations.remove(props)

• move src/server/controller/cycleData/validations/descriptions/notifyDescriptionValidationUpdate.ts
  under src/server/service/dataValidation and expose it DataValidationService.notifyDescriptionValidationUpdate

• remove src/server/controller/cycleData/validations/nationalDataPoint and expose methods in DataValidationService

  • DataValidationService.validateNDPYear
  • DataValidationService.validateNDPNationalClasses

• in src/server/controller/cycleData/nationalDataPoint/remove.ts
  line 83-89 must be grouped in DataValidationService.removeNDPValidation

• remove completely ValidationsController. all exposed methods should be exposed from DataValidationService

• src/tools/validations/nationalDataPoint/validateCountryNationalDataPoints.ts

  • Why do we get current validations if the object here is to validate completely all NDP?
    here we should remove all existing validations and validate from scratch all NDP

• src/tools/validations/tables/validateCountryTables.ts
  line 15: expose the method in DataValidationService.removeTableValidations and rename repos method TableValidationRedisRepository.removeValidations

• Validation repositories should only be used in DataValidationService.

  • Places where they are not:
    • src/server/worker/tasks/verifyLinks/visitDescriptionLinks/utils/refreshDescriptionValidations.ts what is this?
    • src/server/worker/tasks/verifyLinks/visitNationalDataPointLinks/utils/refreshNationalDataPointValidations.ts what is this?

• NDP Validation methods:
  each validation should validate, set and notify client (not get validation first, but return the validation from the repo)

  e.g for national classes:

  1. const validation = NationalDataPointValidator.validateNationalClasses({..})
  2. const validations = NationalDataPointValidationRedisRepository.setNationalClassValidations({...,validation})
  3. notifyNationalDataPointValidationUpdate({...validation})

• To see if we do:

  • Lazy load data and validations slices in country page?
  • Update NDP table data with ODP UUID?