🐛 Program exceptions uploaded before entity level exceptions results in an error

[hknahal]

Description

Uploading a program exception that overwrites existing program exceptions, and then uploading entity level exceptions returns an error. But uploading entity level exception first and then program exception works.

Steps to reproduce error

Example 1 is where an existing set of program exceptions is overwritten by new program exceptions, followed by attempt to upload entity level exception. This results in an error stating Program exception already exists, but this is not true because there is no program level exception on the field we're trying to make an entity level exception on. However, the field did exist previously as a program level exception before being overwritten by a new program exception, so it suggests there is some caching issue that is not cleared after program exceptions are overwritten.

Example 2 demonstrates scenario where the same exceptions files are used but entity level exceptions are uploaded first, followed by program exceptions. This does not result in any errors.

Example 1: Program has existing program exceptions. A new program exception is uploaded first overwriting the initial program exceptions, and then entity level exception is uploaded. This returns an error.

1. Initially TEST-INTL has program exceptions which includes a program exception for the percent_tumour_cells_measurement_method = Unknown (done during previous testing). Output from GET endpoint:

{
  "success": true,
  "exception": {
    "_id": "64874c4a25fec125ea2a51a6",
    "programId": "TEST-INTL",
    "exceptions": [
      {
        "_id": "64877039c741d100095bd181",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "treatment_setting",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "64877039c741d100095bd182",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "treatment_intent",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd183",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "is_primary_treatment",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd184",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd185",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "pathological_tumour_staging_system",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "64877039c741d100095bd186",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "pathological_stage_group",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "64877039c741d100095bd187",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "tumour_grading_system",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "64877039c741d100095bd188",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "tumour_grade",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "64877039c741d100095bd189",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "lymph_nodes_examined_status",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd18a",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "lymph_nodes_examined_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd18b",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "clinical_tumour_staging_system",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd18c",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "clinical_stage_group",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "64877039c741d100095bd18d",
        "program_name": "TEST-INTL",
        "schema": "chemotherapy",
        "requested_core_field": "chemotherapy_drug_dose_units",
        "requested_exception_value": "Unknown"
      }
    ],
    "__v": 0
  }
}

There are no entity-level exceptions for TEST-INTL at this point.

2. Upload new program exceptions file TEST-INTL_program_exceptions.csv which includes only an exception for treatment_setting = Not applicable. This means the initial program exceptions from Step 1 will be overwritten. Output from GET endpoint after program exceptions uploaded:

{
  "success": true,
  "exception": {
    "_id": "64874c4a25fec125ea2a51a6",
    "programId": "TEST-INTL",
    "exceptions": [
      {
        "_id": "64888166641a7a00099f62ca",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "treatment_setting",
        "requested_exception_value": "Not applicable"
      }
    ],
    "__v": 0
  }
}

Can see that program exception for percent_tumour_cells_measurement_method = Unknown is no longer in the list.

3. Now lets upload an entity-level exception for the "specimen" schema for the percent_tumour_cells_measurement_method = Unknown for 28 donors. First use GET endpoint to confirm that there are no entity-level exceptions for TEST-INTL:

{
  "success": true,
  "exception": {
    "_id": "64874c0e25fec125ea2a50ef",
    "programId": "TEST-INTL",
    "__v": 2,
    "specimen": [],
    "follow_up": []
  }
}

4. Next, upload the entity-level exception file: TEST-INTL_specimen_entity_exceptions.csv. This returns a 400 error message:

Program exception already exists

However, there are no program exceptions for percent_tumour_cells_measurement_method (see output from Step 2 above).

Example 2: Upload entity-level exceptions first, then program level exceptions (reverse of Example 1). This will not generate an error.

1. Start off with same program exceptions we had in Example 1 (Step 1). Upload this program exceptions file: TEST-INTL_program-exceptions_multiple_schema.csv. If you use the GET endpoint after, you will see the same output as Step 1 from Example 1 above.

{
  "success": true,
  "exception": {
    "_id": "6488bcc725fec125ea2e3de5",
    "programId": "TEST-INTL",
    "exceptions": [
      {
        "_id": "6488bfb5c741d100095be052",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "treatment_setting",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "6488bfb5c741d100095be053",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "treatment_intent",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be054",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "is_primary_treatment",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be055",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be056",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "pathological_tumour_staging_system",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "6488bfb5c741d100095be057",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "pathological_stage_group",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "6488bfb5c741d100095be058",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "tumour_grading_system",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "6488bfb5c741d100095be059",
        "program_name": "TEST-INTL",
        "schema": "specimen",
        "requested_core_field": "tumour_grade",
        "requested_exception_value": "Not applicable"
      },
      {
        "_id": "6488bfb5c741d100095be05a",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "lymph_nodes_examined_status",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be05b",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "lymph_nodes_examined_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be05c",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "clinical_tumour_staging_system",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be05d",
        "program_name": "TEST-INTL",
        "schema": "primary_diagnosis",
        "requested_core_field": "clinical_stage_group",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bfb5c741d100095be05e",
        "program_name": "TEST-INTL",
        "schema": "chemotherapy",
        "requested_core_field": "chemotherapy_drug_dose_units",
        "requested_exception_value": "Unknown"
      }
    ],
    "__v": 0
  }
}

2. Verify that there are no entity level exceptions for TEST-INTL using GET endpoint:

{
  "success": true,
  "exception": {
    "_id": "64874c0e25fec125ea2a50ef",
    "programId": "TEST-INTL",
    "__v": 2,
    "specimen": [],
    "follow_up": []
  }
}

3. Upload an entity-level exception for the "specimen" schema for the percent_tumour_cells_measurement_method = Unknown for 28 donors. TEST-INTL_specimen_entity_exceptions.csv. The GET endpoint shows this was successful:

{
  "success": true,
  "exception": {
    "_id": "64874c0e25fec125ea2a50ef",
    "programId": "TEST-INTL",
    "__v": 2,
    "specimen": [
      {
        "_id": "6488bc54641a7a00099f6aa3",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0002",
        "submitter_specimen_id": "8013222",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aa4",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0003",
        "submitter_specimen_id": "8013271",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aa5",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0006",
        "submitter_specimen_id": "8031121",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aa6",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0009",
        "submitter_specimen_id": "8012321",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aa7",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0009",
        "submitter_specimen_id": "8012323",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aa8",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0010",
        "submitter_specimen_id": "8014396",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aa9",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0014",
        "submitter_specimen_id": "8014837",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aaa",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0015",
        "submitter_specimen_id": "8015085",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aab",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0018",
        "submitter_specimen_id": "8015097",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aac",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0021",
        "submitter_specimen_id": "8015109",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aad",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0021",
        "submitter_specimen_id": "8015111",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aae",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0024",
        "submitter_specimen_id": "8012675",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aaf",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0025",
        "submitter_specimen_id": "8012209",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab0",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0025",
        "submitter_specimen_id": "8012211",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab1",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0026",
        "submitter_specimen_id": "8014825",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab2",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0026",
        "submitter_specimen_id": "8014827",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab3",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0031",
        "submitter_specimen_id": "8014807",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab4",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0031",
        "submitter_specimen_id": "8014809",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab5",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0032",
        "submitter_specimen_id": "8050103",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab6",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0033",
        "submitter_specimen_id": "8015141",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab7",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0033",
        "submitter_specimen_id": "8057559",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab8",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0037",
        "submitter_specimen_id": "8015259",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6ab9",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0039",
        "submitter_specimen_id": "8030150",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6aba",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0041",
        "submitter_specimen_id": "8013946",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6abb",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0042",
        "submitter_specimen_id": "8013014",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6abc",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0046",
        "submitter_specimen_id": "8015764",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      },
      {
        "_id": "6488bc54641a7a00099f6abd",
        "program_name": "TEST-INTL",
        "submitter_donor_id": "ICGC_0048",
        "submitter_specimen_id": "8014741",
        "schema": "specimen",
        "requested_core_field": "percent_tumour_cells_measurement_method",
        "requested_exception_value": "Unknown"
      }
    ],
    "follow_up": []
  }
}

Entity level exceptions for 28 donors.

4. Now upload new program exception for treatment_setting = Not applicable. This will overwrite the program exceptions from Step 1. This does not return an error. The GET endpoint shows the program exception upload is successful:

{
  "success": true,
  "exception": {
    "_id": "6488bcc725fec125ea2e3de5",
    "programId": "TEST-INTL",
    "exceptions": [
      {
        "_id": "6488c01e641a7a00099f6ac7",
        "program_name": "TEST-INTL",
        "schema": "treatment",
        "requested_core_field": "treatment_setting",
        "requested_exception_value": "Not applicable"
      }
    ],
    "__v": 0
  }
}

This means if you upload entity level exceptions first, and then program exceptions after, it will work and the program can have both program and entity level exceptions implemented. But if you upload program exceptions first and then entity level exceptions, an error will result that appears to be related to a caching issue.

Expected Result

Should be able to upload program level exceptions first and then entity level exceptions without generating an error (order of which exceptions uploaded first shouldn't matter).