Data Release to File Repository

[rosibaj]

https://wiki.oicr.on.ca/pages/viewpage.action?spaceKey=icgcargotech&title=Data+Release

• Step 1: Initialize the the File Service record/tags for all existing files. Initialize the the File Service record/tags for all existing files. files-service#53
  -------> has all files tagged with Release state public
  -------> otherwise, new files get OWN_PROGRAM
• Step 2: Have file service take over publishing file data (without maestro!) Have file service take over publishing file data (without maestro!) files-service#54
  --- Index must include RELEASE_STAGE file.
  --- When we have fs take over, we will replace teh public index (exact same as now)
  --- Publish the same current index, with file-service, but now has the release_stage in index. complete backup of this index as well.
  ------- At a program level, PUBLIC will go to PUBLIC index, [OWNPROGRAM,FULL,ASSOCIATE] goes to an EMBARGO index
  2.1 File service needs to be able to responsd to Song publish, unpublish, and suppress.

>>> Once this is done, the index will have release stage. This unblocks:

• the aggregation facet can be released
• the download proxy feature for song/score can be released
• the download button from the platform from file repo/file page can go forward.

• Step 3: Modify the mapping to add clinical data to the mapping(s). Put fake data in QA to develop against.
• confirm the shape of the mapping with all current fields: Confirm Platform File_Centric Mapping  argo-metadata-schemas#43
• create a set of fake data to work with: Test data for Development Unblocking: Put Fake data in QA to develop against. #741

>>> Once this is done, the index will have release stage. This unblocks:

• File Entity Page development
• Donor Entity Page Development

• Step 4: Work on FS Release Endpoints. Start a new release, give release summary, confirm/publish release. Release Endpoints: Needs to be Broken Down files-service#55

• Step 5: Manual task: Analysisid (and file list) from Bioinfo/Rosi to Jon to set to QUEUED FOR RELEASE file tag in file service.

>>> Once this is done, the index will have release stage. This unblocks:

Mutect 2 release

• Step 6: Add clinical indexing to the file_centric index.
  -- get data ergonomically out of clinical per program/per donor: Get Donor API argo-clinical#442
  -- index the data out of clinical into the indices. Design: Index Clinical Data with file data  files-service#56,
  -- add new data to to platform-api schema: Feature Request: Update/add queries with donor-clinical information  platform-api#380

• Step 7: Event based clinical data updates [only for embargo'd data]
  -- clinical sends events when a program has new data submitted: Accept clinical update event from Kafka files-service#25, Issue Kafka event from clinical to file-manager argo-clinical#601
  -- file-service listen to events from kafka from clinical to update embargo indices with latest clinical data. Don't update public donor data.

• Step 8: Other needed Actions
  -- Event synthesizer: Reindexing: Create Events synthesizer  #586
  -- re- index when tags updated: File service needs to issue and event to reindex file docs when labels updated files-service#22
  -- re-index when new repository added: Reindex document when a repository is updated in file-service files-service#24