Periodic deletion of Bulkrax /imports directory

[eporter23]

In order to prevent Curate from running out of disk space (1/18/23 occurrence), we need to expand our deletion/cleanup jobs to periodically to clean the /imports directory where we upload files to be used for Bulkrax ingests.

To investigate:

1. What is stored in the /working directory and if it should be periodically cleared
2. How to clear /imports which are stored per release

We currently have an AWX job that clears out the /tmp directory:

• Curate-Prod EBS Local Volume Subdirectory Clean

Recommended frequency: same as the above job

[bwatson78]

Questions:

• For @jcrompton42 or @kbowaterskelly Is /tmp on the local or the mount?
• For @eporter23, moving /imports would mean broken paths to old imports, is that ok?

[jcrompton42]

/tmp is local.

[eporter23]

Yes, that is fine, but will that impact @jcrompton42 's file copy process to move files from EFS to /imports for Bulkrax? See #1985

[bwatson78]

It would just need to be updated, I think/hope.

[jcrompton42]

What @bwatson78 said, when the /imports location changes, can someone make me a ticket to update that?

[bwatson78]

@jcrompton42 Yup, I will.

[bwatson78]

Notes:

• Removing the ENV of 'WORKING_PATH' would change /working to the default of /tmp/uploads, which currently doesn't exist.
• If I unset config.import_path = 'imports' in bulkrax' initializer, it will default out the import folder to /tmp/imports.

[bwatson78]

@rotated8 Knowing all of the info above, should I move forward with bringing both to /tmp in Arch as a trial run?

[eporter23]

@bwatson78 tested work in Curate v2.7.0 by running an export from test and a manually uploaded zip file import into arch. Import worked fine. Here's the path that was generated:

[eporter23]

@bwatson78 related to the above comment, when I exported from test I noticed the CSV had a couple of those extraneous multi-valued field columns again for abstract and publisher.

[eporter23]

@bwatson78 @jcrompton42 do we need another ticket related to updating the EFS to Curate file copy job? (#1985)

[jcrompton42]

I updated the file copy job so we should be good on that.

[eporter23]

Notes from Slack discussion 10/4/23: this is implemented in prod, but each release has its own tmp/imports folder so as we do new releases we leave some stuff hanging out in imports. This means that each time we deploy to prod, there's a separate version with its own /tmp directory. Keeping this open until we determine next steps, which will be a separate ticket.