Implement S3 Lifecycle Policy for Temporary Audio Cleanup and Error Handling (Fixes #165) #169
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implement automatic cleanup of temporary audio files by: - Add S3 lifecycle policy to delete objects after 1 day - Add cleanup for aborted multipart uploads after 1 day - Create new S3Service class for better S3 operations encapsulation - Improve error handling and logging for file cleanup - Add cleanup in finally block to ensure temporary files are removed - Keep lifecycle policy as fallback for failed manual deletions Fixes GroupLang#165
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Title: Implement S3 Lifecycle Policy for Temporary Audio Cleanup and Error Handling
Related Issue: Fixes #165
Summary:
This pull request addresses the issue of orphaned audio files in our S3 storage following failed transcription attempts. The implementation introduces an S3 lifecycle policy and enhances error handling mechanisms to ensure that temporary audio files are managed efficiently, ultimately resolving Issue #165.
Key Changes:
Lifecycle Policy Implementation:
Error Handling Improvements:
transcribe_audiofunction by utilizing afinallyblock to ensure that temporary files are deleted regardless of the success or failure of the transcription process. This guarantees that even if a transcription attempt fails or if the process is interrupted, no orphaned files will remain.Multipart Upload Cleanup:
Logging Enhancements:
Outcome:
The changes made in this pull request effectively address the challenges related to unnecessary storage costs, potential security risks, and compliance with data retention policies. By ensuring that temporary audio files do not remain indefinitely in S3, we optimize our storage system and maintain better control over our resources.
All necessary changes have been implemented and tested successfully. Thank you for your attention to this important enhancement!
Please review and merge at your earliest convenience.