Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
e343196
WIP: Enable bucket versioning
nss10 Jul 7, 2026
efdeabb
[UNTESTED]Add initial s3files setup + separate out s3 files into a ne…
nss10 Jul 21, 2026
41926ab
Update EKS security group filter to use EKS group name
nss10 Jul 21, 2026
6e9abd5
Update docstring
nss10 Jul 21, 2026
6e368cd
Add a TODO comment regarding security group creation, maybe can reduc…
nss10 Jul 21, 2026
f4f2b75
Fix minor bugs and add more TODOs.
nss10 Jul 21, 2026
0248c3d
Add file system status implementation and waiter
nss10 Jul 22, 2026
e172079
Update import path
nss10 Jul 22, 2026
73ad87c
Move boto3 clients to a different file to avoid circular imports
nss10 Jul 22, 2026
1ac0aa4
Update exception type
nss10 Jul 22, 2026
f363791
Fix get_s3_files_system and add/update TODO comments
nss10 Jul 22, 2026
15182e5
Create dynamic IAM role for S3Files
nss10 Jul 23, 2026
fbc7545
Add STORAGE_TYPE config variable, add documentation for s3files
nss10 Jul 23, 2026
80cd425
Make S3Files a configurable setup
nss10 Jul 23, 2026
0eaf83e
Merge branch 'master' into feat/s3Files
nss10 Jul 23, 2026
2cf8c10
Make mount target provisioning a bgTask + Update docs
nss10 Jul 23, 2026
fc32d15
Update comments
nss10 Jul 23, 2026
bb84de3
Add missing function call.
nss10 Jul 23, 2026
df89174
Add more comments
nss10 Jul 24, 2026
00f569f
Re-organize aws_utils into aws.bucket, aws.aws_utils, aws.clients. (T…
nss10 Jul 24, 2026
3f56313
Fix patching in unit tests. All current tests pass
nss10 Jul 24, 2026
28614f8
Add more TODOs
nss10 Jul 24, 2026
aeb3071
Add unit tests for S3Files
nss10 Jul 27, 2026
3dd0630
Add fixes related to review comments
nss10 Jul 29, 2026
db7fd5e
Fix lint errors in unit tests
nss10 Jul 29, 2026
44e8406
Fix more PR suggestions and add more unit tests
nss10 Jul 29, 2026
039af05
Merge branch 'master' of https://github.com/uc-cdis/gen3-workflow int…
nss10 Jul 29, 2026
fed3b31
Fix lint issues
nss10 Jul 29, 2026
c6d7202
Make security group field configurable and add docs on how to fetch them
nss10 Jul 30, 2026
1e45803
Update the link in config.py
nss10 Jul 30, 2026
6f9c69e
remove TODO comment
nss10 Jul 30, 2026
bc4e9b8
Remove unused imports
nss10 Jul 30, 2026
8385645
S3 files `_create_file_system` hotfix
nss10 Jul 30, 2026
8e15edf
Merge branch 'master' of https://github.com/uc-cdis/gen3-workflow int…
nss10 Jul 30, 2026
26dcbb0
Fix create alias AlreadyExistsException
paulineribeyre Jul 30, 2026
7f0a7db
Update docstring
paulineribeyre Jul 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions gen3workflow/aws/bucket.py
Original file line number Diff line number Diff line change
Expand Up @@ -421,8 +421,10 @@ async def create_user_bucket(user_id: str) -> Tuple[str, str, str]:
Wrapper for `_create_user_bucket` that handles caching and retries.

Gracefully handles race conditions, for example:
`An error occurred (OperationAborted) when calling the PutBucketEncryption operation:
A conflicting conditional operation is currently in progress against this resource.`
- `An error occurred (OperationAborted) when calling the PutBucketEncryption operation:
A conflicting conditional operation is currently in progress against this resource.`
- `An error occurred (AlreadyExistsException) when calling the CreateAlias operation:
An alias with the name XYZ already exists`
"""
if USER_BUCKET_CACHE.has(user_id):
return USER_BUCKET_CACHE.get(user_id)
Expand All @@ -437,7 +439,8 @@ async def create_user_bucket(user_id: str) -> Tuple[str, str, str]:
return bucket_info
except ClientError as e:
if (
e.response["Error"]["Code"] != "OperationAborted"
e.response["Error"]["Code"]
not in ["OperationAborted", "AlreadyExistsException"]
or attempt == max_tries
):
raise
Expand Down
4 changes: 2 additions & 2 deletions gen3workflow/aws/s3_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@ def _create_s3_files_system(bucket_name: str, role_arn: str) -> str:
roleArn=role_arn,
tags=[
{
"Key": "Name",
"Value": get_safe_name_from_hostname(user_id=None),
"key": "Name",
"value": get_safe_name_from_hostname(user_id=None),
}
],
)
Expand Down
2 changes: 1 addition & 1 deletion tests/test_s3_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ def test_create_s3_files_system_success(mock_aws_services):
bucket="arn:aws:s3:::test-bucket",
prefix="funnel-temp-files/",
roleArn="arn:aws:iam::123456789012:role/s3files-role",
tags=[{"Key": "Name", "Value": "gen3wf-localhost"}],
tags=[{"key": "Name", "value": "gen3wf-localhost"}],
)


Expand Down
Loading