Skip to content

[PB-5942]: feat/better-concurrency-handling#31

Merged
jzunigax2 merged 1 commit into
masterfrom
feat/better-concurrency-handling
Feb 26, 2026
Merged

[PB-5942]: feat/better-concurrency-handling#31
jzunigax2 merged 1 commit into
masterfrom
feat/better-concurrency-handling

Conversation

@jzunigax2
Copy link
Copy Markdown
Collaborator

@jzunigax2 jzunigax2 commented Feb 20, 2026

Implements fs.OpenChunkWriter so rclone controls upload concurrency and buffering,
fixing OOM kills on large file uploads (rclone#9162).

  • Files >= chunk_size use multipart upload via lib/multipart.UploadMultipart()

  • Smaller files continue using single-part UploadFileStreamAuto()

  • All encryption stays in the SDK — zero crypto/* imports in the backend

    New options

    Option Default Description
    --internxt-chunk-size 30Mi Chunk size for multipart uploads
    --internxt-upload-concurrency 4 Concurrent chunks per file

@jzunigax2 jzunigax2 requested a review from larryrider February 20, 2026 21:31
@jzunigax2 jzunigax2 self-assigned this Feb 20, 2026
@jzunigax2 jzunigax2 force-pushed the feat/better-concurrency-handling branch from 1ab1ea0 to be89bf6 Compare February 23, 2026 20:31
@jzunigax2 jzunigax2 force-pushed the feat/better-concurrency-handling branch from be89bf6 to 7c01911 Compare February 23, 2026 21:28
@jzunigax2 jzunigax2 marked this pull request as ready for review February 24, 2026 01:56
@jzunigax2 jzunigax2 changed the title Feat/better-concurrency-handling [PB-5942]: feat/better-concurrency-handling Feb 24, 2026
@giovariot
Copy link
Copy Markdown

Niiiiice :) when could we expect this to be integrated into the official rclone? Thanks a lot

@jzunigax2
Copy link
Copy Markdown
Collaborator Author

hey @giovariot, I'd hope to get this in for rclone's next 1.74 release. I'll be submitting a PR there after this goes through some internal QA

@jzunigax2 jzunigax2 merged commit b6534d1 into master Feb 26, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants