Add --max-concurrent-downloads flag for parallel layer downloads #716

sbhavani · 2025-10-04T05:12:38Z

Adds --max-concurrent-downloads flag to container image pull for configurable concurrent layer downloads.

Fixes #715
Depends on apple/containerization#311

Usage:

container image pull nginx:latest --max-concurrent-downloads 6

Changes:

Add CLI flag (default: 3)
Thread parameter through XPC stack
Update to use forked containerization with configurable concurrency

Performance: ~1.2-1.3x faster pulls for multi-layer images with higher concurrency

Tests: Included standalone tests verify concurrency behavior and parameter flow

jglogan · 2025-10-07T19:19:04Z

test_concurrency.swift

@@ -0,0 +1,109 @@
+#!/usr/bin/env swift


Why is this and test_parameter_flow.swift in the top level directory?

should they belong in Tests/CLITests/Subcommands/Images/?

Yes, this might be a good place for the tests.

Package.swift

dcantah · 2025-10-10T00:55:28Z

Sources/ContainerClient/Core/ClientImage.swift

            guard err.isCode(.notFound) else {
                throw err
            }
-            return try await Self.pull(reference: reference, platform: platform, scheme: scheme, progressUpdate: progressUpdate)


I'm somewhat confused by the commit description however. I thought we had found we do pull layers in parallel, but that it's just a hardcoded 8.

sorry I originally thought it wasn't pulling in parallel but I noticed it was hardcoded later

updated original issue to more accurately reflect this: #715

Implements concurrent layer downloads to improve image pull performance by fetching multiple layers simultaneously instead of sequentially.

sbhavani · 2025-10-17T15:46:19Z

@dcantah anything else needed to merge this?

dcantah · 2025-10-21T05:17:52Z

@sbhavani Yes. We need a tag of Containerization that has the changes. I'm making some changes over there but should have a tag out within the next day or so and then we can bump to this. We should not have our dependency on Containerization be to a branch, and especially not main 😄

jglogan

@sbhavani Just a couple of nits if you still have time to move this forward. Thank you for contributing!

jglogan · 2025-11-13T18:58:20Z

Package.swift

        .package(url: "https://github.com/orlandos-nl/DNSClient.git", from: "2.4.1"),
        .package(url: "https://github.com/Bouke/DNS.git", from: "1.2.0"),
-        .package(url: "https://github.com/apple/containerization.git", exact: Version(stringLiteral: scVersion)),
+        .package(url: "https://github.com/apple/containerization.git", branch: "main"),


@sbhavani As Danny mentioned this needs to be reverted. If you rebase the PR, scVersion should be at the point where your ImageStore.pull() call will build without issue.

Yes, this has to be reverted, and scVersion should be updated to 0.11.0. Services/ContainerSandboxService/SandboxService.swift will need to be updated after that.

jglogan · 2025-11-13T19:04:04Z

Sources/ContainerClient/Flags.swift


+        public init(disableProgressUpdates: Bool, maxConcurrentDownloads: Int) {
+            self.disableProgressUpdates = disableProgressUpdates
+            self.maxConcurrentDownloads = maxConcurrentDownloads


What's the reason behind putting this in the progress group?

The problem I see here is that any command that wants to present the --progress flag (--disable-progress-updates is gone, apologies for letting this PR go so long so as to force a rebase here) will also see --max-concurrent-downloads whether downloads are relevant to the other command or not. Is there a better home for this? Or do we need to create another reusable group especially for download-related options?

I don't see any harm in copy/pasting this option into each command that needs it either, unless there's simply too many of them.

sbhavani · 2025-11-13T21:29:09Z

@sbhavani Just a couple of nits if you still have time to move this forward. Thank you for contributing!

no problem! I'll take a look and make the changes

dkovba

Thank you for the contribution 👍
The default value 3 seems reasonable.

dkovba · 2025-11-14T01:43:06Z

test_concurrency.swift

@@ -0,0 +1,109 @@
+#!/usr/bin/env swift


The project is currently using camelCase for filenames.

dkovba · 2025-11-14T01:55:35Z

Package.swift

        .package(url: "https://github.com/orlandos-nl/DNSClient.git", from: "2.4.1"),
        .package(url: "https://github.com/Bouke/DNS.git", from: "1.2.0"),
-        .package(url: "https://github.com/apple/containerization.git", exact: Version(stringLiteral: scVersion)),
+        .package(url: "https://github.com/apple/containerization.git", branch: "main"),


Yes, this has to be reverted, and scVersion should be updated to 0.11.0. Services/ContainerSandboxService/SandboxService.swift will need to be updated after that.

dkovba · 2025-11-14T01:57:27Z

Sources/TerminalProgress/ProgressTaskCoordinator.swift

+    /// Starts multiple concurrent tasks and returns them.
+    /// - Parameter count: The number of concurrent tasks to start.
+    /// - Returns: An array of ProgressTask instances.
+    public func startConcurrentTasks(count: Int) -> [ProgressTask] {


This method is unused and can be removed.

dkovba · 2025-11-14T01:58:02Z

Sources/TerminalProgress/ProgressTaskCoordinator.swift

+    /// - Parameters:
+    ///   - task: The task whose progress is being updated.
+    ///   - progressUpdate: The handler to invoke when progress updates are received.
+    public static func concurrentHandler(for task: ProgressTask, from progressUpdate: @escaping ProgressUpdateHandler) -> ProgressUpdateHandler {


This method is unused and can be removed.

dkovba · 2025-11-14T02:00:29Z

Sources/ContainerClient/Core/ClientImage.swift


        let insecure = try scheme.schemeFor(host: host) == .http
        request.set(key: .insecureFlag, value: insecure)
+        request.set(key: .maxConcurrentDownloads, value: Int64(maxConcurrentDownloads))


Should we check that maxConcurrentDownloads > 0?

dkovba · 2025-11-14T02:07:16Z

test_concurrency.swift

@@ -0,0 +1,109 @@
+#!/usr/bin/env swift


Yes, this might be a good place for the tests.

dkovba · 2025-11-14T03:38:37Z

Sources/TerminalProgress/ProgressTaskCoordinator.swift

+public struct ProgressTask: Sendable, Equatable, Hashable {
    private var id = UUID()
-    private var coordinator: ProgressTaskCoordinator
+    internal var coordinator: ProgressTaskCoordinator


If the unused methods are removed, coordinator can be kept private.

dkovba · 2025-11-14T03:40:16Z

Sources/TerminalProgress/ProgressTaskCoordinator.swift

 /// A type that coordinates progress tasks to ignore updates from completed tasks.
 public actor ProgressTaskCoordinator {
    var currentTask: ProgressTask?
+    var activeTasks: Set<ProgressTask> = []


It seems the concurrent download occurs inside a single task. Can we simplify the logic and remove activeTasks and hash()?

kmk142789 approved these changes Oct 6, 2025

View reviewed changes

jglogan reviewed Oct 7, 2025

View reviewed changes

dcantah marked this pull request as draft October 10, 2025 00:54

dcantah reviewed Oct 10, 2025

View reviewed changes

Package.swift Outdated Show resolved Hide resolved

dcantah reviewed Oct 10, 2025

View reviewed changes

sbhavani force-pushed the feature/parallel-layer-downloads branch from c843892 to cc15745 Compare October 14, 2025 06:21

sbhavani marked this pull request as ready for review October 14, 2025 06:23

sbhavani force-pushed the feature/parallel-layer-downloads branch from cc15745 to 26ebe63 Compare October 14, 2025 06:25

Add parallel layer downloads support for image pull operations

53e6c45

Implements concurrent layer downloads to improve image pull performance by fetching multiple layers simultaneously instead of sequentially.

sbhavani force-pushed the feature/parallel-layer-downloads branch from 26ebe63 to 53e6c45 Compare October 14, 2025 06:27

Merge branch 'main' into feature/parallel-layer-downloads

903beb8

jglogan requested changes Nov 13, 2025

View reviewed changes

dkovba requested changes Nov 14, 2025

View reviewed changes

dkovba reviewed Nov 14, 2025

View reviewed changes

Add --max-concurrent-downloads flag for parallel layer downloads #716

Are you sure you want to change the base?

Add --max-concurrent-downloads flag for parallel layer downloads #716

Conversation

sbhavani commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbhavani Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbhavani commented Oct 17, 2025

Uh oh!

dcantah commented Oct 21, 2025

Uh oh!

jglogan left a comment

Choose a reason for hiding this comment

Uh oh!

jglogan Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkovba Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbhavani commented Nov 13, 2025

Uh oh!

dkovba left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkovba Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sbhavani commented Oct 4, 2025 •

edited

Loading

sbhavani Oct 10, 2025 •

edited

Loading

jglogan Nov 13, 2025 •

edited

Loading

dkovba Nov 14, 2025 •

edited

Loading

dkovba left a comment •

edited

Loading

dkovba Nov 14, 2025 •

edited

Loading