Skip to content

Conversation

arthurbdiniz
Copy link

@arthurbdiniz arthurbdiniz commented Aug 29, 2025

This commit introduces parallel package building to ratt.

  • New flag -parallel enables parallel builds.
  • New flag -jobs controls the number of concurrent workers (default: number of CPU cores, must be > 0).
  • Added buildPackagesSequential and buildPackagesParallel helpers.
  • Refactored main build loop to support sequential or parallel execution.

This allows faster builds by utilizing multiple CPU cores.


Example:

~/go/bin/ratt -parallel golang-github-go-logr-zapr_1.3.0-1_amd64.changes
2025/09/12 17:09:43 Loading changes file "golang-github-go-logr-zapr_1.3.0-1_amd64.changes"
2025/09/12 17:09:43  - 1 binary packages: golang-github-go-logr-zapr-dev
2025/09/12 17:09:43 Corresponding .debs (will be injected when building):
2025/09/12 17:09:43     golang-github-go-logr-zapr-dev_1.3.0-1_all.deb
2025/09/12 17:09:43 Setting -dist=sid (from .changes file)
2025/09/12 17:09:43 Figuring out reverse build dependencies using dose-ceve(1). This might take a while
2025/09/12 17:10:14 Found 56 reverse build dependencies
2025/09/12 17:10:14 Setting -sbuild_dist=unstable (from .changes file)
2025/09/12 17:10:14 Building packages in parallel using 6 workers
2025/09/12 17:10:14 Building package 1 of 56: golang-github-containerd-stargz-snapshotter
2025/09/12 17:10:14 Building package 2 of 56: sigstore-go
2025/09/12 17:10:14 Building package 3 of 56: golang-github-containers-common
2025/09/12 17:10:14 Building package 4 of 56: golang-github-containerd-nydus-snapshotter
2025/09/12 17:10:14 Building package 4 of 56: golang-github-openshift-imagebuilder
2025/09/12 17:10:14 Building package 6 of 56: docker-compose

ratt.go Outdated
func buildPackagesParallel(builder *sbuild, rebuild map[string][]version.Version, numJobs int) (map[string]*buildResult, []dryRunBuild) {
jobs := make(chan buildJob, len(rebuild))
results := make(chan *buildResult, len(rebuild))
var wg sync.WaitGroup
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using an errgroup instead of a WaitGroup makes for clearer/simpler code in this case.

In particular, we could use https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit to limit the number of active goroutines to numJobs and then iterate the same way as in buildPackagesSequential, but use eg.Go like so:

for src, versions := range rebuild {
	eg.Go(func() error {
		sort.Sort(sort.Reverse(version.Slice(versions)))
		newest := versions[0]
		log.Printf("Building package %d of %d: %s\n", cnt, len(rebuild), src)
		cnt++
		result := builder.build(src, &newest)
		if result.err != nil {
			log.Printf("building %s failed: %v\n", src, result.err)
		}
  		resultsMu.Lock()
  		defer resultsMu.Unlock()
  		results[src] = result
  		return nil
	})
}

This commit introduces parallel package building to ratt.
- New flag `-parallel` enables parallel builds.
- New flag `-jobs` controls the number of concurrent workers
  (default: number of CPU cores, must be > 0).
- Added `buildPackagesSequential` and `buildPackagesParallel` helpers.
- Refactored main build loop to support sequential or parallel execution.

This allows faster builds by utilizing multiple CPU cores.

Signed-off-by: Arthur Diniz <[email protected]>
cnt := 1

for src, versions := range rebuild {
src, versions := src, versions // capture loop variables
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this line, capturing loop variables has not been necessary since Go 1.22 (https://go.dev/blog/loopvar-preview).

By the way, your editor should show you a warning about this and offer a code action to remove the line. If you don’t see this, maybe you use an old version of gopls, or something is wrong with your editor or IDE’s LSP setup…?

})
}

eg.Wait()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While currently, the function we run on the errgroup never returns an error, please add error handling for this eg.Wait() call regardless (to make future changes less surprising).

log.Printf("Building packages in parallel using %d workers\n", *jobs)
buildresults, dryRunBuilds = buildPackagesParallel(builder, rebuild, *jobs)
} else {
buildresults, dryRunBuilds = buildPackagesSequential(builder, rebuild)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buildPackagesSequential(builder, rebuild) could be replaced with buildPackagesParallel(builder, rebuild, 1), right? Then, we can delete the entire buildPackagesSequential function and remove all the duplicate code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants