-
Notifications
You must be signed in to change notification settings - Fork 16
Add parallel build support with -parallel and -jobs options #34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
ratt.go
Outdated
func buildPackagesParallel(builder *sbuild, rebuild map[string][]version.Version, numJobs int) (map[string]*buildResult, []dryRunBuild) { | ||
jobs := make(chan buildJob, len(rebuild)) | ||
results := make(chan *buildResult, len(rebuild)) | ||
var wg sync.WaitGroup |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think using an errgroup instead of a WaitGroup makes for clearer/simpler code in this case.
In particular, we could use https://pkg.go.dev/golang.org/x/sync/errgroup#Group.SetLimit to limit the number of active goroutines to numJobs
and then iterate the same way as in buildPackagesSequential
, but use eg.Go
like so:
for src, versions := range rebuild {
eg.Go(func() error {
sort.Sort(sort.Reverse(version.Slice(versions)))
newest := versions[0]
log.Printf("Building package %d of %d: %s\n", cnt, len(rebuild), src)
cnt++
result := builder.build(src, &newest)
if result.err != nil {
log.Printf("building %s failed: %v\n", src, result.err)
}
resultsMu.Lock()
defer resultsMu.Unlock()
results[src] = result
return nil
})
}
This commit introduces parallel package building to ratt. - New flag `-parallel` enables parallel builds. - New flag `-jobs` controls the number of concurrent workers (default: number of CPU cores, must be > 0). - Added `buildPackagesSequential` and `buildPackagesParallel` helpers. - Refactored main build loop to support sequential or parallel execution. This allows faster builds by utilizing multiple CPU cores. Signed-off-by: Arthur Diniz <[email protected]>
4d57a16
to
30878ea
Compare
cnt := 1 | ||
|
||
for src, versions := range rebuild { | ||
src, versions := src, versions // capture loop variables |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove this line, capturing loop variables has not been necessary since Go 1.22 (https://go.dev/blog/loopvar-preview).
By the way, your editor should show you a warning about this and offer a code action to remove the line. If you don’t see this, maybe you use an old version of gopls, or something is wrong with your editor or IDE’s LSP setup…?
}) | ||
} | ||
|
||
eg.Wait() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While currently, the function we run on the errgroup never returns an error, please add error handling for this eg.Wait()
call regardless (to make future changes less surprising).
log.Printf("Building packages in parallel using %d workers\n", *jobs) | ||
buildresults, dryRunBuilds = buildPackagesParallel(builder, rebuild, *jobs) | ||
} else { | ||
buildresults, dryRunBuilds = buildPackagesSequential(builder, rebuild) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
buildPackagesSequential(builder, rebuild)
could be replaced with buildPackagesParallel(builder, rebuild, 1)
, right? Then, we can delete the entire buildPackagesSequential
function and remove all the duplicate code.
This commit introduces parallel package building to ratt.
-parallel
enables parallel builds.-jobs
controls the number of concurrent workers (default: number of CPU cores, must be > 0).buildPackagesSequential
andbuildPackagesParallel
helpers.This allows faster builds by utilizing multiple CPU cores.
Example: