Skip to content

Commit 24f4323

Browse files
gregcottengregcotten
authored
Fix Linux Process deadlocks (#67) (thanks @gregc !!!)
* initial attempt at adding async/await to `Process` * ParsableCommand -> AsyncParsableCommand * oops, wasn't calling async main * remove all waitUntilExit() remnants * change static let gitURL to gregcotten/swift-bundler for testing * more async/await * don't use DispatchSemaphore which can deadlock no clue if this works or fixes anything * simplify getOutputData * bring back fileHandleForReading.readToEnd() * move fileHandleForReading.readToEnd to the data consuming Task * add more getOutputData debugging * more debug logging * close inputPipe fileHandleForWriting after writing context? * signal debug log * readLine using async iterator if possible * I guess only Darwin has AsyncBytes... * readLineAsync always * have readLineAsync respect `strippingNewline` * builder subprocess input pipe should not be the parent's (?) * flailing * better builder debugging * try using SwiftCommand in BuilderContextImpl * shim in SwiftCommand in some flailing attempt to fix deadlock * Update Package.swift * Update ProjectBuilder.swift * hardcode builder context as test * try using Shwift in Builder * Revert "try using Shwift in Builder" This reverts commit a945c78. * Revert "hardcode builder context as test" This reverts commit b3ea18c. * try calling "swift run Builder" instead of running exe directly? * hardcoded context once again * use AsyncProcess * oops * Update Package.resolved * Update Package.resolved * Update Package.resolved * use tagged `AsyncProcess` * remove extraneous logging * Squashed commit of the following: commit ea59e6e Merge: 4cfdd0e 5bd25e0 Author: Greg Cotten <[email protected]> Date: Wed Mar 5 11:48:59 2025 -0800 Merge branch 'main' into async-process commit 4cfdd0e Author: Greg Cotten <[email protected]> Date: Wed Mar 5 11:46:58 2025 -0800 remove extraneous logging commit b1f4f39 Author: Greg Cotten <[email protected]> Date: Wed Mar 5 11:15:36 2025 -0800 use tagged `AsyncProcess` commit 80c4429 Author: Greg Cotten <[email protected]> Date: Wed Mar 5 10:38:54 2025 -0800 Update Package.resolved commit 1243bed Author: Greg Cotten <[email protected]> Date: Wed Mar 5 10:26:24 2025 -0800 Update Package.resolved commit cbcbe46 Author: Greg Cotten <[email protected]> Date: Wed Mar 5 09:52:30 2025 -0800 Update Package.resolved commit e68cd8d Author: Greg Cotten <[email protected]> Date: Wed Mar 5 09:42:04 2025 -0800 oops commit c1066c1 Author: Greg Cotten <[email protected]> Date: Wed Mar 5 09:41:34 2025 -0800 use AsyncProcess commit 2c171f7 Author: Greg Cotten <[email protected]> Date: Tue Mar 4 09:03:17 2025 -0800 hardcoded context once again commit ba4190c Author: Greg Cotten <[email protected]> Date: Mon Mar 3 17:05:07 2025 -0800 try calling "swift run Builder" instead of running exe directly? commit 7b6d3e3 Author: Greg Cotten <[email protected]> Date: Mon Mar 3 14:58:08 2025 -0800 Revert "hardcode builder context as test" This reverts commit b3ea18c. commit 1062101 Author: Greg Cotten <[email protected]> Date: Mon Mar 3 14:58:00 2025 -0800 Revert "try using Shwift in Builder" This reverts commit a945c78. commit a945c78 Author: Greg Cotten <[email protected]> Date: Sun Mar 2 07:15:53 2025 -0800 try using Shwift in Builder commit b3ea18c Author: Greg Cotten <[email protected]> Date: Sat Mar 1 07:09:46 2025 -0800 hardcode builder context as test commit 959f678 Author: Greg Cotten <[email protected]> Date: Fri Feb 28 13:23:17 2025 -0800 Update ProjectBuilder.swift commit 086f9f8 Author: Greg Cotten <[email protected]> Date: Fri Feb 28 13:22:23 2025 -0800 Update Package.swift commit d46600f Author: Greg Cotten <[email protected]> Date: Fri Feb 28 13:21:12 2025 -0800 shim in SwiftCommand in some flailing attempt to fix deadlock commit f699a44 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 21:03:10 2025 -0800 try using SwiftCommand in BuilderContextImpl commit 065e3c0 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 19:55:42 2025 -0800 better builder debugging commit b6202f5 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 15:00:59 2025 -0800 flailing commit 65a0c7a Author: Greg Cotten <[email protected]> Date: Thu Feb 27 14:31:08 2025 -0800 builder subprocess input pipe should not be the parent's (?) commit 295bee1 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 14:16:28 2025 -0800 have readLineAsync respect `strippingNewline` commit 1bb3c00 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 14:06:19 2025 -0800 readLineAsync always commit 3360f60 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 14:03:41 2025 -0800 I guess only Darwin has AsyncBytes... commit 16fa1cb Author: Greg Cotten <[email protected]> Date: Thu Feb 27 13:52:35 2025 -0800 readLine using async iterator if possible commit 533fb18 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 12:53:26 2025 -0800 signal debug log commit 8b5f0fd Author: Greg Cotten <[email protected]> Date: Thu Feb 27 12:52:23 2025 -0800 close inputPipe fileHandleForWriting after writing context? commit cf242e7 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 12:36:30 2025 -0800 more debug logging commit 6e0b8fd Author: Greg Cotten <[email protected]> Date: Thu Feb 27 12:05:55 2025 -0800 add more getOutputData debugging commit 8957ddc Author: Greg Cotten <[email protected]> Date: Thu Feb 27 11:59:11 2025 -0800 move fileHandleForReading.readToEnd to the data consuming Task commit 0f2e019 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 11:50:33 2025 -0800 bring back fileHandleForReading.readToEnd() commit f6e8f81 Author: Greg Cotten <[email protected]> Date: Thu Feb 27 11:45:21 2025 -0800 simplify getOutputData commit 14775ee Author: Greg Cotten <[email protected]> Date: Thu Feb 27 10:01:51 2025 -0800 don't use DispatchSemaphore which can deadlock no clue if this works or fixes anything commit e9d9225 Author: Greg Cotten <[email protected]> Date: Wed Feb 26 22:29:49 2025 -0800 more async/await commit 4a0a3a5 Author: Greg Cotten <[email protected]> Date: Wed Feb 26 22:15:34 2025 -0800 change static let gitURL to gregcotten/swift-bundler for testing commit 25f773e Author: Greg Cotten <[email protected]> Date: Wed Feb 26 21:50:33 2025 -0800 remove all waitUntilExit() remnants commit d5169d8 Author: Greg Cotten <[email protected]> Date: Wed Feb 26 20:38:10 2025 -0800 oops, wasn't calling async main commit 52a5bd2 Author: Greg Cotten <[email protected]> Date: Wed Feb 26 20:30:49 2025 -0800 ParsableCommand -> AsyncParsableCommand commit ecb5d62 Author: Greg Cotten <[email protected]> Date: Wed Feb 26 20:04:40 2025 -0800 initial attempt at adding async/await to `Process` * back to stackotter swift-bundler * swift-format -> swift format * oneshot swift format * no need for swift-system import remnant * get rid of swift 6 Result extension stuff * format * log `Process.run` everywhere possible * address concerns * formatting * update AsyncProcess to 0.0.4 * AsyncProcess 0.0.5
1 parent 5bd25e0 commit 24f4323

File tree

61 files changed

+863
-515
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

61 files changed

+863
-515
lines changed

‎Package.resolved

+38-2
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,24 @@
99
"version" : "4.6.1"
1010
}
1111
},
12+
{
13+
"identity" : "async-collections",
14+
"kind" : "remoteSourceControl",
15+
"location" : "https://github.com/adam-fowler/async-collections.git",
16+
"state" : {
17+
"revision" : "726af96095a19df6b8053ddbaed0a727aa70ccb2",
18+
"version" : "0.1.0"
19+
}
20+
},
21+
{
22+
"identity" : "asyncprocess",
23+
"kind" : "remoteSourceControl",
24+
"location" : "https://github.com/gregcotten/AsyncProcess",
25+
"state" : {
26+
"revision" : "3a506bb36e4aa54eb3f1b13f15e636c68ee3eb19",
27+
"version" : "0.0.5"
28+
}
29+
},
1230
{
1331
"identity" : "jsonutilities",
1432
"kind" : "remoteSourceControl",
@@ -77,8 +95,17 @@
7795
"kind" : "remoteSourceControl",
7896
"location" : "https://github.com/apple/swift-async-algorithms.git",
7997
"state" : {
80-
"revision" : "9cfed92b026c524674ed869a4ff2dcfdeedf8a2a",
81-
"version" : "0.1.0"
98+
"revision" : "4c3ea81f81f0a25d0470188459c6d4bf20cf2f97",
99+
"version" : "1.0.3"
100+
}
101+
},
102+
{
103+
"identity" : "swift-atomics",
104+
"kind" : "remoteSourceControl",
105+
"location" : "https://github.com/apple/swift-atomics.git",
106+
"state" : {
107+
"revision" : "cd142fd2f64be2100422d658e7411e39489da985",
108+
"version" : "1.2.0"
82109
}
83110
},
84111
{
@@ -162,6 +189,15 @@
162189
"version" : "1.6.1"
163190
}
164191
},
192+
{
193+
"identity" : "swift-nio",
194+
"kind" : "remoteSourceControl",
195+
"location" : "https://github.com/apple/swift-nio.git",
196+
"state" : {
197+
"revision" : "c51907a839e63ebf0ba2076bba73dd96436bd1b9",
198+
"version" : "2.81.0"
199+
}
200+
},
165201
{
166202
"identity" : "swift-overture",
167203
"kind" : "remoteSourceControl",

‎Package.swift

+16-2
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,13 @@ let package = Package(
3030
.package(url: "https://github.com/apple/swift-asn1", from: "1.1.0"),
3131
.package(url: "https://github.com/apple/swift-crypto", from: "3.10.0"),
3232
.package(url: "https://github.com/CoreOffice/XMLCoder", from: "0.17.1"),
33+
.package(url: "https://github.com/adam-fowler/async-collections.git", from: "0.1.0"),
34+
.package(url: "https://github.com/gregcotten/AsyncProcess", from: "0.0.5"),
3335

3436
// File watcher dependencies
3537
.package(url: "https://github.com/sersoft-gmbh/swift-inotify", "0.4.0"..<"0.5.0"),
3638
.package(url: "https://github.com/apple/swift-system", from: "1.2.0"),
37-
.package(url: "https://github.com/apple/swift-async-algorithms", from: "0.1.0"),
39+
.package(url: "https://github.com/apple/swift-async-algorithms", from: "1.0.3"),
3840
],
3941
targets: [
4042
.executableTarget(
@@ -55,6 +57,12 @@ let package = Package(
5557
.product(name: "SwiftSyntax", package: "swift-syntax"),
5658
.product(name: "SwiftSyntaxBuilder", package: "swift-syntax"),
5759
.product(name: "Overture", package: "swift-overture"),
60+
.product(name: "AsyncCollections", package: "async-collections"),
61+
.product(
62+
name: "ProcessSpawnSync",
63+
package: "AsyncProcess",
64+
condition: .when(platforms: [.linux])
65+
),
5866

5967
// Xcodeproj related dependencies
6068
.product(
@@ -117,7 +125,13 @@ let package = Package(
117125

118126
.target(
119127
name: "SwiftBundlerBuilders",
120-
dependencies: []
128+
dependencies: [
129+
.product(
130+
name: "ProcessSpawnSync",
131+
package: "AsyncProcess",
132+
condition: .when(platforms: [.linux])
133+
)
134+
]
121135
),
122136

123137
.target(

‎Plugins/SwiftBundlerCommandPlugin/SwiftBundlerCommandPlugin.swift

+18-5
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,19 @@ import PackagePlugin
44
@main
55
struct SwiftBundlerCommandPlugin: CommandPlugin {
66
/// This entry point is called when operating on a Swift package.
7-
func performCommand(context: PluginContext, arguments: [String]) throws {
7+
func performCommand(context: PluginContext, arguments: [String]) async throws {
88
let bundler = try context.tool(named: "swift-bundler")
99

10-
try run(command: bundler.path, with: arguments)
10+
try await run(command: bundler.path, with: arguments)
1111
}
1212
}
1313

1414
extension SwiftBundlerCommandPlugin {
1515
/// Run a command with the given arguments.
16-
func run(command: Path, with arguments: [String]) throws {
16+
func run(command: Path, with arguments: [String]) async throws {
1717
let exec = URL(fileURLWithPath: command.string)
1818

19-
let process = try Process.run(exec, arguments: arguments)
20-
process.waitUntilExit()
19+
let process = try await Process.runAndWait(exec, arguments: arguments)
2120

2221
// Check whether the subprocess invocation was successful.
2322
if process.terminationReason == .exit,
@@ -30,3 +29,17 @@ extension SwiftBundlerCommandPlugin {
3029
}
3130
}
3231
}
32+
33+
extension Process {
34+
class func runAndWait(_ url: URL, arguments: [String]) async throws -> Process {
35+
try await withCheckedThrowingContinuation { c in
36+
do {
37+
_ = try Process.run(url, arguments: arguments) {
38+
c.resume(returning: $0)
39+
}
40+
} catch {
41+
c.resume(throwing: error)
42+
}
43+
}
44+
}
45+
}

‎Sources/SwiftBundlerBuilders/Builder.swift

+32-6
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import Foundation
33
/// A project builder.
44
public protocol Builder {
55
/// Builds the project defined by the given context.
6-
static func build(_ context: some BuilderContext) throws -> BuilderResult
6+
static func build(_ context: some BuilderContext) async throws -> BuilderResult
77
}
88

99
private enum BuilderError: LocalizedError {
@@ -19,20 +19,46 @@ private enum BuilderError: LocalizedError {
1919

2020
extension Builder {
2121
/// Default builder entrypoint. Parses builder context from stdin.
22-
public static func main() {
22+
public static func main() async {
2323
do {
24-
guard let input = readLine(strippingNewline: true) else {
25-
throw BuilderError.noInput
26-
}
24+
let input = try await readLineAsync()
2725

2826
let context = try JSONDecoder().decode(
2927
_BuilderContextImpl.self, from: Data(input.utf8)
3028
)
3129

32-
_ = try build(context)
30+
_ = try await build(context)
3331
} catch {
3432
print(error)
3533
Foundation.exit(1)
3634
}
3735
}
3836
}
37+
38+
func readLineAsync(strippingNewline: Bool = true) async throws -> String {
39+
let readBytesStream = AsyncStream.makeStream(of: Data.self)
40+
41+
FileHandle.standardInput.readabilityHandler = { handle in
42+
readBytesStream.continuation.yield(handle.availableData)
43+
}
44+
45+
defer { FileHandle.standardInput.readabilityHandler = nil }
46+
47+
var accumulatedData = Data()
48+
for await data in readBytesStream.stream {
49+
accumulatedData.append(data)
50+
if let stringData = String(data: accumulatedData, encoding: .utf8),
51+
stringData.contains(where: { $0.isNewline }),
52+
let firstLine = stringData.components(separatedBy: .newlines).first
53+
{
54+
readBytesStream.continuation.finish()
55+
if strippingNewline {
56+
return firstLine
57+
} else {
58+
return firstLine + "\n"
59+
}
60+
}
61+
}
62+
63+
throw BuilderError.noInput
64+
}

‎Sources/SwiftBundlerBuilders/BuilderContextImpl.swift

+24-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
import Foundation
22

3+
#if os(Linux)
4+
import ProcessSpawnSync
5+
typealias Process = PSProcess
6+
#endif
7+
38
//swiftlint:disable type_name
49
// TODO: Use `package` access level when we bump to Swift 5.9
510
/// Implementation detail, may have breaking changes from time to time.
@@ -14,10 +19,11 @@ public struct _BuilderContextImpl: BuilderContext, Codable {
1419
}
1520

1621
enum Error: LocalizedError {
22+
case commandNotFound
1723
case nonZeroExitStatus(Int)
1824
}
1925

20-
public func run(_ command: String, _ arguments: [String]) throws {
26+
public func run(_ command: String, _ arguments: [String]) async throws {
2127
let process = Process()
2228
#if os(Windows)
2329
process.executableURL = URL(fileURLWithPath: "C:\\Windows\\System32\\cmd.exe")
@@ -27,8 +33,7 @@ public struct _BuilderContextImpl: BuilderContext, Codable {
2733
process.arguments = [command] + arguments
2834
#endif
2935

30-
try process.run()
31-
process.waitUntilExit()
36+
try await process.runAndWait()
3237

3338
let exitStatus = Int(process.terminationStatus)
3439
guard exitStatus == 0 else {
@@ -37,3 +42,19 @@ public struct _BuilderContextImpl: BuilderContext, Codable {
3742
}
3843
}
3944
//swiftlint:enable type_name
45+
46+
extension Process {
47+
func runAndWait() async throws {
48+
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
49+
terminationHandler = { _ in
50+
continuation.resume()
51+
}
52+
53+
do {
54+
try run()
55+
} catch {
56+
continuation.resume(throwing: error)
57+
}
58+
}
59+
}
60+
}

‎Sources/SwiftBundlerBuilders/Context.swift

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,5 @@ public protocol BuilderContext {
88

99
/// Runs the given command (either a path or a tool name) with the given
1010
/// arguments.
11-
func run(_ command: String, _ arguments: [String]) throws
11+
func run(_ command: String, _ arguments: [String]) async throws
1212
}

‎Sources/swift-bundler/Bundler/AppImageBundler.swift

+3-3
Original file line numberDiff line numberDiff line change
@@ -33,13 +33,13 @@ enum AppImageBundler: Bundler {
3333
static func bundle(
3434
_ context: BundlerContext,
3535
_ additionalContext: Context
36-
) -> Result<BundlerOutputStructure, AppImageBundlerError> {
36+
) async -> Result<BundlerOutputStructure, AppImageBundlerError> {
3737
let outputStructure = intendedOutput(in: context, additionalContext)
3838
let appDir = context.outputDirectory
3939
.appendingPathComponent("\(context.appName).AppDir")
4040
let bundleName = outputStructure.bundle.lastPathComponent
4141

42-
return GenericLinuxBundler.bundle(
42+
return await GenericLinuxBundler.bundle(
4343
context,
4444
GenericLinuxBundler.Context(cosmeticBundleName: bundleName)
4545
)
@@ -69,7 +69,7 @@ enum AppImageBundler: Bundler {
6969
}
7070
.andThenDoSideEffect { structure in
7171
log.info("Converting '\(context.appName).AppDir' to '\(bundleName)'")
72-
return AppImageTool.bundle(appDir: appDir, to: outputStructure.bundle)
72+
return await AppImageTool.bundle(appDir: appDir, to: outputStructure.bundle)
7373
.mapError { .failedToBundleAppDir($0) }
7474
}
7575
.replacingSuccessValue(with: outputStructure)

‎Sources/swift-bundler/Bundler/AppImageTool.swift

+2-2
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@ import Foundation
33
/// A swifty interface for the `appimagetool` command-line tool for converting AppDirs to
44
/// AppImages.
55
enum AppImageTool {
6-
static func bundle(appDir: URL, to appImage: URL) -> Result<Void, AppImageToolError> {
6+
static func bundle(appDir: URL, to appImage: URL) async -> Result<Void, AppImageToolError> {
77
let arguments = [appDir.path, appImage.path]
8-
return Process.runAppImage("appimagetool", arguments: arguments)
8+
return await Process.runAppImage("appimagetool", arguments: arguments)
99
.mapError { error in
1010
.failedToRunAppImageTool(
1111
command: "appimagetool \(arguments.joined(separator: " "))",

‎Sources/swift-bundler/Bundler/ArchiveTool.swift

+2-2
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,10 @@ enum ArchiveTool {
66
static func createTarGz(
77
of directory: URL,
88
at outputFile: URL
9-
) -> Result<Void, ArchiveToolError> {
9+
) async -> Result<Void, ArchiveToolError> {
1010
let arguments = ["--create", "--file", outputFile.path, directory.lastPathComponent]
1111
let workingDirectory = directory.deletingLastPathComponent()
12-
return Process.create("tar", arguments: arguments, directory: workingDirectory)
12+
return await Process.create("tar", arguments: arguments, directory: workingDirectory)
1313
.runAndWait()
1414
.mapError { error in
1515
.failedToCreateTarGz(directory: directory, outputFile: outputFile, error)

‎Sources/swift-bundler/Bundler/Bundler.swift

+1-1
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ protocol Bundler {
3333
static func bundle(
3434
_ context: BundlerContext,
3535
_ additionalContext: Context
36-
) -> Result<BundlerOutputStructure, Error>
36+
) async -> Result<BundlerOutputStructure, Error>
3737

3838
/// Returns a description of the files that would be produced if
3939
/// ``Bundler/bundle(_:_:)`` were to get called with the provided context.

0 commit comments

Comments
 (0)