Swift Command Line Tool utilizing Process() and multithreading crashes after a certain number of execution rounds (~3148) - swift

Question

I implemented a password generator script in Swift which utilizes Process() to execute Mac OS X command line tasks. The passwords themselves are just random Strings which then are encrypted (bcrypt) by the command line task as follows:
/usr/sbin/htpasswd -bnBC 10 '' this_is_the_password | /usr/bin/tr -d ':\n'
Also multiple threads are used to generate passwords with their hashes in parallel.
Note: Both the multithreading and the command line task (compared to serveral other native Swift libraries I tried) improved performance in terms of execution time drastically.
The Problem
The Programm runs fine for the first ~3148 rounds and always crashes around this number (problably correlated to the number of threads running).
For example, if I configure 2000 passwords the code executes as expected terminates without any errors.
Common Error Messages
Setting a breakpoint in Process+Pipe.swift in the catch block of the execute(...) function at BREAKPOINT_1 results in:
Thread 1: signal SIGCHLD
po error.localizedDescription
"The operation couldn\\U2019t be completed. (NSPOSIXErrorDomain error 9 - Bad file descriptor)"
When uncommenting the four //return self.hash(string, cost: cost) code snippets to ignore the error the following errors finally crash the execution (again in execute(...), but not necessarily in the catch block):
Program stops ...
Thread 32: EXC_BAD_ACCESS (code=2, address=0x700003e6bfd4)
... on manual continue ...
Thread 2: EXC_BAD_ACCESS (code=2, address=0x700007e85fd4)
po process
error: Trying to put the stack in unreadable memory at: 0x700003e6bf40.
Code
The relevant code componets are the Main.swift which initialized and starts (and later stops) the PasswordGenerator and then loops n times to get passwords via nextPassword() from the PasswordGenerator. The PasswordGenerator itself utilized execute(...) from the Process extension to run commandline tasks which generate the hash.
Main.swift
class Main {
private static func generate(...) {
...
PasswordGenerator.start()
for _ in 0..<n {
let nextPassword = PasswordGenerator.nextPassword()
let readablePassword = nextPassword.readable
let password = nextPassword.hash
...
}
PasswordGenerator.stop()
...
}
}
PasswordGenerator.swift
The PasswordGenerator runs multiple Threads in parallel.
nextPassword() tries to get a password (if there is one in the passwords Array) or else waits for 100 seconds.
struct PasswordGenerator {
typealias Password = (readable: String, hash: String)
private static let semaphore = DispatchSemaphore(value: 1)
private static var active = false
private static var passwords: [Password] = []
static func nextPassword() -> Password {
self.semaphore.wait()
if let password = self.passwords.popLast() {
self.semaphore.signal()
return password
} else {
self.semaphore.signal()
sleep(100)
return self.nextPassword()
}
}
static func start(
numberOfWorkers: UInt = 32,
passwordLength: UInt = 10,
cost: UInt = 10
) {
self.active = true
for id in 0..<numberOfWorkers {
self.runWorker(id: id, passwordLength: passwordLength, cost: cost)
}
}
static func stop() {
self.semaphore.wait()
self.active = false
self.semaphore.signal()
}
private static func runWorker(
id: UInt,
passwordLength: UInt = 10,
cost: UInt = 10
) {
DispatchQueue.global().async {
var active = true
repeat {
// Update active.
self.semaphore.wait()
active = self.active
print("numberOfPasswords: \(self.passwords.count)")
self.semaphore.signal()
// Generate Password.
// Important: The bycrypt(cost: ...) step must be done outside the Semaphore!
let readable = String.random(length: Int(passwordLength))
let password = Password(readable: readable, hash: Encryption.hash(readable, cost: cost))
// Add Password.
self.semaphore.wait()
self.passwords.append(password)
self.semaphore.signal()
} while active
}
}
}
Encryption.swift
struct Encryption {
static func hash(_ string: String, cost: UInt = 10) -> String {
// /usr/sbin/htpasswd -bnBC 10 '' this_is_the_password | /usr/bin/tr -d ':\n'
let command = "/usr/sbin/htpasswd"
let arguments: [String] = "-bnBC \(cost) '' \(string)".split(separator: " ").map(String.init)
let result1 = Process.execute(
command: command,//"/usr/sbin/htpasswd",
arguments: arguments//["-bnBC", "\(cost)", "''", string]
)
let errorString1 = String(
data: result1?.error?.fileHandleForReading.readDataToEndOfFile() ?? Data(),
encoding: String.Encoding.utf8
) ?? ""
guard errorString1.isEmpty else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed with error: \(errorString1)")
}
guard let output1 = result1?.output else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed! No output.")
}
let command2 = "/usr/bin/tr"
let arguments2: [String] = "-d ':\n'".split(separator: " ").map(String.init)
let result2 = Process.execute(
command: command2,
arguments: arguments2,
standardInput: output1
)
let errorString2 = String(
data: result2?.error?.fileHandleForReading.readDataToEndOfFile() ?? Data(),
encoding: String.Encoding.utf8
) ?? ""
guard errorString2.isEmpty else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed with error: \(errorString2)")
}
guard let output2 = result2?.output else {
// return self.hash(string, cost: cost)
fatalError("Error: Command \(command) \(arguments.joined(separator: " ")) failed! No output.")
}
guard
let hash = String(
data: output2.fileHandleForReading.readDataToEndOfFile(),
encoding: String.Encoding.utf8
)?.replacingOccurrences(of: "$2y$", with: "$2a$")
else {
fatalError("Hash: String replacement failed!")
}
return hash
}
}
Process+Pipe.swift
extension Process {
static func execute(
command: String,
arguments: [String] = [],
standardInput: Any? = nil
) -> (output: Pipe?, error: Pipe?)? {
let process = Process()
process.executableURL = URL(fileURLWithPath: command)
process.arguments = arguments
let outputPipe = Pipe()
let errorPipe = Pipe()
process.standardOutput = outputPipe
process.standardError = errorPipe
if let standardInput = standardInput {
process.standardInput = standardInput
}
do {
try process.run()
} catch {
print(error.localizedDescription)
// BREAKPOINT_1
return nil
}
process.waitUntilExit()
return (output: outputPipe, error: errorPipe)
}
}
Question(s)
Why does the program crash?
Why does it not crash for also huge numbers like 2000 passwords?
Is the multithreading implemented correct?
Is there a problem in the execute(...) code?

Answer 1

I have found a fix while researching this bug. It seems that, despite what the documentation claims, Pipe will not automatically close its reading filehandle.
So if you add a try outputPipe.fileHandleForReading.close() after reading from it, that will fix the issue.

Answer 2

It seems this is a Swift bug. I did some testing and could reproduce it by just running a lot of Process.run(). Filed an issue against Swift:
https://bugs.swift.org/browse/SR-15522