How to "tee" NSPipe in Swift - swift

I'm trying to tee the standard output NSPipe one of NSTask to get two NSPipes, each of which will go into the standard input of two new NSTasks.
I know I can do this in C with the tee function, but I couldn't find it in neither the Foundation nor Darwin frameworks. How can I achieve this?

I wrote a solution to this and it is working well in my project. I have made a Swift package available on GitHub. You can also find examples of its use there. Here is the code:
/**
Duplicates the data from `input` into each of the `outputs`.
Following the precedent of `standardInput`/`standardOutput`/`standardError` in `Process` from `Foundation`,
we accept the type `Any`, but throw a precondition failure if the arguments are not of type `Pipe` or `FileHandle`.
https://github.com/apple/swift-corelibs-foundation/blob/eec4b26deee34edb7664ddd9c1222492a399d122/Sources/Foundation/Process.swift
When `input` sends an EOF (write of length 0), the `outputs` file handles are closed, so only output to handles you own.
This function sets the `readabilityHandler` of inputs and the `writeabilityHandler` of outputs,
so you should not set these yourself after calling `tee`.
The one exception to this guidance is that you can set the `readabilityHandler` of `input` to `nil` to stop `tee`ing.
After doing so, the `writeabilityHandler`s of the `output`s will be set to `nil` automatically after all in-progress writes complete,
but if desired, you could set them to `nil` manually to cancel these writes. However, this may result in some outputs recieving less of the data than others.
This implementation waits for all outputs to consume a piece of input before more input is read.
This means that the speed at which your processes read data may be bottlenecked by the speed at which the slowest process reads data,
but this method also comes with very little memory overhead and is easy to cancel.
If this is unacceptable for your use case. you may wish to rewrite this with a data deque for each output.
*/
public func tee(from input: Any, into outputs: Any...) {
tee(from: input, into: outputs)
}
public func tee(from input: Any, into outputs: [Any]) {
/// Get reading and writing handles from the input and outputs respectively.
guard let input = fileHandleForReading(input) else {
preconditionFailure(incorrectTypeMessage)
}
let outputs: [FileHandle] = outputs.map({
guard let output = fileHandleForWriting($0) else {
preconditionFailure(incorrectTypeMessage)
}
return output
})
let writeGroup = DispatchGroup()
input.readabilityHandler = { input in
let data = input.availableData
/// If the data is empty, EOF reached
guard !data.isEmpty else {
/// Close all the outputs
for output in outputs {
output.closeFile()
}
/// Stop reading and return
input.readabilityHandler = nil
return
}
for output in outputs {
/// Tell `writeGroup` to wait on this output.
writeGroup.enter()
output.writeabilityHandler = { output in
/// Synchronously write the data
output.write(data)
/// Signal that we do not need to write anymore
output.writeabilityHandler = nil
/// Inform `writeGroup` that we are done.
writeGroup.leave()
}
}
/// Wait until all outputs have recieved the data
writeGroup.wait()
}
}
/// The message that is passed to `preconditionFailure` when an incorrect type is passed to `tee`.
let incorrectTypeMessage = "Arguments of tee must be either Pipe or FileHandle."
/// Get a file handle for reading from a `Pipe` or the handle itself from a `FileHandle`, or `nil` otherwise.
func fileHandleForReading(_ handle: Any) -> FileHandle? {
switch handle {
case let pipe as Pipe:
return pipe.fileHandleForReading
case let file as FileHandle:
return file
default:
return nil
}
}
/// Get a file handle for writing from a `Pipe` or the handle itself from a `FileHandle`, or `nil` otherwise.
func fileHandleForWriting(_ handle: Any) -> FileHandle? {
switch handle {
case let pipe as Pipe:
return pipe.fileHandleForWriting
case let file as FileHandle:
return file
default:
return nil
}
}

Related

Rewriting looping blocking code to SwiftNIO style non-blocking code

I'm working on a driver that will read data from the network. It doesn't know how much is in a response, other than that when it tries to read and gets 0 bytes back, it is done. So my blocking Swift code looks naively like this:
func readAllBlocking() -> [Byte] {
var buffer: [Byte] = []
var fullBuffer: [Byte] = []
repeat {
buffer = read() // synchronous, blocking
fullBuffer.append(buffer)
} while buffer.count > 0
return fullBuffer
}
How can I rewrite this as a promise that will keep on running until the entire result is read? After trying to wrap my brain around it, I'm still stuck here:
func readAllNonBlocking() -> EventLoopFuture<[Byte]> {
///...?
}
I should add that I can rewrite read() to instead of returning a [Byte] return an EventLoopFuture<[Byte]>
Generally, loops in synchronous programming are turned into recursion to get the same effect with asynchronous programming that uses futures (and also in functional programming).
So your function could look like this:
func readAllNonBlocking(on eventLoop: EventLoop) -> EventLoopFuture<[Byte]> {
// The accumulated chunks
var accumulatedChunks: [Byte] = []
// The promise that will hold the overall result
let promise = eventLoop.makePromise(of: [Byte].self)
// We turn the loop into recursion:
func loop() {
// First, we call `read` to read in the next chunk and hop
// over to `eventLoop` so we can safely write to `accumulatedChunks`
// without a lock.
read().hop(to: eventLoop).map { nextChunk in
// Next, we just append the chunk to the accumulation
accumulatedChunks.append(contentsOf: nextChunk)
guard nextChunk.count > 0 else {
promise.succeed(accumulatedChunks)
return
}
// and if it wasn't empty, we loop again.
loop()
}.cascadeFailure(to: promise) // if anything goes wrong, we fail the whole thing.
}
loop() // Let's kick everything off.
return promise.futureResult
}
I would like to add two things however:
First, what you're implementing above is to simply read in everything until you see EOF, if that piece of software is exposed to the internet, you should definitely add a limit on how many bytes to hold in memory maximally.
Secondly, SwiftNIO is an event driven system so if you were to read these bytes with SwiftNIO, the program would actually look slightly differently. If you're interested what it looks like to simply accumulate all bytes until EOF in SwiftNIO, it's this:
struct AccumulateUntilEOF: ByteToMessageDecoder {
typealias InboundOut = ByteBuffer
func decode(context: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
// `decode` will be called if new data is coming in.
// We simply return `.needMoreData` because always need more data because our message end is EOF.
// ByteToMessageHandler will automatically accumulate everything for us because we tell it that we need more
// data to decode a message.
return .needMoreData
}
func decodeLast(context: ChannelHandlerContext, buffer: inout ByteBuffer, seenEOF: Bool) throws -> DecodingState {
// `decodeLast` will be called if NIO knows that this is the _last_ time a decode function is called. Usually,
// this is because of EOF or an error.
if seenEOF {
// This is what we've been waiting for, `buffer` should contain all bytes, let's fire them through
// the pipeline.
context.fireChannelRead(self.wrapInboundOut(buffer))
} else {
// Odd, something else happened, probably an error or we were just removed from the pipeline. `buffer`
// will now contain what we received so far but maybe we should just drop it on the floor.
}
buffer.clear()
return .needMoreData
}
}
If you wanted to make a whole program out of this with SwiftNIO, here's an example that is a server which accepts all data until it sees EOF and then literally just writes back the number of received bytes :). Of course, in the real world you would never hold on to all the received bytes to count them (you could just add each individual piece) but I guess it serves as an example.
import NIO
let group = MultiThreadedEventLoopGroup(numberOfThreads: 1)
defer {
try! group.syncShutdownGracefully()
}
struct AccumulateUntilEOF: ByteToMessageDecoder {
typealias InboundOut = ByteBuffer
func decode(context: ChannelHandlerContext, buffer: inout ByteBuffer) throws -> DecodingState {
// `decode` will be called if new data is coming in.
// We simply return `.needMoreData` because always need more data because our message end is EOF.
// ByteToMessageHandler will automatically accumulate everything for us because we tell it that we need more
// data to decode a message.
return .needMoreData
}
func decodeLast(context: ChannelHandlerContext, buffer: inout ByteBuffer, seenEOF: Bool) throws -> DecodingState {
// `decodeLast` will be called if NIO knows that this is the _last_ time a decode function is called. Usually,
// this is because of EOF or an error.
if seenEOF {
// This is what we've been waiting for, `buffer` should contain all bytes, let's fire them through
// the pipeline.
context.fireChannelRead(self.wrapInboundOut(buffer))
} else {
// Odd, something else happened, probably an error or we were just removed from the pipeline. `buffer`
// will now contain what we received so far but maybe we should just drop it on the floor.
}
buffer.clear()
return .needMoreData
}
}
// Just an example "business logic" handler. It will wait for one message
// and just write back the length.
final class SendBackLengthOfFirstInput: ChannelInboundHandler {
typealias InboundIn = ByteBuffer
typealias OutboundOut = ByteBuffer
func channelRead(context: ChannelHandlerContext, data: NIOAny) {
// Once we receive the message, we allocate a response buffer and just write the length of the received
// message in there. We then also close the channel.
let allData = self.unwrapInboundIn(data)
var response = context.channel.allocator.buffer(capacity: 10)
response.writeString("\(allData.readableBytes)\n")
context.writeAndFlush(self.wrapOutboundOut(response)).flatMap {
context.close(mode: .output)
}.whenSuccess {
context.close(promise: nil)
}
}
func errorCaught(context: ChannelHandlerContext, error: Error) {
print("ERROR: \(error)")
context.channel.close(promise: nil)
}
}
let server = try ServerBootstrap(group: group)
// Allow us to reuse the port after the process quits.
.serverChannelOption(ChannelOptions.socket(.init(SOL_SOCKET), .init(SO_REUSEADDR)), value: 1)
// We should allow half-closure because we want to write back after having received an EOF on the input
.childChannelOption(ChannelOptions.allowRemoteHalfClosure, value: true)
// Our program consists of two parts:
.childChannelInitializer { channel in
channel.pipeline.addHandlers([
// 1: The accumulate everything until EOF handler
ByteToMessageHandler(AccumulateUntilEOF(),
// We want 1 MB of buffering max. If you remove this parameter, it'll also
// buffer indefinitely.
maximumBufferSize: 1024 * 1024),
// 2: Our "business logic"
SendBackLengthOfFirstInput()
])
}
// Let's bind port 9999
.bind(to: SocketAddress(ipAddress: "127.0.0.1", port: 9999))
.wait()
// This will never return.
try server.closeFuture.wait()
Demo:
$ echo -n "hello world" | nc localhost 9999
11

How to use file descriptor to divert write-to-file in swift?

I would like to use some C code that uses a file descriptor.
Background is that I would like to read some data from cgraph library.
public extension UnsafeMutablePointer where Pointee == Agraph_t {
func saveTo(fileName: String) {
let f = fopen(cString(fileName), cString("w"))
agwrite(self,f)
fsync(fileno(f))
fclose(f)
}
}
I would like to have the file output, but without writing to a temp file. Hence, I would like to do something like this:
public extension UnsafeMutablePointer where Pointee == Agraph_t {
var asString: String {
let pipe = Pipe()
let fileDescriptor = UnsafeMutablePointer<Int32>.allocate(capacity: 1)
fileDescriptor.pointee = pipe.fileHandleForWriting.fileDescriptor
agwrite(self, fileDescriptor)
let data = pipe.fileHandleForReading.readDataToEndOfFile()
if let output = String(data: data, encoding: .utf8) {
return output
}
return ""
}
}
But it doesn't work, resulting in a EXC_BAD_ACCESS within agwrite(,). What do I need to do instead?
Many thanks in advance!
File descriptors and file pointers are not the same thing. It's confusing, and made even more frustrating by the fact that FILE * is really hard to Google because of the symbol.
You need to fdopen the file descriptor (pipe.fileHandleForWriting.fileDescriptor), to receive a FILE * (UnsafeMutablePointer<FILE> in Swift). This is what you then pass to agwrite.
It's important to fclose the file pointer when you're done writing to it, otherwise .readDataToEndOfFile() will never terminate. I made a helper function to ensure the fclose can't be forgetten. It's possible that agwrite closes the file pointer itself, internally. If that's the case, you should delete this code and just give it the result of fdopen, plain and simple.
import Foundation
public typealias Agraph_t = Int // Dummy value
public struct AGWriteWrongEncoding: Error { }
func agwrite(_: UnsafeMutablePointer<Agraph_t>, _ filePointer: UnsafeMutablePointer<FILE>) {
let message = "This is a stub."
_ = message.withCString { cString in
fputs(cString, stderr)
}
}
#discardableResult
func use<R>(
fileDescriptor: Int32,
mode: UnsafePointer<Int8>!,
closure: (UnsafeMutablePointer<FILE>) throws -> R
) rethrows -> R {
// Should prob remove this `!`, but IDK what a sensible recovery mechanism would be.
let filePointer = fdopen(fileDescriptor, mode)!
defer { fclose(filePointer) }
return try closure(filePointer)
}
public extension UnsafeMutablePointer where Pointee == Agraph_t {
func asString() throws -> String {
let pipe = Pipe()
use(fileDescriptor: pipe.fileHandleForWriting.fileDescriptor, mode: "w") { filePointer in
agwrite(self, filePointer)
}
let data = pipe.fileHandleForReading.readDataToEndOfFile()
guard let output = String(data: data, encoding: .utf8) else {
throw AGWriteWrongEncoding()
}
return output
}
}
let ptr = UnsafeMutablePointer<Agraph_t>.allocate(capacity: 1) // Dummy value
print(try ptr.asString())
Several other things:
Throwing an error is probably a better choice than returning "". Empty strings aren't a good error handling mechanism. Returning an optional would also work, but it's likely to always be force unwrapped, anyway.
readDataToEndOfFile is a blocking call, which can lead to a bad use experience. It's probably best that this code be run on a background thread, or use a FileHandle.readabilityHandler to asynchronously consume the data as it comes in.

How to combine write characteristics and charcteristics notify using RXBluetoothKit for RxSwift

I am trying to interface a BLE device using RXBluetoothKit for swift. All the data commands of the device follow the following sequence
1. write a command (writeWithResponse)
2. Read the response from notification (on a different characteristics)
The number of notification packets (20 bytes max in a notification packet) will depend on the command. This will be a fixed number or essentially indicated using a end-of-data bit in notif value.
Can this be achieved using writeValue(), monitorValueUpdate() combination?
// Abstraction of your commands / results
enum Command {
case Command1(arg: Float)
case Command2(arg: Int, arg2: Int)
}
struct CommandResult {
let command: Command
let data: NSData
}
extension Command {
func toByteCommand() -> NSData {
return NSData()
}
}
// Make sure to setup notifications before subscribing to returned observable!!!
func processCommand(notifyCharacteristic: Characteristic,
_ writeCharacteristic: Characteristic,
_ command: Command) -> Observable<CommandResult> {
// This observable will take care of accumulating data received from notifications
let result = notifyCharacteristic.monitorValueUpdate()
.takeWhile { characteristic in
// Your logic which know when to stop reading notifications.
return true
}
.reduce(NSMutableData(), accumulator: { (data, characteristic) -> NSMutableData in
// Your custom code to append data?
if let packetData = characteristic.value {
data.appendData(packetData)
}
return data
})
// Your code for sending commands, flatmap with more commands if needed or do something similar
let query = writeCharacteristic.writeValue(command.toByteCommand(), type: .WithResponse)
return Observable.zip(result, query, resultSelector: { (result: NSMutableData, query: Characteristic) -> CommandResult in
// This block will be called after query is executed and correct result is collected.
// You can now return some command specific result.
return CommandResult(command: command, data: result)
})
}
// If you would like to serialize multiple commands, you can do for example:
func processMultipleCommands(notifyCharacteristic: Characteristic,
writeCharacteristic: Characteristic,
commands: [Command]) -> Observable<()> {
return Observable.from(Observable.just(commands))
// concatMap would be more appropriate, because in theory we should wait for
// flatmap result before processing next command. It's not available in RxSwift yet.
.flatMap { command in
return processCommand(notifyCharacteristic, writeCharacteristic, command)
}
.map { result in
return ()
}
}
You can try above. It's just an idea how you could handle it. I tried to comment the most important things. Let me know if it works for you.

Calling Swift's readline to the same variable twice ignores the second call

I'm reading user input from command line, checking if it's a valid file path and—if it's not—asking the user to try again.
In case the user input is nil, it should be treated as any other incorrect input the first time, letting the user enter a new value, but the second time that nil is entered, the program should force quit.
(I assume that a nil value is something a user won’t enter on purpose so if it happens more than twice I assume something has gone wrong and quit the program to avoid a never-ending loop asking for new input. This may or may not be a good way to do it but that doesn’t affect the question.)
The problem is that readLine() doesn’t ask for user input the second time it is called after if it has received an end-of-line input (which yields the nil value). (End-of-line can be input with ^D.)
This means the function where readLine() is located automatically returns nil because that’s the latest value of the variable that receives the readLine().
Questions
Shouldn’t readLine() be called no matter what value the receiving variable already has?
If so, why isn't the user asked for input when nil has been input once?
This is the code:
import Foundation
/**
Ask the user to provide a word list file, check if the file exists. If it doesn't exist, ask the user again.
*/
func askUserForWordList() -> String? {
print("")
print("Please drag a word list file here (or enter its path manually), to use as basis for the statistics:")
var path = readLine(stripNewline: true) // THIS IS SKIPPED IF "PATH" IS ALREADY "NIL".
return path
}
/**
Check the user input // PROBABLY NOT RELEVANT FOR THIS QUESTION
*/
func fileExists(filePath: String) -> Bool {
let fileManager = NSFileManager.defaultManager()
if fileManager.fileExistsAtPath(filePath) {
return true
} else {
return false
}
}
/**
Get the file from the user and make sure it’s valid.
*/
func getFilePathFromUser() throws -> String {
enum inputError: ErrorType {
case TwoConsecutiveEndOfFiles
}
var correctFile = false
var path: String? = ""
var numberOfConsecutiveNilFiles = 0
repeat {
// Check that the user did not enter a nil-value (end-of-file) – if they did so two times in a row, terminate the program as this might be some kind of error (so that we don't get an infinite loop).
if numberOfConsecutiveNilFiles > 1 { // FIXME: entering ^D once is enough to end the program (it should require two ^D). Actually the problem seems to be in function "askUserForWordList()".
throw inputError.TwoConsecutiveEndOfFiles
}
path = askUserForWordList()
if path == nil {
numberOfConsecutiveNilFiles += 1
} else {
numberOfConsecutiveNilFiles = 0
correctFile = fileExists(path!)
if !correctFile {
print("")
print("Oops, I couldn't recognize that file path. Please try again.")
}
}
} while !correctFile
return path!
}
// This is where the actual execution starts
print("")
print("=== Welcome to \"Word Statistics\", command line version ===")
print("")
print("This program will give you some statistics for the list of words you provide.")
do {
let path = try getFilePathFromUser()
} catch {
print("Error: \(error)")
exit(-46) // Using closest error type from http://www.swiftview.com/tech/exitcodes.htm (which may not be standard at all. I could, however, not find any "standard" list of exit values).
}
Notes
When entering any other non-valid path (any string, even an empty one (just press Enter), the loop works as intended.
Originally path in the askUserForWordList() function was declared as a constant (let path = readLine(stripNewline: true)) but I changed it to a var since it’s supposed to be updated everytime the function is called. However, this didn’t affect how the program works.
I tried declaring path separately on the line before calling readLine(), which made no difference.
I tried skipping the path variable completely and let the askUserForWordList() function return the readLine() result directly (return readLine(stripNewline: true)). This made no difference.
I skipped the askUserForWordList() function all together and moved the code asking for user input into the ”main” code of the function "getFilePathFromUser()", but that didn’t change anything.
Modified code:
func getFilePathFromUser() throws -> String {
enum inputError: ErrorType {
case TwoConsecutiveEndOfFiles
}
var correctFile = false
var path: String? = ""
var numberOfConsecutiveNilFiles = 0
repeat {
// Check that the user did not enter a nil-value (end-of-file) – if they did so two times in a row, terminate the program as this might be some kind of error (so that we don't get an infinite loop).
if numberOfConsecutiveNilFiles > 1 { // FIXME: entering ^D once is enough to end the program (it should require two ^D). Actually the problem seems to be in function "askUserForWordList()".
throw inputError.TwoConsecutiveEndOfFiles
}
// MODIFIED – This code was previously located in "askUserForWordList()"
print("")
print("Please drag a word list file here (or enter its path manually), to use as basis for the statistics:")
path = readLine(stripNewline: true)
// END OF MODIFICATION
if path == nil {
numberOfConsecutiveNilFiles += 1
} else {
numberOfConsecutiveNilFiles = 0
correctFile = fileExists(path!)
if !correctFile {
print("")
print("Oops, I couldn't recognize that file path. Please try again.")
}
}
} while !correctFile
return path!
}
readLine() returns nil if (and only if) the standard input file descriptor has reached end-of-file. This happens (for example) when reading the input
from a tty, and Ctrl-D (the "end-of-transmission character") is entered as the first character on a line.
All subsequent readLine() calls then return nil as well, there is no
way detect that "Ctrl-D was entered twice".
In other words, once the standard input is in the end-of-file condition
you cannot read any data from it anymore. If your programs requires
more data then you can only report an error, e.g.
guard let path = readLine(stripNewline: true) else {
throw InputError.UnexpectedEndOfFile
}

How do I wrap up a series of related NSTask operations into discrete functions

I'm writing a Swift command line tool that uses NSTask to interact with git. In the simplest scenario I want to run three commands: init, add ., and commit -m Initial Commit. I intend to use a separate NSTask for each command, and want to house each command in its own function - returning true if the task succeeded or false if it didn't. This set-up would allow my main function to look like this:
func main() {
if runInit() {
if runStage() {
if runCommit() {
NSLog("success!")
}
}
}
}
To accomplish this each of the three functions must do the following before returning (i) launch the task (ii) wait for it to complete, (iii) obtain whatever is in stdout, and (iv) set the return value (true or false). Here's what I've got for the commit stage:
func runCommit() -> Bool {
var retval = false
var commitTask = NSTask()
commitTask.standardOutput = NSPipe()
commitTask.launchPath = gitPath
commitTask.arguments = ["commit", "-m", "Initial Commit"]
commitTask.currentDirectoryPath = demoProjectURL.path!
commitTask.standardOutput.fileHandleForReading.readToEndOfFileInBackgroundAndNotify()
nc.addObserverForName(NSFileHandleReadToEndOfFileCompletionNotification,
object: commitTask.standardOutput.fileHandleForReading,
queue: nil) { (note) -> Void in
// get the output, log it, then...
if commitTask.terminationStatus == EXIT_SUCCESS {
retval = true
}
}
commitTask.launch()
commitTask.waitUntilExit()
return retval
}
My question is essentially about how waitUntilExit works, particularly in conjunction with the notification I sign up for to enable me to get the output. Apple's docs say:
This method first checks to see if the receiver is still running using isRunning. Then it polls the current run loop using NSDefaultRunLoopMode until the task completes.
I'm a bit out of my depth when it comes to run loop mechanics, and was wondering what this means in this context - can I safely assume that my notification block will always be executed before the enclosing function returns?
waitUntilExit returns when the SIGCHILD signal has been received
to indicate that the child process has terminated. The notification
block is executed when EOF is read from the pipe to the child process.
It is not specified which of these events occurs first.
Therefore you have to wait for both. There are several possible solutions,
here is one using a "signalling semaphore", you could also use
a "dispatch group".
Another error in your code is that the observer is never removed.
func runCommit() -> Bool {
let commitTask = NSTask()
commitTask.standardOutput = NSPipe()
commitTask.launchPath = gitPath
commitTask.arguments = ["commit", "-m", "Initial Commit"]
commitTask.currentDirectoryPath = demoProjectURL.path!
commitTask.standardOutput!.fileHandleForReading.readToEndOfFileInBackgroundAndNotify()
let sema = dispatch_semaphore_create(0)
var obs : NSObjectProtocol!
obs = nc.addObserverForName(NSFileHandleReadToEndOfFileCompletionNotification,
object: commitTask.standardOutput!.fileHandleForReading, queue: nil) {
(note) -> Void in
// Get data and log it.
if let data = note.userInfo?[NSFileHandleNotificationDataItem] as? NSData,
let string = String(data: data, encoding: NSUTF8StringEncoding) {
print(string)
}
// Signal semaphore.
dispatch_semaphore_signal(sema)
nc.removeObserver(obs)
}
commitTask.launch()
// Wait for process to terminate.
commitTask.waitUntilExit()
// Wait for semaphore to be signalled.
dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER)
let retval = commitTask.terminationStatus == EXIT_SUCCESS
return retval
}