How to capture stdout & stderr in Swift XCTest unit tests? - swift

I’m writing a command-line tool in Swift. In my unit tests, I want to test the code’s output to stdout (and, secondarily, to stderr).
The examples I’ve found of using a Pipe() to copy or capture stdout output mostly rely on spawning a sub-process and assigning its stdout—and I don’t know where to begin figuring out how to deal with that in XCTest. But I have managed to pull together something that sort of begins to work without spawning a separate process.
What I’ve come up with so far is compiling and running, and with an ugly hack does sort of what I need—but just for one use (if I try to use it across multiple tests in the same run, all but the first fail). The hack is that, after the calls that write to stdout, I have to add a call to write to stderr to have the data already sent to stdout then be written to my capturedStdout variable.
It seems there ought to be a more appropriate way to get this to work.
Here’s what I’ve got so far:
import XCTest
final class StdoutCaptureTests: XCTestCase {
func testCapturePrint() {
let pipe = Pipe()
var capturedStdout = ""
// Setup to copy data going to stdout into capturedStdout.
setvbuf(stdout, nil, _IONBF, 0)
// Assign stdout’s pointer to the pipe:
dup2(pipe.fileHandleForWriting.fileDescriptor, STDOUT_FILENO)
// A closure for receiving the data sent to stdout:
pipe.fileHandleForReading.readabilityHandler = { handle in
if let str = String(data: handle.availableData, encoding: .utf8) {
capturedStdout += str
}
}
// The subject of the test outputs to stdout:
// (In this simplified example, just print "test" to stdout.)
print("test")
// Here’s where it gets weird...
// If none of the following lines are uncommented,
// the XCTAssertEqual test below will fail
// (nothing gets written to capturedStdout).
// Uncommenting the following line makes the XCTAssertEqual work:
//XCTAssert(false) // ⚠️
// (but that means the test is considered failed.)
// And this has no effect:
//XCTAssert(true) // 🛑
// fflush has no effect (when it gets nil as argument here,
// the man(ual) page says it “flushes all open streams.”)
//fflush(nil) // 🛑
// But this works and lets the test pass:
//FileHandle.standardError.write("".data(using: .utf8)!) // ✅
XCTAssertEqual(capturedStdout, "test\n")
// When the above commented lines are still commented:
// 🛑 XCTAssertEqual failed: ("") is not equal to ("test")
}
}
Given that it fails when used in subsequent test methods in the same run, I’m thinking there’s probably some cleanup I should be doing at the end of each test to restore stdout before the next test method is run.
Some of the references I’ve found:
using pipe() in Swift App to redirect stdout into a textView (only runs in simulator, not native)
Redirect Process stdout to Apple System Log Facility in Swift
https://www.hackingwithswift.com/forums/ios/redirecting-output-to-a-log-file-when-not-attached-to-debugger/5766

Related

Swift Process with Psuedo Terminal (PTY)

I'm looking for some insight and guidance on using the Foundation.Process type with a PTY (Psuedo Terminal) so that the subprocess can accept input and behave as if it was running via a terminal.
The reason for needing a PTY is that for programs like ssh or in my case (xcodes) which ask for user input including passwords, running these via Foundation.Process does not display the prompts to the user as the output is usually buffered (this works fine in the Xcode debugger console but when running via a real terminal that is buffered the prompts are never displayed in the terminal)
Looking at other threads it seems like correct approach here is create a PTY and use the filehandles to attach to the Process.
While I've got this to work to the point where prompts are now shown, I cant seem to figure out how to pass input back to the process as these are being controlled by the PTY.
Here is my Process setup:
let process = Process()
// Setup the process with path, args, etc...
// Setup the PTY handles
var parentDescriptor: Int32 = 0
var childDescriptor: Int32 = 0
guard Darwin.openpty(&parentDescriptor, &childDescriptor, nil, nil, nil) != -1 else {
  fatalError("Failed to spawn PTY")
}
parentHandle = FileHandle(fileDescriptor: parentDescriptor, closeOnDealloc: true)
childHandle = FileHandle(fileDescriptor: childDescriptor, closeOnDealloc: true)
process.standardInput = childHandle
process.standardOutput = childHandle
process.standardError = childHandle
With this setup I then read the parent handle and output any result it gets (such as the input prompts):
parentHandle?.readabilityHandler = { handle in
  guard let line = String(data: handle.availableData, encoding: .utf8), !line.isEmpty else {
    return
  }
  logger.notice("\(line)")
}
When process.run() is executed the program runs and I can see it asks for Apple ID: input in my terminal, however, when typing input into the terminal the process does not seem to react to this input.
I've tried forwarding the FileHandle.standardInput:
FileHandle.standardInput.readabilityHandler = { handle in
  parentHandle?.write(handle.availableData)
}
But this doesn't seem to work either.
What is the recommended way to setup a PTY with Foundation.Process for executing arbitrary programs and having them behave as if they were being run in a terminal context?
Most of the resources I found online are about other languages and I'd like to stick with Foundation.Process vs. doing anything custom in C/C++ if possible as it just makes it easier to reason about / maintain. The resources for Swift on this topic are very lacking and I've checked out some open source projects that claim to do this but most require manually sending input to the PTY handle vs. accepting them from the user in a terminal.
Any insight / help is very much appreciated!

ebpf: where verifier prints its messages?

Where does the verifier print its messages? I have a simple code embedded in struct bpf_insn which I load and attach as BPF_PROG_TYPE_SOCKET_FILTER type:
struct bpf_insn prog[] = {
BPF_MOV64_REG(BPF_REG_6, BPF_REG_1),
BPF_EXIT_INSN(),
};
This code is intentionally made wrong (R0 is not initialized before the exit). bpf_prog_load() returns EACCESS error and fails to load, which is expected, but I wanted to the verifier messages (nothing in dmesg or console).
When attempting to load an eBPF program, it is up to the loader to pass a buffer to the kernel verifier and to later print it to get the verifier's output.
The verifier will use this buffer provided by the user space program and print all its logs in it. Excepted for a very few specific messages, it will not print anything to the kernel logs or to the console (which is handled by your shell, not the kernel directly).
Let's have a look at a snippet from samples/bpf/sock_example.c, that you mentioned in the comments.
prog_fd = bpf_load_program(BPF_PROG_TYPE_SOCKET_FILTER, prog, insns_cnt,
"GPL", 0, bpf_log_buf, BPF_LOG_BUF_SIZE);
if (prog_fd < 0) {
printf("failed to load prog '%s'\n", strerror(errno));
goto cleanup;
}
This is the part where we attempt to load the program. We call bpf_load_program() from libbpf, and we pass it, in this order, the program type, the instructions, the number of instructions, the license string, some flag related to kernel versions, and at last: an empty buffer and its size. The size BPF_LOG_BUF_SIZE is non-null (defined in tools/lib/bpf/bpf as (UINT32_MAX >> 8)).
The function bpf_load_program() will pass all this information, including the pointer to the buffer, to the bpf() system call, which will attempt to load the program. The verifier will populate the buffer with logs (whether the load succeeds or not, but see note at the bottom). Then it is up to the loader program, again, to use these logs. The function bpf_load_program() is low-level, it does nothing with the verifier's logs in the buffer, even on failure to load. It leaves it to the caller to process or dump the logs. The sample application that you attempt to run does nothing either; therefore, the buffer is unused, and you don't get to see the logs in the console.
To see the logs, in your case, you probably just need to dump this buffer. Something as simple as the following should work:
...
if (prog_fd < 0) {
printf("failed to load prog '%s'\n", strerror(errno));
printf("%s", bpf_log_buf);
goto cleanup;
}
Note: In addition to the buffer and the size of the buffer, the loader must pass a log_level integer to the verifier, to tell it what level of verbosity it should use. If the value is at 0, the verifier prints nothing to the buffer. In the current case, we do not handle the log_level directly. bpf_load_program() does not either and sets the value to 0, but it ends up calling libbpf__bpf_prog_load() in libbpf. That function tries to load the program a first time without changing the log_level, but in case of failure, it does a new attempt with the log_level set at 1 - See Mark's pointers in the comments for details. The different values for log_level are defined in internal kernel headers and are not part of the user API, meaning the behaviour of the verifier regarding log verbosity may vary between kernel versions.

Why is FileHandle inconsistently returning 'nil'

I have an app which is inconsistently returning 'nil' when using FileHandle to open a file for Read. I'm on OSX (10.13.4), XCode 9.4, Swift 4.1
This OSX app uses the NSOpenPanel() to get a list of files selected by the user. My 'model' class code opens these files to build a collection of data structures The code which does this starts out like this and successfully gets a FileHandle EVERY TIME for any file and is able to read data from the file.
private func getFITHeader(filename: String) {
let file: FileHandle? = FileHandle(forReadingAtPath: filename)
if file == nil {
print("FITFile >>> File open failed for file \(filename)")
}
else {
var databuffer: Data
databuffer = (file?.readData(ofLength: 80))!
:
:
}
The files also contain a block of binary data which I process in another part of the app. While I develop the code for this I'm temporarily hard coding one of the same filenames as works above for test purposes. BUT this code (below) ALWAYS throws an exception 'Thread 1: Fatal error: Unexpectedly found nil while unwrapping an Optional value' when it gets to fileHandle?.seek() - for some reason the attempt to create a FileHandle is always returning 'nil' despite the code being functionally identical to tha above.
#IBAction func btnProcFile(_ sender: Any) {
var data: Data
let filename = "/Users/johncneal/Dropbox/JN Astronomy/Astronomy/Spectroscopy/RSpec_Analyses/Gamma_Cas/20161203/Gamma Cas_065.fit"
let fileHandle: FileHandle? = FileHandle(forReadingAtPath: filename)
fileHandle?.seek(toFileOffset: 2880) //skip past the headers
let dataLenToRead = 1391 * 1039 * 2
data = (fileHandle?.readData(ofLength: dataLenToRead))!
:
:
}
The code in the second function works fine in a Playground (not attaching too much meaning to that) and, wierdly, has also worked when temporarily added to a different project. Probably also worth mentioning the length of the file path doesn't seem to matter - it behaves the same on short paths.
So the question is - why is this behaviour of FileHandle reliably inconsistent?
print()'ing the filenames presented to FileHandle() showed they were identical in each case (see below). So I'm stumped and frustrated by this - any perspectives or workarounds would be appreciated.
/Users/johncneal/Dropbox/JN Astronomy/Astronomy/Spectroscopy/RSpec_Analyses/Gamma_Cas/20161203/Gamma Cas_065.fit
/Users/johncneal/Dropbox/JN Astronomy/Astronomy/Spectroscopy/RSpec_Analyses/Gamma_Cas/20161203/Gamma Cas_065.fit
Found the answer - Sandboxing !!
Darren - coincidentally I did look at the URL based route and discovering it 'throws' put some proper error reporting in the catches. Low and behold they reported I didn't have permissions on the file (which initially surprised me since I'm obviously admin on my Mac's and all the files ar local and under my username.
I bit more research turned up. this article - https://forums.developer.apple.com/thread/96062 which quickly revealed its a sandboxing problem :-) Looks like recent versions of XCode have it turned on in 'Entitlements'. The post also points out that the NSOpenPanel FileOpen dialog returns 'Security scoped urls'. At first I thought this explained why the code in the first function worked but I'm not totally convinced because I was only feeding the url.path property to FileHandle.
However, turning off Sandbox in Entitlements makes everything work just fine. Yes, I know thats not the right thing to do longer term (or if I want this to go to the App Store) so I'll be checking out the right way to do this. At least I can get on now - thanks for the input.
The FileHandle initializers are not well named.
You should use FileHandle(forReadingFrom:URL) instead of FileHandle(forReadingAtPath:String). The former is newer API that throws an error instead of returning nil. You can use the thrown error to see why it is failing, and your variables are guaranteed to be non-nil.
For example:
#IBAction func btnProcFile(_ sender: Any) {
do {
let fileUrl = URL(fileURLWithPath:"/Users/johncneal/Dropbox/JN Astronomy/Astronomy/Spectroscopy/RSpec_Analyses/Gamma_Cas/20161203/Gamma Cas_065.fit")
let fileHandle = try FileHandle(forReadingFrom: fileUrl)
fileHandle.seek(toFileOffset: 2880) //skip past the headers
let dataLenToRead = 1391 * 1039 * 2
let data: Data = fileHandle.readData(ofLength: dataLenToRead)
// etc...
} catch let error as NSError {
print("FITFile >>> File open failed: \(error)")
NSApp.presentError(error)
}
}

Error when running coreml in the background: Error computing NN outputs error

I'm running an mlmodel that is coming from keras on an iPhone 6. The predictions often fails with the error Error computing NN outputs. Does anyone know what could be the cause and if there is anything I can do about it?
do {
return try model.prediction(input1: input)
} catch let err {
fatalError(err.localizedDescription) // Error computing NN outputs error
}
EDIT: I tried apple's sample project and that one works in the background so it seems it's specific to either our project or model type.
I got the same error myself at similar "seemingly random" times. A bit of debug tracing established that it was caused by the app sometimes trying to load its coreml model when it was sent to background, then crashing or freezing when reloaded into foreground.
The message Error computing NN outputs error was preceded by:
Execution of the command buffer was aborted due to an error during execution. Insufficient Permission (to submit GPU work from background) (IOAF code 6)
I didn't need (or want) the model to be used when the app was in background, so I detected when the app was going in / out of background, set a flag and used a guard statement before attempting to call the model.
Detect when going into background using applicationWillResignActive within the AppDelegate.swift file and set a Bool flag e.g. appInBackground = true. See this for more info: Detect iOS app entering background
Detect when app re-enters foreground using applicationDidBecomeActive in the same AppDelegate.swift file, and reset flag appInBackground = false
Then in the function where you call the model, just before calling model, use a statement such as:
guard appInBackground == false else { return } // new line to add
guard let model = try? VNCoreMLModel(for modelName.model) else { fatalError("could not load model") // original line to load model
I doubt this is the most elegant solution, but it worked for me.
I haven't established why the attempt to load the model in background only happens sometimes.
In the Apple example you link to, it looks like their app only ever calls the model in response to a user input, so it will never try to load the model when in background. Hence the difference in my case ... and possibly yours as well?
In the end it was enough for us to set the usesCPUOnly flag. Using the GPU in the background seems prohibited in iOS. Apple actually wrote about this in their documentation as well. To specify this flag we couldn't use the generated model class anymore but had to call the raw coreml classes instead. I can imagine this changing in a future version however. The snippet below is taken from the generated model class, but with the added MLPredictionOptions specified.
let options = MLPredictionOptions()
options.usesCPUOnly = true // Can't use GPU in the background
// Copied from from the generated model class
let input = model_input(input: mlMultiArray)
let output = try generatedModel.model.prediction(from: input, options: options)
let result = model_output(output: output.featureValue(for: "output")!.multiArrayValue!).output

Swift lazy var inconsistent behavior

I have encountered a very unusual problem with lazy initialization.
My application reads metadata from an mp4 file, edits some of the data, and writes it back to the file. In previous versions, I read all the data when the file was opened. In the current version, I have switched to lazy reading of the data to improve performance and lower the memory footprint. However, I have encountered a very sporadic but reproducible bug, in which a few bytes seem to be read incorrectly from the file during lazy initialization.
Here is the initialization code:
lazy var data: Data = { [unowned self] in self.readDataFromFile() }()
func readDataFromFile() -> Data {
guard let file = fileHandle else { return Data() }
file.seek(toFileOffset: dataFilePointer)
let data = file.readData(ofLength: dataCountInFile)
print(Pointer: \(dataFilePointer)" + " Data: " + data.hexBytes(length: 32))
return data
}
The print statement was added to debug this problem. The fileHandle is a FileHandle object which I create when the file is selected. The dataFilePointer and dataCountInFile variables are set when the file is opened, by parsing the file for the pointers to the data.
When I read from the file for the first time, this is what I see in the log:
Pointer: 6808113917 Data: 14600000 1EA70000 12FB0000 19260000 12E20000
If I stop app without writing any data to the file, and then restart, not doing anything else, this is what I see in the log:
Pointer: 6808113917 Data: CFCB378C CFCBCDDB 00015F90 4B4231AA 55C40000
This is actually the correct data. Subsequent running of the app consistently shows the correct data.
I have put breakpoints at all points where the file could be written to verify that no data is written to the file.
Additional observations:
This bug only occurs with a few selected files. The vast majority of
files are read correctly.
This bug only occurs with Release build configuration. It does not occur in Debug configuration. That is why
I have had to use the print statement to observe what is happening.
This bug does not occur if I do not initialize the data var lazily,
i.e. if I immediately read the data from the file when the object is
initialized.
I am having a difficult time understanding what could be causing this strange behavior.