Issue Using Swift's withUnsafeMutableBytes Wrapper - swift

I'm trying to write an extension to Data that allows me to extract it in various casts.
I'm running into a strange problem that I can't quite figure out.
Before I get into generics, I'm trying out fixed data types, and I created this method:
func intValueFromData(_ inData: Data) -> Int64? {
var number = Int64(0)
let len = Swift.min(MemoryLayout<Int64>.size, inData.count)
_ = withUnsafeMutableBytes(of: &number) {
inData.copyBytes(to: $0, from: 0..<len)
}
return number
}
That works. If I do this:
var int_64 = Int64(12345)
var data = Data(bytes: &int_64, count: MemoryLayout<Int64>.size)
let fetched = intValueFromData(data)
fetched becomes "12345" as an Int64.
However, when I try to embed the same method into the Data type, like so:
extension Data {
mutating func intValueFromData() -> Int64? {
var number = Int64(0)
let len = Swift.min(MemoryLayout<Int64>.size, self.count)
_ = withUnsafeMutableBytes(of: &number) {
self.copyBytes(to: $0, from: 0..<len)
}
return number
}
}
I get a compile-time error that says that "withUnsafeMutableBytes(of: &number)" is not supposed to have an argument.
The last time that I encountered something like this, it turned out that Apple explicitly blocked a functionality, but neglected to tell us in a straightforward manner.
I am not an expert at this kind of thing, but I am wondering if anyone could shed any light on why withUnsafeMutableBytes(of: &number) behaves differently inside the extension.

In the context of Data, withUnsafeMutableBytes refers to Data.withUnsafeMutableBytes(_:).
To disambiguate this and refer to the global function, explicitly prefix the module name: _ = Swift.withUnsafeMutableBytes(of: &number) {

Just to complete the set, here's the generic function I created:
/* ###################################################################################################################################### */
// MARK: - Data Extension -
/* ###################################################################################################################################### */
/**
This extension adds the ability to extract data fron a Data instance, cast into various types.
*/
public extension Data {
/* ################################################################## */
/**
This method allows a Data instance to be cast into various standard types.
- parameter inValue: This is an inout parameter, and the type will be used to determine the cast.
- returns: the cast value (the parameter will also be set to the cast value). Can be ignored.
*/
#discardableResult
mutating func castInto<T>(_ inValue: inout T) -> T {
// Makes sure that we don't try to read past the end of the data.
let len = Swift.min(MemoryLayout<T>.size, self.count)
_ = Swift.withUnsafeMutableBytes(of: &inValue) {
self.copyBytes(to: $0, from: 0..<len)
}
return inValue
}
}

Related

How to share array memory between JavaScriptCore and Swift?

I am trying to write a Swift program that runs JS via JavaScriptCore. I wish to share memory between both parts of my program such that the JS writes to a typed array buffer created in Swift, and Swift reads and writes to it afterwards. This will be a sort of command buffer.
For example, here is some pseudocode that approximately represents what I'm planning to do:
// js
let buf;
let i = 0;
setup() {
buf = new Uint8Array(mem.alloc(N_BYTES));
}
frame() {
i = 0;
buf[i++] = some_command_enum;
}
// swift
func alloc(bytes : Int) -> the_memory {
// allocate bytes uints and save the memory here
// save a reference to the memory here
// return the memory to use in JS
}
The problem is that whenever I try actually adding the implementation to alloc, JS reports via exception that the function is undefined, meaning that something is off with the way I'm doing things. Non-returning functions are fine, so I have that down.
This is my faulty implementation (please see the comments):
// swift
#objc protocol JSMemoryExports: JSExport {
static func alloc(_ byte_count: Int) -> JSObjectRef
static func free(_ memory: JSObjectRef)
}
class JSMemory: NSObject, JSMemoryExports {
// What is the correct return type?
class func alloc(_ byte_count: Int) -> JSObjectRef {
// temp
let jsContext = JS_Controller.js.ctx!
print("BYTE_COUNT", byte_count)
// allocating a typed array
let arr = JSObjectMakeTypedArray(jsContext.jsGlobalContextRef!, kJSTypedArrayTypeUint8Array, byte_count, nil)
// just testing here to see how I'd write to this buffer (Note: is this the fastest way, or is all this memory binding slow?:
// getting the raw bytes
let ptr = JSObjectGetTypedArrayBytesPtr(jsContext.jsGlobalContextRef!, arr, nil)
//let buf = JSObjectGetTypedArrayBuffer(jsContext.jsGlobalContextRef, arr, nil)
let u8Ptr = ptr!.bindMemory(to: UInt8.self, capacity: byte_count)
//u8Ptr[0] = 5
return arr!
}
}
...
jsContext["mem"] = JSMemory.self
// js
const buf = new Uint8Array(mem.alloc(8)) // JS Exception: TypeError: mem.alloc is not a function. (In 'mem.alloc(8)', 'mem.alloc' is undefined)
I've seen variants of function binding that uses some sort of #convention attribute. Am I meant to use that instead?
What is the correct thing to do?
The documentation isn't very helpful unless you piece together a lot of information from separate sources. The seemingly working solution involves using parts of the older C API that are callable in Swift, unsafe pointers, and making sure the return values of the bound functions are JSValue?s. That makes sense since JavaScript functions all return an object, null, or undefined. An optional type mirrors this behavior.
Here is my work-in-progress code for anyone who might need some leads:
Just for an update, I've figured out how to mix the old C API with the new more limited Swift-specific APIs. I haven't yet made sure I'm not leaking memory, but it looks like I've found what I needed, hopefully.
In case you ever wanted to know:
#objc protocol JSMemoryExports: JSExport {
// note that I'm returning an optional
static func Uint8ArrayMake(_ count : JSValue) -> JSValue?
}
class JSMemory: NSObject, JSMemoryExports {
class func UInt8ArrayMake(_ count : JSValue) -> JSValue? {
guard !count.isUndefined && !count.isNull else {
return nil
}
let ref : JSValueRef = JSObjectMakeTypedArray(
JS_Controller.js.ctx.jsGlobalContextRef!,
kJSTypedArrayTypeUint8Array,
Int(count.toInt32()),
nil
)!
// if you want to modify the data
// let ptr = JSObjectGetTypedArrayBytesPtr(
// JS_Controller.js.ctx.jsGlobalContextRef!, ref, nil
// )
return JSValue(jsValueRef: ref, in: JS_Controller.js.ctx)
}
}
Here are a couple helpful references:
pointers in Swift
manual memory management in Swift

'UnsafePointer<Int8>' is not convertible to 'UnsafePointer<_>

I am trying to implement to write a wrapper around libssh2 using Swift. The following code is for removing a file via SFTP.
func removeFile(_ path: String) {
let data = path.data(using: String.Encoding.utf8)!
let result = data.withUnsafeBytes { (pointer: UnsafePointer<Int8>) -> Int in
return libssh2_sftp_unlink_ex(sftpHandle, pointer, data.count)
}
}
For pointer: UnsafePointer<Int8> I am getting the following error message:
'UnsafePointer<Int8>' is not convertible to 'UnsafePointer<_>
I found this thread about a similar Problem with UInt8. I tried removing the cast but was just getting the next error:
'Swift.UnsafePointer<_>' is not convertible to 'Swift.UnsafePointer<_>'
Running the libssh2_sftp_unlink_ex(sftpHandle, pointer, data.count) outside the closure with a dummy pointer works.
I also found this answer on converting a String to UInt8, the problem is that I wasn't able to port it to Int8. Any ideas on how to correctly convert the pointer?
data.withUnsafeBytes calls the closure with an UnsafeRawBufferPointer, this has to be “bound” to an UnsafePointer<Int8>. Also data.count must be converted to an UInt32 (aka CUnsignedInt) because that is how the C type unsigned integer is imported to Swift:
func removeFile(_ path: String) {
let data = path.data(using: String.Encoding.utf8)!
let result = data.withUnsafeBytes {
libssh2_sftp_unlink_ex(sftpHandle,
$0.bindMemory(to: Int8.self).baseAddress,
UInt32(data.count))
}
}
Alternatively, use the withCString() method of String:
func removeFile(_ path: String) {
let result = path.withCString {
libssh2_sftp_unlink_ex(sftpHandle, $0, UInt32(strlen($0)))
}
}
Even simpler: use the variant with needs only a C string and not an explicit string length. Here the compiler automatically creates the code to convert the Swift string to a temporary C string:
func removeFile(_ path: String) {
let result = libssh2_sftp_unlink(sftpHandle, path)
}
(Does not work because libssh2_sftp_unlink is a macro and not imported to Swift.)

Reference Types/Subclassing, and Changes to Swift 4 Codable & encoder/decoders

I'm struggling to understand class/reference type behavior and how this relates to changes as I try to upgrade and reduce code using Codable in Swift 4.
I have two classes – a SuperClass with all of the data that will be persistent and that I save to UserDefaults (a place name & string with coordinates), and a SubClass that contains additional, temporary info that I don't need (weather data for the SuperClass coordinates).
In Swift 3 I used to save data like this:
func saveUserDefaults() {
var superClassArray = [SuperClass]()
// subClassArray is of type [SubClass] and contains more data per element.
superClassArray = subClassArray
let superClassData = NSKeyedArchiver.archivedData(withRootObject: superClassArray)
UserDefaults.standard.set(superClassData, forKey: " superClassData")
}
SuperClass conformed to NSObject & NSCoding
It also included the required init decoder & the encode function.
It all worked fine.
In trying to switch to Swift 4 & codable I've modified SuperClass to conform to Codable.
SuperClass now only has one basic initializer and none of the encoder/decoder stuff from Swift 3. There is no KeyedArchiving happening with this new approach (below). SubClass remains unchanged. Unfortunately I crash on the line where I try? encoder.encode [giving a Thread 1: EXC_BAD_ACCESS (code=1, address=0x10)]. My assumption is that the encoder is getting confused with identical reference types where one is SuperClass and one SubClass (subClassArray[0] === superClassArray[0] is true).
I thought this might work:
func saveUserDefaults() {
var superClassArray = [SuperClass]()
superClassArray = subClassArray
// assumption was that the subclass would only contain parts of the superclass & wouldn't produce an error when being encoded
let encoder = JSONEncoder()
if let encoded = try? encoder.encode(superClassArray){
UserDefaults.standard.set(encoded, forKey: " superClassArray ")
} else {
print("Save didn't work!")
}
}
Then, instead of creating an empty superClassArray, then using:
superClassArray = subClassArray, as shown above, I replace this with the single line:
let superClassArray: [SuperClass] = subClassArray.map{SuperClass(name: $0.name, coordinates: $0.coordinates)}
This works. Again, assumption is because I'm passing in the values inside of the class reference type & haven't made the superClassArray = subClassArray. Also, as expected, subClassArray[0] === superClassArray[0] is false
So why did the "old stuff" in Swift 3 work, even though I used the line superClassArray = subClassArray before the let superClassData = NSKeyedArchiver.archivedData(withRootObject: superClassArray)
? Am I essentially achieving the same result by creating the array in Swift 4 that was happening with the old Swift 3 encoder/decoder? Is the looping / recreation
Thanks!
Polymorphic persistence appears to be broken by design.
The bug report SR-5331 quotes the response they got on their Radar.
Unlike the existing NSCoding API (NSKeyedArchiver), the new Swift 4 Codable implementations do not write out type information about encoded types into generated archives, for both flexibility and security. As such, at decode time, the API can only use the concrete type your provide in order to decode the values (in your case, the superclass type).
This is by design — if you need the dynamism required to do this, we recommend that you adopt NSSecureCoding and use NSKeyedArchiver/NSKeyedUnarchiver
I am unimpressed, having thought from all the glowing articles that Codable was the answer to some of my prayers. A parallel set of Codable structs that act as object factories is one workaround I'm considering, to preserve type information.
Update I have written a sample using a single struct that manages recreating polymorphic classes. Available on GitHub.
I was not able to get it to work easily with subclassing. However, classes that conform to a base protocol can apply Codable for default encoding. The repo contains both keyed and unkeyed approaches. The simpler is unkeyed, copied below
// Demo of a polymorphic hierarchy of different classes implementing a protocol
// and still being Codable
// This variant uses unkeyed containers so less data is pushed into the encoded form.
import Foundation
protocol BaseBeast {
func move() -> String
func type() -> Int
var name: String { get }
}
class DumbBeast : BaseBeast, Codable {
static let polyType = 0
func type() -> Int { return DumbBeast.polyType }
var name:String
init(name:String) { self.name = name }
func move() -> String { return "\(name) Sits there looking stupid" }
}
class Flyer : BaseBeast, Codable {
static let polyType = 1
func type() -> Int { return Flyer.polyType }
var name:String
let maxAltitude:Int
init(name:String, maxAltitude:Int) {
self.maxAltitude = maxAltitude
self.name = name
}
func move() -> String { return "\(name) Flies up to \(maxAltitude)"}
}
class Walker : BaseBeast, Codable {
static let polyType = 2
func type() -> Int { return Walker.polyType }
var name:String
let numLegs: Int
let hasTail: Bool
init(name:String, legs:Int=4, hasTail:Bool=true) {
self.numLegs = legs
self.hasTail = hasTail
self.name = name
}
func move() -> String {
if numLegs == 0 {
return "\(name) Wriggles on its belly"
}
let maybeWaggle = hasTail ? "wagging its tail" : ""
return "\(name) Runs on \(numLegs) legs \(maybeWaggle)"
}
}
// Uses an explicit index we decode first, to select factory function used to decode polymorphic type
// This is in contrast to the current "traditional" method where decoding is attempted and fails for each type
// This pattern of "leading type code" can be used in more general encoding situations, not just with Codable
//: **WARNING** there is one vulnerable practice here - we rely on the BaseBeast types having a typeCode which
//: is a valid index into the arrays `encoders` and `factories`
struct CodableRef : Codable {
let refTo:BaseBeast //In C++ would use an operator to transparently cast CodableRef to BaseBeast
typealias EncContainer = UnkeyedEncodingContainer
typealias DecContainer = UnkeyedDecodingContainer
typealias BeastEnc = (inout EncContainer, BaseBeast) throws -> ()
typealias BeastDec = (inout DecContainer) throws -> BaseBeast
static var encoders:[BeastEnc] = [
{(e, b) in try e.encode(b as! DumbBeast)},
{(e, b) in try e.encode(b as! Flyer)},
{(e, b) in try e.encode(b as! Walker)}
]
static var factories:[BeastDec] = [
{(d) in try d.decode(DumbBeast.self)},
{(d) in try d.decode(Flyer.self)},
{(d) in try d.decode(Walker.self)}
]
init(refTo:BaseBeast) {
self.refTo = refTo
}
init(from decoder: Decoder) throws {
var container = try decoder.unkeyedContainer()
let typeCode = try container.decode(Int.self)
self.refTo = try CodableRef.factories[typeCode](&container)
}
func encode(to encoder: Encoder) throws {
var container = encoder.unkeyedContainer()
let typeCode = self.refTo.type()
try container.encode(typeCode)
try CodableRef.encoders[typeCode](&container, refTo)
}
}
struct Zoo : Codable {
var creatures = [CodableRef]()
init(creatures:[BaseBeast]) {
self.creatures = creatures.map {CodableRef(refTo:$0)}
}
func dump() {
creatures.forEach { print($0.refTo.move()) }
}
}
//: ---- Demo of encoding and decoding working ----
let startZoo = Zoo(creatures: [
DumbBeast(name:"Rock"),
Flyer(name:"Kookaburra", maxAltitude:5000),
Walker(name:"Snake", legs:0),
Walker(name:"Doggie", legs:4),
Walker(name:"Geek", legs:2, hasTail:false)
])
startZoo.dump()
print("---------\ntesting JSON\n")
let encoder = JSONEncoder()
encoder.outputFormatting = .prettyPrinted
let encData = try encoder.encode(startZoo)
print(String(data:encData, encoding:.utf8)!)
let decodedZoo = try JSONDecoder().decode(Zoo.self, from: encData)
print ("\n------------\nAfter decoding")
decodedZoo.dump()
Update 2020-04 experience
This approach continues to be more flexible and superior to using Codable, at the cost of a bit more programmer time. It is used very heavily in the Touchgram app which provides rich, interactive documents inside iMessage.
There, I need to encode multiple polymorphic hierarchies, including different Sensors and Actions. By storing signatures of decoders, it not only provides with subclassing but also allows me to keep older decoders in the code base so old messages are still compatible.

Delegating to another initializer from a closure

I'm looking at ways to create a DispatchData instance out of a Data instance. I started with a static function:
static func dispatchData(fromData data: Data) -> DispatchData {
return data.withUnsafeBytes { (typedPtr: UnsafePointer<UInt8>) -> DispatchData in
let bufferPtr = UnsafeRawBufferPointer(start: UnsafeRawPointer(typedPtr), count: data.count)
return DispatchData(bytes: bufferPtr)
}
}
This works fine (and is coincidentally very similar to this answer). I then decided I'd like to add this as an initializer to DispatchData in an extension and wrote:
private extension DispatchData {
init(data: Data) {
data.withUnsafeBytes { (typedPtr: UnsafePointer<UInt8>) in // 1
let bufferPtr = UnsafeRawBufferPointer(start: UnsafeRawPointer(typedPtr), count: data.count)
self.init(bytes: bufferPtr)
}
}
}
At this, the compiler balked on the line marked // 1:
Variable 'self.__wrapped' captured by a closure before being initialized
It makes sense — the compiler doesn't want self to be captured before I've delegated to init(bytes: UnsafeRawBufferPointer), since stored variables — like __wrapped, in this case — aren't yet initialized. But I can't see another way to get the UnsafeRawBufferPointer; I can't return it from data.withUnsafeBytes, per the documentation:
Warning
The byte pointer argument should not be stored and used outside of the lifetime of the call to the closure.
I know I can just go back to using a static function to achieve what I want, but I'd prefer it if there were a way to add this initializer. Is there a way to make this initializer work as intended?
The problem with your extension is that although the compiler knows you're passing the closure as non-escaping argument to withUnsafeBytes(_:); it doesn't have any guarantee that it will be called once and only once (which is required for initialisation).
One simple solution is to, rather than chain to another initialiser, just construct a new instance of DispatchData yourself, and then assign this to self, thus completing initialisation:
import Foundation
private extension DispatchData {
init(data: Data) {
self = data.withUnsafeBytes { (typedPtr: UnsafePointer<UInt8>) in
DispatchData(bytes:
UnsafeRawBufferPointer(start: typedPtr, count: data.count)
)
}
}
}
Note also that there's an implicit conversion from UnsafePointer<UInt8> to UnsafeRawPointer? when passing it as an argument; so we don't need to wrap it in an initialiser call.
And in Swift 5, withUnsafeBytes gives you a UnsafeRawBufferPointer directly:
private extension DispatchData {
init(data: Data) {
self = data.withUnsafeBytes(DispatchData.init)
}
}

Using a Type Variable in a Generic

I have this question except for Swift. How do I use a Type variable in a generic?
I tried this:
func intType() -> Int.Type {
return Int.self
}
func test() {
var t = self.intType()
var arr = Array<t>() // Error: "'t' is not a type". Uh... yeah, it is.
}
This didn't work either:
var arr = Array<t.Type>() // Error: "'t' is not a type"
var arr = Array<t.self>() // Swift doesn't seem to even understand this syntax at all.
Is there a way to do this? I get the feeling that Swift just doesn't support it and is giving me somewhat ambiguous error messages.
Edit: Here's a more complex example where the problem can't be circumvented using a generic function header. Of course it doesn't make sense, but I have a sensible use for this kind of functionality somewhere in my code and would rather post a clean example instead of my actual code:
func someTypes() -> [Any.Type] {
var ret = [Any.Type]()
for (var i = 0; i<rand()%10; i++) {
if (rand()%2 == 0){ ret.append(Int.self) }
else {ret.append(String.self) }
}
return ret
}
func test() {
var ts = self.someTypes()
for t in ts {
var arr = Array<t>()
}
}
Swift's static typing means the type of a variable must be known at compile time.
In the context of a generic function func foo<T>() { ... }, T looks like a variable, but its type is actually known at compile time based on where the function is called from. The behavior of Array<T>() depends on T, but this information is known at compile time.
When using protocols, Swift employs dynamic dispatch, so you can write Array<MyProtocol>(), and the array simply stores references to things which implement MyProtocol — so when you get something out of the array, you have access to all functions/variables/typealiases required by MyProtocol.
But if t is actually a variable of kind Any.Type, Array<t>() is meaningless since its type is actually not known at compile time. (Since Array is a generic struct, the compiler needs know which type to use as the generic parameter, but this is not possible.)
I would recommend watching some videos from WWDC this year:
Protocol-Oriented Programming in Swift
Building Better Apps with Value Types in Swift
I found this slide particularly helpful for understanding protocols and dynamic dispatch:
There is a way and it's called generics. You could do something like that.
class func foo() {
test(Int.self)
}
class func test<T>(t: T.Type) {
var arr = Array<T>()
}
You will need to hint the compiler at the type you want to specialize the function with, one way or another. Another way is with return param (discarded in that case):
class func foo() {
let _:Int = test()
}
class func test<T>() -> T {
var arr = Array<T>()
}
And using generics on a class (or struct) you don't need the extra param:
class Whatever<T> {
var array = [T]() // another way to init the array.
}
let we = Whatever<Int>()
jtbandes' answer - that you can't use your current approach because Swift is statically typed - is correct.
However, if you're willing to create a whitelist of allowable types in your array, for example in an enum, you can dynamically initialize different types at runtime.
First, create an enum of allowable types:
enum Types {
case Int
case String
}
Create an Example class. Implement your someTypes() function to use these enum values. (You could easily transform a JSON array of strings into an array of this enum.)
class Example {
func someTypes() -> [Types] {
var ret = [Types]()
for _ in 1...rand()%10 {
if (rand()%2 == 0){ ret.append(.Int) }
else {ret.append(.String) }
}
return ret
}
Now implement your test function, using switch to scope arr for each allowable type:
func test() {
let types = self.someTypes()
for type in types {
switch type {
case .Int:
var arr = [Int]()
arr += [4]
case .String:
var arr = [String]()
arr += ["hi"]
}
}
}
}
As you may know, you could alternatively declare arr as [Any] to mix types (the "heterogenous" case in jtbandes' answer):
var arr = [Any]()
for type in types {
switch type {
case .Int:
arr += [4]
case .String:
arr += ["hi"]
}
}
print(arr)
I would break it down with the things you already learned from the first answer. I took the liberty to refactor some code. Here it is:
func someTypes<T>(t: T.Type) -> [Any.Type] {
var ret = [Any.Type]()
for _ in 0..<rand()%10 {
if (rand()%2 == 0){ ret.append(T.self) }
else {
ret.append(String.self)
}
}
return ret
}
func makeArray<T>(t: T) -> [T] {
return [T]()
}
func test() {
let ts = someTypes(Int.self)
for t in ts {
print(t)
}
}
This is somewhat working but I believe the way of doing this is very unorthodox. Could you use reflection (mirroring) instead?
Its possible so long as you can provide "a hint" to the compiler about the type of... T. So in the example below one must use : String?.
func cast<T>(_ value: Any) -> T? {
return value as? T
}
let inputValue: Any = "this is a test"
let casted: String? = cast(inputValue)
print(casted) // Optional("this is a test")
print(type(of: casted)) // Optional<String>
Why Swift doesn't just allow us to let casted = cast<String>(inputValue) I'll never know.
One annoying scenerio is when your func has no return value. Then its not always straightford to provide the necessary "hint". Lets look at this example...
func asyncCast<T>(_ value: Any, completion: (T?) -> Void) {
completion(value as? T)
}
The following client code DOES NOT COMPILE. It gives a "Generic parameter 'T' could not be inferred" error.
let inputValue: Any = "this is a test"
asyncCast(inputValue) { casted in
print(casted)
print(type(of: casted))
}
But you can solve this by providing a "hint" to compiler as follows:
asyncCast(inputValue) { (casted: String?) in
print(casted) // Optional("this is a test")
print(type(of: casted)) // Optional<String>
}