I'm trying to initialize Data from an UnsafeBufferPointer, but it's throwing an EXC_BAD_ACCESS when it hits the third line. Help appreciated.
let pointer = UnsafePointer<UInt8>(UnsafePointer<UInt8>(bitPattern: 15)!)
let buffer = UnsafeBufferPointer<UInt8>(start: pointer, count: 1)
let data = Data(bytes: Array(buffer)) // EXC_BAD_ACCESS
My end goal is to convert bits of data to some human readable format (eg., convert a bit pattern of 15 to "F"). I was hoping to initialize a String from the data object as a hex value. I'm open to better and correct ways of going about this.
Related
let str = "This is a swift bug"
let data = Data(str.utf8)
print("data size = ", data.endIndex, data.count)
let trimmed = data[2..<data.endIndex]
print("trimmed size = ", trimmed.endIndex, trimmed.count)
The result is
data size = 19 19
trimmed size = 19 17
According to the Apple doc about endIndex:
This is the “one-past-the-end” position, and will always be equal to the count.
Is it a bug? or I'm missing something?
You should open an Apple Feedback for the documentation of Data.endIndex. It's incorrect.
The startIndex of Data is not promised to be zero, and this is an example of when it isn't. Using the Int subscript on Data is unfortunately very dangerous unless you know precisely how the Data was constructed (and specifically that it has a zero index).
Data uniquely mixes two facts that make it tricky to use correctly:
It is its own Slice
Its Index is Int
For some discussion of this, and suggested patterns, see Data.popFirst(), removeFirst() adjust indices. Also see Data ranged subscribe strange behavior for another version of this question.
When you use an expression like array[2..<array.endIndex] you are creating a slice. A slice is a sort of window onto an array (or something similar to an array). Its startIndex is not necessarily 0 and its endIndex is not necessarily one after the last index of the original.
Example:
let arr = Array(1...10)
print(arr.startIndex) // 0
print(arr.endIndex) // 10
let slice = arr[2...4]
print(slice.startIndex) // 2
print(slice.endIndex) // 5
print(slice.count) // 3
You see how this works? The slice has its own logic. Its size (count) is the size of the slice, but its index numbers come from the original array, because the slice is nothing but a pointer into a section of the original array. It has no independent existence; it is just a way of seeing, as it were.
An important consequence is that slice[0] will crash: the first available index of slice is 2, as we have already been told. This is why it is crucial to know whether you're dealing with an original array or a slice.
However, at least you have reason to know that this issue might exist, because slice has a special type — Array<Int>.SubSequence, meaning an ArraySlice. But the fact that you are encountering this by way of Data makes it more tricky, because trimmed is typed as a Data, not as a DataSlice! It is in fact a Data.SubSequence, but you have no simple way of finding that out! That's because Data.SubSequence is typealiased to Data itself. This is to be regarded as a flaw in the Data implementation.
Nevertheless, it is exactly the same phenomenon. These answers should look strangely familiar:
let str = "This is a swift bug"
let data = Data(str.utf8)
let trimmed = data[2...4]
print(trimmed.startIndex) // 2
print(trimmed.endIndex) // 5
print(trimmed.count) // 3
The best way to solve this is Don't Do That. To take a subrange of a Data as a true Data, use subdata:
let trimmed2 = data.subdata(in: 2..<5)
print(trimmed2.startIndex) // 0, and so on; it's an independent copy
I would like to know how to read a binary file into memory (writing it to memory like an "Array Buffer" from JavaScript), and write to different parts of memory 8-bit, 16-bit, 32-bit etc. values, even 5 bit or 10 bit values.
extension Binary {
static func readFileToMemory(_ file) -> ArrayBuffer {
let data = NSData(contentsOfFile: "/path/to/file/7CHands.dat")!
var dataRange = NSRange(location: 0, length: ?)
var ? = [Int32](count: ?, repeatedValue: ?)
data.getBytes(&?, range: dataRange)
}
static func writeToMemory(_ buffer, location, value) {
buffer[location] = value
}
static func readFromMemory(_ buffer, location) {
return buffer[location]
}
}
I have looked at a bunch of places but haven't found a standard reference.
https://github.com/nst/BinUtils/blob/master/Sources/BinUtils.swift
https://github.com/apple/swift/blob/master/stdlib/public/core/ArrayBuffer.swift
https://github.com/uraimo/Bitter/blob/master/Sources/Bitter/Bitter.swift
In Swift, how do I read an existing binary file into an array?
Swift - writing a byte stream to file
https://apple.github.io/swift-nio/docs/current/NIO/Structs/ByteBuffer.html
https://github.com/Cosmo/BinaryKit/blob/master/Sources/BinaryKit.swift
https://github.com/vapor-community/bits/blob/master/Sources/Bits/Data%2BBytesConvertible.swift
https://academy.realm.io/posts/nate-cook-tryswift-tokyo-unsafe-swift-and-pointer-types/
https://medium.com/#gorjanshukov/working-with-bytes-in-ios-swift-4-de316a389a0c
I would like for this to be as low-level as possible. So perhaps using UnsafeMutablePointer, UnsafePointer, or UnsafeMutableRawPointer.
Saw this as well:
let data = NSMutableData()
var goesIn: Int32 = 42
data.appendBytes(&goesIn, length: sizeof(Int32))
println(data) // <2a000000]
var comesOut: Int32 = 0
data.getBytes(&comesOut, range: NSMakeRange(0, sizeof(Int32)))
println(comesOut) // 42
I would basically like to allocate a chunk of memory and be able to read and write from it. Not sure how to do that. Perhaps using C is the best way, not sure.
Just saw this too:
let rawData = UnsafeMutablePointer<UInt8>.allocate(capacity: width * height * 4)
If you're looking for low level code you'll need to use UnsafeMutableRawPointer. This is a pointer to a untyped data. Memory is accessed in bytes, so 8 chunks of at least 8 bits. I'll cover multiples of 8 bits first.
Reading a File
To read a file this way, you need to manage file handles and pointers yourself. Try the the following code:
// Open the file in read mode
let file = fopen("/Users/joannisorlandos/Desktop/ownership", "r")
// Files need to be closed manually
defer { fclose(file) }
// Find the end
fseek(file, 0, SEEK_END)
// Count the bytes from the start to the end
let fileByteSize = ftell(file)
// Return to the start
fseek(file, 0, SEEK_SET)
// Buffer of 1 byte entities
let pointer = UnsafeMutableRawPointer.allocate(byteCount: fileByteSize, alignment: 1)
// Buffer needs to be cleaned up manually
defer { pointer.deallocate() }
// Size is 1 byte
let readBytes = fread(pointer, 1, fileByteSize, file)
let errorOccurred = readBytes != fileByteSize
First you need to open the file. This can be done using Swift strings since the compiler makes them into a CString itself.
Because cleanup is all for us on this low level, a defer is put in place to close the file at the end.
Next, the file is set to seek the end of the file. Then the distance between the start of the file and the end is calculated. This is used later, so the value is kept.
Then the program is set to return to the start of the file, so the application starts reading from the start.
To store the file, a pointer is allocated with the amount of bytes that the file has in the file system. Note: This can change inbetween the steps if you're extremely unlucky or the file is accessed quite often. But I think for you, this is unlikely.
The amount of bytes is set, and aligned to one byte. (You can learn more about memory alignment on Wikipedia.
Then another defer is added to make sure no memory leaks at the end of this code. The pointer needs to be deallocated manually.
The file's bytes are read and stored in the pointer. Do note that this entire process reads the file in a blocking manner. It can be more preferred to read files asynchronously, if you plan on doing that I'll recommend looking into a library like SwiftNIO instead.
errorOccurred can be used to throw an error or handle issues in another manner.
From here, your buffer is ready for manipulation. You can print the file if it's text using the following code:
print(String(cString: pointer.bindMemory(to: Int8.self, capacity: fileByteSize)))
From here, it's time to learn how to read manipulate the memory.
Manipulating Memory
The below demonstrates reading byte 20..<24 as an Int32.
let int32 = pointer.load(fromByteOffset: 20, as: Int32.self)
I'll leave the other integers up to you. Next, you can alos put data at a position in memory.
pointer.storeBytes(of: 40, toByteOffset: 30, as: Int64.self)
This will replace byte 30..<38 with the number 40. Note that big endian systems, although uncommon, will store information in a different order from normal little endian systems. More about that here.
Modifying Bits
As you notes, you're also interested in modifying five or ten bits at a time. To do so, you'll need to mix the previous information with the new information.
var data32bits = pointer.load(fromByteOffset: 20, as: Int32.self)
var newData = 0b11111000
In this case, you'll be interested in the first 5 bits and want to write them over bit 2 through 7. To do so, first you'll need to shift the bits to a position that matches the new position.
newData = newData >> 2
This shifts the bits 2 places to the right. The two left bits that are now empty are therefore 0. The 2 bits on the right that got shoved off are not existing anymore.
Next, you'll want to get the old data from the buffer and overwrite the new bits.
To do so, first move the new byte into a 32-bits buffer.
var newBits = numericCast(newData) as Int32
The 32 bits will be aligned all the way to the right. If you want to replace the second of the four bytes, run the following:
newBits = newBits << 16
This moves the fourth pair 16 bit places left, or 2 bytes. So it's now on position 1 starting from 0.
Then, the two bytes need to be added on top of each other. One common method is the following:
let oldBits = data32bits & 0b11111111_11000001_11111111_11111111
let result = oldBits | newBits
What happens here is that we remove the 5 bits with new data from the old dataset. We do so by doing a bitwise and on the old 32 bits and a bitmap.
The bitmap has all 1's except for the new locations which are being replaced. Because those are empty in the bitmap, the and operator will exclude those bits since one of the two (old data vs. bitmap) is empty.
AND operators will only be 1 if both sides of the operator are 1.
Finally, the oldBits and the newBits are merged with an OR operator. This will take each bit on both sides and set the result to 1 if the bits at both positions are 1.
This will merge successfully since both buffers contain 1 bits that the other number doesn't set.
Say I have the following code...
let x = "ABCDE"
// 'x' is a String
var y = x[1...3]
// 'y' is a Substring that equals "BCD"
If you only have access to y, is it possible to access x, or specifically parts of x which are outside the range of y? (i.e. can you access 'A' or 'E', or grow the range of y?)
So here's what Apple says:
Important
Don’t store substrings longer than you need them to perform a specific
operation. A substring holds a reference to the entire storage of the
string it comes from, not just to the portion it presents, even when
there is no other reference to the original string. Storing substrings
may, therefore, prolong the lifetime of string data that is no longer
otherwise accessible, which can appear to be memory leakage.
Now I find their use of the word "otherwise" in the last sentence rather interesting. It seems to me to keep the door open on this question - could a substring be manipulated to be expanded to include memory on either side that we know still exists as part of the original string?
So here's what I'd think is a fair test:
let x = "ABCDEFGH"
let substr = x.prefix(3)
var substrIndex = substr.startIndex
substr.formIndex(&substrIndex, offsetBy: 4) // offset beyond the substring
let prefix = substr.prefix(through:substrIndex)
print(prefix)
So what'cha think that would print?
Actually we never get to the print. We get a runtime fatal error instead.
Thread 1: Fatal error: Operation results in an invalid index
BTW, even trying the following results in an EXC_BAD_ACCESS crash:
let x = "ABCDEFGH"
var substr = x.prefix(3)
withUnsafePointer(to: &substr)
{ substrPointer in
let z = substrPointer.advanced(by: 3)
print(z.pointee)
}
So I don't think there's a way to get to the rest of the string if you just have a substring... from within Substring or String classes anyhow, or even dealing with unsafe pointers. I'm sure there's a way using direct memory access, for Apple claims the rest of the String's memory is there... but you'd probably have to fall back to C or C++.
I am trying to do some computation on the raw PCM samples of a mp3 files I'm playing with an AVAudioEngine graph. I have a closure every 44100 samples that provides an AVAudioPCMBuffer. It has a property channelData of type UnsafePointer<UnsafeMutablePointer<Float>>?. I have not worked with pointers in Swift 3 and so I'm unclear how to access these Float values.
I have the following code but there are many issues:
audioPlayerNode.installTap(onBus: 0,
bufferSize: 1024,
format: audioPlayerNode.outputFormat(forBus: 0)) { (pcmBuffer, time) in
let numChans = Int(pcmBuffer.format.channelCount)
let frameLength = pcmBuffer.frameLength
if let chans = pcmBuffer.floatChannelData?.pointee {
for a in 0..<numChans {
let samples = chans[a]// samples is type Float. should be pointer to Floats.
for b in 0..<flength {
print("sample: \(b)") // should be samples[b] but that gives error as "samples" is Float
}
}
}
For instance, how do I iterate through the UnsafeMutablePointer<Floats which are N float pointers where N is the number of channels in the buffer. I could not find discussion on accessing buffer samples in the Apple Docs on this Class.
I think the main problem is let samples = chans[a]. Xcode says chans is of type UnsafeMutablePointer<Float>. But that should be NumChannels worth of those pointers. Which is why I use a in 0..<numChans to subscript it. Yet I get just Float when I do.
EDIT:
hm, seems using chans.advanced(by: a) instead of subscripting fixed things
Here is what I've found:
let arraySize = Int(buffer.frameLength)
let samples = Array(UnsafeBufferPointer(start: buffer.floatChannelData![0], count:arraySize))
This is assuming buffer is the name of your AVAudioPCMBuffer.
This way you can avoid pointers, which is likely much simpler. Now you can actually search through the data using a for loop.
I think I get the basic idea behind ASN.1 parsing. Walk the bytes, interpret them and do something useful with them. Alas I am stuck on the implementation.
Apple has no sample code (that I could find), probably for security reasons. OpenSSL is not very well documented, so I can only guess at what the functions actually do. The only sample code in swift I did find doesn't handle case 17 (the in-app purchases), which is the one thing I am interested in.
I tried figuring out where in the data stream the pointer is located, but I always get the same nonsensical result of 49.
let receiptContents = NSData(contentsOfURL: receiptLocation)!
let receiptBIO = BIO_new(BIO_s_mem())
BIO_write(receiptBIO, receiptContents.bytes, Int32(receiptContents.length))
contents = d2i_PKCS7_bio(receiptBIO, nil)
//parsing
var currentIndex = UnsafePointer<UInt8>()
var endIndex = UnsafePointer<UInt8>()
let octets = pkcs7_d_data(pkcs7_d_sign(self.contents).memory.contents)
var ptr = UnsafePointer<UInt8>(octets.memory.data)
let end = ptr.advancedBy(Int(octets.memory.length))
println(ptr.memory) //always 49 ???
println(end.memory) //always 0 ???
println(octets.memory.length)
I tried parsing the NSData myself, but well, what is the type of the binaire data?
receiptContents = NSData(contentsOfURL: receiptLocation)!
//get bytes
let count = receiptContents.length / sizeof(UInt8)
var bytes = [UInt8](count: count, repeatedValue: 0)
receiptContents.getBytes(&bytes, length:count * sizeof(UInt8))
//parsing
for index in 0...5
{
let value = Int(bytes[index])
println(value)
}
I get this output:
48
130
21
57
6
9
but if understand the ASN.1 format correctly, it's supposed to start with a value of 17 (set), then 3 bytes for the length (Int24?), then a value of 16 (first sequence), sequence length (1 byte), sequence type (1 byte), sequence payload, (repeat).
Other types like Int32, Int16 makes even less sense to me.
Not sure how to proceed here. Any suggestions?
From the outset, I suppose I should disclaim this by saying I'm not very familiar with many of the underlying technologies (Swift, OpenSSL, the Biometrics standards that I think you're working with).
That said, you're probably running afoul of BER's tagging rules. The Wiki article on X.690 has some introductory comments about how BER tags are constructed, but really you'll want to consult Annex A of X.690 for an example encoding and X.680 §8 for information about tagging.
A SET type can appear in several different forms; but in your case 49 = 0x31 = 0b00110001 = UNIVERSAL 17 (SET, constructed). Other forms may occur, but this is the only one that's explicitly identified as a SET from the tag itself: the specification may use a different tag for a SET type.