Splitting Data into chunks in Swift 3 - swift

I need to send images read from the Photo Library over the wire - in chunks of 5MB.
I read an image from the library using: PHImageManager.requestImageData(for:options:resultHandler:) and get a Data object. I would then like to efficiently split the data into chunks (without copying the memory). What would be the best way to do that?
This is what I have so far:
imageData.withUnsafeBytes { (unsafePointer: UnsafePointer<UInt8>) -> Void in
let totalSize = data.endIndex
var offset = 0
while offset < totalSize {
let chunkSize = offset + uploadChunkSize > totalSize ? totalSize - offset : uploadChunkSize
let chunk = Data(bytesNoCopy: unsafePointer, count: chunkSize, deallocator: Data.Deallocator.none)
// send the chunk...
offset += chunkSize
}
}
However I get this error at compile time:
Cannot convert value of type 'UnsafePointer' to expected argument type 'UnsafeMutableRawPointer'
If I use mutableBytes:
data.withUnsafeMutableBytes { (unsafePointer: UnsafeMutablePointer<UInt8>) -> Void in... }
Then I get the compile-time error:
Cannot use mutating member on immutable value: 'data' is a 'let' constant
Which is correct, since I do not really want to make changes to the image data. I only want to send one chunk of it at a time.
Is there a better way to do this?

Hi there!
I need the same behaviour and came up with this solution. You pointed the error right and the solution is just to create the UnsafeMutableRawPointer with the UnsafePointer address. It's the fastest solution I found yet.
One other thing is to add the offset to the base address of the mutRawPointer when you create a chunk.
50MB data in 2MB chunks takes ~ 0.009578s
func createChunks(forData: Data) {
imageData.withUnsafeBytes { (u8Ptr: UnsafePointer<UInt8>) in
let mutRawPointer = UnsafeMutableRawPointer(mutating: u8Ptr)
let uploadChunkSize = 2097152
let totalSize = imageData.count
var offset = 0
while offset < totalSize {
let chunkSize = offset + uploadChunkSize > totalSize ? totalSize - offset : uploadChunkSize
let chunk = Data(bytesNoCopy: mutRawPointer+offset, count: chunkSize, deallocator: Data.Deallocator.none)
offset += chunkSize
}
}
}

The Data(bytesNoCopy: ... initializer requires a mutable pointer. Change your code to the following to make it work:
imageData.withUnsafeMutableBytes { (unsafePointer: UnsafeMutablePointer<UInt8>) -> Void in
// your code
}

Related

How to ignore cache when repeatedly reading from disk

I am writing an app that contains a small benchmark for I/O operations.
For write operations, I am using a 'FileHandle' which works pretty well. I am testing my old USB stick and my calculation results in values of roughly 20MB/s which seems correct.
However, when reading, the values jump up to 8 GB/s. Although I would love to have an USB stick that fast...I think this has to do with some sort of cacheing.
Here is the code that I am using (some bits were removed):
guard let handle = FileHandle(forUpdatingAtPath: url.path) else { return }
let data = Data(repeating: 0, count: 2 * 1024 * 1024)
var startTime = Date.timestamp
// Write Test
while Date.timestamp - startTime < 5.0
{
handle.write(data)
try? handle.synchronize()
// ...
}
// Go back to beginning of file.
try? handle.seek(toOffset: 0)
// Remove everything at the end of the file
try? handle.truncate(atOffset: blockSize)
startTime = Date.timestamp
// Read Test
while Date.timestamp - startTime < 5.0
{
autoreleasepool
{
if let handle = try? FileHandle(forReadingFrom: fileUrl), let data = try? handle.readToEnd()
{
let count = UInt64(data.count)
self.readData += count
self.totalReadData += count
handle.close()
}
// I also tried FileManager.default.contents(atPath: ) - same result
}
}
I also tried this piece of code (it's either from Martin R. here on SO or from Quinn on the Apple forums):
let fd = open(fileUrl.path, O_RDONLY)
_ = fcntl(fd, F_NOCACHE, 1)
var buffer = Data(count: 1024 * 1024)
buffer.withUnsafeMutableBytes { ptr in
let amount = read(fd, ptr.baseAddress, ptr.count)
self.readData += UInt64(amount)
self.totalReadData += UInt64(amount)
}
close(fd)
The code itself works...but there is still cacheing.
TL;DR How can I disable cacheing when writing to and reading from a file using Swift?
Regards

withUnsafeBytes' is deprecated: use `withUnsafeBytes<R>(_: (UnsafeRawBufferPointer) throws -> R) rethrows -> R` instead [duplicate]

I previously used this code in Swift 4.2 to generate an id:
public static func generateId() throws -> UInt32 {
let data: Data = try random(bytes: 4)
let value: UInt32 = data.withUnsafeBytes { $0.pointee } // deprecated warning!
return value // + some other stuff
}
withUnsafeBytes is deprecated on Swift 5.0. How can I solve this?
In Swift 5 the withUnsafeBytes() method of Data calls the closure with an (untyped) UnsafeRawBufferPointer, and you can load() the value from the raw memory:
let value = data.withUnsafeBytes { $0.load(as: UInt32.self) }
(compare How to use Data.withUnsafeBytes in a well-defined manner? in the Swift forum). Note that this requires that the memory is aligned on a 4-byte boundary. For alternatives see round trip Swift number types to/from Data.
Note also that as of Swift 4.2 you can create a random 32-bit integer simply using the new Random API:
let randomId = UInt32.random(in: .min ... .max)
On Xcode 10.2, Swift 5, using $0.load(as:) didn't work for me, both when reading from the pointer or writing to it.
Instead, using $0.baseAddress?.assumingMemoryBound(to:) seems to work well.
Example reading from the pointer buffer (code is unrelated to the question):
var reachability: SCNetworkReachability?
data.withUnsafeBytes { ptr in
guard let bytes = ptr.baseAddress?.assumingMemoryBound(to: Int8.self) else {
return
}
reachability = SCNetworkReachabilityCreateWithName(nil, bytes)
}
Example writing to the buffer pointer (code is unrelated to the question):
try outputData.withUnsafeMutableBytes { (outputBytes: UnsafeMutableRawBufferPointer) in
let status = CCKeyDerivationPBKDF(CCPBKDFAlgorithm(kCCPBKDF2),
passphrase,
passphrase.utf8.count,
salt,
salt.utf8.count,
CCPseudoRandomAlgorithm(kCCPRFHmacAlgSHA1),
rounds,
outputBytes.baseAddress?.assumingMemoryBound(to: UInt8.self),
kCCKeySizeAES256)
guard status == kCCSuccess else {
throw Error.keyDerivationError
}
}
The code from the question would look like:
let value = data.withUnsafeBytes {
$0.baseAddress?.assumingMemoryBound(to: UInt32.self)
}
In cases where the 'withUnsafeBytes' is deprecated: use withUnsafeBytes<R>(…) warning persists, it seems like the compiler can get confused when the closure has only one line. Making the closure have two or more lines might remove the ambiguity.
One more way to fix this warning to use bindMemory(to:).
var rawKey = Data(count: rawKeyLength)
let status = rawKey.withUnsafeMutableBytes { rawBytes -> Int32 in
guard let rawBytes = rawBytes.bindMemory(to: UInt8.self).baseAddress else {
return Int32(kCCMemoryFailure)
}
return CCSymmetricKeyUnwrap(alg, ivBytes, iv.count, keyBytes, key.count, wrappedKeyBytes, wrappedKey.count, rawBytes, &rawKeyLength)
}
I got this error as I was trying to figure out a compression stream tutorial. To get it to work, I added a step of converting the raw buffer pointer to a UnsafePointer
Original code from a tutorial I was working on.
--> where input: Data
--> where stream: compression_stream
//Method that shows the deprecation alert
return input.withUnsafeBytes { (srcPointer: UnsafePointer<UInt8>) in
//holder
var output = Data()
//Source and destination buffers
stream.src_ptr = srcPointer //UnsafePointer<UInt8>
stream.src_size = input.count
… etc.
}
Code with a conversion to make the above code work with a valid method
return input.withUnsafeBytes { bufferPtr in
//holder
var output = Data()
//Get the Raw pointer at the initial position of the UnsafeRawBuffer
let base: UnsafeRawPointer? = bufferPtr.baseAddress
//Unwrap (Can be combined with above, but kept it separate for clarity)
guard let srcPointer = base else {
return output
}
//Bind the memory to the type
let count = bufferPtr.count
let typedPointer: UnsafePointer<UInt8> = srcPointer.bindMemory(to: UInt8.self, capacity: count)
// Jump back into the original method
stream.src_ptr = typedPointer //UnsafePointer<UInt8>
}

Problem accessing MTLBuffer via typed UnsafeMutualPointer

I have a function that is passed an optional MTLBuffer. My goal is to iteratively change values in that buffer using an index into a Typed pointer to the same buffer. However, when I run the code, I'm getting an error "Thread 1: EXC_BAD_ACCESS (code=2, address=0x1044f1000)".
Am I converting to the Typed UnsafeMutablePointer correctly?
Would it be better to covert to a Typed UnsafeMutableBufferPointer? If so, how would I convert from the MTLBuffer to Typed UnsafeMutableBufferPointer?
Any idea why I'm getting this error?
Note: I've removed most guard checks to keep this simple. I've confirmed the MTLDevice (via device), bufferA allocation, dataPtr and floatPtr are all non-nil. floatPtr and dataPtr do point to the same memory address.
This is how I allocate the buffer:
bufferSize = 16384
bufferA = device?.makeBuffer(length: bufferSize, options: MTLResourceOptions.storageModeShared)`
Here's my code operating on the buffer:
guard let dataPtr = bufferA?.contents() else {
fatalError("error retrieving buffer?.contents() in generateRandomFloatData()")
}
let floatPtr = dataPtr.bindMemory(to: Float.self, capacity: bufferA!.length)
for index in 0...bufferSize - 1 {
floatPtr[index] = 1.0 // Float.random(in: 0...Float.greatestFiniteMagnitude)
}
Thank you!
Am I converting to the Typed UnsafeMutablePointer correctly?
NO.
When you call makeBuffer(length:options:) you pass the length in bytes.
But, a Float occupies 4 bytes in memory.
So, you may need to modify some parts related to number of elements:
let floatPtr = dataPtr.bindMemory(to: Float.self, capacity: bufferA!.length/MemoryLayout<Float>.stride)
for index in 0..<bufferSize/MemoryLayout<Float>.stride {
floatPtr[index] = 1.0 // Float.random(in: 0...Float.greatestFiniteMagnitude)
}

subdata method in Swift doesn't seem to be doing as I want

I have the following code in a playground (Swift 5)
import Foundation
let array : [UInt8] = [0,1,2,3,4,5,6,7,8,9,10,11,12]
public extension Data {
func uint32( offset:Int)-> UInt32{
let range = offset..<(offset+4)
let copy = self.subdata(in: range)
print(copy as NSData) // Prints <02030405>
return copy.withUnsafeBytes{
$0.load(fromByteOffset: 0, as: UInt32.self).bigEndian
}
}
}
let data = Data(array)
let datadropped = data.dropFirst(2)
print(data as NSData) // Prints <00010203 04050607 08090a0b 0c>
print(datadropped as NSData) // Prints <02030405 06070809 0a0b0c>
let sub = data.subdata(in: 4..<8 ) // gives 4,5,6,7
let sub2 = datadropped.subdata(in: 4..<8) // also gives 4,5,6,7
data.uint32(offset: 2)
Now if I set the offset in the final line as 0 or 1 it crashes. An offset of 2 works but returns a uint constructed using the bytes 02,03,04,05 which is not what I would expect. The documentation states the dropFirst() and subdata() return copies of the data.
I did get my uint32 function working with the following code. But I would like to know why the ranges of bytes in the initial function are not working. How do I force a genuine new copy of the Data? If someone could explain it to me I'd be grateful.
extension Data
func uint32( offset:Int)-> UInt32{
let array = Array(0...3).map {
uint8(offset: $0+ offset)
}
return array.withUnsafeBytes{
$0.load(fromByteOffset: 0, as: UInt32.self).bigEndian
}
}
func uint8( offset:Int)-> UInt8 {
return self.withUnsafeBytes{
$0.load(fromByteOffset: offset, as: UInt8.self).bigEndian
}
}
}
datadropped is a Slice
It contains the subset of the data but it shares the same indices with the original collection. It crashes because the first index of datadropped is 2, not 0.
To get a new Data object you have to write
let datadropped = Data(data.dropFirst(2))
For more information about slices please watch WWDC 2018: Using Collections Effectively (from 11:00)
Note: You can drop the fromByteOffset parameter
return copy.withUnsafeBytes{
$0.load(as: UInt32.self).bigEndian
}

Pointers, Pointer Arithmetic, and Raw Data in Swift

My application uses a somewhat complex inmutable data structure that is encoded in a binary file. I need to have access to it at the byte level, avoiding any copying. Normally, I would use C or C++ pointer arithmetic and typecasts, to access and interpret the raw byte values. I would like to do the same with Swift.
I have found that the following works:
class RawData {
var data: NSData!
init(rawData: NSData) {
data = rawData
}
func read<T>(byteLocation: Int) -> T {
let bytes = data.subdataWithRange(NSMakeRange(byteLocation, sizeof(T))).bytes
return UnsafePointer<T>(bytes).memory
}
func example_ReadAnIntAtByteLocation5() -> Int {
return read(5) as Int
}
}
However, I am not sure how efficient it is. Do data.subdataWithRange and NSMakeRange allocate objects every time I call them, or are they just syntactic sugar for dealing with pointers?
Is there a better way to do this in Swift?
EDIT:
I have created a small Objective-C class that just encapsulates a function to offset a pointer by a given number of bytes:
#implementation RawDataOffsetPointer
inline void* offsetPointer(void* ptr, int bytes){
return (char*)ptr + bytes;
}
#end
If I include this class in the bridging header, then I can change my read method to
func read<T>(byteLocation: Int) -> T {
let ptr = offsetPointer(data.bytes, CInt(byteLocation))
return UnsafePointer<T>(ptr).memory
}
which will not copy data from my buffer, or allocate other objects.
However, it would still be nice to do some pointer arithmetic from Swift, if it were possible.
If you just want to do it directly, UnsafePointer<T> can be manipulated arithmetically:
let oldPointer = UnsafePointer<()>
let newPointer = oldPointer + 10
You can also cast a pointer like so (UnsafePointer<()> is equivalent to void *)
let castPointer = UnsafePointer<MyStruct>(oldPointer)
I would recommend looking into NSInputStream, which allows you to read NSData as a series of bytes (UInt8 in Swift).
Here is a little sample I put together in the playground:
func generateRandomData(count:Int) -> NSData
{
var array = Array<UInt8>(count: count, repeatedValue: 0)
arc4random_buf(&array, UInt(count))
return NSData(bytes: array, length: count)
}
let randomData = generateRandomData(256 * 1024)
let stream = NSInputStream(data: randomData)
stream.open() // IMPORTANT
var readBuffer = Array<UInt8>(count: 16 * 1024, repeatedValue: 0)
var totalBytesRead = 0
while (totalBytesRead < randomData.length)
{
let numberOfBytesRead = stream.read(&readBuffer, maxLength: readBuffer.count)
// Do something with the data
totalBytesRead += numberOfBytesRead
}
You can create an extension to read primitive types like so:
extension NSInputStream
{
func readInt32() -> Int
{
var readBuffer = Array<UInt8>(count:sizeof(Int32), repeatedValue: 0)
var numberOfBytesRead = self.read(&readBuffer, maxLength: readBuffer.count)
return Int(readBuffer[0]) << 24 |
Int(readBuffer[1]) << 16 |
Int(readBuffer[2]) << 8 |
Int(readBuffer[3])
}
}
I would recommend the simple way to use UnsafeArray.
let data = NSData(contentsOfFile: filename)
let ptr = UnsafePointer<UInt8>(data.bytes)
let bytes = UnsafeBufferPointer<UInt8>(start:ptr, count:data.length)