I've set up this script to loop through a bunch of data in the background and I've successfully set up a semaphore to keep everything (the array that will populate the table) in order but I cannot exactly understand how or why the semaphore keeps the array in order. The dispatchGroup is entered, the loop stops and waits until the image is downloaded, once the image is gotten the dispatchSemaphore is set to 1 and immediately the dispatchGroup is exited and the semaphore set back to 0. The semaphore is toggled so fast from 0 to 1 that I don't understand how it keeps the array in order.
let dispatchQueue = DispatchQueue(label: "someTask")
let dispatchGroup = DispatchGroup()
let dispatchSemaphore = DispatchSemaphore(value: 0)
dispatchQueue.async {
for doc in snapshot.documents {
// create data object for array
dispatchGroup.enter()
// get image with asynchronous completion handler
Storage.storage().reference(forURL: imageId).getData(maxSize: 1048576, completion: { (data, error) in
defer {
dispatchSemaphore.signal()
dispatchGroup.leave()
}
if let imageData = data,
error == nil {
// add image to data object
// append to array
}
})
dispatchSemaphore.wait()
}
// do some extra stuff in background after loop is done
}
dispatchGroup.notify(queue: dispatchQueue) {
DispatchQueue.main.async {
self.tableView.reloadData()
}
}
The solution is in your comment get image with asynchronous completion handler. Without the semaphore all image downloads would be started at the same time and race for completion, so the image that downloads fastest would be added to the array first.
So after you start your download you immediately wait on your semaphore. This will block until it is signaled in the callback closure from the getData method. Only then the loop can continue to the next document and download it. This way you download one file after another and block the current thread while the downloads are running.
Using a serial queue is not an option here, since this would only cause the downloads to start serially, but you can’t affect the order in which they finish.
This is a rather inefficient though. Your network layer probably can run faster if you give it multiple requests at the same time (think of parallel downloads and HTTP pipelining). Also you're 'wasting' a thread which could do some different work in the meantime. If there is more work to do at the same time GCD will spawn another thread which wastes memory and other resources.
A better pattern would be to skip the semaphore, let the downloads run in parallel and store the image directly at the correct index in your array. This of course means you have to prepare an array of the appropriate size beforehand, and you have to think of a placeholder for missing or failed images. Optionals would do the trick nicely:
var images: [UIImage?] = Array(repeating: nil, count: snapshot.documents.count)
for (index, doc) in snapshot.documents.enumerated() {
// create data object for array
dispatchGroup.enter()
// get image with asynchronous completion handler
Storage.storage().reference(forURL: imageId).getData(maxSize: 1048576) { data, error in
defer {
dispatchGroup.leave()
}
if let imageData = data,
error == nil {
// add image to data object
images[index] = image
}
}
}
The DispatchGroup isn't really doing anything here. You have mutual exclusion granted by the DispatchSemaphor, and the ordering is simply provided by the iteration order of snapshot.documents
Related
I have a JSONEncoder encoding a 20mb file, which takes ages to process. If the data it's processing changes, I'd like to cancel the encoding, and restart the encoding process but I can't think of a way to do this. Any ideas?
I could call JSONEncoder.encode again, but now I would have two 30 second processes running, and double the amount of memory and processor overhead.
It would be lovely to be able cancel the previous one.
EDIT: Some of you requested to see my encoder. Here's the one which I'd say causes the biggest bottleneck...
func encode(to encoder: Encoder) throws {
try autoreleasepool {
var container = encoder.container(keyedBy: CodingKeys.self)
try container.encode(brush, forKey: .brush)
if encoder.coderType == CoderType.export {
let bezierPath = try NSKeyedUnarchiver.unarchivedObject(ofClass: UIBezierPath.self, from: beziersData)
let jsonData = try UIBezierPathSerialization.data(with: bezierPath, options: UIBezierPathWritingOptions.ignoreDrawingProperties)
let bezier = try? JSONDecoder().decode(DBBezier.self, from: jsonData)
try container.encodeIfPresent(bezier, forKey: .beziersData)
} else {
try container.encodeIfPresent(beziersData, forKey: .beziersData)
}
}
}
You can use OperationQueue and add your long running task into that operation queue.
var queue: OperationQueue?
//Initialisation
if queue == nil {
queue = OperationQueue()
queue?.maxConcurrentOperationCount = 1
}
queue?.addOperation {
//Need to check the isCanceled property of the operation for stopping the ongoing execution in any case.
self.encodeHugeJSON()
}
You can also cancel the task whenever you want using the following code:
//Whenever you want to cancel the task, you can do it like this
queue?.cancelAllOperations()
queue = nil
What is an Operation Queue:
An operation queue invokes its queued Operation objects based on their
priority and readiness. After you add an operation to a queue, it
remains in the queue until the operation finishes its task. You can’t
directly remove an operation from a queue after you add it.
Reference links:
https://developer.apple.com/documentation/foundation/operationqueue
https://www.hackingwithswift.com/example-code/system/how-to-use-multithreaded-operations-with-operationqueue
The code is in a simple way, only read and parse an xml file into an array. I did not notice the problem until one day I tried to open a big xml file.
I added a blur view with NSProgressIndicator when the data is parsing, but the blur view did not show up until the parsing was completed.
self.addBlurView()
let file = HandleFile.shared.openFile(filePath)
self.removeBlurView()
guard let name = file.name, let path = file.path, let data = file.data else {
return
}
So I tried to delay parsing data. The blur view can be showed up, and removed when completed.
self.addBlurView()
DispatchQueue.main.asyncAfter(deadline: .now() + 0.3, execute: {
let file = HandleFile.shared.openFile(filePath)
self.removeBlurView()
guard let name = file.name, let path = file.path, let data = file.data else {
return
}
})
I thought it might be a problem fo thread, so I tried this in func addBlurView(), failed. I also tried to add an counting in addBlurView(), it counted to a certain number and paused, and continue counting after parsing data.
DispatchQueue.main.async {
self.blurView.isHidden = false
self.spinner.startAnimation(self)
}
Have no idea why this happen. Can anyone help to solve this problem?
Thanks.
As I mentioned in the comments above, main queue is a serial queue and all the tasks assigned to it are executed serially by main thread. In general, You should not perform any heavy lifting task (like loading file to memory) on main thread as it would block the main thread and render UI unresponsive.
Typically all the heavy lifting tasks like loading a file to a memory (anything which does not deal with UI rendering directly) should be delegated to one of dispatch queues. Try wrapping your openFile(filePath) call inside DispatchQueue
self.addBlurView()
DispatchQueue.global(qos: .default).async {
let file = HandleFile.shared.openFile(filePath)
}
Personally I would expect openFile function to have a completion block which is triggered on main queue when it finished loading file so that you can remove your blurView, but in your case it seems like its a synchronous statement so you can try
self.addBlurView()
DispatchQueue.global(qos: .default).async {
let file = HandleFile.shared.openFile(filePath)
DispatchQueue.main.async {
self.removeBlurView()
}
}
Basically, at the launch of my app, I want to load the latest data from firebase from about 4-5 different documents. Then I also want to set up a listener to monitor data changes. I do this by calling 4-5 similar functions that take a dispatchGroup as an argument. I may be approaching this completely wrong but I could not think of any other way to do it. I just want to load those documents, set up listeners, and take certain action whenever those docs are loaded at the launch of the app.
// app launch
let dispatch = DispatchGroup()
getFirebaseDocument1(dispatch: dispatch)
getFirebaseDocument2(dispatch: dispatch)
getFirebaseDocument3(dispatch: dispatch)
getFirebaseDocument4(dispatch: dispatch)
getFirebaseDocument5(dispatch: dispatch)
dispatch.notify(queue:main) {
// execute some code to execute after all the documents are fetched
}
// typical getFirebaseDocument code
dispatch.enter()
let ref = someFirestoreReference
ref.addSnapshotListener { (snapshot, error) in
if let error = error{
// handle error
} else {
// load the document
}
dispatch.leave()
}
The code works fine when it's launched but crashes whenever the listener receives an update. I know this is because the dispatch.leave() is called in the listener function. However, I cannot seem to figure out a clever solution to where I can asynchronously load the data from firebase at launch while also setting up listeners. I would also prefer not to nest closures within one another as it wouldn't be asynchronous and it would also be a pain.
I may be wrong, but you should leave the group inside your block, here is the example how I do it using group
class func getRates(completion: #escaping EmptyBlock) {
let eurRequest = APIConfigs.request(part: "rs/price/history")
let usdRequest = APIConfigs.request(part: "rs/price/history/usd")
let group = DispatchGroup()
group.enter()
sendRequest(request: eurRequest, method: .get, parameters: ServerParameters.rates()) { response in
Course.current.addRate(rates: ratesRequest(response: response), type: .eur)
group.leave()
}
group.enter()
sendRequest(request: usdRequest, method: .get, parameters: ServerParameters.rates()) { response in
Course.current.addRate(rates: ratesRequest(response: response), type: .usd)
group.leave()
}
group.notify(queue: .main) {
completion()
}
}
This is the code I use to retrieve image files from Firebase storage:
let group = DispatchGroup()
print("starting ImageSetting")
group.enter()
for query in friendArray {
if imageList[query.uid] == nil {
print("going through iteration")
self.profpicRef.child("profile_pic/" + query.uid + ".jpeg").getData(maxSize: 1
* 1024 * 1024) { (data, error) in
print("accessing image")
if let error = error {
self.imageList[query.uid] = self.defaultImage
} else {
self.imageList[query.uid] = UIImage(data: data!)
}
}
}
}
group.leave()
I call this method in ViewWillAppear. I also tried ViewDIdAppear but the result did not change.
This is the result I get from calling this method on the first run
starting ImageSetting
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
going through iteration
So first run getData() is not going through.
However, on second run, the function works properly and I get all the images
Is there any way to fix this issue?
I suspect the problem is that you're not really using dispatch group properly. The issue here is that the for loop is essentially executed and completed immediately -- yes, those callbacks will be called at a later point, but that's not the point where the code telling the dispatch group to leave.
(Also, I don't see a notify call in your sample code, but I'm assuming that's in a part of the code that's being called later.)
So if you're doing something in your code that's dependent upon having those images already loaded, it'll give you an error. And I suspect it probably works the second time because you're grabbing cached data, which probably does execute quick enough for your purposes.
One way to fix it would be to make sure you're adding the dispatch group elements at the right places. Maybe something like this...
let group = DispatchGroup()
print("starting ImageSetting")
for query in friendArray {
if imageList[query.uid] == nil {
print("going through iteration")
group.enter()
self.profpicRef.child("profile_pic/" + query.uid + ".jpeg").getData(maxSize: 1
* 1024 * 1024) { (data, error) in
print("accessing image")
if let error = error {
self.imageList[query.uid] = self.defaultImage
} else {
self.imageList[query.uid] = UIImage(data: data!)
}
group.leave()
}
}
}
group.notify(queue: .main) {
print("Images done loading")
}
DispatchGroup and OperationQueue have methods wait() and waitUntilAllOperationsAreFinished() which wait for all operations in respective queues to complete.
But even when I call cancelAllOperations it just changes the flag isCancelled in every running operation and stop the queue from executing new operations. But it still waits for the operations to complete. Therefore running the operations must be stopped from the inside. But it is possible only if operation is incremental or has an inner cycle of any kind. When it's just long external request (web request for example), there is no use of isCancelled variable.
Is there any way of stopping the OperationQueue or DispatchGroup waiting for the operations to complete if one of the operations decides that all queue is now outdated?
The practical case is: mapping a request to a list of responders, and it is known that only one may answer. If it happens, queue should stop waiting for other operations to finish and unlock the thread.
Edit: DispatchGroup and OperationQueue usage is not obligatory, these are just tools I thought would fit.
OK, so I think I came up with something. Results are stable, I've just tested. The answer is just one semaphore :)
let semaphore = DispatchSemaphore(value: 0)
let group = DispatchGroup()
let queue = DispatchQueue(label: "map-reduce", qos: .userInitiated, attributes: .concurrent)
let stopAtFirst = true // false for all results to be appended into one array
let values: [U] = <some input values>
let mapper: (U) throws -> T? = <closure>
var result: [T?] = []
for value in values {
queue.async(group: group) {
do {
let res = try mapper(value)
// appending must always be thread-safe, otherwise you end up with race condition and unstable results
DispatchQueue.global().sync {
result.append(res)
}
if stopAtFirst && res != nil {
semaphore.signal()
}
} catch let error {
print("Could not map value \"\(value)\" to mapper \(mapper): \(error)")
}
}
}
group.notify(queue: queue) { // this must be declared exactly after submitting all tasks, otherwise notification fires instantly
semaphore.signal()
}
if semaphore.wait(timeout: .init(secondsFromNow: 5)) == .timedOut {
print("MapReduce timed out on values \(values)")
}