What i am doing wrong? At playground it runs as it should. But as soon as i deploy it on iOS simulator it returns the wrong sequence.
#objc func buttonTapped(){
let group = DispatchGroup()
let dispatchQueue = DispatchQueue.global(qos: .default)
for i in 1...4 {
group.enter()
dispatchQueue.async {
print("๐น \(i)")
}
group.leave()
}
for i in 1...4 {
group.enter()
dispatchQueue.async {
print("โ \(i)")
}
group.leave()
}
group.notify(queue: DispatchQueue.main) {
print("jobs done by group")
}
}
Console Output:
I don't get it. ๐
You should put the group.leave() statement in the dispatchQueue.async block as well, otherwise it will be executed synchronously before the async block would finish execution.
#objc func buttonTapped(){
let group = DispatchGroup()
let dispatchQueue = DispatchQueue.global(qos: .default)
for i in 1...4 {
group.enter()
dispatchQueue.async {
print("๐น \(i)")
group.leave()
}
}
for i in 1...4 {
group.enter()
dispatchQueue.async {
print("โ \(i)")
group.leave()
}
}
group.notify(queue: DispatchQueue.main) {
print("jobs done by group")
}
}
As Dรกvid said, properly employed dispatch groups only ensure that the notification takes place after all of the tasks finish, which you can achieve by calling leave from within the dispatched blocks, as he showed you. Or alternatively, since your dispatched tasks are, themselves, synchronous, you don't have to manually enter and leave the group, but can use group parameter of async method:
let group = DispatchGroup()
let queue = DispatchQueue.global(qos: .default)
for i in 1...4 {
queue.async(group: group) {
print("๐น \(i)")
}
}
for i in 1...4 {
queue.async(group: group) {
print("โ \(i)")
}
}
group.notify(queue: .main) {
print("jobs done by group")
}
Use group.enter() and group.leave() when calling some asynchronous method, but in the case of these print statements, you can just use async(group:execute:) as shown above.
Now, we've solved the problem where the "jobs done by group" block didn't wait for all of the dispatched tasks. But, because you're doing all of this dispatching to a concurrent queue (all the global queues are concurrent queues), you have no assurances that your tasks will be performed in the order that you requested. They're queued up in a strict FIFO manner, but because they're concurrent, you have no assurances when you'll hit the respective print statements.
If you need it to print the messages in order, you will have to use a serial queue. For example, if you create your own queue, in the absence of the .concurrent attribute, the following will create a serial queue:
// create serial queue
let queue = DispatchQueue(label: "...")
// but not your own concurrent queue:
//
// let queue = DispatchQueue(label: "...", attributes: .concurrent)
//
// nor one of the global concurrent queues:
//
// let queue = DispatchQueue.global(qos: .default)
//
And if you run the above code with this serial queue, you'll see what you were looking for:
๐น 1
๐น 2
๐น 3
๐น 4
โ 1
โ 2
โ 3
โ 4
jobs done by group
But, then, again, if you were using a serial queue, the group would be completely unnecessary (you could just add the "completion" task as yet another dispatched task at the end of the serial queue). I only show the use of serial queues as a way to avoid the race condition of dispatching eight tasks to a concurrent queue.
Related
I want to speed up some process so I wrote a swift CLI script that process thousands of files in parallel and write the process result of each file into a single file. (The order of the files does not really matter)
So I wrote below code and it works in the Xcode unit tests (Even with a list of approx 1200 files!) However when I execute the program from the command line without Xcode and with the same list of files it never ends. It looks like it is stuck near the end.
I read that sometimes spanning too many threads will cause the program to stop because it runs out of resources but I thought DispatchQueue.concurrentPerform will take care of that... I have no clue why this works in XCTests and does not work in the terminal.
I have tried DispatchGroup and Semaphore approach and both have the same problem...
Any help is highly appreciated.
let filePaths: [String] = Array with thousands of file paths to process
let group = DispatchGroup()
let concurrentQueue = DispatchQueue(label: "my.concurrent.queue", qos: .userInitiated, attributes: .concurrent)
let serialQueue = DispatchQueue(label: "my.serial.queue", qos: .userInitiated)
group.enter()
concurrentQueue.async {
DispatchQueue.concurrentPerform(iterations: filePaths.count) { (fileIndex) in
let filePath = filePaths[fileIndex]
let result = self.processFile(path: filePath)
group.enter()
serialQueue.async {
self.writeResult(result)
group.leave()
}
}
group.leave()
}
group.wait()
First, a few simplifications:
You have code that is
group.enter()
serialQueue.async {
self.writeResult(result)
group.leave()
}
That can be simplified to:
serialQueue.async(group: group) {
self.writeResult(result)
}
Consider:
group.enter()
concurrentQueue.async {
DispatchQueue.concurrentPerform(iterations: filePaths.count) { (fileIndex) in
...
}
group.leave()
}
That concurrentQueue is redundant. This can be simplified to:
DispatchQueue.concurrentPerform(iterations: filePaths.count) { (fileIndex) in
...
}
That reduces your code to:
let group = DispatchGroup()
let writeQueue = DispatchQueue(label: "serial.write.queue", qos: .userInitiated)
DispatchQueue.concurrentPerform(iterations: filePaths.count) { [self] index in
let result = processFile(path: filePaths[index])
writeQueue.async(group: group) {
writeResult(result)
}
}
group.wait()
That begs the question as to why you are dispatching asynchronously to a serial queue for the write operations. That can introduce problems (e.g. if it gets backlogged, you will holding all unwritten result values in memory at the same time).
One option is to write synchronously (you have to wait for the write operations in the end, anyway):
let writeQueue = DispatchQueue(label: "serial.write.queue", qos: .userInitiated)
DispatchQueue.concurrentPerform(iterations: filePaths.count) { [self] index in
let result = processFile(path: filePaths[index])
writeQueue.sync {
writeResult(result)
}
}
Or you can probably just write from the various concurrent threads, themselves:
let writeQueue = DispatchQueue(label: "serial.write.queue", qos: .userInitiated)
DispatchQueue.concurrentPerform(iterations: filePaths.count) { [self] index in
let result = processFile(path: filePaths[index])
writeResult(result)
}
When learning OperationQueue, is is possible to use OperationQueue instead of gcd barrier?
Here is the case:
upload 3 images , then upload other 3 images
With barrier, gcd works perfect
func workFlow(){
let queue = DispatchQueue(label: "test.concurrent.queue", qos: .background, attributes: .concurrent, autoreleaseFrequency: .workItem)
queue.async {
self.uploadImg(idx: "A_0")
}
queue.async {
Thread.sleep(forTimeInterval: 2)
self.uploadImg(idx: "A_1")
}
queue.async {
self.uploadImg(idx: "A_2")
}
queue.async(qos: queue.qos, flags: .barrier) {
print("group A done")
}
print("A: should not be hanged")
queue.async {
self.uploadImg(idx: "B_0")
}
queue.async {
self.uploadImg(idx: "B_1")
}
queue.async {
self.uploadImg(idx: "B_2")
}
queue.async(qos: queue.qos, flags: .barrier) {
print("group B done")
}
print("B: should not be hanged")
}
func uploadImg(idx info: String){
Thread.sleep(forTimeInterval: 1)
print("img \(info) uploaded")
}
While with OperationQueue, there is a little flaw here
The main queue gets hanged, just check the print
"A/B: should not be hanged"
lazy var uploadQueue: OperationQueue = {
var queue = OperationQueue()
queue.name = "upload queue"
queue.maxConcurrentOperationCount = 5
return queue
}()
func workFlow(){
let one = BlockOperation {
self.uploadImg(idx: "A_0")
}
let two = BlockOperation {
Thread.sleep(forTimeInterval: 3)
self.uploadImg(idx: "A_1")
}
let three = BlockOperation {
self.uploadImg(idx: "A_2")
}
uploadQueue.addOperations([one, two, three], waitUntilFinished: true)
print("A: should not be hanged")
uploadQueue.addOperation {
print("group A done")
}
let four = BlockOperation {
self.uploadImg(idx: "B_0")
}
let five = BlockOperation {
self.uploadImg(idx: "B_1")
}
let six = BlockOperation {
self.uploadImg(idx: "B_2")
}
uploadQueue.addOperations([four, five, six], waitUntilFinished: true)
print("B: should not be hanged")
uploadQueue.addOperation {
print("group B done")
}
}
How to do it better with OperationQueue?
If you do not want the operations added to the queue to block the current thread, waitUntilFinished must be false. But if you set it to true, it will block the current thread until the added operations finish.
Obviously, if you do not wait, it will not block the the main thread, but you will also lose the barrier behavior. But iOS 13 and macOS 10.15 introduced addBarrierBlock. If you really need barriers and must support earlier OS versions, then you will have to use dependencies. But if you were previously using GCD barriers simply to constrain the degree of concurrency, then maxConcurrentOperationCount might render the barrier moot. It all depends upon why you were using barriers with these uploads/downloads. (It is a little unusual to see barriers with upload/download queues as it reduces efficiency.)
How to do it better with OperationQueue?
I assume that uploadImg downloads the image synchronously. I would refactor it to be its own Operation subclass, that does the necessary Operation KVO such as shown here. That wraps download task in operation, but you can do the same with upload or data tasks, too (though the memory impact with data tasks is much greater).
But it is always advisable to avoid having synchronous network requests to (a) make sure you do not tie up worker threads; and (b) to make the requests cancelable.
In the following code, is it safe to append to an array? Is the order guaranteed to be maintained?
let processedData: [SomeType] = []
let dispatchGroup = DispatchGroup()
for _ in 0..<N {
dispatchGroup.enter()
startSomeAsyncTaskXYZ { (data, error) in
// handle error and process data
// add processed data to an array
processedData.append(..)
dispatchGroup.leave()
}
}
dispatchGroup.notify(queue: .main) {
// update UI
}
To stick with DispatchGroup while preserving the desired asynchronous nature and the expected ordering, make your array an array of optionals and populate it in whatever order the tasks complete:
var processedData: [SomeType?] = Array(repeating: nil, count: N)
let dispatchGroup = DispatchGroup()
for idx in 0..<N {
dispatchGroup.enter()
startSomeAsyncTaskXYZ { (data, error) in
// Ensure we always .leave() after we're done
// handling the completion of the task
defer { dispatchGroup.leave() }
guard let data = data,
error == nil else {
// TODO: Actual error handling
return
}
// This needs to be .sync now (not .async) to ensure
// the deferred dispatchGroup.leave() is not called
// until *after* we've updated the array
DispatchQueue.main.sync {
processedData[idx] = SomeType(data: data)
}
}
}
dispatchGroup.notify(queue: .main) {
// update UI
}
DispatchGroup has no relevance to execution order. It's merely a way of tracking completion of groups of tasks.
Whether the group's constituent tasks run async or sync, and in what order, is entirely dependant on how you use DispatchQueues.
Just went through this with my app. I have some async tasks that need to be done in order. The best way to accomplish that is through dispatchGroup() and semaphore().
The basic idea is that dispatchGroup fetches data in no particular order, but semaphore goes in a specific order if you need it to.
Here is a good video that demonstrates it: https://www.youtube.com/watch?v=6rJN_ECd1XM&ab_channel=LetsBuildThatApp
Some sample code would look like this:
let dispatchQueue = DispatchQueue.global(qos: .userInitiated)
let dispatchGroup = DispatchGroup()
let semaphore = DispatchSemaphore(value: 0)
override func viewDidLoad() {
dispatchQueue.async {
// --------------
// Family members
// --------------
// Get members and household information first (esp. payday time and time zone), then everything else
self.dispatchGroup.enter()
MPUser.loadFamilyMembers {
print("We got us some fambly members!")
self.dispatchGroup.leave()
self.semaphore.signal()
}
// ^^^ Wait for above code to finish ('signal') before moving on (in other words, get users first)
self.semaphore.wait()
// --------------
// Household info
// --------------
self.dispatchGroup.enter()
FamilyData.observeHouseholdInformation {
self.dispatchGroup.leave()
self.semaphore.signal()
}
// ^^^ Wait for above code to finish ('signal') before moving on (in other words, get users first, then household info)
self.semaphore.wait()
// ---------------
// Everything else
// ---------------
self.dispatchGroup.enter()
Setup.observeProgress {
self.dispatchGroup.leave()
}
self.dispatchGroup.enter()
OutsideIncome.observeUserOutsideIncomeBudget {
self.dispatchGroup.leave()
}
self.dispatchGroup.enter()
MPUser.observeCurrentEarnings {
self.dispatchGroup.leave()
}
self.dispatchGroup.notify(queue: .main) {
let end = Date()
print("\nAll functions completed in \(end.timeIntervalSince(self.start!).rounded(toPlaces: 2)) seconds!\n")
self.sendUserToCorrectPage()
}
}
}
I'm using a very simple swift project created with SPM where it includes Alamofire.
main.swift:
import Alamofire
Alamofire.request("https://google.com").responseString(queue: queue) { response in
print("\(response.result.isSuccess)")
}
The closure is never executed if I don't use a lock.
Is there a way to instruct to wait for all threads or that specific thread before exiting?
I'm aware this can be easily achieved using Playgrounds.
Simplest way to wait for an async task is to use a semaphore:
let semaphore = DispatchSemaphore(value: 0)
doSomethingAsync {
semaphore.signal()
}
semaphore.wait()
// your code will not get here until the async task completes
Alternatively, if you're waiting for multiple tasks, you can use a dispatch group:
let group = DispatchGroup()
group.enter()
doAsyncTask1 {
group.leave()
}
group.enter()
doAsyncTask2 {
group.leave()
}
group.wait()
// You won't get here until all your tasks are done
For Swift 3
let group = DispatchGroup()
group.enter()
DispatchQueue.global(qos: .userInitiated).async {
// Do work asyncly and call group.leave() after you are done
group.leave()
}
group.notify(queue: .main, execute: {
// This will be called when block ends
})
This code will be helpful when you need to execute some code after some task is done.
Please add details about your question, then I can help you more.
DispatchGroup and OperationQueue have methods wait() and waitUntilAllOperationsAreFinished() which wait for all operations in respective queues to complete.
But even when I call cancelAllOperations it just changes the flag isCancelled in every running operation and stop the queue from executing new operations. But it still waits for the operations to complete. Therefore running the operations must be stopped from the inside. But it is possible only if operation is incremental or has an inner cycle of any kind. When it's just long external request (web request for example), there is no use of isCancelled variable.
Is there any way of stopping the OperationQueue or DispatchGroup waiting for the operations to complete if one of the operations decides that all queue is now outdated?
The practical case is: mapping a request to a list of responders, and it is known that only one may answer. If it happens, queue should stop waiting for other operations to finish and unlock the thread.
Edit: DispatchGroup and OperationQueue usage is not obligatory, these are just tools I thought would fit.
OK, so I think I came up with something. Results are stable, I've just tested. The answer is just one semaphore :)
let semaphore = DispatchSemaphore(value: 0)
let group = DispatchGroup()
let queue = DispatchQueue(label: "map-reduce", qos: .userInitiated, attributes: .concurrent)
let stopAtFirst = true // false for all results to be appended into one array
let values: [U] = <some input values>
let mapper: (U) throws -> T? = <closure>
var result: [T?] = []
for value in values {
queue.async(group: group) {
do {
let res = try mapper(value)
// appending must always be thread-safe, otherwise you end up with race condition and unstable results
DispatchQueue.global().sync {
result.append(res)
}
if stopAtFirst && res != nil {
semaphore.signal()
}
} catch let error {
print("Could not map value \"\(value)\" to mapper \(mapper): \(error)")
}
}
}
group.notify(queue: queue) { // this must be declared exactly after submitting all tasks, otherwise notification fires instantly
semaphore.signal()
}
if semaphore.wait(timeout: .init(secondsFromNow: 5)) == .timedOut {
print("MapReduce timed out on values \(values)")
}