How to process an array of task asynchronously with swift combine - swift

I have a publisher which takes a network call and returns an array of IDs. I now need to call another network call for each ID to get all my data. And I want the final publisher to have the resulting object.
First network result:
"user": {
"id": 0,
"items": [1, 2, 3, 4, 5]
}
Final object:
struct User {
let id: Int
let items: [Item]
... other fields ...
}
struct Item {
let id: Int
... other fields ...
}
Handling multiple network calls:
userPublisher.flatMap { user in
let itemIDs = user.items
return Future<[Item], Never>() { fulfill in
... OperationQueue of network requests ...
}
}
I would like to perform the network requests in parallel, since they are not dependent on each other. I'm not sure if Future is right here, but I'd imagine I would then have code to do a
DispatchGroup or OperationQueue and fulfill when they're all done. Is there more of a Combine way of doing this?
Doe Combine have a concept of splitting one stream into many parallel streams and joining the streams together?

Combine offers extensions around URLSession to handle network requests unless you really need to integrate with OperationQueue based networking, then Future is a fine candidate. You can run multiple Futures and collect them at some point, but I'd really suggest looking at URLSession extensions for Combine.
struct User: Codable {
var username: String
}
let requestURL = URL(string: "https://example.com/")!
let publisher = URLSession.shared.dataTaskPublisher(for: requestURL)
.map { $0.data }
.decode(type: User.self, decoder: JSONDecoder())
Regarding running a batch of requests, it's possible to use Publishers.MergeMany, i.e:
struct User: Codable {
var username: String
}
let userIds = [1, 2, 3]
let subscriber = Just(userIds)
.setFailureType(to: Error.self)
.flatMap { (values) -> Publishers.MergeMany<AnyPublisher<User, Error>> in
let tasks = values.map { (userId) -> AnyPublisher<User, Error> in
let requestURL = URL(string: "https://jsonplaceholder.typicode.com/users/\(userId)")!
return URLSession.shared.dataTaskPublisher(for: requestURL)
.map { $0.data }
.decode(type: User.self, decoder: JSONDecoder())
.eraseToAnyPublisher()
}
return Publishers.MergeMany(tasks)
}.collect().sink(receiveCompletion: { (completion) in
if case .failure(let error) = completion {
print("Got error: \(error.localizedDescription)")
}
}) { (allUsers) in
print("Got users:")
allUsers.map { print("\($0)") }
}
In the example above I use collect to collect all results, which postpones emitting the value to the Sink until all of the network requests successfully finished, however you can get rid of the collect and receive each User in the example above one by one as network requests complete.

Related

Swift Combine complex api request

I just started to learn Combine and therefore I can't figure out how to make a complex request to the API.
It is necessary to create an application where the user can enter the name of the company's GitHub account in the input field and get a list of open repositories and their branches.
There are two API methods:
https://api.github.com/orgs/<ORG_NAME>/repos This method returns a list of organization account repositories by name. For example, you can try to request a list of Apple's repositories https://api.github.com/orgs/apple/repos
struct for this method
struct Repository: Decodable {
let name: String
let language: String?
enum Seeds {
public static let empty = Repository(name: "", language: "")
}
}
https://api.github.com/repos/<ORG_NAME>/<REPO_NAME>/branches This method will be needed to get the branch names in the specified repository.
struct for this method
struct Branch: Decodable {
let name: String
}
As a result, I need to get an array of such structures.
struct BranchSectionModel {
var name: Repository
var branchs: [Branch]
}
For this I have two functions:
func loadRepositorys(orgName: String) -> AnyPublisher<[Repository], Never> {
guard let url = URL(string: "https://api.github.com/orgs/\(orgName)/repos" ) else {
return Just([])
.eraseToAnyPublisher()
}
return URLSession.shared.dataTaskPublisher(for: url)
.map { $0.data }
.decode(type: [Repository].self, decoder: JSONDecoder())
.replaceError(with: [])
.receive(on: RunLoop.main)
.eraseToAnyPublisher()
}
and
func loadBranchs(orgName: String, repoName: String) -> AnyPublisher<[Branch], Never> {
guard let url = URL(string: "https://api.github.com/repos/\(orgName)/\(repoName)/branches") else {
return Just([])
.eraseToAnyPublisher()
}
return URLSession.shared.dataTaskPublisher(for: url)
.map { $0.data }
.decode(type: [Branch].self, decoder: JSONDecoder())
.replaceError(with: [])
.receive(on: RunLoop.main)
.eraseToAnyPublisher()
}
Both of these functions work separately, but I don't know how to end up with an [BranchSectionModel] . I guess to use flatMap and sink, but don't understant how.
I do not understand how to combine these two requests in one thread.
When you're looking to convert one publisher into another, .map and .switchToLatest. In this case, since you're also looking to turn one publisher into many (and then back down into one), MergeMany will also be a useful tool:
loadRepositorys(orgName: orgName)
.map { repos in
Publishers.MergeMany(repos.map { repo in
loadBranchs(orgName: orgName, repoName: repo.name)
.map { branches in
BranchSectionModel(name: repo, branchs: branches)
}
})
.collect(repos.count)
}
.switchToLatest()
.sink { result in
print("---")
print(result)
}
.store(in: &cancellables)
Although I'm a big fan of Combine, I don't think it's particularly well suited to this task, compared with async/await, which will probably be a little less confusing and look cleaner. As a learning exercise, it's a great one, but if you were to tackle this problem in the real world, async/await would likely be my go-to.

Swift combine chain requests

I'm attempting to use combine to chain two requests together. The code is pretty rough, but I need to call two api requests. One to get the schedule data than one for live data. I'm able to get the live data (second request) but how do I get the schedule data (first request)? I'm having a hard time understanding how to use combine to chain two requests together, this is my first need to use combine for a widget I'm working on. I'm still fresh to Swift, so my terminology may be lacking.
My last code example wasn't correct and my question was unclear. I have two publishers and the second one depends on the first one. My understanding is still unclear on how to handle the data from my first publisher as well as in .flatMap for the second data. Does it need to be ObservableObject class and have #Published variables for the data? Do I use .assign or .sink to get data from my codable data Schedule and Live? Articles seem a bit too advance for myself as they create custom extensions and changing the API data to nested types.
New example code
import Foundation
import Combine
class DataGroup {
// How to get data from Schedule and Live codable data, do I use a variable and .assign or .sink?
// Where do I put the subscriber?
func requestSchedule(_ teamID : Int) -> AnyPublisher<Schedule, Error> {
let url = URL(string: "https://statsapi.web.nhl.com/api/v1/schedule?teamId=\(teamID)")!
return URLSession
.shared.dataTaskPublisher(for: url)
.map(\.data)
.decode(type: Schedule.self, decoder: JSONDecoder())
.flatMap {self.fetchLiveFeed($0.dates.first?.games.first?.link ?? "")}
/*
.flatMap {URLSession.shared.dataTaskPublisher(for: URL(string: $0.dates.first?.games.first?.link ?? "")!)}
*/
.eraseToAnyPublisher()
}
// Remove and put into flatMap URLSession.shared.dataTaskPublisher?
func fetchLiveFeed(_ link: String) -> AnyPublisher<Live, Error> {
let url = URL(string: "https://statsapi.web.nhl.com\(link)")!
return URLSession.shared.dataTaskPublisher(for: url)
.map(\.data)
.decode(type: Live.self, decoder: JSONDecoder())
.eraseToAnyPublisher()
}
}
OLD
import Foundation
import Combine
class CombineData {
var schedule: Schedule? // Get schedule data alongside live data
var live: Live?
private var cancellables = Set<AnyCancellable>()
func fetchSchedule(_ teamID: Int, _ completion: #escaping (/* Schedule, */Live) -> Void) {
let url = URL(string: "https://statsapi.web.nhl.com/api/v1/schedule?teamId=\(teamID)")!
URLSession.shared.dataTaskPublisher(for: url)
.map { $0.data }
.decode(type: Schedule.self, decoder: JSONDecoder())
.flatMap { self.fetchLiveFeed($0.dates.first?.games.first?.link ?? "") }
.receive(on: DispatchQueue.main)
.sink(receiveCompletion: { _ in }) { data in
// How to get both schedule data and live data here?
//self.schedule = ?
self.live = data
print(data)
completion(self.schedule!, self.live!)
}.store(in: &cancellables)
}
func fetchLiveFeed(_ link: String) -> AnyPublisher<Live, Error> {
let url = URL(string: "https://statsapi.web.nhl.com\(link)")!
return URLSession.shared.dataTaskPublisher(for: url)
.map(\.data)
.decode(type: Live.self, decoder: JSONDecoder())
.eraseToAnyPublisher()
}
}
The general idea is to use a flatMap for chaining, which is what you did, but if you also need the original value, you can return a Zip publisher (with a .zip operator) that puts two results into a tuple.
One of the publishers is the second request, and the other should just emit the value. You can typically do this with Just(v), but you have to make sure that its failure type (which is Never) matches with the other publisher. You can match its failure type with .setFailureType(to:):
publisher1
.flatMap { one in
Just(one).setFailureType(to: Error.self) // error has to match publisher2
.zip(publisher2(with: one))
}
.sink(receiveCompletion: { completion in
// ...
}, receiveValue: { (one, two) in
// ...
})
Alternatively, you can use Result.Publisher which would infer the error (but might look somewhat odd):
.flatMap { one in
Result.Publisher(.success(one))
.zip(publisher2)
}
So, in your case it's going to be something like this:
URLSession.shared.dataTaskPublisher(for: url)
.map(\.data)
.decode(type: Schedule.self, decoder: JSONDecoder())
.flatMap {
Result.Publisher(.success($0))
.zip(self.fetchLiveFeed($0.dates.first?.games.first?.link ?? ""))
}
.sink(receiveCompletion: { completion in
// ...
}, receiveValue: { (schedule, live) in
// ...
})
.store(in: &cancellables)

Swift Combine - Accessing separate lists of publishers

I have two lists of URLs that return some links to images.
The lists are passed into a future like
static func loadRecentEpisodeImagesFuture(request: [URL]) -> AnyPublisher<[RecentEpisodeImages], Never> {
return Future { promise in
print(request)
networkAPI.recentEpisodeImages(url: request)
.sink(receiveCompletion: { _ in },
receiveValue: { recentEpisodeImages in
promise(.success(recentEpisodeImages))
})
.store(in: &recentImagesSubscription)
}
.eraseToAnyPublisher()
}
Which calls:
/// Get a list of image sizes associated with a featured episode .
func featuredEpisodeImages(featuredUrl: [URL]) -> AnyPublisher<[FeaturedEpisodeImages], Error> {
let featuredEpisodesImages = featuredUrl.map { (featuredUrl) -> AnyPublisher<FeaturedEpisodeImages, Error> in
return URLSession.shared
.dataTaskPublisher(for: featuredUrl)
.map(\.data)
.decode(type: FeaturedEpisodeImages.self, decoder: decoder)
.receive(on: networkApiQueue)
.catch { _ in Empty<FeaturedEpisodeImages, Error>() }
.print("###Featured###")
.eraseToAnyPublisher()
}
return Publishers.MergeMany(featuredEpisodesImages).collect().eraseToAnyPublisher()
}
/// Get a list of image sizes associated with a recent episode .
func recentEpisodeImages(recentUrl: [URL]) -> AnyPublisher<[RecentEpisodeImages], Error> {
let recentEpisodesImages = recentUrl.map { (recentUrl) -> AnyPublisher<RecentEpisodeImages, Error> in
return URLSession.shared
.dataTaskPublisher(for: recentUrl)
.map(\.data)
.decode(type: RecentEpisodeImages.self, decoder: decoder)
.receive(on: networkApiQueue)
.catch { _ in Empty<RecentEpisodeImages, Error>() }
.print("###Recent###")
.eraseToAnyPublisher()
}
return Publishers.MergeMany(recentEpisodesImages).collect().eraseToAnyPublisher()
}
and is attached to the app state:
/// Takes an action and returns a future mapped to another action.
static func recentEpisodeImages(action: RequestRecentEpisodeImages) -> AnyPublisher<Action, Never> {
return loadRecentEpisodeImagesFuture(request: action.request)
.receive(on: networkApiQueue)
.map({ images in ResponseRecentEpisodeImages(response: images) })
.replaceError(with: RequestFailed())
.eraseToAnyPublisher()
}
It seems that:
return Publishers.MergeMany(recentEpisodes).collect().eraseToAnyPublisher()
doesn't give me a reliable downstream value as whichever response finishes last overwrites the earlier response.
I am able to log the responses of both series of requests. Both are processing the correct arrays and returning the proper json.
I would like something like:
return recentEpisodeImages
but currently this gives me the error
Cannot convert return expression of type '[AnyPublisher<RecentEpisodeImages, Error>]' to return type 'AnyPublisher<[RecentEpisodeImages], Error>'
How can I collect the values of the inner publisher and return them as
AnyPublisher<[RecentEpisodeImages], Error>
Presuming that the question is how to turn an array of URLs into an array of what you get when you download and process the data from those URLs, the answer is: turn the array into a sequence publisher, process each URL by way of flatMap, and collect the result.
Here, for instance, is how to turn an array of URLs representing images into an array of the actual images (not identically what you're trying to do, but probably pretty close):
func publisherOfArrayOfImages(urls:[URL]) -> AnyPublisher<[UIImage],Error> {
urls.publisher
.flatMap { (url:URL) -> AnyPublisher<UIImage,Error> in
return URLSession.shared.dataTaskPublisher(for: url)
.compactMap { UIImage(data: $0.0) }
.mapError { $0 as Error }
.eraseToAnyPublisher()
}.collect().eraseToAnyPublisher()
}
And here's how to test it:
let urls = [
URL(string:"http://www.apeth.com/pep/moe.jpg")!,
URL(string:"http://www.apeth.com/pep/manny.jpg")!,
URL(string:"http://www.apeth.com/pep/jack.jpg")!,
]
let pub = publisherOfArrayOfImages(urls:urls)
pub.sink { print($0) }
receiveValue: { print($0) }
.store(in: &storage)
You'll see that what pops out the bottom of the pipeline is an array of three images, corresponding to the array of three URLs we started with.
(Note, please, that the order of the resulting array is random. We fetched the images asynchronously, so the results arrive back at our machine in whatever order they please. There are ways around that problem, but it is not what you asked about.)

What's the proper method for creating re-usable pipelines with Combine?

I'm relatively new to the Functional Reactive programming world, and still trying to wrap my head around the concepts. I'm utilizing an SDK to make some network requests - specifically to query a remote database. The SDK returns a publisher, and I have a working pipeline that transforms that result into model objects. Here's that working pipeline:
let existingClaimQuery = "SELECT Id, Subject, CaseNumber FROM Case WHERE Status != 'Closed' ORDER BY CaseNumber DESC"
let requestForOpenCases = RestClient.shared.request(forQuery: existingClaimQuery, apiVersion: RestClient.apiVersion)
caseCancellable = RestClient.shared
.publisher(for: requestForOpenCases)
.receive(on: RunLoop.main)
.tryMap({restresponse -> [String:Any] in
let json = try restresponse.asJson() as? [String:Any]
return json ?? RestClient.JSONKeyValuePairs()
})
.map({json -> [[String:Any]] in
let records = json["records"] as? [[String:Any]]
return records ?? [[:]]
})
.map({
$0.map{(item) -> Claim in
return Claim(
id: item["Id"] as? String ?? "None Listed",
subject: item["Subject"] as? String ?? "None Listed",
caseNumber: item["CaseNumber"] as? String ?? "0"
)
}
})
.mapError{error -> Error in
print(error)
return error
}
.catch{ error in
return Just([])
}
.assign(to: \.claims, on: self)
I went to work on another section of the code, and realized I often need to do this same process - write a query, create a request for that query, and process it through a pipeline that ultimately returns a [[String:Any]].
So here's the million dollar question. What's the right way to encapsulate this pipeline such that I can re-use it without having to copy/pasta the entire pipeline all over the code base? This is my ... attempt at it, but it feels ...wrong?
class QueryStream: ObservableObject {
var query: String = ""
private var queryCancellable: AnyCancellable?
#Published var records: [[String:Any]] = [[String:Any]]()
func execute(){
let queryRequest = RestClient.shared.request(forQuery: query, apiVersion: RestClient.apiVersion)
queryCancellable = RestClient.shared.publisher(for: queryRequest)
.receive(on: RunLoop.main)
.tryMap({restresponse -> [String:Any] in
let json = try restresponse.asJson() as? [String:Any]
return json ?? [String:Any]()
})
.map({json -> [[String:Any]] in
let records = json["records"] as? [[String:Any]]
return records ?? [[:]]
})
.mapError{error -> Error in
print(error)
return error
}
.catch{ error in
return Just([])
}
.assign(to: \.records, on: self)
}
}
This still requires a pipeline to be written for each use. I feel like there should be some way to have a one off promise like pipeline that would allow for
let SomeRecords = QueryStream("Query here").execute()
Am I too n00b? overthinking it? What's the stack's wisdom?
Entire pipelines are not reusable. Publishers are reusable. When I say "publisher" I mean an initial publisher plus operators attached to it. (Remember, an operator is itself a publisher.) A publisher can exist as a property of something, so you can subscribe to it, or it can be generated for a particular case (like a particular query request) by a function.
To illustrate, here's a one-off pipeline:
let s = "https://photojournal.jpl.nasa.gov/tiff/PIA23172.tif"
let url = URL(string:s)!
let eph = URLSessionConfiguration.ephemeral
let session = URLSession(configuration: eph)
session.dataTaskPublisher(for: url)
.map {$0.data}
.replaceError(with: Data())
.compactMap { UIImage(data:$0) }
.receive(on: DispatchQueue.main)
.assign(to: \.image, on: self.iv)
.store(in:&self.storage)
That pipeline tries to download the data from a URL, tests to see whether it's image data, and if it is, turns the image data into an image and displays it in an image view in the interface.
Let's say I want to do this for various different remote images. Obviously it would be ridiculous to repeat the whole pipeline everywhere. What differs up front is the URL, so let's encapsulate the first part of the pipeline as a publisher that can be generated on demand based on the URL:
func image(fromURL url:URL) -> AnyPublisher<UIImage,Never> {
let eph = URLSessionConfiguration.ephemeral
let session = URLSession(configuration: eph)
return session.dataTaskPublisher(for: url)
.map {$0.data}
.replaceError(with: Data())
.compactMap { UIImage(data:$0) }
.receive(on: DispatchQueue.main)
.eraseToAnyPublisher()
}
Now the only thing that needs to be repeated in various places in our code is the subscriber to that publisher:
let s = "https://photojournal.jpl.nasa.gov/tiff/PIA23172.tif"
let url = URL(string:s)!
image(fromURL:url)
.map{Optional($0)}
.assign(to: \.image, on: self.iv)
.store(in:&self.storage)
You see? Elsewhere, we might have a different URL, and we might do something different with the UIImage that comes popping out of the call to image(fromURL:), and that's just fine; the bulk of the pipeline has been encapsulated and doesn't need to be repeated.
Your example pipeline's publisher is susceptible of that same sort of encapsulation and reuse.
Let me first note that I think you are dispatching to main to early in your pipeline. As far as I can tell, all of your map transforms are pure functions (no side effects or references to mutable state), so they can just as well run on the background thread and thus not block the UI.
Second, as Matt said, a Publisher is generally reusable. Your pipeline builds up a big complex Publisher, and then subscribes to it, which produces an AnyCancellable. So factor out the big complex Publisher but not the subscribing.
You can factor it out into an extension method on your RestClient for convenience:
extension RestClient {
func records<Record>(
forQuery query: String,
makeRecord: #escaping ([String: Any]) throws -> Record)
-> AnyPublisher<[Record], Never>
{
let request = self.request(forQuery: query, apiVersion: RestClient.apiVersion)
return self.publisher(for: request)
.tryMap { try $0.asJson() as? [String: Any] ?? [:] }
.map { $0["records"] as? [[String: Any]] ?? [] }
.tryMap { try $0.map { try makeRecord($0) } }
.mapError { dump($0) } // dump is a Swift standard function
.replaceError(with: []) // simpler than .catch
.eraseToAnyPublisher()
}
}
Then you can use it like this:
struct Claim {
var id: String
var subject: String
var caseNumber: String
}
extension Claim {
static func from(json: [String: Any]) -> Claim {
return .init(
id: json["Id"] as? String ?? "None Listed",
subject: json["Subject"] as? String ?? "None Listed",
caseNumber: json["CaseNumber"] as? String ?? "0")
}
}
class MyController {
var claims: [Claim] = []
var caseCancellable: AnyCancellable?
func run() {
let existingClaimQuery = "SELECT Id, Subject, CaseNumber FROM Case WHERE Status != 'Closed' ORDER BY CaseNumber DESC"
caseCancellable = RestClient.shared.records(forQuery: existingClaimQuery, makeRecord: Claim.from(json:))
.receive(on: RunLoop.main)
.assign(to: \.claims, on: self)
}
}
Note that I've put the receive(on: RunLoop.main) operator in the method that subscribes to the publisher, rather than building it in to the publisher. This makes it easy to add additional operators that run on a background scheduler before dispatching to the main thread.
UPDATE
From your comment:
In promise syntax, i could say execute run() as defined above, and .then(doSomethingWithThatData()) knowing that the doSomethingWithThatData wouldn't run until the intial work had completed successfully. I'm trying to develop a setup where I need to use this records(fromQuery:) method runs, and then (and only then) do soemthing with that data. I'm struggling with how to bolt that on to the end.
I don't know what promise implementation you're using, so it's difficult to know what your .then(doSomethingWithThatData()) does. What you've written doesn't really make much sense in Swift. Perhaps you meant:
.then { data in doSomething(with: data) }
In which case, the doSomething(with:) method cannot possibly be called until the data is available, because doSomething(with:) takes the data as an argument!

Swift Combine - prefix publisher on array

I'm playing around with publishers in Swift/Combine, I have a function that fetches 100 records and returns them as an array.
As a test I want to return just the first two items, but it's not working as I expected it to, it always returns 100, my feeling is that it's because, the first item is an array of 100 items, if so, how do I split them out?
import UIKit
import Combine
struct Post : Decodable {
let userId: Int
let id: Int
let title: String
let body: String
}
//let url = URL(string: "https://jsonplaceholder.typicode.com/todos/1")!
let url = URL(string: "https://jsonplaceholder.typicode.com/posts")!
var subscriptions: Set<AnyCancellable> = []
func fetch() -> AnyPublisher<[Post], Never> {
return URLSession.shared.dataTaskPublisher(for: url)
.tryCompactMap{ (arg) -> [Post]? in
let (data, _) = arg
return try JSONDecoder().decode([Post].self, from: data)
}
//.print("here")
.replaceError(with: [])
.eraseToAnyPublisher()
}
fetch()
.prefix(2)
.sink(receiveCompletion: { (comp) in
print("comp: \(comp)")
}) { (res) in
print("Res: \(res.count)")
}.store(in: &subscriptions)
Update, this seems to work, not sure on the syntax though:
fetch()
.flatMap { Publishers.Sequence(sequence: $0) }
.prefix(2)
.sink(receiveCompletion: { (comp) in
print("comp: \(comp)")
}) { (res) in
print("Res: \(res)")
}.store(in: &subscriptions)
You can use map to take the full array and extract only what you need. Take a look at the following example:
[Array(0..<100)].publisher.map { array in
return Array(array[..<2])
}.sink(receiveValue: { items in
print(items)
})
This is a publisher that publishes an array with 100 values. I then use array[..<2] to create an ArraySlice that contains the first two items. This slice is then converted to Array so it's easier to use later.
The items argument received in sink is an array with just two items.
.flatMap(maxPublishers: 2)
Could be a better approach for you depending on what you want to accomplish overall.