Combine - Remove pipeline from container - swift

Assume the following simplified code-snippet
import Foundation
import Combine
public class NetworkFetch {
fileprivate var networkPipelines : Set<AnyCancellable> = []
public func loadDataFor(url : URL)
{
URLSession.shared.dataTaskPublisher(for: url)
.map { $0.data }
.decode(type: City.self, decoder: JSONDecoder())
.eraseToAnyPublisher()
.sink(receiveCompletion: {_ in print("Finish")},
receiveValue: { v in
print("\(c)\n")
}
)
.store(in: &networkPipelines)
}
}
For each call of loadDataFor a new combine-pipeline is generated and added to the networkPipelines container. This container grows over time.
What is the correct way to remove such an URLSession-pipeline from this container as soon as all data is fetched by the URLSession-pipeline?

One thing you could do is remove your own subscription from inside sink:
But perhaps the better approach is to subscribe to a PassthroughSubject one time and send the requested URL and the callback through it:
private let subject = PassthroughSubject<(URL, (City) -> Void)), Never>()
private var c: Set<AnyCancellable> = []
init() {
subject
.flatMap { (url, callback) in
URLSession.shared.dataTaskPublisher(for: url)
.map(\.data)
.decode(type: City.self, decoder: JSONDecoder())
.zip(Just(callback).setFailureType(to: Error.self))
}
.sink(receiveCompletion: {_ in print("Finish")},
receiveValue: { (city, callback) in
callback(city)
}
)
.store(in: &c)
}
public func loadDataFor(url : URL, callback: #escaping (City) -> Void) {
subject.send(url, callback)
}
So, a single subscription can handle multiple requests by sending a pair of the requested URL and the callback through the subject.

Related

Chaining multiple requests in Combine

I am trying to nest multiple request together to create the object that I need. Just a short description about the API and what I try to accomplish. I am basically fetching an array of exercises. These exercise however don't have every information that I need so I have to take the id and do another request to get the whole object. From there on I get an array of images which contains the url that I need. Then I need another attribute called variations which consists of an array of exercise ids.
So I basically need to do 4 requests:
Fetch all exercises
fetch exercise by id
From there I take the image urls and fetch the images
Then I want to fetch all the variations using the ids from the array
So my new object should be an array of type:
struct ExerciseDetails: Hashable {
let id: Int
let title: String
let image: [UIImage]
let variations: [ExerciseDetails]
}
Here is what I am trying to do:
URLSession.shared.dataTaskPublisher(for: URL(string: "https://wger.de/api/v2/exercise/?limit=70&offset=40"))
.map { $0.data }
.decode(type: [Exercise].self, decoder: JSONDecoder())
.map { $0.map { $0.id } }
.flatMap(\.publisher)
.flatMap { id in
URLSession.shared.dataTaskPublisher(for: URL(string: "https://wger.de/api/v2/exerciseinfo/\(id)")!)
.map(\.data)
.decode(type: Exercise.self, decoder: JSONDecoder())
.map { $0.images }
.flatMap { image in
URLSession.shared.dataTaskPublisher(for: URL(string: image.imageUrl))
.map { return UIImage(data: $0.data) }
}
}
.map { $0.map { $0.variations } }
.flatMap(\.publisher)
.flatMap { id in
URLSession.shared.dataTaskPublisher(for: URL(string: "https://wger.de/api/v2/exerciseinfo/\(id)")!)
.map(\.data)
.decode(type: Exercise.self, decoder: JSONDecoder())
.map { return $0 }
}
Now I don't know how to map the output to -> [ExerciseInfo]
I think I am on a good way but there is something missing.
Can anybody help?

Chaining with flatMap

I'm having difficulty trying to figure out how to loop TMDb results from my publisher 1 and use the value I'm getting from key id for my second publisher. It's not looping through each result, it does get the proper URL inside my func fetchVideos(_ id: Int) but it's not calling each URL from TMDb's results array. I'm not sure if it's because I'm also getting an Array of results from Videos Codable data?
I attempted to use Publishers.MergeMany in my first publisher's flatMap. I'm still definitely at a novice level for combine, any tips would help. I'm trying to get a list of movies, then from the movies get the id key then use that to fetch the movie trailer data for each movie.
print output
https://api.themoviedb.org/3/movie/602223/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/459151/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/385128/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/522478/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/637693/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/529203/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/578701/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/631843/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/645856/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/581644/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/436969/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/568620/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/522931/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/681260/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/630586/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/671/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/618416/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/646207/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/550988/videos?api_key=API_KEY&language=en-US
https://api.themoviedb.org/3/movie/482373/videos?api_key=API_KEY&language=en-US
Videos(id: 459151, results: [themoviedb_demo.Video(id: 3F24D610-4261-4AEB-8906-A9D0E5FE8E4D, iso639_1: "en", iso3166_1: "US", key: "CK6xdYIsaa0", name: "DreamWorks\' The Boss Baby: Family Business | Official Trailer #3 | Peacock", site: "YouTube", size: 1080, type: "Trailer"), themoviedb_demo.Video(id: 417EF49C-B983-4CC3-B435-7A902DECE917, iso639_1: "en", iso3166_1: "US", key: "-rF2j6K5FoM", name: "The Boss Baby 2: Family Business – Official Trailer 2 (Universal Pictures) HD", site: "YouTube", size: 1080, type: "Trailer"), themoviedb_demo.Video(id: C34A3F5F-9429-4267-86F0-5506EF3E8281, iso639_1: "en", iso3166_1: "US", key: "QPzy8Ckza08", name: "THE BOSS BABY: FAMILY BUSINESS | Official Trailer", site: "YouTube", size: 1080, type: "Trailer")])
Codable data
struct Videos: Codable {
let id: Int
let results: [Video]
}
struct Video: Codable {
let id = Int
let key: String
let name: String
enum CodingKeys: String, CodingKey {
case id
case key, name
}
}
struct TMDb: Codable {
let results: [Results]?
}
struct Results: Codable {
let id: Int
let releaseDate, title: String?
let name: String?
enum CodingKeys: String, CodingKey {
case id
case releaseDate = "release_date"
case title
case name
}
}
#Published var movies = TMDb(results: Array(repeating: Results(id: 1, releaseDate: "", title: "", name: "") , count: 5))
#Published var videos = Videos(id: 1, results: Array(repeating: Video(id: 1, key: "", name: "") , count: 5))
func getUpcoming() {
var request = URLRequest(url:URL(string:"https://api.themoviedb.org/3/movie/upcoming?api_key=API_KEY&language=en-US&page=1")!)
request.httpMethod = "GET"
let publisher = URLSession.shared.dataTaskPublisher(for: request)
.map{ $0.data }
.decode(type: TMDb.self, decoder: JSONDecoder())
let publisher2 = publisher
.flatMap{
// loop results from TMDb for id for publisher 2, only one is called
Publishers.MergeMany($0.results!.map { item in
return self.fetchVideos(item.id)
.map { $0 as Videos }
.replaceError(with: nil)
})
}
// Publishers.CombineLatest
Publishers.Zip(publisher, publisher2)
.receive(on: DispatchQueue.main)
.sink(receiveCompletion: {_ in
}, receiveValue: { movies, videos in
self.movies = movies
self.videos = videos
}).store(in: &cancellables)
}
func fetchVideos(_ id: Int) -> AnyPublisher<Videos, Error> {
let url = URL(string: "https://api.themoviedb.org/3/movie/\(id)/videos?api_key=API_KEY&language=en-US")!
return URLSession.shared.dataTaskPublisher(for: url)
.mapError { $0 as Error }
.map{ $0.data }
.decode(type: Videos.self, decoder: JSONDecoder())
.eraseToAnyPublisher()
}
Hi #cole for this operation you don't need either the merge or the zip, because you are not subscribing to two publishers, you are attempting to do an action after your first publisher emitted an event.
For this you only need a map .handleEvents in my opinion.
So lets try to enhance your code, we want to update both movies and videos separately, but we still need videos to be dependent of movies
First we will create the publisher to request the movies:
var request = URLRequest(url:URL(string:"https://api.themoviedb.org/3/movie/upcoming?api_key=API_KEY&language=en-US&page=1")!)
request.httpMethod = "GET"
let publisher = URLSession.shared.dataTaskPublisher(for: request)
.map{ $0.data }
.decode(type: TMDb.self, decoder: JSONDecoder())
Now we enhance this publisher handling by assigning movies:
var request = URLRequest(url:URL(string:"https://api.themoviedb.org/3/movie/upcoming?api_key=API_KEY&language=en-US&page=1")!)
request.httpMethod = "GET"
let publisher = URLSession.shared.dataTaskPublisher(for: request)
.map{ $0.data }
.decode(type: TMDb.self, decoder: JSONDecoder())
.sink(receiveCompletion: { print ($0) },
receiveValue: { self.movies = $0.results })
Now we will add .handleEvent in order to iterate through our movies to create all the publishers which emit videos events and append videos for the videos array:
var request = URLRequest(url:URL(string:"https://api.themoviedb.org/3/movie/upcoming?api_key=API_KEY&language=en-US&page=1")!)
request.httpMethod = "GET"
let publisher = URLSession.shared.dataTaskPublisher(for: request)
.map{ $0.data }
.decode(type: TMDb.self, decoder: JSONDecoder())
.sink(receiveCompletion: { print ($0) },
receiveValue: { self.movies = $0.results })
.handleEvents(receiveSubscription:nil, receiveOutput: { [weak self] movies in guard let self = self else {return}
self.videos = [Videos]()
for movie in movies.results {
self.fetchVideos(movie.id)
}, receiveCompletion:nil, receiveCancel:nil, receiveRequest:nil)
})
.store(in: &cancellables)
Now for the last step lets update the fetchVideos accordingly:
func fetchVideos(_ id: Int) {
let url = URL(string: "https://api.themoviedb.org/3/movie/\(id)/videos?api_key=API_KEY&language=en-US")!
return URLSession.shared.dataTaskPublisher(for: url)
.mapError { $0 as Error }
.map{ $0.data }
.decode(type: Videos.self, decoder: JSONDecoder())
.sink(receiveCompletion: { print ($0) },
receiveValue: { [weak self] videos in guard let self = self else {return}
self.videos.append(videos)
})
.store(in: &cancellables)
}
Solved my own question. I needed to create an array for my #Published variable instead of single item. Then I needed to call .collect() and .append() in my publisher's flatMap which will append to my #Published variable videos.

Chain network calls sequentially in Combine

My goal is to chain multiple (two at this time) network calls with Combine, breaking chain if first call fails.
I have two object types: CategoryEntity and SubcategoryEntity. Every CategoryEntity has a property called subcategoriesIDS.
With first call I need to fetch all subcategories, with second I will fetch all categories and then I will create an array of CategoryEntityViewModel.
CategoryEntityViewModel contains an array of SubcategoryEntityViewModel based on CategoryEntity's subcategoriesIDS.
Just to be clearer:
Fetch subcategories
Fetch categories
Create a SubcategoryEntityViewModel for every fetched subcategory and store somewhere
CategoryEntityViewModel is created for every category fetched. This object will be initialized with a CategoryEntity object and an array of SubcategoryEntityViewModel, found filtering matching ids between subcategoriesIDS and stored SubcategoryEntityViewModel array
My code right now is:
class CategoriesService: Service, ErrorManager {
static let shared = CategoriesService()
internal let decoder = JSONDecoder()
#Published var error: ServerError = .none
private init() {
decoder.dateDecodingStrategyFormatters = [ DateFormatter.yearMonthDay ]
}
func getAllCategories() -> AnyPublisher<[CategoryEntity], ServerError> {
let request = self.createRequest(withUrlString: "\(AppSettings.api_endpoint)/categories/all", forMethod: .get)
return URLSession.shared.dataTaskPublisher(for: request)
.receive(on: DispatchQueue.main)
.tryMap { data, response -> Data in
guard let httpResponse = response as? HTTPURLResponse, 200..<300 ~= httpResponse.statusCode else {
switch (response as! HTTPURLResponse).statusCode {
case (401):
throw ServerError.notAuthorized
default:
throw ServerError.unknown
}
}
return data
}
.map { $0 }
.decode(type: NetworkResponse<[CategoryEntity]>.self, decoder: self.decoder)
.map { $0.result}
.mapError { error -> ServerError in self.manageError(error: error)}
.receive(on: RunLoop.main)
.eraseToAnyPublisher()
}
func getAllSubcategories() -> AnyPublisher<[SubcategoryEntity], ServerError> {
let request = self.createRequest(withUrlString: "\(AppSettings.api_endpoint)/subcategories/all", forMethod: .get)
return URLSession.shared.dataTaskPublisher(for: request)
.receive(on: DispatchQueue.main)
.tryMap { data, response -> Data in
guard let httpResponse = response as? HTTPURLResponse, 200..<300 ~= httpResponse.statusCode else {
switch (response as! HTTPURLResponse).statusCode {
case (401):
throw ServerError.notAuthorized
default:
throw ServerError.unknown
}
}
return data
}
.map { $0 }
.decode(type: NetworkResponse<[SubcategoryEntity]>.self, decoder: self.decoder)
.map { $0.result }
.mapError { error -> ServerError in self.manageError(error: error)}
.receive(on: RunLoop.main)
.eraseToAnyPublisher()
}
}
These methods are working (sink is called in another class, don't think it is useful so not copied here) but I cannot find the correct way to chain them.
The way to chain async operations with Combine is flatMap. Produce the second publisher inside the map function. Be sure to pass any needed info as a value down into the map function so the second publisher can use it. See How to replicate PromiseKit-style chained async flow using Combine + Swift for a basic example.

Apple Combine framework: How to execute multiple Publishers in parallel and wait for all of them to finish?

I am discovering Combine. I wrote methods that make HTTP requests in a "combine" way, for example:
func testRawDataTaskPublisher(for url: URL) -> AnyPublisher<Data, Error> {
var request = URLRequest(url: url,
cachePolicy: .useProtocolCachePolicy,
timeoutInterval: 15)
request.httpMethod = "GET"
return urlSession.dataTaskPublisher(for: request)
.tryMap {
return $0.data
}
.eraseToAnyPublisher()
}
I would like to call the method multiple times and do a task after all, for example:
let myURLs: [URL] = ...
for url in myURLs {
let cancellable = testRawDataTaskPublisher(for: url)
.sink(receiveCompletion: { _ in }) { data in
// save the data...
}
}
The code above won't work because I have to store the cancellable in a variable that belongs to the class.
The first question is: is it a good idea to store many (for example 1000) cancellables in something like Set<AnyCancellable>??? Won't it cause memory leaks?
var cancellables = Set<AnyCancellable>()
...
let cancellable = ...
cancellables.insert(cancellable) // ???
And the second question is: how to start a task when all the cancellables are finished? I was thinking about something like that
class Test {
var cancellables = Set<AnyCancellable>()
func run() {
// show a loader
let cancellable = runDownloads()
.receive(on: RunLoop.main)
.sink(receiveCompletion: { _ in }) { _ in
// hide the loader
}
cancellables.insert(cancellable)
}
func runDownloads() -> AnyPublisher<Bool, Error> {
let myURLs: [URL] = ...
return Future<Bool, Error> { promise in
let numberOfURLs = myURLS.count
var numberOfFinishedTasks = 0
for url in myURLs {
let cancellable = testRawDataTaskPublisher(for: url)
.sink(receiveCompletion: { _ in }) { data in
// save the data...
numberOfFinishedTasks += 1
if numberOfFinishedTasks >= numberOfURLs {
promise(.success(true))
}
}
cancellables.insert(cancellable)
}
}.eraseToAnyPublisher()
}
func testRawDataTaskPublisher(for url: URL) -> AnyPublisher<Data, Error> {
...
}
}
Normally I would use DispatchGroup, start multiple HTTP tasks and consume the notification when the tasks are finished, but I am wondering how to write that in a modern way using Combine.
You can run some operations in parallel by creating a collection of publishers, applying the flatMap operator and then collect to wait for all of the publishers to complete before continuing. Here's an example that you can run in a playground:
import Combine
import Foundation
func delayedPublisher<Value>(_ value: Value, delay after: Double) -> AnyPublisher<Value, Never> {
let p = PassthroughSubject<Value, Never>()
DispatchQueue.main.asyncAfter(deadline: .now() + after) {
p.send(value)
p.send(completion: .finished)
}
return p.eraseToAnyPublisher()
}
let myPublishers = [1,2,3]
.map{ delayedPublisher($0, delay: 1 / Double($0)).print("\($0)").eraseToAnyPublisher() }
let cancel = myPublishers
.publisher
.flatMap { $0 }
.collect()
.sink { result in
print("result:", result)
}
Here is the output:
1: receive subscription: (PassthroughSubject)
1: request unlimited
2: receive subscription: (PassthroughSubject)
2: request unlimited
3: receive subscription: (PassthroughSubject)
3: request unlimited
3: receive value: (3)
3: receive finished
2: receive value: (2)
2: receive finished
1: receive value: (1)
1: receive finished
result: [3, 2, 1]
Notice that the publishers are all immediately started (in their original order).
The 1 / $0 delay causes the first publisher to take the longest to complete. Notice the order of the values at the end. Since the first took the longest to complete, it is the last item.

Loop over Publisher Combine framework

I have the following function to perform an URL request:
final class ServiceManagerImpl: ServiceManager, ObservableObject {
private let session = URLSession.shared
func performRequest<T>(_ request: T) -> AnyPublisher<String?, APIError> where T : Request {
session.dataTaskPublisher(for: self.urlRequest(request))
.tryMap { data, response in
try self.validateResponse(response)
return String(data: data, encoding: .utf8)
}
.mapError { error in
return self.transformError(error)
}
.eraseToAnyPublisher()
}
}
Having these 2 following functions, I can now call the desired requests from corresponded ViewModel:
final class AuditServiceImpl: AuditService {
private let serviceManager: ServiceManager = ServiceManagerImpl()
func emptyAction() -> AnyPublisher<String?, APIError> {
let request = AuditRequest(act: "", nonce: String.randomNumberGenerator)
return serviceManager.performRequest(request)
}
func burbleAction(offset: Int) -> AnyPublisher<String?, APIError> {
let request = AuditRequest(act: "burble", nonce: String.randomNumberGenerator, offset: offset)
return serviceManager.performRequest(request)
}
}
final class AuditViewModel: ObservableObject {
#Published var auditLog: String = ""
private let auditService: AuditService = AuditServiceImpl()
init() {
let timer = Timer(timeInterval: 5, repeats: true) { _ in
self.getBurbles()
}
RunLoop.main.add(timer, forMode: .common)
}
func getBurbles() {
auditService.emptyAction()
.flatMap { [unowned self] offset -> AnyPublisher<String?, APIError> in
let currentOffset = Int(offset?.unwrapped ?? "") ?? 0
return self.auditService.burbleAction(offset: currentOffset)
}
.receive(on: RunLoop.main)
.sink(receiveCompletion: { [unowned self] completion in
print(completion)
}, receiveValue: { [weak self] burbles in
self?.auditLog = burbles!
})
.store(in: &cancellableSet)
}
}
Everything is fine when I use self.getBurbles() for the first time. However, for the next calls, print(completion) shows finished, and the code doesn't perform self?.auditLog = burbles!
I don't know how can I loop over the getBurbles() function and get the response at different intervals.
Edit
The whole process in a nutshell:
I call getBurbles() from class initializer
getBurbles() calls 2 nested functions: emptyAction() and burbleAction(offset: Int)
Those 2 functions generate different requests and call performRequest<T>(_ request: T)
Finally, I set the response into auditLog variable and show it on the SwiftUI layer
There are at least 2 issues here.
First when a Publisher errors it will never produce elements again. That's a problem here because you want to recycle the Publisher here and call it many times, even if the inner Publisher fails. You need to handle the error inside the flatMap and make sure it doesn't propagate to the enclosing Publisher. (ie you can return a Result or some other enum or tuple that indicates you should display an error state).
Second, flatMap is almost certainly not what you want here since it will merge all of the api calls and return them in arbitrary order. If you want to cancel any existing requests and only show the latest results then you should use .map followed by switchToLatest.