I'm fetching some data from an endpoint, every time I requested for 10 items. These items contains a URL that should be scraping, which slow down the process a bit.
Due to some bug probably, the received data from scraping is not the same number of the original data, which something i will deal with later. So I don't know when all data is ready.
Now, I want to make sure when I get all the data, then refresh the collection view. the only way that I can think of it, is using the timestamp of data and if the timestamp is not updated for about a sec, means, I have all the data that I need than I can refresh collection view. Not the best way of course
I added a timestamp variable
private lazy var fetchedTimestamp = Date().nanosecondsSince1970
then, whenever the data is recived I updated this timestamp with the the current time
Than, I use DispatchQueue.main.asyncAfter to check the timestamp on sec later and if the the timestamp one sec later is 1 sec after the last fetchedTimestampupdate, means the data hasn't been updated for about a sec, so I know it's ready to reload
KingfisherManager.shared.retrieveImage(with: imageURL) { result in
if let image = try? result.get().image {
scraper.setBaseImage(with: image)
DispatchQueue.main.async { [weak self] in
guard let this = self else { return }
this.feedsCollectionView.feeds.append(scraper)
this.fetchedTimestamp = Date().nanosecondsSince1970
DispatchQueue.main.asyncAfter(deadline: .now() + 1, execute: {
let currentTime = Date().nanosecondsSince1970
if currentTime >= this.fetchedTimestamp + Int64(1e+9) {
this.feedsCollectionView.reload()
}
})
}
}
}
Is there any way to handle it better? Thank you so much
Related
I have a limit of 40 URL Session calls to my API per minute.
I have timed the number of calls in any 60s and when 40 calls have been reached I introduced sleep(x). Where x is 60 - seconds remaining before new minute start. This works fine and the calls don’t go over 40 in any given minute. However the limit is still exceeded as there might be more calls towards the end of the minute and more at the beginning of the next 60s count. Resulting in an API error.
I could add a:
usleep(x)
Where x would be 60/40 in milliseconds. However as some large data returns take much longer than simple queries that are instant. This would increase the overall download time significantly.
Is there a way to track the actual rate to see by how much to slow the function down?
Might not be the neatest approach, but it works perfectly. Simply storing the time of each call and comparing it to see if new calls can be made and if not, the delay required.
Using previously suggested approach of delay before each API call of 60/40 = 1.5s (Minute / CallsPerMinute), as each call takes a different time to produce response, total time taken to make 500 calls was 15min 22s. Using the below approach time taken: 11min 52s as no unnecessary delay has been introduced.
Call before each API Request:
API.calls.addCall()
Call in function before executing new API task:
let limit = API.calls.isOverLimit()
if limit.isOver {
sleep(limit.waitTime)
}
Background Support Code:
var globalApiCalls: [Date] = []
public class API {
let limitePerMinute = 40 // Set API limit per minute
let margin = 2 // Margin in case you issue more than one request at a time
static let calls = API()
func addCall() {
globalApiCalls.append(Date())
}
func isOverLimit() -> (isOver: Bool, waitTime: UInt32)
{
let callInLast60s = globalApiCalls.filter({ $0 > date60sAgo() })
if callInLast60s.count > limitePerMinute - margin {
if let firstCallInSequence = callInLast60s.sorted(by: { $0 > $1 }).dropLast(2).last {
let seconds = Date().timeIntervalSince1970 - firstCallInSequence.timeIntervalSince1970
if seconds < 60 { return (true, UInt32(60 + margin) - UInt32(seconds.rounded(.up))) }
}
}
return (false, 0)
}
private func date60sAgo() -> Date
{
var dayComponent = DateComponents(); dayComponent.second = -60
return Calendar.current.date(byAdding: dayComponent, to: Date())!
}
}
Instead of using sleep have a counter. You can do this with a Semaphore (it is a counter for threads, on x amount of threads allowed at a time).
So if you only allow 40 threads at a time you will never have more. New threads will be blocked. This is much more efficient than calling sleep because it will interactively account for long calls and short calls.
The trick here is that you would call a function like this every sixty second. That would make a new semaphore every minute that would only allow 40 calls. Each semaphore would not affect one another but only it's own threads.
func uploadImages() {
let uploadQueue = DispatchQueue.global(qos: .userInitiated)
let uploadGroup = DispatchGroup()
let uploadSemaphore = DispatchSemaphore(value: 40)
uploadQueue.async(group: uploadGroup) { [weak self] in
guard let self = self else { return }
for (_, image) in images.enumerated() {
uploadGroup.enter()
uploadSemaphore.wait()
self.callAPIUploadImage(image: image) { (success, error) in
uploadGroup.leave()
uploadSemaphore.signal()
}
}
}
uploadGroup.notify(queue: .main) {
// completion
}
}
This is a little snip it from my code.
db.collection("tasks").document(Auth.auth().currentUser?.uid ?? "").collection("currentUser").whereField("Date", isEqualTo: date).addSnapshotListener{ (querySnapshot, err) in
self.task = []
if querySnapshot!.documents.isEmpty{
self.firstView.alpha = 1
self.labelText.alpha = 1
}
else{
self.firstView.alpha = 0
self.labelText.alpha = 0
if let snapshotDocuments = querySnapshot?.documents{
for doc in snapshotDocuments{
let data = doc.data()
print("xx")
if let descS = data["Task Description"] as? String, let dateS = data["Date"] as? String, let titleS = data["Task Title"] as? String, let type1 = data["Type"] as? String{
let newTask = Tasks(title: titleS, desc: descS, date: dateS, type: type1, docId: doc.documentID)
self.task.append(newTask)
DispatchQueue.main.async {
self.taskTableView.reloadData()
let indexPath = IndexPath(row: self.task.count - 1, section: 0)
self.taskTableView.scrollToRow(at: indexPath, at: .top, animated: false)
}
}
}
}
}
}
The problem is that when I go to add another task I perform a segue to another view controller. From there I add a document but I need to perform another segue to go back because I am using a hamburger menu.
I did try using getDocuments(source: cache), which did reduce writes when the user did not add a task. But when they did add a task it reloads all the documents, adding tons of reads. The goal of using a snapshotListner is to reduce reads, however, I'm not sure if it will reread data when I perform a segue to the screen again. Thank-You!
The snapshot listener doesn't care at all about anything going on the UI, such as segues. The snapshot listener only fires when you first add it and when there are updates in the remote data. You can finetune the snapshot listener to omit or include cached or presumed data (using the snapshot's metadata property). But this is the nature of realtime data, it may get updated a lot as the user does things. The only workaround is coming up with the most efficient data architecture possible (on the server side) to reduce your read cost.
As an aside, your code is very dangerously written. You should first check if there is a valid userId before attaching a listener to a userId that even your code suggests could be an empty string. You should never force unwrap snapshots like you are doing because it will crash the entire app when there is even a slight network error.
I've never really gotten the nuances of async operations so time and again, I get stymied. And I just can't figure it out.
I'm trying to do some very simple web scraping.
My local volleyball association has a page (verbose HTML, not responsive, not mobile-friendly, yaddah, yaddah, yaddah) which shows the refs assigned to each game of the season. I'm trying to write a silly little app which will scrape that page (no API, no direct access to db, etc.) and display the data in a grouped table. The first group will show today's matches (time, home team, away team). The second group will show tomorrow's matches. Third group shows the entire season's matches.
Using code I found elsewhere, my viewDidLoad loads the page, scrapes the data and parses it into an array. Once I've parsed the data, I have three arrays: today, tomorrow, and matches, all are [Match].
override func viewDidLoad() {
super.viewDidLoad()
let url = URL(string: urlString)!
let request = NSMutableURLRequest(url: url)
let task = URLSession.shared.dataTask(with: request as URLRequest) {
data, response, error in
if let error = error {
print (error)
} else {
if let unwrappedData = data {
// scrape, scrape, parse, parse
matchRow = ...
self.matches.append(matchRow)
if matchRow.date == todaysDate {
self.today.append(matchRow)
} else if matchRow.date == tomorrowsDate {
self.tomorrow.append(matchRow)
}
}
}
}
task.resume()
}
As I'm sure is no surprise to anyone who understands async operations, my table is empty. I've checked and I see the the data is there and properly parsed, etc. But I can't for the life of me figure out how get the data in my table. The way I have it now, the data is not ready when numberOfSections or numberOfRowsInSection is called.
I've found the Ray Wenderlich tutorial on URLSession and I also have a Udemy course (Rob Percival) that builds an app to get the weather using web scraping, but in both those instances, the app starts and waits for user input before going out to the web to get the data. I want my app to get the data immediately upon launch, without user interaction. But I just can't figure out what changes I need to make so that those examples work with my program.
Help, please.
You can simply reload the tableviews once the data arrays are getting populated from the URLSession completion block. Have you tried that. Sample snippet may be like the one follows.
let task = URLSession.shared.dataTask(with: request as URLRequest) {
data, response, error in
if let error = error {
print (error)
} else {
if let unwrappedData = data {
// scrape, scrape, parse, parse
matchRow = ...
self.matches.append(matchRow)
if matchRow.date == todaysDate {
self.today.append(matchRow)
} else if matchRow.date == tomorrowsDate {
self.tomorrow.append(matchRow)
}
}
DispatchQueue.main.async { [weak self] in
self?.todayTableView.reloadData()
self?.tomorrowTableView.reloadData()
}
}
}
I want my app to periodically fetch new records and stores them in Core Data. I have a label on my UI that should display the number of elements for a particular record and I want that number to be updated as more records are added into the database. As an exercise, I want to use Combine to accomplish it.
I'm able to display the number of elements in the database when the app launches, but the number doesn't get updated when new data enters into the database (I verified that new data was being added by implementing a button that would manual refresh the UI).
Here's the code that displays the correct number of elements on launch but doesn't update when new records are added:
let replayRecordFetchRequest: NSFetchRequest<ReplayRecord> = ReplayRecord.fetchRequest()
_ = try? persistentContainer.viewContext.fetch(replayRecordFetchRequest).publisher.count().map { String(format: Constants.Strings.playsText, $0) }.assign(to: \.text, on: self.playsLabel)
Here's a code snippet from the WWDC 2019 Session 230 talk that I adapted but this doesn't work at all (the subscriber is never fired):
let replayRecordFetchRequest: NSFetchRequest<ReplayRecord> = ReplayRecord.fetchRequest()
if let replayRecords = try? replayRecordFetchRequest.execute() {
_ = replayRecords.publisher.count().map { String(format: Constants.Strings.playsText, $0) }.assign(to: \.text, on: self.playsLabel)
}
So, I didn't know this until now, but not all publishers are infinitely alive.
And the problem was that the NSFetchRequest.publisher is not a long-living publisher. It simply provides a way to iterate through the sequence of elements in the fetch request. As a result, the subscriber will cancel after the elements are iterated. In my case, I was counting the elements published until cancellation then assigning that value onto the UI.
Instead, I should be subscribing to changes to the managed object context and assigning that pipeline to my UI. Here's some example code:
extension NotificationCenter.Publisher {
func context<T>(fetchRequest: NSFetchRequest<T>) -> Publishers.CompactMap<NotificationCenter.Publisher, [T]> {
return compactMap { notification -> [T]? in
let context = notification.object as! NSManagedObjectContext
var results: [T]?
context.performAndWait {
results = try? context.fetch(fetchRequest)
}
return results
}
}
}
let playFetchRequest: NSFetchRequest<ReplayRecord> = ReplayRecord.fetchRequest()
let replayVideoFetchRequest: NSFetchRequest<ReplayVideo> = ReplayVideo.fetchRequest()
let playsPublisher = contextDidSavePublisher.context(fetchRequest: playFetchRequest).map(\.count)
let replayVideoPublisher = contextDidSavePublisher.context(fetchRequest: replayVideoFetchRequest).map(\.count)
playsSubscription = playsPublisher.zip(replayVideoPublisher).map {
String(format: Constants.Strings.playsText, $0, $1)
}.receive(on: RunLoop.main).assign(to: \.text, on: self.playsLabel)
I am currently developing an iOS app with a google cloud firestore as a backend and I am using a few listeners to find out if data is updated and then pushing it to my device accordingly. I wrote this function that listens for a value if true or not and according to so will update an animation in my app. The trouble is I don't know if I wrote it properly and don't want to incur unnecessary reads from my database if I don't have to.
func dingAnimation() {
let identifier = tempDic![kBOUNDIDENTIFIER] as! String
if identifier != "" {
dingListener = reference(.attention).document(identifier).addSnapshotListener({ (snapshot, error) in
if error != nil {
SVProgressHUD.showError(withStatus: error!.localizedDescription)
return
}
guard let snapshot = snapshot else { return }
let data = snapshot.data() as! NSDictionary
for dat in data {
let currentId = FUser.currentId() as! String
let string = dat.key as! String
if string == currentId {
} else {
let value = dat.value as! Bool
self.shouldAnimate = value
self.animateImage()
}
}
})
}
}
This might help you.
From Firestore DOCS - Understand Cloud Firestore billing
https://firebase.google.com/docs/firestore/pricing
Listening to query results
Cloud Firestore allows you to listen to the results of a query and get realtime updates when the query results change.
When you listen to the results of a query, you are charged for a read each time a document in the result set is added or updated. You are also charged for a read when a document is removed from the result set because the document has changed. (In contrast, when a document is deleted, you are not charged for a read.)
Also, if the listener is disconnected for more than 30 minutes (for example, if the user goes offline), you will be charged for reads as if you had issued a brand-new query.