I need to export data from a dictionary of 12 thousand items to Cloudkit. I tried to use the convenience API but I keep hitting the rate limit while trying to save to the public database. I then tried the Operation API and I got a similar error. My question is: how to save a very large amount of data to cloudkit without hitting the limits?
According to the docs for the CKErrorLimitExceeded error code, you should
Try refactoring your request into multiple smaller batches.
So if your CKModifyRecordsOperation operation results in the CKErrorLimitExceeded error, you can just create two CKModifyRecordsOperation objects, each with half the data from the failed operation. If you do that recursively (so any of the split operations could also fail with the limit exceeded error, splitting again in two) then you should eventually get a number of CKModifyRecordsOperation objects that have a small enough number of records to avoid the error.
If you have your own server, you could try the CloudKit Web Service API.
In iOS 10, the maximum permitted record count per operation is 400.
/// The system allowed maximum record modifications count.
///
/// If excute a CKModifyRecordsOperation with more than 400 record modifications, system will return a CKErrorLimitExceeded error.
private let maximumRecordModificationsLimit = 400
private func modifyRecords(recordsToSave: [CKRecord], recordIDsToDelete: [CKRecordID], previousRetryAfterSeconds: TimeInterval = 0, completion: ((Bool) -> Void)? = nil) {
guard !recordsToSave.isEmpty || !recordIDsToDelete.isEmpty else {
completion?(true)
return
}
func handleLimitExceeded() {
let recordsToSaveFirstSplit = recordsToSave[0 ..< recordsToSave.count / 2]
let recordsToSaveSecondSplit = recordsToSave[recordsToSave.count / 2 ..< recordsToSave.count]
let recordIDsToDeleteFirstSplit = recordIDsToDelete[0 ..< recordIDsToDelete.count / 2]
let recordIDsToDeleteSecondSplit = recordIDsToDelete[recordIDsToDelete.count / 2 ..< recordIDsToDelete.count]
self.modifyRecords(recordsToSave: Array(recordsToSaveFirstSplit), recordIDsToDelete: Array(recordIDsToDeleteFirstSplit))
self.modifyRecords(recordsToSave: Array(recordsToSaveSecondSplit), recordIDsToDelete: Array(recordIDsToDeleteSecondSplit), completion: completion)
}
if recordsToSave.count + recordIDsToDelete.count > maximumRecordModificationsLimit {
handleLimitExceeded()
return
}
// run CKModifyRecordsOperation at here
}
Related
I'm fetching some data from an endpoint, every time I requested for 10 items. These items contains a URL that should be scraping, which slow down the process a bit.
Due to some bug probably, the received data from scraping is not the same number of the original data, which something i will deal with later. So I don't know when all data is ready.
Now, I want to make sure when I get all the data, then refresh the collection view. the only way that I can think of it, is using the timestamp of data and if the timestamp is not updated for about a sec, means, I have all the data that I need than I can refresh collection view. Not the best way of course
I added a timestamp variable
private lazy var fetchedTimestamp = Date().nanosecondsSince1970
then, whenever the data is recived I updated this timestamp with the the current time
Than, I use DispatchQueue.main.asyncAfter to check the timestamp on sec later and if the the timestamp one sec later is 1 sec after the last fetchedTimestampupdate, means the data hasn't been updated for about a sec, so I know it's ready to reload
KingfisherManager.shared.retrieveImage(with: imageURL) { result in
if let image = try? result.get().image {
scraper.setBaseImage(with: image)
DispatchQueue.main.async { [weak self] in
guard let this = self else { return }
this.feedsCollectionView.feeds.append(scraper)
this.fetchedTimestamp = Date().nanosecondsSince1970
DispatchQueue.main.asyncAfter(deadline: .now() + 1, execute: {
let currentTime = Date().nanosecondsSince1970
if currentTime >= this.fetchedTimestamp + Int64(1e+9) {
this.feedsCollectionView.reload()
}
})
}
}
}
Is there any way to handle it better? Thank you so much
I have a limit of 40 URL Session calls to my API per minute.
I have timed the number of calls in any 60s and when 40 calls have been reached I introduced sleep(x). Where x is 60 - seconds remaining before new minute start. This works fine and the calls don’t go over 40 in any given minute. However the limit is still exceeded as there might be more calls towards the end of the minute and more at the beginning of the next 60s count. Resulting in an API error.
I could add a:
usleep(x)
Where x would be 60/40 in milliseconds. However as some large data returns take much longer than simple queries that are instant. This would increase the overall download time significantly.
Is there a way to track the actual rate to see by how much to slow the function down?
Might not be the neatest approach, but it works perfectly. Simply storing the time of each call and comparing it to see if new calls can be made and if not, the delay required.
Using previously suggested approach of delay before each API call of 60/40 = 1.5s (Minute / CallsPerMinute), as each call takes a different time to produce response, total time taken to make 500 calls was 15min 22s. Using the below approach time taken: 11min 52s as no unnecessary delay has been introduced.
Call before each API Request:
API.calls.addCall()
Call in function before executing new API task:
let limit = API.calls.isOverLimit()
if limit.isOver {
sleep(limit.waitTime)
}
Background Support Code:
var globalApiCalls: [Date] = []
public class API {
let limitePerMinute = 40 // Set API limit per minute
let margin = 2 // Margin in case you issue more than one request at a time
static let calls = API()
func addCall() {
globalApiCalls.append(Date())
}
func isOverLimit() -> (isOver: Bool, waitTime: UInt32)
{
let callInLast60s = globalApiCalls.filter({ $0 > date60sAgo() })
if callInLast60s.count > limitePerMinute - margin {
if let firstCallInSequence = callInLast60s.sorted(by: { $0 > $1 }).dropLast(2).last {
let seconds = Date().timeIntervalSince1970 - firstCallInSequence.timeIntervalSince1970
if seconds < 60 { return (true, UInt32(60 + margin) - UInt32(seconds.rounded(.up))) }
}
}
return (false, 0)
}
private func date60sAgo() -> Date
{
var dayComponent = DateComponents(); dayComponent.second = -60
return Calendar.current.date(byAdding: dayComponent, to: Date())!
}
}
Instead of using sleep have a counter. You can do this with a Semaphore (it is a counter for threads, on x amount of threads allowed at a time).
So if you only allow 40 threads at a time you will never have more. New threads will be blocked. This is much more efficient than calling sleep because it will interactively account for long calls and short calls.
The trick here is that you would call a function like this every sixty second. That would make a new semaphore every minute that would only allow 40 calls. Each semaphore would not affect one another but only it's own threads.
func uploadImages() {
let uploadQueue = DispatchQueue.global(qos: .userInitiated)
let uploadGroup = DispatchGroup()
let uploadSemaphore = DispatchSemaphore(value: 40)
uploadQueue.async(group: uploadGroup) { [weak self] in
guard let self = self else { return }
for (_, image) in images.enumerated() {
uploadGroup.enter()
uploadSemaphore.wait()
self.callAPIUploadImage(image: image) { (success, error) in
uploadGroup.leave()
uploadSemaphore.signal()
}
}
}
uploadGroup.notify(queue: .main) {
// completion
}
}
I have an OperationQueue with multiple custom Operations which all append to the same array on completion (each operation downloads a file from user's iCloud and when it's done it appends the file to the array)
This, sometimes, causes the app to crash, because several operations try to edit the array at the same time.
How can I prevent this and only edit the array 1 operation at a time but running all operations simultaneously?
I must use OperationQueue because I need the operations to be cancelable.
func convertAssetsToMedias(assets: [PHAsset],
completion: #escaping (_ medias: [Media]) ->()) {
operationQueue = OperationQueue()
var medias: [Media] = []
operationQueue?.progress.totalUnitCount = Int64(assets.count)
for asset in assets {
// For each asset we start a new operation
let convertionOperation = ConvertPHAssetToMediaOperation(asset)
convertionOperation.qualityOfService = .userInteractive
convertionOperation.completionBlock = { [unowned convertionOperation] in
let media = convertionOperation.media
medias.append(media) // CRASH HERE (sometimes)
self.operationQueue?.progress.completedUnitCount += 1
if let progress = self.operationQueue?.progress.fractionCompleted {
self.delegate?.onICloudProgressUpdate(progress: progress)
}
convertionOperation.completionBlock = nil
}
operationQueue?.addOperation(convertionOperation)
}
operationQueue?.addBarrierBlock {
completion(medias)
}
}
Edit 1:
The Media file itself is nothing big, just a bunch of metadata and a url to an actual file at documents directory. There are usually about 24 medias max at 1 run. The memory is barely increasing during those operations. The crash never occured due to a lack of memory.
The operation ConvertPHAssetToMediaOperation is a subclass of AsyncOperation where isAsynchronous propery is set to true.
That's how I construct the Media object in the end of each operation:
self.media = Media(type: mediaType, url: resultURL, creationDate: date)
self.finish()
Edit 2: The crash is always the same:
I want my app to periodically fetch new records and stores them in Core Data. I have a label on my UI that should display the number of elements for a particular record and I want that number to be updated as more records are added into the database. As an exercise, I want to use Combine to accomplish it.
I'm able to display the number of elements in the database when the app launches, but the number doesn't get updated when new data enters into the database (I verified that new data was being added by implementing a button that would manual refresh the UI).
Here's the code that displays the correct number of elements on launch but doesn't update when new records are added:
let replayRecordFetchRequest: NSFetchRequest<ReplayRecord> = ReplayRecord.fetchRequest()
_ = try? persistentContainer.viewContext.fetch(replayRecordFetchRequest).publisher.count().map { String(format: Constants.Strings.playsText, $0) }.assign(to: \.text, on: self.playsLabel)
Here's a code snippet from the WWDC 2019 Session 230 talk that I adapted but this doesn't work at all (the subscriber is never fired):
let replayRecordFetchRequest: NSFetchRequest<ReplayRecord> = ReplayRecord.fetchRequest()
if let replayRecords = try? replayRecordFetchRequest.execute() {
_ = replayRecords.publisher.count().map { String(format: Constants.Strings.playsText, $0) }.assign(to: \.text, on: self.playsLabel)
}
So, I didn't know this until now, but not all publishers are infinitely alive.
And the problem was that the NSFetchRequest.publisher is not a long-living publisher. It simply provides a way to iterate through the sequence of elements in the fetch request. As a result, the subscriber will cancel after the elements are iterated. In my case, I was counting the elements published until cancellation then assigning that value onto the UI.
Instead, I should be subscribing to changes to the managed object context and assigning that pipeline to my UI. Here's some example code:
extension NotificationCenter.Publisher {
func context<T>(fetchRequest: NSFetchRequest<T>) -> Publishers.CompactMap<NotificationCenter.Publisher, [T]> {
return compactMap { notification -> [T]? in
let context = notification.object as! NSManagedObjectContext
var results: [T]?
context.performAndWait {
results = try? context.fetch(fetchRequest)
}
return results
}
}
}
let playFetchRequest: NSFetchRequest<ReplayRecord> = ReplayRecord.fetchRequest()
let replayVideoFetchRequest: NSFetchRequest<ReplayVideo> = ReplayVideo.fetchRequest()
let playsPublisher = contextDidSavePublisher.context(fetchRequest: playFetchRequest).map(\.count)
let replayVideoPublisher = contextDidSavePublisher.context(fetchRequest: replayVideoFetchRequest).map(\.count)
playsSubscription = playsPublisher.zip(replayVideoPublisher).map {
String(format: Constants.Strings.playsText, $0, $1)
}.receive(on: RunLoop.main).assign(to: \.text, on: self.playsLabel)
I have some kind of CloudKit indexing issue. When I save records to iCloud using CKModifyRecordsOperation, modifyRecordsCompletionBlock returns no errors. When I query those records using NSPredicate(value: true) or Dashboard, most of the time it misses one or two records.
So say I upload 5 records (no errors), wait some time (~15 secs) to make sure that indexes are updated, and then query them (through dashboard or app's CKQueryOperation). Most of the time it will show 4 records out 5. Again, no errors. Records are in privateDB in customZone.
Here is what's strange: I'm always able to get records that query didn't return by manually typing recordNames in Dashboard (development) under 'Fetch' menu. So it stores them, just doesn't query. When I delete indexes in a dashboard and reassign them, dashboard query will start to return all the results (with previously omitted records too), but after a few more uploads, some will start to be missing from query again.
Here is my CKModifyRecordsOperation:
let operation = CKModifyRecordsOperation(recordsToSave: records, recordIDsToDelete: [])
operation.modifyRecordsCompletionBlock =
{ [weak self] savedRecords, deletedRecordIDs, error in
guard error == nil else { // no errors here ... }
...
//for each item uploaded to iCloud, mark as synced
if let savedRecords = savedRecords{ // all attempted to save records are here
PendingCloudOperations.shared.markAsUploaded(
savedRecords.map{ $0.recordID.recordName })
}
completion(...)
}
operation.savePolicy = .changedKeys // tried .allKeys too
operation.qualityOfService = .userInitiated
self.privateDB.add(operation)
I experimented with record fields (originally date, asset, and reference) trying to see if any of the fields make a problem. But even if I remove all field's (creating a record with no extra fields, just system meta), problem persists. I didn't include CKQueryOperation code, because Dashboard acts same way as the app.
Any ideas?
EDIT:
Here are bare-bones of my fetching function:
var receipts:[FS_Receipt] = []
let query = CKQuery(recordType: myRecordType, predicate: NSPredicate(value: true))
let operation = CKQueryOperation(query: query)
//completion block
operation.queryCompletionBlock = { [weak self] cursor, error in
guard error == nil else {
// doesn't have any errors here
}
completion(...)
}
operation.recordFetchedBlock = { record in
// doesn't return all records here most of the time.
}
operation.qualityOfService = .userInitiated // without this, 'no internet' will NOT return error
operation.resultsLimit = 5000
operation.zoneID = customZoneID
self.privateDB.add(operation)
}