Core-data insert multiple objects - swift

i am struggling again to solve a core Data task which keeps failing randomly on me
The following code is building my initial database, which is necessary for the app to work properly
(...)
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext
(...)
for person in groupOfPeople { //singlePerson (codable Struct) <> SinglePerson (NSManagedObject)
saveSinglePerson(sPerson: person, context: context)
counter+=1
}
logComment = logComment + "..success!(\(counter))"
func saveSinglePerson(sPerson: singlePerson, context: NSManagedObjectContext) {
let newSinglePerson = SinglePerson(context: context)
newSinglePerson.id = sPerson.ID
newSinglePerson.name = sPerson.name
newSinglePerson.age = sPerson.age
(...)
context.performAndWait {
do {
try context.save()
}catch let error {
print(error)
Logging.insertError(message: "IMPORT ERROR: \(error.localizedDescription)", location: "buildDatabase20")
}
}
}
Now here's my problem:
At first i didn't even notice there is a problem, because everything is working fine and all objects get saved as they are supposed to be,
but indeed there is one, because: i am randomly getting an error like so:
error= Error Domain=NSCocoaErrorDomain Code=134030 "An error occurred while saving." UserInfo={NSAffectedObjectsErrorKey=(
"<AppName.SinglePerson: 0x60000104e7b0> (entity: SinglePerson; id: 0x600003337ce0 <x-coredata:///SinglePerson/t3081F988-C5D1-4532-AD81-46F3B4B10215139>; data: {\n id = 138;\n name = testname;\n age = \"25\";\n })"
and i get this error multiple times (20x-150x), with just this one single ID, in this example 138, but it is a different id each time...
i investigate this situation for days now, and i just can't wrap my head around this..
what i found out by now is:
the method should insert 150 rows, and if this error occurs it is not just a count of 149, it's like 87, or 127, or whatever
seems like an object gets stuck in the context, and every execution after the first error fails and is throwing the (same) error..
i tried to fetch those new written data directly after i inserted them, and i always get the same (wrong) count of 150..
i know that this count is not legit because if i take a look at the sqllite file, is see just 87, or 127 or whatever row count..
i do this fetch again with the same context, this is why i think that the issue is within my NSManaged context..
why is this happening on me? and why does this happen sometimes but not all the time?
How do i solve it?

i've found a solution to fix this issue, even though i now know that i will have rework all Core Data interactions from the ground up, to make it real stable and reliable..
this is my first swift project, so along the way things got pretty messy tbh :)
fix: the fact that i save all created objects at once now instead of saving each item on its own, did work and solved the issue for me at this very moment :)
maybe this is helpful for somebody else too ;)
(...)
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext
(...)
var personArray = [SinglePerson]()
for person in groupOfPeople {
let newSinglePerson = SinglePerson(context: context)
newSinglePerson.id = sPerson.ID
newSinglePerson.name = sPerson.name
newSinglePerson.age = sPerson.age
(...)
personArray.append(newSinglePerson)
}
context.performAndWait {
do {
try context.save()
} catch let error {
print(error.localizedDescription)
}
}

Related

Debugging async CoreData calls

i am adding await and concurrency methods to my existing project and ran into some strange behaviours
which i am unable to debug because they don't happen each and every time, just randomly sometimes, somewhere down the road (inside buildDataStructureNew)
func buildDataStructureNew() async -> String {
var logComment:String = ""
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext
// Step 1 get JSON data
let jsonData = try? await getRemoteFoo()
guard jsonData != nil else {...}
let feedback2 = await step2DeleteAllEntities(context: context) // Step 2
{...}
let feedback3 = await step3saveJSONtoFoo(context: context, remoteData: remoteData) // Step3
{...}
let sourceData = await step41getAllSavedFoo(context: context) // Step 4.1
{...}
let feedback42 = await step42deleteAllSingleFoos(context: context) //Step 4.2
{...}
let feedback43 = await step43splitRemoteDataIntoSingleDataFoos(context: context, sourceData: sourceData) // Step 4.3
{...}
let feedback5 = await step5createDataPacks() // Step 5
return logComment
}
as you see i executed each step with the same context, expecting it to work properly
but i started to receive fatal errors from step 4.3, which i could not explain myself..
CoreData: error: Serious application error. Exception was caught during Core Data change processing.
This is usually a bug within an observer of NSManagedObjectContextObjectsDidChangeNotification.
-[__NSCFSet addObject:]: attempt to insert nil with userInfo (null)
so i tried using a different context just for this single step
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.newBackgroundContext() // just to test it
Step 3 as an example:
func step3saveJSONtoFoo(context: NSManagedObjectContext, remoteData:[remoteData]) async -> String {
var feedback:String = ""
var count:Int = 0
for i in 0...remoteFoo.count-1 {
let newFoo = remoteFoos(context: context)
newFoo.a = Int16(remoteFoo[i].a)
newFoo.b = remoteFoo[i].b
newFoo.c = remoteFoo[i].c
newFoo.Column1 = remoteFoo[i].Column1
newFoo.Column2 = remoteFoo[i].Column2
newFoo.Column3 = remoteFoo[i].Column3
newFoo.Column4 = remoteFoo[i].Column4
newFoo.Column15 = remoteFoo[i].Column4
count = i
do {
try context.save()
// print("DEBUG - - ✅ save JSON to RemoteFoo successfull")
} catch {
feedback = "\n-- !!! -- saving JSON Data to CoreData.Foos failed(Step 3), error:\(error)"
Logging.insertError(message: "error saving JSON to CoreData.Foos", location: "buildDataStructureNew")
}
}
// print("DEBUG - - ✅ Step3 \(count+1) records saved to RemoteFoo")
feedback = feedback + "\n-- ✅ \(count+1) records saved to RemoteFoo"
return feedback
}
and this solved the issue, but i then got the same error from step 5, so i added the background context to this step as well
and on first sight this solved it again
i thought this is it, but a few minutes later the method crashed on me again, but now on step 3, with again the same exact error msg..
as i understood it it has something to do with the context i use, but i don't really get what's really the issue here.
i did not have any issues with this method before, this started to happen on me when i rewrote that method as async..
right now the method is working fine without any issues, or at least i can't reproduce it at the moment, but it might come back soon..
i guess i am missing some understanding here, i hope you guys can help me out
The problems you're seeing are because Core Data has its own ideas about concurrency that don't directly map to any other concurrency technique you might use. Bugs that crop up inconsistently, sometimes but not always, are a classic sign of a concurrency problem.
For concurrent Core Data use, you must use either the context's perform { ... } or performAndWait { ... } with all Core Data access. Async/await is not a substitute, nor is DispatchQueue or anything else you would use for concurrency in other places in an iOS app. This includes everything that touches Core Data in any way-- fetching, saving, merging, accessing properties on a managed object, etc. You're not doing that, which is why you're having these problems.
The only exception to this is if your code is running on the main queue and you have a main-queue context.
While you're working on this you should turn on Core Data concurrency debugging by using -com.apple.CoreData.ConcurrencyDebug 1 as an argument to your app. That's described in various blog posts including this one.
with the help of Tom's explanations i was able to fix this issue
to make sure everything runs in the correct order,
it is important to do those core Data calls within a .perform or performAndWait call
and as step 1 is the only real async action (network API access), this is the only step marked as "async/await",
all other steps are hopefully now correctly queued via CoreData's own queue..
here's my final working code
func buildDataStructureNew() async -> String {
var logComment:String = ""
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext
// Step 1 get JSON data
let jsonData = try? await getRemoteFoo()
guard jsonData != nil else {...}
let feedback2 = step2DeleteAllEntities(context: context) // Step 2
{...}
let feedback3 = step3saveJSONtoFoo(context: context, remoteData: remoteData) // Step3
{...}
let sourceData = step41getAllSavedFoo(context: context) // Step 4.1
{...}
let feedback42 = step42deleteAllSingleFoos(context: context) //Step 4.2
{...}
let feedback43 = step43splitRemoteDataIntoSingleDataFoos(context: context, sourceData: sourceData) // Step 4.3
{...}
let feedback5 = step5createDataPacks() // Step 5
return logComment
}
and again step 3 as an example
func step3saveJSONtoFoo(context: NSManagedObjectContext, remoteData:[remoteData]) -> String {
var feedback:String = ""
var count:Int = 0
for i in 0...remoteFoo.count-1 {
let newFoo = remoteFoos(context: context)
newFoo.a = Int16(remoteFoo[i].a)
newFoo.b = remoteFoo[i].b
{...}
count = i
context.performAndWait {
do { try context.save() }
catch let error {
feedback = "\n-- !!! -- saving JSON Data to CoreData.Foos failed(Step 3), error:\(error)"
Logging.insertError(message: "error saving JSON to CoreData.Foos", location: "buildDataStructureNew")
}
}
}
feedback = feedback + "\n-- ✅ \(count+1) records saved to RemoteFoo"
return feedback
}
one last note: Xcode initially told me that "performAndWait" is only available in iOS 15 and above,
that is true because apple released a new version of this function which is marked as "rethrow",
but there is also the old version of this function available and this version of "perform" is available from iOS 5 and above ;)
the trick is to not forget to use a own catch block after the do statement ;)

SwiftUI + CoreData: Insert more than 1000 entities with relationships

I have a SwiftUI project in which I'm using CoreData to save data fetched from an API into the device. I was trying to insert the entities in batches which worked fine until I realized that the relationships when inserting in batched are "untouched":
An entity Job, which has a one-to-many relationship with the entity Tag.
A one-to-one relationship with Category.
A one-to-one relationship with Type.
What I'm doing now is inserting the entities manually in a background task:
container.performBackgroundTask { context in
for job in jobs {
let jobToInsert = Job(context: context)
let type = JobType(context: context)
let category = Category(context: context)
jobToInsert.id = Int32(job.id)
....
do {
print("Inserting jobs")
try context.save()
} catch {
// log any errors
}
}
Is there any way to improve the performance by perhaps doing this in a way that I don't know? Because for the user, when they start the app, inserting the jobs one by one isn't a very nice experience because first, it takes a long time (more than 2 minutes) and second because my UI isn't automatically updated as I'm inserting the entities.
Thanks a lot in advance!
EDIT: I also see the memory increasing and after taking the screenshot and before I stopped the process, I saw the memory in 1.09 GB
EDIT 2: This is the code I used when trying to insert the jobs in batch:
private func newBatchInsertRequest(with jobs: [JobCodable]) -> NSBatchInsertRequest {
var index = 0
let total = jobs.count
let batchInsert = NSBatchInsertRequest(
entity: Job.entity()) { (managedObject: NSManagedObject) -> Bool in
guard index < total else { return true }
if let job = managedObject as? Job {
let type = JobType(context: self.container.viewContext)
let category = Category(context: self.container.viewContext)
let data: JobCodable = jobs[index]
job.id = Int32(data.id)
...
return batchInsert
Unfortunately, the relationship can't be built due probably to the context? since I'm getting Thread 13: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) in the line let category = Category(context: self.container.viewContext)
Are you calling save() on the context for every job that you insert? If so, I would start there — if you want to load a bunch of jobs all at once, you can insert all your Job instances, set up the relationships, and then only save() once at the very end (outside the loop).
This will not only save you time, but also make it appear as if all the objects show up at the same time to your UI. If that's not what you want, you can experiment with saving in batches within the loop — only call save() every thousand Job objects, for example.

Ambiguous reference to member 'save(_:completionHandler:)' with CloudKit save attempt

I'm trying to save back to CloudKit after updating a reference list and getting the error on the first line of this code block.
Error: Ambiguous reference to member 'save(_:completionHandler:)'
CKContainer.default().publicCloudDatabase.save(establishment) { [unowned self] record, error in
DispatchQueue.main.async {
if let error = error {
print("error handling to come")
} else {
print("success")
}
}
}
This sits within a function where the user going to follow a given location (Establishment). We're taking the existing establishment, and its record of followers, checking to see if the selected user is in it, and appending them to the list if not (or creating it if the list of followers is null).
Edit, in case helpful
//Both of these are passed in from the prior view controller
var establishment: Establishment?
var loggedInUserID: String?
#objc func addTapped() {
// in here, we want to take the logged in user's ID and append it to the list of people that want to follow this establishment
// which is a CK Record Reference
let userID = CKRecord.ID(recordName: loggedInUserID!)
var establishmentTemp: Establishment? = establishment
var followers: [CKRecord.Reference]? = establishmentTemp?.followers
let reference = CKRecord.Reference(recordID: userID, action: CKRecord_Reference_Action.none)
if followers != nil {
if !followers!.contains(reference) {
establishmentTemp?.followers?.append(reference)
}
} else {
followers = [reference]
establishmentTemp?.followers = followers
establishment = establishmentTemp
}
[this is where the CKContainer.default.....save block pasted at the top of the question comes in]
I've looked through the various posts on 'ambiguous reference' but haven't been able to figure out the source of my issue. tried to explicitly set the types for establisthmentTemp and followers in case that was the issue (based on the solutions to other related posts) but no luck.
Afraid I'm out of ideas as a relatively inexperienced newbie!
Help appreciated.
Documenting the solution that I figured out:
Combination of two issues:
I was trying to save an updated version of a CK Record instead of updating
I was not passing a CK Record to the save() call - but a custom object
(I believe point two was the cause of the 'ambiguous reference to member'
error)
I solved it by replacing the save attempt (first block of code in the question) with:
//first get the record ID for the current establishment that is to be updated
let establishmentRecordID = establishment?.id
//then fetch the item from CK
CKContainer.default().publicCloudDatabase.fetch(withRecordID: establishmentRecordID!) { updatedRecord, error in
if let error = error {
print("error handling to come")
} else {
//then update the 'people' array with the revised one
updatedRecord!.setObject(followers as __CKRecordObjCValue?, forKey: "people")
//then save it
CKContainer.default().publicCloudDatabase.save(updatedRecord!) { savedRecord, error in
}
}
}

Inserting child records is slow in coredata

I have close to 7K items stored in a relation called Verse. I have another relation called Translation that needs to load 7K related items with a single call from a JSON file.
Here is my code:
let container = getContainer()
container.performBackgroundTask() { (context) in
autoreleasepool {
for row in translations{
let t = Translation(context: context)
t.text = (row["text"]! as? String)!
t.lang = (row["lang"]! as? String)!
t.contentType = "Verse"
t.verse = VerseDao.findById(row["verse_id"] as! Int16, context: context)
// this needs to make a call to the database to retrieve the approparite Verse instance.
}
}
do {
try context.save()
} catch {
fatalError("Failure to save context: \(error)")
}
context.reset()
}
Code for the findById method.
static func findById(_ id: Int16, context: NSManagedObjectContext) -> Verse{
let fetchRequest: NSFetchRequest<Verse>
fetchRequest = Verse.fetchRequest()
fetchRequest.predicate = NSPredicate(format: "verseId == %#", id)
fetchRequest.includesPropertyValues = false
fetchRequest.fetchLimit = 1
do {
let results =
try context.fetch(fetchRequest)
return results[0]
} catch let error as NSError {
print("Could not fetch \(error), \(error.userInfo)")
return Verse()
}
}
This works fine until I add the VerseDao.findById, which makes the whole process really slow because it has to make a request for each object to the Coredata database.
I did everything I could by limiting the number of fetched properties and using NSFetchedResultsController for data fetching but no luck.
I wonder if there's any way to insert child records in a more efficient way? Thanks.
Assuming your store type is persistent store type is sqlite (NSSQLiteStoreType):
The first thing you should check is whether you have an Core Data fetch index on the Verse objects verseId property. See this stack overflow answer for some introductory links on fetch indexes.
Without that, the fetch in your VerseDao.findById function may be scanning the whole database table every time.
To see if your index is working properly you may inspect the SQL queries generated by adding -com.apple.CoreData.SQLDebug 1 to the launch arguments in your Xcode scheme.
Other improvements:
Use NSManagedObjectContext.fetch or NSFetchRequest.execute (equivalent) instead of NSFetchedResultsController. The NSFetchedResultsController is typically used to bind results to a UI. In this case using it just adds overhead.
Don't set fetchRequest.propertiesToFetch, instead set fetchRequest.includesPropertyValues = false. This will avoid fetching the Verse object property values which you don't need to establish the relation to the Translation object.
Don't specify a sortDescriptor on the fetch request, this just complicates the query

Swift 3 Completion Handler on Google Places Lookup. Due to delay how do I know when Im "done"?

Sorry, newbie here and Ive read extensively about completion handlers, dispatch queues and groups but I just can't get my head around this.
My app loads an array of Google Place IDs and then wants to query Google to get full details on each place. The problem is, due to async processing the Google Lookup Place returns immediately and the callback happens much further down the line so whats the "proper way" to know when the last bit of data has come in for my inquiries because the function ends almost immedately ?
Code is attached. Thanks in advance.
func testFunc() {
let googlePlaceIDs = ["ChIJ5fTXDP8MK4cRjIKzek6L6NM", "ChIJ9Wd6mGYGK4cRiWd0_bkohHg", "ChIJaeXT08ASK4cRkCGpGgzYpu8", "ChIJkRkS4BapK4cRXCT8-SJxNDI", "ChIJ3wDV_2zX5IkRtd0hg2i1LhE", "ChIJb4wUsI5w44kRnERe7ywQaJA"]
let placesClient = GMSPlacesClient()
for placeID in googlePlaceIDs {
placesClient.lookUpPlaceID(placeID, callback: { (place, error) in
if let error = error {
print("lookup place id query error: \(error.localizedDescription)")
return
}
guard let place = place else {
print("No place details for \(placeID)")
return
}
print("Place Name = \(place.name)")
})
}
print("Done")
}