SwiftUI + CoreData: Insert more than 1000 entities with relationships - swift

I have a SwiftUI project in which I'm using CoreData to save data fetched from an API into the device. I was trying to insert the entities in batches which worked fine until I realized that the relationships when inserting in batched are "untouched":
An entity Job, which has a one-to-many relationship with the entity Tag.
A one-to-one relationship with Category.
A one-to-one relationship with Type.
What I'm doing now is inserting the entities manually in a background task:
container.performBackgroundTask { context in
for job in jobs {
let jobToInsert = Job(context: context)
let type = JobType(context: context)
let category = Category(context: context)
jobToInsert.id = Int32(job.id)
....
do {
print("Inserting jobs")
try context.save()
} catch {
// log any errors
}
}
Is there any way to improve the performance by perhaps doing this in a way that I don't know? Because for the user, when they start the app, inserting the jobs one by one isn't a very nice experience because first, it takes a long time (more than 2 minutes) and second because my UI isn't automatically updated as I'm inserting the entities.
Thanks a lot in advance!
EDIT: I also see the memory increasing and after taking the screenshot and before I stopped the process, I saw the memory in 1.09 GB
EDIT 2: This is the code I used when trying to insert the jobs in batch:
private func newBatchInsertRequest(with jobs: [JobCodable]) -> NSBatchInsertRequest {
var index = 0
let total = jobs.count
let batchInsert = NSBatchInsertRequest(
entity: Job.entity()) { (managedObject: NSManagedObject) -> Bool in
guard index < total else { return true }
if let job = managedObject as? Job {
let type = JobType(context: self.container.viewContext)
let category = Category(context: self.container.viewContext)
let data: JobCodable = jobs[index]
job.id = Int32(data.id)
...
return batchInsert
Unfortunately, the relationship can't be built due probably to the context? since I'm getting Thread 13: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) in the line let category = Category(context: self.container.viewContext)

Are you calling save() on the context for every job that you insert? If so, I would start there — if you want to load a bunch of jobs all at once, you can insert all your Job instances, set up the relationships, and then only save() once at the very end (outside the loop).
This will not only save you time, but also make it appear as if all the objects show up at the same time to your UI. If that's not what you want, you can experiment with saving in batches within the loop — only call save() every thousand Job objects, for example.

Related

Core-data insert multiple objects

i am struggling again to solve a core Data task which keeps failing randomly on me
The following code is building my initial database, which is necessary for the app to work properly
(...)
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext
(...)
for person in groupOfPeople { //singlePerson (codable Struct) <> SinglePerson (NSManagedObject)
saveSinglePerson(sPerson: person, context: context)
counter+=1
}
logComment = logComment + "..success!(\(counter))"
func saveSinglePerson(sPerson: singlePerson, context: NSManagedObjectContext) {
let newSinglePerson = SinglePerson(context: context)
newSinglePerson.id = sPerson.ID
newSinglePerson.name = sPerson.name
newSinglePerson.age = sPerson.age
(...)
context.performAndWait {
do {
try context.save()
}catch let error {
print(error)
Logging.insertError(message: "IMPORT ERROR: \(error.localizedDescription)", location: "buildDatabase20")
}
}
}
Now here's my problem:
At first i didn't even notice there is a problem, because everything is working fine and all objects get saved as they are supposed to be,
but indeed there is one, because: i am randomly getting an error like so:
error= Error Domain=NSCocoaErrorDomain Code=134030 "An error occurred while saving." UserInfo={NSAffectedObjectsErrorKey=(
"<AppName.SinglePerson: 0x60000104e7b0> (entity: SinglePerson; id: 0x600003337ce0 <x-coredata:///SinglePerson/t3081F988-C5D1-4532-AD81-46F3B4B10215139>; data: {\n id = 138;\n name = testname;\n age = \"25\";\n })"
and i get this error multiple times (20x-150x), with just this one single ID, in this example 138, but it is a different id each time...
i investigate this situation for days now, and i just can't wrap my head around this..
what i found out by now is:
the method should insert 150 rows, and if this error occurs it is not just a count of 149, it's like 87, or 127, or whatever
seems like an object gets stuck in the context, and every execution after the first error fails and is throwing the (same) error..
i tried to fetch those new written data directly after i inserted them, and i always get the same (wrong) count of 150..
i know that this count is not legit because if i take a look at the sqllite file, is see just 87, or 127 or whatever row count..
i do this fetch again with the same context, this is why i think that the issue is within my NSManaged context..
why is this happening on me? and why does this happen sometimes but not all the time?
How do i solve it?
i've found a solution to fix this issue, even though i now know that i will have rework all Core Data interactions from the ground up, to make it real stable and reliable..
this is my first swift project, so along the way things got pretty messy tbh :)
fix: the fact that i save all created objects at once now instead of saving each item on its own, did work and solved the issue for me at this very moment :)
maybe this is helpful for somebody else too ;)
(...)
let context = await (UIApplication.shared.delegate as! AppDelegate).persistentContainer.viewContext
(...)
var personArray = [SinglePerson]()
for person in groupOfPeople {
let newSinglePerson = SinglePerson(context: context)
newSinglePerson.id = sPerson.ID
newSinglePerson.name = sPerson.name
newSinglePerson.age = sPerson.age
(...)
personArray.append(newSinglePerson)
}
context.performAndWait {
do {
try context.save()
} catch let error {
print(error.localizedDescription)
}
}

Inserting child records is slow in coredata

I have close to 7K items stored in a relation called Verse. I have another relation called Translation that needs to load 7K related items with a single call from a JSON file.
Here is my code:
let container = getContainer()
container.performBackgroundTask() { (context) in
autoreleasepool {
for row in translations{
let t = Translation(context: context)
t.text = (row["text"]! as? String)!
t.lang = (row["lang"]! as? String)!
t.contentType = "Verse"
t.verse = VerseDao.findById(row["verse_id"] as! Int16, context: context)
// this needs to make a call to the database to retrieve the approparite Verse instance.
}
}
do {
try context.save()
} catch {
fatalError("Failure to save context: \(error)")
}
context.reset()
}
Code for the findById method.
static func findById(_ id: Int16, context: NSManagedObjectContext) -> Verse{
let fetchRequest: NSFetchRequest<Verse>
fetchRequest = Verse.fetchRequest()
fetchRequest.predicate = NSPredicate(format: "verseId == %#", id)
fetchRequest.includesPropertyValues = false
fetchRequest.fetchLimit = 1
do {
let results =
try context.fetch(fetchRequest)
return results[0]
} catch let error as NSError {
print("Could not fetch \(error), \(error.userInfo)")
return Verse()
}
}
This works fine until I add the VerseDao.findById, which makes the whole process really slow because it has to make a request for each object to the Coredata database.
I did everything I could by limiting the number of fetched properties and using NSFetchedResultsController for data fetching but no luck.
I wonder if there's any way to insert child records in a more efficient way? Thanks.
Assuming your store type is persistent store type is sqlite (NSSQLiteStoreType):
The first thing you should check is whether you have an Core Data fetch index on the Verse objects verseId property. See this stack overflow answer for some introductory links on fetch indexes.
Without that, the fetch in your VerseDao.findById function may be scanning the whole database table every time.
To see if your index is working properly you may inspect the SQL queries generated by adding -com.apple.CoreData.SQLDebug 1 to the launch arguments in your Xcode scheme.
Other improvements:
Use NSManagedObjectContext.fetch or NSFetchRequest.execute (equivalent) instead of NSFetchedResultsController. The NSFetchedResultsController is typically used to bind results to a UI. In this case using it just adds overhead.
Don't set fetchRequest.propertiesToFetch, instead set fetchRequest.includesPropertyValues = false. This will avoid fetching the Verse object property values which you don't need to establish the relation to the Translation object.
Don't specify a sortDescriptor on the fetch request, this just complicates the query

How to deal with concurrency on core data

I've been wandering around google, stackoverflow and internet trying to understand how to work with core data and deal with the concurrency.
Consider that we have 2 tables, Events and Rooms.
An Event can have 1+ Rooms.
FunctionA - AddEvent
FunctionB - AddRoom
FunctionC - SearchRoom -> returns RoomEntity or nil
My problem, I keep getting these errors
Error Domain=NSCocoaErrorDomain Code=133020 "Could not merge changes." UserInfo={conflictList=(
"NSMergeConflict (0x10a507160) for NSManagedObject (0x1092f00c0) with objectID '0xd000000000040000 <x-coredata://A34C65BD-F9F0-4CCC-A9FB-1B1F5E48C70E/Rooms/p1>' with oldVersion = 116 and newVersion = 124 and old object snapshot = {\n location = Lisboa;\n name = \"\\U00cdndico LX\";\n} and new cached row = {\n location = Lisboa;\n name = \"\\U00cdndico LX\";\n}"
Notice the information of the Rooms is equal
my approach is the following.
1- I call the webservice once ( it brings a json with data of 3 types of Events ) These 3 all have the same json structure and share the same managedObjectContext passed by parameter
2- I create a managedObject
var managedObjectContext: NSManagedObjectContext = NSManagedObjectContext(concurrencyType: .mainQueueConcurrencyType)
managedObjectContext = persistentContainer.viewContext
managedObjectContext.parent?.mergePolicy = NSMergePolicyType.mergeByPropertyObjectTrumpMergePolicyType
3-
managedObjectContext.perform(
{
do
{
try self.deleteAllEventsFromDb()
FunctionA(eventList, managedObjectContext) -> save
FunctionA(eventList2, managedObjectContext) -> save
FunctionA(eventList3, managedObjectContext) -> save
self.DatabaseDispatchGroup.enter()
try managedObjectContext.save()
self.DatabaseDispatchGroup.leave()
completion(Result.Success(true))
}
catch let error as NSError
{
print("Could not save. \(error), \(error.userInfo)")
completion(Result.Success(false))
}
})
4- For each Event I execute the same FunctionA to create and save the data in database (managedObjectContext.insert(eventEntity)) . This will work over several tables but lets only consider Events and Rooms(FunctionB).
5- FunctionA contains functionB. Function B search for an existing Room(FunctionC->returns entity?) if it doesn't exists(nil), it creates the entity ( should I save here? )
6- If a Room exists, gets the entity and tries to update the data
Not sure if its making any difference but when I save I do these saves I do it between a dispatchGroup
DatabaseDispatchGroup.enter()
try managedObjectContext.save()
DatabaseDispatchGroup.leave()
I was using a static managedObjectContext which was used for all the database requests but now I decided to create a objectContext per function which accesses the database.
I do keep the same persistentContainer and the same DispatchGroup for all requests
private override init() {
persistentContainer = NSPersistentContainer(name: "DataModel")
persistentContainer.loadPersistentStores() { (description, error) in
if let error = error {
fatalError("Failed to load Core Data stack: \(error)")
}
}
}
It seems to me that my problem is that I am Storing the changes in memory, and always doing updates over the initial data meaning that when I execute save() the context the data is not updated for the next operation?
How/when am I suppose to execute the save function?
Thank you
Once context is being saved, global notification is being posted: ContextDidSave notification.
When using multiple contexts (and not using parent-child approach) you should use this notification and either:
Re-fetch/refresh data in case you need to update view or perform some operation on new data set (using either fetch request or refreshObjects: API).
Merge changes to other contexts (remember about thread confinement! do that only on proper context queues). (merge doc)
There are many articles about it, check for instance this tutorial
and documentation

How to improve performance for large datasets with Realm?

My database has 500,000 records. The tables don't have a primary key because Realm doesn't support compound primary keys. I fetch data in background thread, then I want to display it in the UI on the main thread. But since Realm objects cannot be shared across threads I cannot use the record I fetched in the background. Instead I need to refetch the record on main thread? If I fetch a record out of the 500,000 records it will block the main thread. I don't know how to deal with it. I use Realm because it said it's enough quick. If I need refetch the record many times, is it really faster than SQLite? I don't want to create another property that combine other columns as primary key because the Realm database is already bigger than a SQLite file.
#objc class CKPhraseModel: CKBaseHMMModel{
dynamic var pinyin :String!
dynamic var phrase :String = ""
class func fetchObjects(apinyin :String) -> Results<CKPhraseModel> {
let realm = Realm.createDefaultRealm()
let fetchString = generateQueryString(apinyin)
let phrases = realm.objects(self).filter(fetchString).sorted("frequency", ascending: false)
return phrases
}
func save(needTransition :Bool = true) {
if let realm = realm {
try! realm.write(needTransition) {[unowned self] in
self.frequency += 1
}
}
else {
let realm = Realm.createDefaultRealm()
if let model = self.dynamicType.fetchObjects(pinyin).filter("phrase == %#", phrase).first {
try! realm.write(needTransition) {[unowned self] in
model.frequency += self.frequency
}
}
else {
try! realm.write(needTransition) {[unowned self] in
realm.add(self)
}
}
}
}
}
then I store fetched records in Array
let userInput = "input somthing"
let phraseList = CKPhraseModel().fetchObjects(userInput)
for (_,phraseModel) in phraseList.enumerate() {
candidates.append(phraseModel)
}
Then I want to display candidates information in UI when the user clicks one of these. I will call CKPhraseModel's save function to save changes. This step is on main thread.
Realm is fast if you use its lazy loading capability, which means that you create a filter that would return your candidates directly from the Realm, because then you'd need to only retrieve only the elements you index in the results.
In your case, you copy ALL elements out. That's kinda slow, which is why you end up freezing.

Can't save or load objects

I am new to Realm and its the first time I am using it. I followed every step from the guide and its inserted in my project just fine. I created a model and a function to insert the object into the realm database.
Somehow I keep getting errors. Here is what I do.
my function
do {
let realm = try Realm()
let proposition = Proposition()
proposition.name = (currentProposition.name)
proposition.energyType = (currentProposition.energyType)
proposition.lifetime = (currentProposition.lifetime)
proposition.saving = (currentProposition.saving)
proposition.investing = (currentProposition.investing)
if let _ = propositionsArray.indexOf(proposition) {
try! realm.write {
realm.delete(proposition)
loadPropositions()
}
} else {
try! realm.write {
realm.add(proposition)
loadPropositions()
}
}
} catch let error as NSError {
print("Add proposition error \(error)")
}
Here is my model
import RealmSwift
import Foundation
class Proposition : Object {
dynamic var name: String = ""
dynamic var energyType: String = ""
dynamic var lifetime = 0
dynamic var saving = 0
dynamic var investing = 0
}
Somehow I keep getting the following error
Can someone tell me what I am doing wrong?
The errors you're seeing indicate that the data model defined by your application does not match the data model of the Realm you're opening. This is usually due to changing your data model. In this case, the errors mention that you've added the lifetime, saving, and investing properties, and changed name and energyType to be non-nullable.
There are two ways to accommodate changes to your data model:
If you're in early development and don't need to support your old data model, you can simply remove the Realm files and start over with empty data.
You can perform a migration to have Realm update the data model of the Realm file. See the Migrations section of the Realm documentation for information about how to perform a migration.