I'm trying to create a method that will automatically create something I can save and use later, which will let me automate data migration from one version of a class to the next.
When I compile a normal persistent class, a "storage" gets created, and I would like to be able to archive some code that would let me recreate this "old storage" so I could access an "older version" from a global and map the values into the current mapping of the class. I have objectgenerator methods that will save a version with every record, and detect if the "current version" of the class code is different from the version saved in the data record itself, but I'm not sure how to retain what the "old" version actually looked like so I can auto-migrate the data from the "old to the new". To do this, I'm thinking that being able to read what the current storage is at compile time, I should be able to save that value elsewhere and create a durable "version reader" so I can migrate the data forward without having to actually have the programmer do the work.
Does this seem like a reasonable approach? and if so, can anyone point me to where, at compile time, the storage values are so I can do the saving of the data?
(or should I be looking at cloning some part of the generated %Save() method chain at compile time to a version specific save attached to "something else" (not sure what, but 'something'))
This question and answer originally appeared in the InterSystems Developer Community https://community.intersystems.com/post/how-do-you-access-storage-map-cache-class-programmatically
In a database, your most important asset is your data -- business logic and code comes a close second. There should at least be someone paying attention to the data slots for sanity, even though in most cases, there is nothing to do. At the very least, any developer adding or modifying properties should "diff" the storage definition when submitting code to the source repository. The class compiler by default will handle everything correctly.
The simplest and best way to move from one version of a class to the next is to keep the same storage definition. The class compiler will create slots for any new properties and is completely non-destructive and conservative. When I retire properties, I generally manually rename the storage slot and give it a prefix such as zzz. This allows me to explicitly clean up and reuse these slots later if I so choose.
For the type of before/after compile triggers you are looking for, I would use the RemoveProjection/CreateProjection methods of a projection class which provide "a way to customize what happens when a class is compiled or removed".
You can use %Dictionary.CompiledStorage and related classes(Data) to have full access to the compiled storage definition of a class.
3a. An alternate approach is to use XSLT or XML parsing to read the storage definition from the exported class definition. I would only use this alternative if you need to capture details for separate source control purposes.
The simplest storage definitions use $ListBuild slots and a single global node. When subclasses and collection properties are used, the default storage definition gets more complicated -- that is where you will really want to stick to the simple approach (item 1).
Related
I'm trying to create a note-taking like app that uses NSPersistentCloudKitContainer and core data.
The store uses the NSMergeByPropertyObjectTrumpMergePolicy, which is fine for almost every property. For example, if the name of a file is changed on two different devices, then it's fine to use the latest value.
The problem is that the note text cannot be overridden by the latest value if it's changed on two devices at once. It needs to be detected as a conflict so the user can choose which version they want to keep.
I can replicate the behavior by turning off wifi on one device and writing content, then writing content on a different device at the same time. When I turn the wifi back on, whichever device saved the changes last completely overrides the other device's text.
What I'd like to accomplish is detect when there is a conflict of text, then create a duplicate file called "Conflicted Copy." Bonus points if someone can tell me how Apple Notes magically merges text without ever creating a conflict. I really only need a simple solution though that prevents data loss.
Any help in the right direction would be appreciated!
The conflict arises when CoreData & CloudKit sync an object to the persistent store while a managed context has an updated version of the object that has not yet been saved.
Any merge policy, including a custom merge policy, is used to create a single object that will be stored in the persistent store, but you want to have both conflicting objects so that the user can choose one of them.
Thus automatic conflict resolution, including with a custom merge policy, cannot be applied.
Instead, use the default merge policy of type error. The docs say
If a save fails because of conflicting objects, you can find the IDs
of those objects in error’s userInfo dictionary. Use the
NSInsertedObjectsKey and NSUpdatedObjectsKey keys to extract the
object IDs.
This means, you can keep the properties of the not-yet-stored conflicting object in the managed context, and re-fetch the properties of the conflicting object from the persistent store. This will overwrite the conflicting in-context version and "resolve" the conflict, but you still have now the conflicting properties and can present them to the user.
If the user selects the version in the persistent store, you are done. Otherwise update the objects properties with the kept values as required and save the context.
PS: Here is for your information how to implement a custom merge policy, although it cannot be applied here, just because such info is difficult to find.
EDIT:
I can imagine 2 ways to have one merge policy for your "normal" objects, and the error policy for the text objects:
A merge policy is set for a managed context. So you could have one context with NSMergeByPropertyObjectTrumpMergePolicy, and another one with NSErrorMergePolicy. So if it is feasible to fetch most objects into the 1st context, but the text storing objects into the 2nd one, you could apply both conflict resolution strategies.
During conflict resolution, i.e. in func resolve(optimisticLockingConflicts…, one has to call super.resolve(optimisticLockingConflicts: (see my example implementation cited above). So if you set NSMergeByPropertyObjectTrumpMergePolicy to your context, but call super only for your "normal" objects, this merge policy would not be applied to your text objects, and you could handle the conflict by your own. Warning: As far as I know this is non-standard, and I am not sure what happens if you don't call super.
I've searched all over the web for definitions, but I'm still confused. I've narrowed it all down to two different defintions:
"A data structure is persistent if it support saccess to multiple version" and "Persistence is the ability of an object to survive the lifetime of the OS process in which it resides".
To me, these mean different things, but maybe I'm just not getting it. Could someone please explaing to me in a basic way what exactly persistence means?
This word means different things in different contexts:
Persistent data structures create new copies of themselves to incorporate changes (all versions are accessible AND modifiable at any time).
Persistence in your second example refers to the ability of objects to be stored in non-volatile memory, such as a hard disk. Otherwise, they would be destroyed when the OS ends its session.
I'm attempting to write up a Yesod app as a replacement for a Ruby JSON service that uses MongoDB on the backend and I'm running into some snags.
the sql=foobar syntax in the models file does not seem too affect which collection Persistent.MongoDB uses. How can I change that?
is there a way to easily configure mongodb (preferably through the yaml file) to be explicitly read only? I'd take more comfort deploying this knowing that there was no possible way the app could overwrite or damage production data.
Is there any way I can get Persistent.MongoDB to ignore fields it doesn't know about? This service only needs a fraction of the fields in the collection in question. In order to keep the code as simple as possible, I'd really like to just map to the fields I care about and have Yesod ignore everything else. Instead it complains that the fields don't match.
How does one go about defining instances for models, such as ToJSON. I'd like to customize how that JSON gets rendered but I get the following error:
Handler/ProductStat.hs:8:10:
Illegal instance declaration for ToJSON Product'
(All instance types must be of the form (T t1 ... tn)
where T is not a synonym.
Use -XTypeSynonymInstances if you want to disable this.)
In the instance declaration forToJSON Product'
1) seems that sql= is not hooked up to mongo. Since sql is already doing this it shouldn't be difficult for Mongo.
2) you can change the function that runs the queries
in persistent/persistent-mongoDB/Database/Persist there is a runPool function of PersistConfig. That gets used in yesod-defaults. We should probably change the loadConfig function to check a readOnly setting
3) I am ok with changing the reorder function to allow for ignoring, although in the future (if MongoDB returns everything in ordeR) that may have performance implications, so ideally you would list the ignored columns.
4) This shouldn't require changes to Persistent. Did you try turning on TypeSynonymInstances ?
I have several other Yesod/Persistent priorities to attend to before these changes- please roll up your sleeves and let me know what help you need making them. I can change 2 & 3 myself fairly soon if you are committed to testing them.
I am fairly new to Core Data and have run into an issue which others must have encountered.
My data model includes images and I am keeping those external to the database and simply storing a path/URL to the image. (as recommended in one of Apple's Core Data presentations)
When deleting my image object I can manually traverse relationships and remove the image files but I am wondering if there is a more elegant way to do this.
The ideal solution would be tied to the image object somehow and would work with Core Data undo/redo.
In your "image" entity class, implement willSave. Check [self isDeleted] and delete the file if so. This postpones actual deletion of the file until the store is saved, which gives you some undo-ability. Set up appropriate cascade rules to delete the image entities when their owner goes away, and there you go.
[eta.: Phil Calvin's comment below is right - didSave is probably a better place, if you're using multiple contexts.]
[eta. much later:] MartinW raises an excellent point - if the object has never been saved, will/did save won't get called. Deleting an un-saved object just un-inserts it from the context and throws it away, with no special lifecycle events. And just to complicate matters more, you can "undo" and "redo" a deletion, including (I think) this kind.
A few approaches that come to mind:
This might be a case for overriding prepareForDeletion:, plus awakeFromSnapshotEvents: to catch un-deletion and re-deletion. To support undo/redo, you'll need to not simply delete the file on the spot, but use some kind of "to be removed" registry (e.g. a shared mutable set of filenames to clean up when a save notification is posted). Then will/didSave are out of the picture.
Or, if you can live with BLOB fields instead of files, you could check the "allows external storage" box on a binary property, put the jpeg data there, and get some (not all) of the advantages of file storage without (most of) the headaches. Little binary will be kept in the db; anything bigger than a private threshold will be shunted out into a separate hidden core-data-managed file. Core data still has to load the whole thing into an NSData when faulting in the object, though. I use this approach for "user avatar"-type small images.
Finally, if nothing else is kept in the images directory, you could register for didSave notifications and manually clean up after any save. Run a fetch request for all the Image.filename properties, compare it to a directory listing of the images dir in question, delete as appropriate. I use this approach as my "big stick" during development to make sure everything else is doing what it should.
[Let me know of successes or hardships with these approaches and I'll keep this up to date.]
In your Image class, implement -didSave. In this method, check whether [self isDeleted] and if it's YES, delete the image files from disk.
It's important to do this in -didSave rather than -willSave particularly if you have multiple managed object contexts associated with your persistent store. willSave is sent before Core Data discovers and reports (as a save error) any merge conflicts. This means you could potentially receive willSave either:
Multiple times if you implement a merge strategy that merges objects and re-tries the save
Before a save that never commits to the persistent store due to a merge conflict
Deleting the image files prematurely could cause a crash later when those images are accessed from another managed object context.
I would suggest overriding the image entities prepareForDeletion method to delete the image file on disk. The method will only be called with the image entity object is actually deleted.
The best practice would be (which I think would be the same to what #Rog implied) to have an entity for stored images, and make your objects have a relationship to that entity, rather than storing paths in each object. In that case, you can just find one object which represents the picture to be deleted and delete it, than the reverse relationship can be automatically nullified.
For my current app I'm using the NSKeyedArchiver approach to persist my objects to the iPhone between launches. I'm wondering if it makes sense to tag this file when it's created with a version number so that in the future when the app revs we'll know what kind of data we're dealing with should migration be necessary. Is this recommended for something as simple as an NSArray of custom objects that are serialized with NSKeyedArchiver? If so how/where to stash the version number?
Thanks!
If you feel like the format will change, then yes, you'll want some way to detect which version of the file you're working with. A simple way to do this (since you're using NSArray) would be to simply add the version number as an NSNumber to either the beginning or end of the array. Alternatively, for a more complicated approach that doesn't actually change the files, you could store the information in a separate file (say, using an NSDictionary that links the file name and version number). Or, you could use the user defaults system to store it, and if it's frome an older version, update the file and the user defaults key.
I wouldn't use the second method, unless you're dealing with multiple keyed data files. Of those three, I'd probably choose the last, but there may be some other way to do it that makes more sense.