I've a bunch of collections (12) and I need to rename many fields of them. I can't do it live in the database, all I can do is download and reupload a dump of it.
So I've downloaded the collection with mongodump manipulated the data and I'm planning to use mongorestore to push it back on the database.
I'm wondering what will happen then with ObjectIds.. I know that an objectId is unique throughout the database so I'm thinking about deleting all the old data right before using mongorestore, is it ok or will I still have problems with the ids?
You can specify any value for MongoID whatever you want. You even can use string instead of MongoID.
If you have production app you need to perform upgrade and migrate data by application itself step by step.
If you have one proccess singlethreaded application or if you can run your app in that way - it is most simple case. Else you need synchronization service.
Be carefull with async/await and promises and so on asyncronous processes. They receive and have in memory one the data in one time and continue process with that data in another time, and it need to have in mind that.
You need to do:
modify service to be ready to both data format
create modification code which will go through all the data and migrate it
modify service to be ready only to new data format once all the data migrate done
Related
Is there a possible way of writing database migrations for Parse Server?
My use case is: I want some tables with pre-populated data whenever I connect my application to a fresh mongodb server. Say for example, setup a staging environment, setup a local development environment etc etc.
I could not really find anything in the docs.
Am i going in the right direction or am I missing something?
Part of the Parse.com -> Parse-Server migration is that you now have to manage your own database. Parse-Server gives you the necessary tools to connect to your database, but you have to do things like manage indexes (mlab gives you weekly tips on what indexes you could add for improvement!), uploading large amounts of data, etc.
So, if your question is "Does Parse-Server do this?" No, and they won't.
If your question is "Can this be done?" Well, yes! MongoDB has an upload feature that takes in JSON or CSV, if I recall correctly. I know you can upload a collection, I'm not sure if you can do multiple collections at a time. One caveat is that you need to set createdAt and updatedAt and objectId yourself for this, but yes you can do this.
If you aren't too familiar with working with raw data that mongo needs, you could always setup the tables you want if they aren't too big, export all the data, and then use that data export as the import for all the fresh databases thereafter. Only issue there is that updatedAt and createdAt will show old dates on new instances.
MongoDB contains data ready for client-side apps. The raw data being stored in Google BigQuery (GBQ). Each day a lot of new data being added into GBQ and once a day pretty much everything in MongoDB needs to be updated according to the most recent data in GBQ. All outdated (not updated) records must be deleted.
What is the right way to handle MongoDB update with close to 0 downtime?
Among the crazy solutions: may be i should have two instances of MongoDB, one is in production, another is being updated. Once the second db updated, i'll run Google Kubernetes Engine deploy with changed configs, so all clients will be smoothly moved from previous data to the updated one without messing up with partially updated data and without downtime. Though, i have never heard about such solutions, so i'm not sure if this is the right one.
Another solution is to have two versions of each collection under a single instance of MongoDB. Once collection is updated, server switches to that collection.
The 2nd solution seems a good option, if you know the trigger for the update, you can have minimum downtime by creating a new collection (named by date or a unique serial maybe) and update your code accordingly.
I had some good experience doing this for a fashion website sometime back, where we scraped data (using scrapinghub) and imported them into mongodb (collections stored by date) and used accordingly. So our scraping ran early morning (5-6AM) and when our editors/curators came in the office, they would start using the current dated collection (via the Web Interface of course :) )
I'm trying to set up an app that will act as a front end to an externally updated mongo database. The data will be pushed into the database by another process.
I so far have the app connecting to the external mongo instance and pulling data out with on issues, but its not reactive (not seeing any of the new data going into the mongo database).
I've done some digging and it so far can only find that I might need to set up replica sets and use oplog, is there a way to do this without going to replica sets (or is that the best way anyway)?
The code so far is really simple, a single collection, a single publication (pulling out the last 10 records from the database) and a single template just displaying that data.
No deps that I've written (not sure if that's what I'm missing).
Thanks.
Any reason not to use Oplog? For what I've read it is the recommended approach even if your DB isn't updated by an external process, and a must if it does.
Nevertheless, without Oplog your app should see the changes on the DB made by the external process anyway. It should take longer (up to 10 seconds), but it should update.
I have written a client that uses the SyncFramework to coordinate the consolidating of data in a hub and spoke architecture warehousing application.
When the sync transactions process the sync framework updates a specified anchor table with the value of ##DBTS, indicating when the last sync was processed and uploaded to the server.
I would like to offer as part of this scenario the ability to allow one client to relay the data on behalf of one of the others.
This would be used in cases where one client may not be able to make contact with the warehouse; its database could be retrieved and synchronized by a client that does have access to the warehouse (Exchanged as a database backup on DVD or USB flash media).
The problem with this theory is that without the SentAnchor being set on the client database when the snapshot is retrieved, the next time this process is performed, the whole database is replicated in a second time.
What I would like to do, is when I grab a snapshot of the client database, update its SentAnchor so the next time I grab a copy the sync framework will know its SentAnchor as if it had actually communicated with the server.
So my first impulse was to simply update the anchor table, set the SentAnchor to ##DBTS, however the problem with that is sync framework inserts the same value in a different format, it runs it through the BinaryFormatter first.
So same intrinsic value, different headers, and when I try just updating with the value of ##DBTS, the SyncFramework errors trying to convert that back from the format it anticipates to have set itself.
What I would like to do is set via a TSQL statement, the same format for ##DBTS that the sync framework uses; I do not want to have to write an application to execute a single SQL statement if this can be done in the statement already being executed to create the backup.
Something like...
USE MyDB
GO
BACKUP DATABASE MyDb
TO DISK = 'F:\01032012MyDb.bak'
WITH FORMAT,
NAME = '20120103 Full Backup of MyDb'
GO
UPDATE Anchor SET SentAnchor = ##DBTS
GO
Essentially replacing ##DBTS above with whatever is needed to get the same value into the correct fromat that the SyncFramework will use.
Servers are 2008R2 Express.
the problem with setting the SentAnchor is that you might actually miss uploading changes. by setting the value, you have effectively told Sync Framework it has sent changes up to that value of ##DBTS.
i suggest you explore using the SqlSyncProvider instead.
So I'm planning an app that will involve having a master db on a server, lets say 3,000 CDs, with the columns Title, Artist, and Release Date.
1)When a user adds a CD to their collection, it will add it to the apps local SQLite DB. But lets say I spelled a CD title wrong, so I make an update to it. When the user goes to sync, how should I go about handling an updated row? Should I have a column 'IsUpdated' that is just a numeric value that increase by one every time I update that row? That way when the app sees IsUpdated on the server is larger than the local IsUpdated for that particular item, it will now to replace the contents. Does that make sense? Is it even practical? What other option would there be?
2) How would I do about handling the addition of brand new columns? Like adding a Barcode or Price? Do I just push an update for the app that adds the new columns locally, then do the same on the server, and let the rest take its run? Which would also trickle to number 1 with the syncing issue.
First you have to give more detail than that. Is the entire 3000 master list also replicated down to the remote db?
Sounds like it.
Ok so if that the case, this isn't a DB design issue so much as it is replication.
It's a bad idea to update every row in a table, especially one that makes the row longer. You'll be better off just dropping the table and recreating. <--- that's how it works in RDBMS on servers, no idea if that concept changes on a client db. And now we get into more iPhone questions of replication than simple db replication. Would it be better to just republish the app? Is the user data segregated from the server data. Can DDL be done on the local/remote tables after published?
Instead of searching the entire list for changes as you outline in #1. I would keep a dated delta table. The local app would store a last_updated_Datetime, any records in the delta table after that datetime would need to be brought down. Once downloaded the local system can determine how to apply them. Again this is inappropriate for mass changes.