Currently I am trying to clone a cosmos db collection from one database to another database within the cosmos db. The API of the cosmos db is set to Mongo API.
I already tried to use Azure Data factory, but it looks like that there is no support for the Mongo API so far.
Has anyone an idea how to do this respective to efficiency, automation and performance?
Any ideas are appreciated.
You can use data Migration tool suggested by Microsoft to do the same.
There is no way to take a backup and import cosmosdb.
EDIT:
With the new Cosmic Clone tool, you can take a clone/backup with data/stored procedures/triggers/udf, etc. Read my blog on the same.
I already tried to use Azure Data factory, but it looks like that
there is no support for the Mongo API so far.
Actually, Cosmos DB Mongo API and SQL API are all belong to Azure Cosmos DB service.So , you still can create cosmos db linked service and dataset in the azure data factory for your database.
Then you could create copy activity to import data from one collection to another collection.
If you want to make it as an automation task, I suggest using following 2 ways to run the copy activity.
1.Azure Time Trigger Function.
2.Web job which is run in the background of Azure Web App.
Hope it helps you.Any concern, please feel free to let me know.
I used mongodump and mongorestore to copy my database (with mongodb version 4.0.9 installed). From the windows command line I ran the following commands from my mongodb bin directory (c:\Program Files\MongoDB\Server\4.0\bin in my case).
This will copy all the collections, including indexes, in the DB to the specified /out directory as .json files.
mongodump.exe /uri:URI /out:A_DIRECTORY_TO_DUMP_TO
I then ran the following command to take everything in the /out directory and write it to the target DB:
mongorestore.exe /uri:URI /dir:DIRECTORY_TO_RESTORE_FROM
NOTE: Before importing I also had to increase the throughput for the collection, otherwise I ran into rate limiting errors. If you've set throughput at the database level this may need to be changed.
Related
I would be happy to get your help / feedback re data load.
Goal:
Load source data from a Postgres database, which is located in GCP project A to another Postgres database, which is located in GCP project B.
Challenge:
Get a connection (I have an IAM account with sufficient rights to run a COPY TO / COPY FROM command) to the Postgres DB in GCP Project A and copy the table either to a CSV or create a dump that can be used in order to be inserted to another Postgres DB in GCP Project B.
How do I connect to the database (e.g. if I create a key, where shall I store the json keyfile and would that approach even be feasible?) with this IAM email account?
Other ways I've researched were to use psycopg2 (thus I could use the function cursor.copy_expert (which doesn’t need any superuser right or Postgres user credentials and copy the data), but I didn’t succeed in connecting to the database with psycopg2 due to challenges with cloud proxy.
Another idea was to use pg_dump or gcloud sql export csv.
I would be curious if some of you were facing a similar challenge and how did you solve it and what might be the best way/practice
You can have a try out database migration service. You can set up a continuous migration configuration and use Cloud SQL for PostgreSQL.
Hello after a lot of searching I've come to these solutions:
If you have continuous copy, you need to use the database migration service, check this documentation.
If you have one shot copy:
you can restore your instance, see the bottom page of this documentation
you can create a bucket and backup your instance on it, then import it from the other project
The issue is that mirgating data from MongoDB to DynamoDB/S3/Redshift currently, as I unterstand for us is not available via AWS DMS Service, as it does not support all data types. Or maybe I'm wrong.
The probelm is that our Mongo object contain not scalar fields(arrays, maps).
So when I make a mirgation task via AWS DMS with table mode, it pull data badly.Buy some reason only selection works. Transformation rules are ignored by DMS(tried renaming and removing).
In the doc mode is all ok, but how can I run migration with some custom script for transformation? As storing data this way still need transformation.
We need some modifications like: rename, remove fields and flatting some fields(for example we ahve a map object and it should be flatten into several scalar fields).
Migration should be done into one of the sources: S3, Dyanamo, Redshift
Will be thankfull for any help and suggestions.
use the following below script to take a backup of the MongoDB DB
mongodump -h localhost:27017 -d my_db_name -o $DEST
use the below command to sync your backup to S3 bucket
aws s3 sync ~/db_backups s3://my-bucket-name
Once your data in S3, you can load very easily to Redshift using copy command
I have installed an extension for having access to a Microsoft Azure Cosmos DB in VS Code, and I'm able to see the documents inside as expected.
However, I wasn't able to figure out how to run a query that allow me to filter results based on different conditions e.g: SELECT * FROM c WHERE c.DocumentId = 123
Is there a way to run SQL queries in VS Code on a Cosmos DB? I couldn't find any helpful tutorial and by installing "mssql" extension seems that is mainly focused on ADO.NET connections.
Based on the statements in the cosmos db extension:
Browse and query your MongoDB databases both locally and in the cloud
using scrapbooks with rich Intellisense then connect to Azure to
manage your Cosmos DB databases with support for MongoDB, Graph
(Gremlin), and SQL (previously known as DocumentDB).
You could follow below official example of Scrapbooks to query your sql api db with mongo shell syntax.
E.G.
db.coll.find({"DocumentId" :123})
Update:
Sorry for the late updates. I got 2 tools for you:
1.CosmosDbExplorer :https://www.bruttin.com/CosmosDbExplorer/
2.Azure Storage Explorer: https://azure.microsoft.com/en-us/features/storage-explorer/
This tool might help: https://github.com/microsoft/vscode-cosmosdb
The issue is that they do not explain how to use and manipulate SQL API. Please share if anyone has an idea for using SQL API like the Mango DB and this tool.
We are creating Meteor-based Mongo database manager and we need the ability to "unmount" (remove from system) all collections when we switch databases.
Example:
I'm managing database called dbA. We have all collections for that database created using Mongo.Collection() on server and on client side.
I want to switch database to dbB. I need to unmount all Collections of dbA and mount those of dbB. Reason: dbB could have a collection of the same name as dbA (and usually does)
Is there a way to do this?
Thanks!
You may be able to accomplish this by publishing the necessary data from the new database.
Here's a discussion from a similar question on the Meteor Forums (note the proposed solution at the end):
https://forums.meteor.com/t/switch-database-while-meteor-is-running/4361/5
hi i think you can do with
db.copyDatabase()
run the shell command in the backend from meteor server and execute the copy database command. and after the database is copied you can remove the previous collection.
more detail about copyDatabase() is here
https://docs.mongodb.org/manual/reference/method/db.copyDatabase/
I have separate development and production MongoDB servers and I want to keep actual data in development server for sometime. What I should use for it: mongodump, mongoimport or something else?
Clarification: I want to copy data from production to development.
If it's a one time-thing
and you want fine control over parameters such as which collections to sync, you should use:
mongodump to dump bson files of your Production DB to your local machine
mongorestore to then, retrieve the dumped BSON files in your Local DB
Otherwise you should check out mongo-sync
It's a script I wrote for my self when I had to constantly copy my Local MongoDB database to and from my Production DB for a Project (I know it's stupid).
Once you put your DB details in config.yml, you can start syncing using two simple commands:
./mongo-sync push # Push DB to Remote
./mongo-sync pull # Pull DB to Local
If you use it inside some project, it's a good idea to add config.yml to .gitignore
You can use the db.copyDatabase(...) or db.cloneDatabase(...) commands:
http://www.mongodb.org/display/DOCS/Copy+Database+Commands
This is faster than mongodump / mongorestore because it skips creating the bson representation on disk.
When you want the dev database to look exactly like the production database, you can just copy the files. I am currently running a setup where I synchronize my MongoDB database between my desktop and my notebook with dropbox - even that works flawless.