I a developing a POC to import some data from MongoDB to MS SQL Server 2012. For e g. there are 5 collections n Mongo which need to be loaded to 5 tables in MS SQL Server tables. The data needs to be dumped as is.
What is the best approach to achieve this ? Wil any ETL Tool (SSIS) suffice or shall we have to write code in Node jS? Can anyone point me in the right direction.
Please advise.
Thanks in advance.
I use ZappySys' SSIS PowerPack, which includes MongoDB Source and Destination connections.
Related
Currently i have a database with almost 50kk record and an collection with almost 20kk that i need to connect with PowerBI, I tried using the MongoDB BI Connector, but it is impossible slow, are anything I can do to speed things up? Or should i move to something more robust to help my BI Team?
Need to identify the performance bottle neck in detail.
If data import is slow, then Extract the data from Mongo DB into a file. (as the data volume that you specified seems to be low - in thousands). Then import the file into Power BI model.
If the slicing and dicing is slower after data import into PowerBI, then tuning the model is required.
Thanks.
Anything going through ODBC (I am guessing that's what you are using), is going to be slow. One way to do this is to use r from Power BI to get the data from MongoDB. The following link gives details about how you can connect to MongoDB from r:
Connect R to remote mongoDB with rmongodb
Once your query is ready, you can choose "R Script" from the "Get data" window to use the r query. Hope this helps.
I have created custom script in Express that actually migrates SQL Server database to MongoDB.
But I am facing problems in live syncing between the two databases.
Currently I have added a column updated_by in both the databases.
Then I fetch the latest updated_by row from MongoDb and SQL Server database.
Then I check the date difference and based on it I update my MongoDB database.
There are lots of db tables and I am finding it difficult to identify that, which table is being updated.
Is there any log in SQL Server 2008 R2 that states which table is updated and at what time?
I need a mechanism like, any data update in the db table should immediately sync that rows into my MongoDB.
Any more suggestions on live data syncing is also welcome.
Thanks in advance. :)
When i have such requirement to Sync between Relational DB say (MYSQL) and Non-Relational DB (Mongodb).
I had followed following steps which may help others in future. and the concept is generally called as Change Data Capture
Capture changes (For MYSQL iam using triggers.)
Transform changes to a suitable changes
ie RDBMS to Non RDBMS
Update changes
Remember to sync the structural changes of database and corresponding implementaions.
Following links may help
https://www.flydata.com/blog/what-change-data-capture-cdc-is-and-why-its-important/
I am receiving .dmp and .mdb files from a customer & need to get that data into MongoDB.
Is there any way to straight import these file types into Mongo?
The goal is to programmatically ingest these into mongo in any way I can. The only rule is that customer will not change their method of data delivery, meaning I'm stuck with the .dmp and .mdb files as a source.
Any assistance would be greatly appreciated.
Here are a few options/ideas:
Convert mdb to csv, then use mongoimport --type csv to import into MongoDB.
Use an ETL tool, e.g. Pentaho, Informatica, etc. This will give you much more flexibility for doing any necessary transformation/conversion of data.
Write a custom ETL tool, using libraries that know how to read mdb and dmp files.
You don't mention how you plan to use this data, how many tables are in the database, and how normalized the tables are. Depending on the specifics of your use case, it's very possible that loading the data from Access "as is" will not be a good choice since normalized schemas are not a good fit for MongoDB and MongoDB does not natively support joins. This is where an ETL tool can help, by extracting the source data and transforming it into an appropriate JSON structure.
MongoDB has released ODBC drivers. Go Here MongoDB ODBC Drivers connect MSAccess directly to MongoDB through ODBC. Voila!
I am trying to load data from falt file source to Sybase IQ 15.1 destination using SSIS 2008. I am using ADO.NET destination component with Sybase IQ ODBC data provider to connect to the Sybase IQ db. Though the package is able to load the data, performance of the package is extremely slow. Its just loading 150 rows per minute. I am executing the package through SQL Server Agent. There are no data transformations in the package.
Can any one please tell me how can i improve the performance?
Thanks in advance
ODBC drivers are generally slow. How many columns are you pulling in? Also, have you checked your buffer sizes and stuff?
Use BulkInsertWheneverPossible property to true, it will improve the performance.
To access that property right click->ADO.NET destination-> choose properties or ShowAdvanceEditor
(MSDN Reference)
http://technet.microsoft.com/en-us/library/ms135971(v=sql.105).aspx#adonet
A value that specifies whether to use the System.Data.SqlClient.SqlBulkCopy interface to improve the performance of bulk insert operations. Only certain ADO.NET providers support this interface.
The default value is True.
I am looking for a tool/command which can compare data between two PostgreSQL databases. The reason to do this is to have some external verification that the SQL script responsible for data migration from one PostgreSQL database to the other have been written correctly.
Any pointers would be appreciated
regards
Sameer
EMS makes a tool which can do this.
http://sqlmanager.net/en/products/postgresql/dbcomparer
The PostgreSQL site has a nice Software catalogue. Peruse the Administration/development tools category.