I have a couchbase installation onpremise and the same data at real time needs to be replicated to AWS Document DB, what are the possible approaches?
Can Synch Gateway help?
You can use Couchbase's Kafka connector for that (https://github.com/couchbase/kafka-connect-couchbase). Basically, it allows you to listen to all document changes in CB and push it to third party systems.
Related
Please what are the options for me in the following scenario.
We have a SQL managed instance on Azure, our client has requested that they want their data (managed instance) to sync on a weekly basis with their on premise SQL server. I suggested using Azure Data Sync but that will be costly for us and using transactional replication based on what was described here https://learn.microsoft.com/en-us/azure/azure-sql/managed-instance/replication-transactional-overview?view=azuresql is also costly as it requires that we have another managed instance to act as a broker.
Please, I'm very welcoming to ideas you may have on how to go about this.
Thank you.
Hi I'd like to move some data from an AWS DocumentDB to a PostgreSQL, I've found on the web ToroDB that would be the best tool for my task. It's implemented for MongoDB I'd like to know if I can use it also in AWS DocumentDB (with MongoDB compatibility)
Thank you in advice!
From what I can see in ToroDB documentation , is using the oplog to tail the changes in MongoDB, not change streams. Amazon DocumentDB doesn't have the oplog collection, but does supports change streams. Unless ToroDB gets updated to support change streams (which I doubt it will, documentation says only MongoDB 3.2 and 3.4 are supported and last Github update is from 5 years ago), is not going to be able to live replicate from DocumentDB.
However, if the destination is AWS RDS PostgreSQL or AWS Aurora PostgreSQL, then AWS DMS can be used to replicate from DocumentDB.
Is there an easy way to sync MongoDB data with Elastic if we are using a Loopback 4 application? (In a NodeJS Express application we can easily do this using mongoosastic plugin.)
Currently in loopback a model is bound to one data source which is either a mongo or elastic. My ultimate goal is to write POST, PUT, DELETE methods in mongo (and replicate in elastic) and GET methods to use Elastic as datasource.
I would highly recommend to decouple the syncoronization part from the core routes of your application. You could leverage the MongoDB Change streams to listen for change and update your elasticsearch. This could be a simple app running on a totally different server which does the syncronization while you carry on serving your requests. This will make the overall process and architecture more durable and fault tolerant. You can read more about this on following link
Mongodb to elasticsearch
Actually I am not very familiar with Dynamodb and I would like to launch a NoSQL database with local Dynamodb (downloadable version) but not hosted on amazon AWS. I would appreciate it if someone could let me know is it possible to make such a clustered or does downloadable version of Dynamodb support to be cluster locally ?
You can very easily run DynamoDB locally, but it only supports running a single instance—not a cluster. It's intended to be used for local testing/debugging.
DynamoDB is provided as a hosted service. Does not exist a DynamoDB code that you can download and install to use as a host or service provider.
As part of SDK for a lot of languages, AWS Team developed some wrappers that permits you to execute local versions of DynamoDB to test your particular code. These wrappers respect the DynamoDB API contract. In that case you can code to the DynamoDB interface and get the responses like it were hosted in AWS environment. But you can't host any database or even serve data as a service using this solutions.
I want to keep a windows azure hdinsight cluster always running so that I can periodically write updates from my master data store (which is mongodb) and have it process map-reduce jobs on demand.
How can periodically sync data from mongodb with the hdinsight service? I'm trying to not have to upload all data whenever a new query is submitted which anytime, but instead have it somehow pre-warmed.
Is that possible on hdinsight? Is it even possible with hadoop?
Thanks,
It is certainly possible to have that data pushed from Mongo into Hadoop.
Unfortunately HDInsight does not support HBase (yet) otherwise you could use something like ZeroWing which is a solution from Stripe that reads the MongoDB Op log used by Mongo for replication and then writes that our to HBase.
Another solution might be to write out documents from your Mongo to Azure Blob storage, this means you wouldn't have to have the cluster up all the time, but would be able to use it to do periodic map reduce analytics against the files in the storage vault.
Your best method is undoubtedly to use the Mongo Hadoop connector. This can be installed in HDInsight, but it's a bit fiddly. I've blogged a method here.