I'm new-ish to MongoDB Atlas and I have my dev environment all set up and working with my collections, rules, and some functions in use by HTTPS Endpoints. I'd like to clone all of that into a second cluster on a serverless cluster instance for production.
Is there a best-practice way of doing this and then updating it on an ongoing basis in both directions? Copying functions/endpoints etc from dev to prod, and copying collections from prod to dev?
Related
We are in the process of creating a new Database using Mongo DB Atlas and we are checking the possibility of setting up the entire setup (Project, Cluster, Database, Collections, Users etc.) using Infrastructure as Code. This will enable us to setup a new environment easily using the scripts. We saw few options like Terraform Templates (https://docs.mongodb.com/mongodb-vscode/create-cluster-terraform/). But this will setup only the project, cluster and database users. We do not see a way in setting up databases or collections. Is there any documentation or guidance around how to spin up Mongo DB using IaC?
The following is the mongodb atlas terraform provider: https://registry.terraform.io/providers/mongodb/mongodbatlas/latest/docs/resources/database_user this can be used to interact with the resources supported by MongoDB Atlas.
There you can check all the available provider resources and data sources.
I'm working on trying to get a MongoDB replicaset deployed into Kubernetes with a default set of collections and data. The Kubernetes piece isn't too pertinent but I wanted to provide that for background.
Essentially in our environment we have a set of collections and data in the form of .js scripts that we currently build into our MongoDB image by copying them into /docker-entrypoint-initdb.d/. This works well in our current use case where we're only deploying MongoDB as a single container using Docker. Along with revamping our entire deployment process to deploy our application into Kubernetes, I need to get MongoDB deployed in a replicaset (with persistent storage) for obvious reasons such as failover.
The issue I've run into and found recognized elsewhere such as this issue https://github.com/docker-library/mongo/issues/339 is that scripts in /docker-entrypoint-initdb.d/ do not run in the same manner when configuring a replicaset. I've attempted a few other things such as running a seed container after the mongo replicaset is initialized, building our image with the collections and data on a different volume (such as /data/db2) so that it persists once the build is finished, and a variety of scripts such as those in the github link above. All of these either don't work or feel very "hacky" and I don't particularly feel comfortable deploying these to customer environments.
Unfortunately I'm a bit limited with toolsets and have not been approved to use a cloud offering like MongoDB Atlas or tooling such as the Enterprise Kubernetes Operator. Is there any real supported method for this use case or is the supported method to use a cloud offering or one of the MondoDB operators?
Thanks in advance!
I am building my own webapp which requires a huge database. I want to build and manage my own Mongo database on AWS rather than using Mongo Atlas. Which will be more cost saving? And whether I should go for Mongo Atlas? What will be its advantage over my own database?
There are pros and cons for both approaches:
Running MongoDB on AWS
Pros:
Complete control over how you run the database and how resources are allocated on the server. This could even be together with an application server on the same EC2 instance depending on your traffic and load. This might help with cost saving if your database is huge but isn't likely to see much traffic.
Cons:
You will be responsible for ensuring database availability and applying security patches as and when they are available. You may also have to setup firewalls and protect the EC2 instance and database in other ways that would be trivial to do on a hosted service like Atlas.
Data sharding and clustering can be a real pain to manage by yourself.
Running on Atlas
Pros:
Completely managed service where you don't have to be concerned about performance optimization or scalability. You pay for the services and Mongodb takes care of the rest.
You can focus on building a great application instead of spending your time on administering the database and the EC2 instance on which the database runs.
Cons:
You will be constrained by the options offered by Atlas. For most use cases this should be fine, but if you really want a specific change, it would be difficult to implement it if Mongodb doesn't already support it as a part of Atlas.
Think running your application on EC2 vs buying a server on-premise and running your application on that.
Being a managed service, costs might also be higher if your database does not see much traffic.
HOSTING yourself: You can get one or more AWS ec2 instances(which are VMs) where you can install and run Mongo DB yourself and manage it like you wanted to, making sure that you spin up more instances when the workload becomes large and there are instances up and running at all times to enable high availability.
Cost (high) - Management responsibilities (lots) - Full MongoDB functionality
MongoDB Atlas is a managed service, you don't need to worry about management tasks like scaling of your database and high availability when a single/more instances die... You pay a very low cost for it - this is run by MongoDb themselves on AWS, Azure, Google cloud;
Cost (low) - Management responsibilities (some) - Full MongoDB functionality
Now AWS has its own Mongo compatible database called DocumentDB - this is also a managed database, so you don't need to worry about scalability, high availability etc. This is only available on AWS so super simple and convinient.
Cost (low) - Management responsibilities (minimal) - Limited MongoDB functionality
The idea here is, I have mongo cluster deployed in managed cloud service atlas. I have enabled Continuous Backup.
Now what I want to do is :
1) I want to use existing backup.
2) Using this existing backup I want to create similar cluster
(having same data form backup)
3) Automate this process so that every day my new cluster gets upto date from original cluster.
Note: The idea here for cloning cluster is, The original cluster is production data. I want to create a db which has similar data on which I can plug and play using any analytic tools and perform diffrent operations without affecting production data and load.
So far what I have found is to use mongorestore and mongodump.But here mongodump is putting load on production db even though my backup is enabled. I want to use same backup to clone this to another db cluster.
Deployed on Atlas, your server must have replica set.
Here are 2 solutions :
You need only reading data : connect your tools to a secondary server (ideally dedicated with priority 0 for becoming primary)
You need to read/write data : on the same server than above, play your mongodump command with --oplog option. By this way, you're dumping your data from a read-only server, preventing slowing performances of your main servers.
In this last case, what you need will find its solution in backup strategies, take a look at the doc to know more.
There's an offering for this purpose in ATLAS called analytic node.Link.
Analytic node is read replica of your database. Plus it will not interfere with your production traffic which makes it safer.
Also, you can connect BI connectors to this node and create your analytic platform.
We used redash.
We are at early stages with running our services on AWS. We have our server hosted in AWS, in a VPC, having private and public subnets and have multiple instances in private and public subnets using ELB and autoscaling setup (using AMIs) for frontend web servers. The whole environement(VPC, security groups, EC2 instances, DB instances, S3 buckets, cloudfront) are setup manually using AWS console at first.
Application servers host jboss and war files are deployed on the servers.
As per AWS best practices we want to create whole infrastructure using cloudformation and have setup test/stage/prod environment.
-Would it be a good idea to have all the above componenets (VPC, security groups, EC2 instances, DB instances, S3 buckets, cloudfront etc) using one cloudformation stack/template? Or we should we create two stacks 1) having network replated components and 2) having EC2 related components?
-Once we have a prod envoronemtn running with cloudformation stact and In case we want to update the new AMIs on prod in future, how can we update the live running EC2 instances using cloudformation without interruptions?
-What are the best practices/multiple ways for code deployment to multiple EC2 notes when a new release is done? We dont use Contineus integration at the moment.
It's a very good idea to separate your setup into multiple stacks. One obvious reason is that stacks have certain limits that you may reach eventually. A more practical reason is that you don't really need to update, say, your VPC every time you just want to deploy a new version. The network architecture typically changes less frequently. Another reason to avoid having one huge template, or to make changes to an "important" template needlessly, is that you always run the risk of messing things up. If there's an error in your template and you remove an important resource by accident (e.g. commented out) you'll be very sorry. So separating stacks out of sheer caution is probably a good idea.
If you want to update your application you can simply update the template with the new AMIs and CFN will know what needs to be recreated or updated. You can read about rolling updates here. However, I'd recommend considering using something a bit more straightforward for deploying your actual code, like Ansible or Chef.
I'd also recommend you look into Docker for packaging and deploying your application's nodes. Very handy.