We are trying to setup microservice architecture on Amazon ECS using Fargate. When it comes to database, we are not able to spin up instance of mongodb. The database automatically switches off after 3 minutes.
The log states
{"t":{"$date":"2021-03-15T15:34:17.913+00:00"},"s":"I", "c":"REPL", "id":4784900, "ctx":"SignalHandler","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
My question is ,
a) What could be the possible reason of auto shutdown of the db after 3 minutes?
b) Is this the right approach to spin up database in Amazon fargate ? Or is there a better way to achieve the same ?
I'm not certain about the core MongoDB issue. My first guess would be a failing or misconfigured health check on ECS; if this were the case, the health check would appear in your ECS event history (in the (new) UI: Cluster > Select the service > Notifications at the bottom).
If there are no bad health check notifications, then that would rule this out, and more information may be necessary to fully diagnose, such as the full ECS/Fargate Service/Task configuration.
Generally, and more to your second question, running databases on Fargate is not a recommended use-case. Fargate is a better fit for stateless services, like web APIs, which are more tolerant to being stopped and started frequently, and receiving different IP addresses upon each start. Within AWS, MongoDB would be a better fit on simple EC2, or via their DocumentDB MongoDB-like managed service. There are also several MongoDB managed hosting providers which can provide the low-management, serverless feel of Fargate, like MongoDB Atlas.
Related
CONTEXT: I have been learning Kubernetes and trying to get some hands-on experience. I have been using AKS to abstract the complexity of having to deal with the control plane (and because I have a free student azure account). I am deploying a NodeJS app that connects to the MongoDB database. So far the deployment has been successful but I am using MongoDB Atlas and connecting to it.
Based on the little I have learned about Stateful sets, the MongoDB Atlas service seems a lot easier and more convenient but my question is, when would it be a better idea to consider deploying a stateful set with MongoDB database? (running on the pod) What's more cost-effective? More easily scalable?
I realize the questions might be a little bit vague but I am just getting started with Kubernetes..
disclaimer: This is not a production application, just something simple I am using to learn K8S
Official docs docs uses statefullset and that would make sense. Generally all DB kind of applications deployed as statefullset. Because there can be states that nodes are not sync with each other and that would create data inconsistencies between nodes(mongodb nodes not kubernetes).
You can deploy MongoDB as deployment. I have seen it deployed. But most clients use a connection string to connect(a string of multiple node addresses). And since kubernetes exposes statefullsets with headless services you should be okay.
For learning purpose, I advice you to deploy your MongoDB in a StatefulSet. Then you can learn how it works and what problem you could encounter with this Kubernetes object.
For production application, I advice to never deploy a database in a StatefulSet if you don't need it. In fact, StatefulSet will come with a lot of problematics that you might not need to manage.
Sometimes, companies rules restrict to host their data on external company storage.
To know if you need to put your database in a StatefulSet, the question I try to answer is:
Should my DB be hosted on premise (for privacy)?
Should my DB be scalable?
Should my DB be updated frequently?
You can find a list of pros/cons on the documentation.
I am building my own webapp which requires a huge database. I want to build and manage my own Mongo database on AWS rather than using Mongo Atlas. Which will be more cost saving? And whether I should go for Mongo Atlas? What will be its advantage over my own database?
There are pros and cons for both approaches:
Running MongoDB on AWS
Pros:
Complete control over how you run the database and how resources are allocated on the server. This could even be together with an application server on the same EC2 instance depending on your traffic and load. This might help with cost saving if your database is huge but isn't likely to see much traffic.
Cons:
You will be responsible for ensuring database availability and applying security patches as and when they are available. You may also have to setup firewalls and protect the EC2 instance and database in other ways that would be trivial to do on a hosted service like Atlas.
Data sharding and clustering can be a real pain to manage by yourself.
Running on Atlas
Pros:
Completely managed service where you don't have to be concerned about performance optimization or scalability. You pay for the services and Mongodb takes care of the rest.
You can focus on building a great application instead of spending your time on administering the database and the EC2 instance on which the database runs.
Cons:
You will be constrained by the options offered by Atlas. For most use cases this should be fine, but if you really want a specific change, it would be difficult to implement it if Mongodb doesn't already support it as a part of Atlas.
Think running your application on EC2 vs buying a server on-premise and running your application on that.
Being a managed service, costs might also be higher if your database does not see much traffic.
HOSTING yourself: You can get one or more AWS ec2 instances(which are VMs) where you can install and run Mongo DB yourself and manage it like you wanted to, making sure that you spin up more instances when the workload becomes large and there are instances up and running at all times to enable high availability.
Cost (high) - Management responsibilities (lots) - Full MongoDB functionality
MongoDB Atlas is a managed service, you don't need to worry about management tasks like scaling of your database and high availability when a single/more instances die... You pay a very low cost for it - this is run by MongoDb themselves on AWS, Azure, Google cloud;
Cost (low) - Management responsibilities (some) - Full MongoDB functionality
Now AWS has its own Mongo compatible database called DocumentDB - this is also a managed database, so you don't need to worry about scalability, high availability etc. This is only available on AWS so super simple and convinient.
Cost (low) - Management responsibilities (minimal) - Limited MongoDB functionality
can anyone please give me a high level difference between MongoDB Cloud Manager and Mongodb Atlas. My main aim is to monitor mongodb instances in AWS.
Thanks.
Cloud Manager is used when you want to manage your own infrastructure (you spin up the nodes where MongoDB runs) but still have the benefits of automated backups and monitoring.
Atlas goes one step further by automating everything for you including provisioning the infrastructure. It's a true database as a service fully managed by MongoDB. They hide the complexity of managing servers so all you have to worry about it using MongoDB. It's interesting to note they use AWS (with plans to support Azure and Google) to spin up nodes, perform monitoring, and backups.
The Major difference between Atlas and Cloud manager is that :
Cloud manager is used for monitoring your database deployment and providing the automated back ups in the self hosted environment.
While MongoDB Atlas is used when your deployments are hosted on the MongoDB Servers. So each and ever task is managed by the MongoDB staff. This is basically the database as a service. In case you encounter any issue all you need to open a case with the mongodb and they will help in the investigations of the issue occurred.
Here is an up-to-date answer to this question which explains differences between Atlas, Cloud Manager and also the Ops-Manager:
MongoDB Atlas handles all the complexity of deploying, managing, and healing your deployments on the cloud service provider of your choice (AWS, Azure, and GCP). Atlas pricing details are here 4.
Cloud Manager is a platform for managing MongoDB on the infrastructure of your choice. Cloud Manager pricing details are here 7.
Ops Manager automate, monitor, and back up your MongoDB infrastructure.
Here is the original article and additional resources in the MongoDB community forum: https://www.mongodb.com/community/forums/t/cloud-manager-vs-ops-manager-vs-atlas/42624
Would really appreciate some suggestions for resources on how to properly deploy with Elastic Beanstalk with the following stack:
MongoDB
Rails (Puma)
Sidekiq/Redis
Elasticsearch
Do I need to get all these things setup in ebextension files? Or is it a matter of settings things up manually in AWS and then routing them together properly somewhere?
You definitely don't want to run all those on your Elastic Beanstalk servers. Elastic Beanstalk will automatically add or remove servers based on your traffic/server load. You don't want your database to be on one of those servers when it gets deleted.
Elastic Beanstalk is a Platform as a Service that is great for running web servers. There are other services on AWS such as ElastiCache (Redis/Memcached as a service) and Elasticsearch as a service. There are also third parties that provide services that run on AWS such as RedisLabs (Redis as a service) and MongoLab (MongoDB as a service).
You can decide to use any of these services to reduce the amount of system administration work you have to do yourself. Or you can manually setup EC2 Linux servers (outside of Elastic Beanstalk) and install things like Rails and MongoDB and ElasticSearch on them and manage them yourself.
For your case I would recommend something like the following:
Rails: ElasticBeanstalk
MongoDB: MongoLab
Redis: RedisLabs
Elasticsearch: AWS Elasticsearch Service
You would want to setup each of those services and then simply add the connection information for each of them to your Elastic Beanstalk environment so Rails can use them.
Edit:
Here are the best instructions on setting up MongoDB on EC2 manually: https://docs.mongodb.org/ecosystem/platforms/amazon-ec2/
For ElastiCache and Elasticsearch, you just click around in the AWS console to provision a Redis server and get the URLs to connect to. Once you have set all these things up, you just need to put the connection parameters in your ElasticBeanstalk environments as custom environment variables, something like:
MONGO_DB_URL="Your MongoDB EC2 internal IP address"
REDIS_URL="the url ElastiCache provided you"
Then read those environment variables in your application when creating connections to those services.
Also, you are going to have to learn about setting up your VPN and security groups to enable everything to connect. For example you will want your Elastic Beanstalk servers in one security group, and MongoDB server(s) in another group. Then you will have to configure the MongoDB security group to allow access from the beanstalk group on the MongoDB port. It's similar for ElastiCache. I think for Elasticsearch you will have to create an IAM role with access to the Elasticsearch API, and then assign that role to your Beanstalk servers.
Of course there is also the administrative tasks of setting up Linux servers for your MongoDB cluster, configuring clustering, fail-over, automated backups, log archives, periodic security updates, etc. I know you have all this AWS credit, but you should weigh moving everything over to AWS versus the cost of all the administrative tasks you will be spending time on. Elastic Beanstalk, Elasticsearch and ElasticCache are a no-brainer if you are getting them for free, but my MongoLab bill would have to be fairly high to justify setting all that up and managing it myself.
Was wondering if anyone out there has any experience in deploying a Zend community app to the cloud (e.g. AWS or similar)?
I'm new to cloud hosting having always been fortunate enough in the past to work for folks who have dedicated servers, my main concern (non-zend specific) is how you manage resilience at the database level? FOr example I would in a traditional setup have 2 boxes running the DB (Mysql) in Master/Slave mode with the master replicating to the slave. Assuming any HD failure of the Master I could swap the DB connection over from the Master to the slave and rebuild master at a later point? is this done differently in the cloud?
Any help/pointers greatly appreciated?
It depends on the type of cloud service that you use. If you're using AWS to get your own virtual machine ( Amazon EC2 ) then it's basically the same as having a dedicated server and you can keep a master slave setup and work them much the same way.
However, if you plan on using Amazon's cloud database service ( Amazon Simple DB ) then you don't have to worry about masters and slaves since Amazon does this for you and makes sure that you always have access to your data. The only thing is that it's in beta.
One of the points of the cloud is to take your mind off the hardware. Amazon worries about that.
You might still want to have two virtual machines in case amazon is doing maintenance that might cause your vm to become unavailable, however, Amazon stresses that it would be highly available and never go down really, so long as you pay.