I am searching for tool, which should provide collections overview, queries to them, replicas configurations, instance performance dashboards.
You can look for zabbix as elementary monitoring. However for querying the DB for data you need to use custom shell scripts or metric reporting tools to get that information
Related
We have a company with a HDP 2.6.4 cluster and an outsource / offshore team that handles different ops tasks. Due to quite strict data access policies, that cannot have access to quite a few datasets. However, we do need them to be able to 24/7 monitor and (ideally) execute different jobs.
So I'm in position as someone in Big data team to enable them to do so, but without access to data. Not being sure what would HDP 2.6 have to offer, I do know that there are certain tools that enable devs to develop all kind of API endpoints, which could then be mapped to different ETL jobs / shell scripts etc.
Would this be optimal approach from an architectural standpoint?
I was thinking of getting us something like Dreamfactory, but opensource and something I can run on premise. Any ideas?
Cheers!
DreamFactory can generate REST APIs for a multitude of databases, among them MySQL, Microsoft SQL Server, Oracle, PostgreSQL, and MongoDB. Along with the API, DreamFactory will also auto-generate an extensive set of interactive Swagger documentation for your API. Are you looking to connect databases?
I know that we can monitor the infrastructure and OS level metrics using Newrelic's integration with AWS. But how can we monitor the queries and DB level parameters using newrelic.
This feature was requested but it's not implemented by new relic yet, which basically pulling rds performance insights to newrelic
https://discuss.newrelic.com/t/add-rds-performance-insights-data-to-aws-integration/60821
I am building my own webapp which requires a huge database. I want to build and manage my own Mongo database on AWS rather than using Mongo Atlas. Which will be more cost saving? And whether I should go for Mongo Atlas? What will be its advantage over my own database?
There are pros and cons for both approaches:
Running MongoDB on AWS
Pros:
Complete control over how you run the database and how resources are allocated on the server. This could even be together with an application server on the same EC2 instance depending on your traffic and load. This might help with cost saving if your database is huge but isn't likely to see much traffic.
Cons:
You will be responsible for ensuring database availability and applying security patches as and when they are available. You may also have to setup firewalls and protect the EC2 instance and database in other ways that would be trivial to do on a hosted service like Atlas.
Data sharding and clustering can be a real pain to manage by yourself.
Running on Atlas
Pros:
Completely managed service where you don't have to be concerned about performance optimization or scalability. You pay for the services and Mongodb takes care of the rest.
You can focus on building a great application instead of spending your time on administering the database and the EC2 instance on which the database runs.
Cons:
You will be constrained by the options offered by Atlas. For most use cases this should be fine, but if you really want a specific change, it would be difficult to implement it if Mongodb doesn't already support it as a part of Atlas.
Think running your application on EC2 vs buying a server on-premise and running your application on that.
Being a managed service, costs might also be higher if your database does not see much traffic.
HOSTING yourself: You can get one or more AWS ec2 instances(which are VMs) where you can install and run Mongo DB yourself and manage it like you wanted to, making sure that you spin up more instances when the workload becomes large and there are instances up and running at all times to enable high availability.
Cost (high) - Management responsibilities (lots) - Full MongoDB functionality
MongoDB Atlas is a managed service, you don't need to worry about management tasks like scaling of your database and high availability when a single/more instances die... You pay a very low cost for it - this is run by MongoDb themselves on AWS, Azure, Google cloud;
Cost (low) - Management responsibilities (some) - Full MongoDB functionality
Now AWS has its own Mongo compatible database called DocumentDB - this is also a managed database, so you don't need to worry about scalability, high availability etc. This is only available on AWS so super simple and convinient.
Cost (low) - Management responsibilities (minimal) - Limited MongoDB functionality
can anyone please give me a high level difference between MongoDB Cloud Manager and Mongodb Atlas. My main aim is to monitor mongodb instances in AWS.
Thanks.
Cloud Manager is used when you want to manage your own infrastructure (you spin up the nodes where MongoDB runs) but still have the benefits of automated backups and monitoring.
Atlas goes one step further by automating everything for you including provisioning the infrastructure. It's a true database as a service fully managed by MongoDB. They hide the complexity of managing servers so all you have to worry about it using MongoDB. It's interesting to note they use AWS (with plans to support Azure and Google) to spin up nodes, perform monitoring, and backups.
The Major difference between Atlas and Cloud manager is that :
Cloud manager is used for monitoring your database deployment and providing the automated back ups in the self hosted environment.
While MongoDB Atlas is used when your deployments are hosted on the MongoDB Servers. So each and ever task is managed by the MongoDB staff. This is basically the database as a service. In case you encounter any issue all you need to open a case with the mongodb and they will help in the investigations of the issue occurred.
Here is an up-to-date answer to this question which explains differences between Atlas, Cloud Manager and also the Ops-Manager:
MongoDB Atlas handles all the complexity of deploying, managing, and healing your deployments on the cloud service provider of your choice (AWS, Azure, and GCP). Atlas pricing details are here 4.
Cloud Manager is a platform for managing MongoDB on the infrastructure of your choice. Cloud Manager pricing details are here 7.
Ops Manager automate, monitor, and back up your MongoDB infrastructure.
Here is the original article and additional resources in the MongoDB community forum: https://www.mongodb.com/community/forums/t/cloud-manager-vs-ops-manager-vs-atlas/42624
I m developing an application using SignalR to manage websockets and allow my clients to dialog between each other.
I m planning to host this back-office on an Azure worker role. As my SignalR requests carry data that is most of the time saved in the database, I m wondering if NoSQL's MongoDB instead of the classic SQL Server/Entity Framework couple should be a good approach.
Assuming that my application's data types will be strings for most of them, I think MongoDB will be a reliable and a performant solution, and it will allow me to get rid of Azure's SQL's database costs.
For information, the Azure worker role will be running on a machine with the following hardware: 1 core CPU, 3.5GB RAM and 50GB SSD storage.
Do you think I m on a good start with this architecture ?
Thanks
Do you think I m on a good start with this architecture?
In a word, no.
A user asked a similar question regarding running Redis on Worker Roles - Setting up Redis on Azure cloud service worker role - all of the content on that Q/A is relevant in the MongoDb context.
I'd suggest that you read my answer as it goes into more detail, but as an overview of why this is a bad architectural approach:
You cannot guarantee when a Worker Role will be restarted by the Azure Service Fabric.
In a real-world implementation of Mongo, you would run multiple nodes within a cluster, with a single Worker Role (as you have suggested in your question) this won't be possible.
You will need to manage your MongoDb installation within the Worker Role and they simply aren't designed for this.
If you are really fixed on using Mongo, I would suggest that you use a hosted solution such as MongoLabs (as suggested in earlier answers), or consider hosting it on Azure IaaS VM's.
If you are not fixed on using Mongo, I would sincerely suggest that you look at Azure DocumentDb (also suggested above), Microsoft's Azure NoSQL offering - I have used it in several production systems already and it is certainly a capable NoSQL solution; granted, it may not have all of the features available with MongoDb.
If you are looking at a NoSQL solution for caching of data (i.e. not long term storage), I would suggest you take a look at Azure Redis Cache, which is a very capable Redis offering.
Azure has its own native NoSQL Document database called DocumentDB, have you had a look at it? If I were you I would use DocumentDB unless there are some special requirements that you have that you have not mentioned, but from what little requirement info that you have posted DocumentDB would do just fine. I don't think that it is quite similar to MongoDB in terms of the basic functionality, see this article for a comparison between Azure DocumentDB and MongoDB.