I come across the following phase and I had fair idea of distributed systems like hdfs and elasticsearch etc ..
A replica set in MongoDB is a group of mongod processes that maintain
the same data set. Replica sets provide redundancy and high
availability, and are the basis for all production deployments. This
section introduces replication in MongoDB as well as the components
and architecture of replica sets. The section also provides tutorials
for common tasks related to replica sets.
In all those distributed systems like hdfs , elasticsearch - we can set the replication factor at the file level or index level. It seems it is not possible to do with mongo db, the only way to do the replication with mongodb is at the instance / process level - which means the machines in the replicaset group will have similar data no matter what.
Isnt it possible to create the replication at the db level ?
In a MongoDB replica set, each document is stored on each node (hence the replication factor is the number of nodes, and it is not configurable).
The benefit of this design is each node can answer any query from its locally stored data, without needing to retrieve data from other nodes.
Related
Currently we are working with standalone mongodb without any replication or sharding, Now we are considering moving to replica-set for production purposes.
Will an application written for standalone mongodb will work for replica-set or sharded replica-set without any changes or are there some standalone/replica-set specific features in mongodb ?
Provided the MongoDB uses the default ports (27017 for standalone mongod and mongos) you don't need to touch your client application at all, it will work in either case.
Of course, when you connect to a MongoDB then a sharded cluster has more options, but the defaults are fine.
Will an application written for standalone mongodb will work for
replica-set or sharded replica-set without any changes or are there
some standalone/replica-set specific features in mongodb ?
Here are some things to think about when an application is to run on a replica-set or a sharded cluster. In addition, replica-sets and sharded clusters has some features not available in standalone deployment (see the Transactions and Change Streams topic at the bottom).
Replica Sets
A replica-set is cluster with multiple database servers - with replicated data on each server. The topology of the replica-set has one primary node (or member) and remaining members are secondaries (there can be other special purpose nodes like arbiters).
The data redundancy and failover features of replica-sets give your applications additional capabilties - for example, an application always runs even if a server is down.
The data is always written to the primary and read from it, by default. You can configure that the data can be read from the secondary nodes also from your application - this the Read Preference. This configuration can be used by the applications accessing a replica-set in some scenarios (see Read Preference Use Cases). This is for replica-sets and has no usage for standalone deployment.
Also, see Replica Set Read and Write Semantics:
From the perspective of a client application, whether a MongoDB
instance is running as a single server (i.e. “standalone”) or a
replica set is transparent. However, MongoDB provides additional read
and write configurations for replica sets.
Then, there are some things like, the Connection String URI, which uses different format for replica-set and sharded clusters - this is used by the applications to connect.
Sharded Cluster
The application should not be run in sharded cluster deployment as it is. It will require design level changes - and will affect the queries. Sharding is about distributing the data among shards. Note that in sharded cluster each shard is a replica-set. A sharded database can have sharded and un-sharded collections. Sharded collections are the distributed data.
To create a sharded collection, you must figure a shard key - this is the most important aspect of your application accessing a sharded collection. Shard key determines how the queries access particular shard to get the data. So, your application must take into consideration the shard key - the queries need to be created with shard key usage. Shard key affects the performance of your application queries, primarily.
Also, in the sharded cluster environment the application accesses the database via a mongos router - not the servers directly.
There are many other finer aspects when working with sharded databases and accessing for applications - the topic is too broad to discuss here. Changing from standalone to sharded cluster is an architectural change. Some aspects that can affect the application due to migrating from standalone to a replica-set also apply here (as each shard is a replica-set).
Also, see Operational Restrictions in Sharded Clusters - these are specific to sharded clusters and not applicable to standalone deployments.
Transactions and Change Streams
Features like transactions and change streams are available with replica-sets and sharded clusters only (and not on single standalone servers). This gives additional capabilites to your applications and can solve complex business logic and scenarios.
As per MongoDB documentation, transactions only works for replica sets and not single node. Why such requirement? Isn't it is easier to do transaction stuff on a single node rather than a distributed system?
Implementation of transactions uses sessions which in turn require an oplog. Oplog is provided by replica sets for data synchronization between nodes.
Isn't it is easier to do transaction stuff on a single node rather than a distributed system?
This is true but in practice, MongoDB positions itself as a high-availability database therefore there are rather few production deployments using a standalone server (as far as I know this isn't even an option in Atlas, for example). Hence lack of transaction support on standalone servers typically doesn't affect anything.
Conversely, implementing transactions only on standalone servers would not address the needs of the vast majority of MongoDB deployments/customers that use replica sets and sharded clusters.
For development purposes you can run a single-node replica set which gives you an oplog required for sessions and transactions but still only one mongod process.
I have a few questions about MongoDB standalone and Replica sets, I don't really get it.
When should I use either of them
Why all the replica sets tutorials show 3 connections, is there a reason?
Can I create a replica set for 1 instance only? and in that case how is it different than the standalone mongodb instance?
How to Migrate data from a standalone instance to replica sets?
All these questions I'm asking because recently I was trying to implement transactions and sessions can only start on "replica sets" I don't really get the difference at all.
When should I use either of them?
Replication is the process of synchronizing data across multiple servers. Replication provides redundancy and increases data availability with multiple copies of data on different database servers. Replication protects a database from the loss of a single server. Replication also allows you to recover from hardware failure and service interruptions. With additional copies of the data, you can dedicate one to disaster recovery, reporting, or backup.
To keep your data safe
High (24*7) availability of data
Disaster recovery
No downtime for maintenance (like backups, index rebuilds, compaction) Read scaling (extra copies to read from)
Replica set is transparent to the application
Why all the replica sets tutorials show 3 connections, is there a reason?
The basic implementation to take full advantage of replication specifies you
should have at least one primary node with two secondary nodes. So the
examples are always with 3 nodes. Not only this if from 3 the
Primary node goes down you still have 2 nodes (mongoDB will assign
using arbiter rule) and one primary and one secondary for high availability
Can I create a replica set for 1 instance only? and in that case how is it different than the standalone mongodb instance?
It does not make sense to have single instance with mongo replication.
How to Migrate data from a standalone instance to replica sets?
Convert a Standalone to a Replica Set . Your existing data will be migrated to all replication instances once they are up and running when converted to replication sets from standalone.
Technically is it supported to begin with just one shard for a shard cluster? So we can be ready for adding additional one(s) at anytime, at the same time save the cost of additional shard(s) before we really need it(them)?
To go further, is it possible to have a shard running on one single instance, instead of having to be based off of a 3 instance replica set?
From here, sharding is:
A database architecture that partitions data by key ranges and
distributes the data among two or more database instances.
A shard will be either a replica set or a standalone mongod instance. It is possible for you to use a single machine by using different ports to establish distinct communication endpoints for the config, mongod and mongos processes on the single machine. Also, yes, you may add a shard at a later time when you need to expand.
However, the point of providing sharding is to support horizontal scaling. Additionally, the point of sharded clustering is to provide failover and redundancy support. By using a single shard on a single server, you are losing the benefits of scaling and certainly failover.
The recommended production architecture includes:
Three config servers on separate machines for each sharded cluster.
Two or more replica sets as shards.
One or more query routers (mongos); typically, one mongos instance per application server.
Peruse the Sharded Cluster Requirements section in the documentation to get a feel for whether or not your environment needs sharding and sharded clusters since there is complexity in establishing such an architecture.
I would like to create a development project on EC2 cluster. Current design suggest accessing mongo database files stored on EBS volume. If that is possible to run distributed computing and access same files in /data/db/ simultaneously from different nodes?
No, that will not work. You cannot access the same mongodb database files from different processes on different nodes.
The way you use mongoDB in a distributed environment is with replica sets and sharding. In both cases you have mongodb instances running on each node. Replica sets duplicate the same data across all the nodes in the set, for data redundancy and fault tolerance. Sharding allows you to distribute different sets of data on different nodes to provide horizontal scaling. Large production environments use both replica sets and sharding.
Best place to read up on all of this is:
http://docs.mongodb.org/manual/administration/replica-sets/
http://docs.mongodb.org/manual/sharding/
http://docs.mongodb.org/ecosystem/platforms/amazon-ec2/