So I have a production sharded MongoDB cluster that has 8 shards (replica sets) managed by mongos. Let's say I have 20 servers which are running my application and each of the servers runs a mongos process that manages the 8 shards.
Given this setup, when I check the number of ops on each of my mongos on the 20 servers, I can see that my number of inserts and deletes are in proportion - which is in accordance with my application logic. However, when I run mongostat --discover on the individual shards, I see that deletes are nearly 4x the number of inserts which violates both my application logic as well as the 1:1 ratio indicated by mongos. Straightforward intuition supports that mongos would write to only one shard and so the average ratio of inserts and deletes across individual shards should be the same as that on mongos (which the application directly writes to) unless mongos does something different internally with the shards.
Could anyone point me to any relevant info on why this would happen or let me know if something could possibly wrong with my infra?
Thanks
The reason for this is that I was running the remove() queries to mongos without specifying my shard key. In that case, mongos does not know which shard to direct the query to and thus broadcasts the query to all the shards effectively performing more deletes than a targeted query.
Check documentation for more information.
Related
From the mongodb docs:
mongos uses "majority" for the write concern of the shardCollection command and its helper sh.shardCollection().
In replica sets majority is Nodes/2 and round up.
Is it the majority of the config servers or what exactly?
I guess mongo doesn't specify very specifically what happens. If you dug one more page down, Write Concern, you'll read that "In sharded clusters, mongos instances will pass the write concern on to the shards."
Assuming your shard cluster is also a replica set e.g. P-S-S (primary-secondary-secondary) it should follow the same behavior as if you only had a single unsharded replica set. "For this replica set, calculated majority is two, and the write must propagate to the primary and one secondary to acknowledge the write concern to the client". The client here technically being mongos.
The other shard clusters do not need to acknowledge the write if they are not being written to. If you are writing to multiple shard clusters, then I'd assume both shard clusters need to ack the write similarly.
I know this may sounds like a naive question, but I don't seem to see much on google. Is it ok to insert / upsert documents into a mongodb (3.6) database, while sharding is going on (i.e. the chunk balancer is running)?
Thanks
The documentation is pretty clear on this:
Sharding improves concurrency by distributing collections over
multiple mongod instances, allowing shard servers (i.e. mongos
processes) to perform any number of operations concurrently to the
various downstream mongod instances.
In a sharded cluster, locks apply to each individual shard, not to the
whole cluster; i.e. each mongod instance is independent of the others
in the sharded cluster and uses its own locks. The operations on one
mongod instance do not block the operations on any others.
So, yes, you can run any query against a cluster at any time. This should be completely transparent to your client and MongoDB will internally manage potentially required locks.
I have a problem on Sharded Cluster. I'm testing performance to compare between Sharded and Replica Set.
I have inserted data to Shard 1 directly without mongos and then query it by aggregate query but I cannot found it. I checked in explain plan that shows "SHARDING_FILTER" in stage on Primary shard but doesn't have that in Secondary when I checked explain plan.
What's configuration to control about it?
MongoDB version : 3.0.12
I have inserted data to Shard 1 directly without mongos and then query it by aggregate query but I cannot found it.
It's not entirely clear what your performance comparison is, but irrespective you should always interact with data via mongos for a sharded cluster.
The role of mongos includes keeping track of the sharded cluster metadata (as cached from the config servers), observing data inserts/updates/deletions, and routing requests. Bypassing mongos will lead to potential complications in collection/data visibility (as you have observed) because you are skipping some of the expected data management infrastructure for your sharded deployment.
I checked in explain plan that shows "SHARDING_FILTER" in stage on Primary shard but doesn't have that in Secondary when I checked explain plan.
Secondary reads are eventually consistent, so the state of data on a given secondary may not necessarily match the current sharded cluster metadata. This becomes more problematic with many shards: with a secondary read preference results can potentially be combined from secondaries with significant differences in replication lag.
For consistent queries for a sharded cluster you should always use primary reads (which is the default behaviour) via mongos. Queries against primaries through mongos may include a SHARDING_FILTER stage which filters result documents that are not owned by the current shard (for example, due to migrations in progress where documents need to transiently exist on both a donor and target shard).
As at MongoDB 3.4, secondaries do not have the ability to filter results because they'd need to maintain a separate view of the cluster metadata which matches their eventually consistent state. There's a relevant Jira issue to watch/upvote: SERVER-5931 - Secondary reads in sharded clusters need stronger consistency. I currently would not recommend secondary reads in a sharded cluster (or in general) without careful consideration of the impact of eventual consistency on your use case. For the general case, please read Can I use more replica nodes to scale?.
What's configuration to control about it?
Use the default read preference (primary reads) and always interact with your sharded deployment through mongos.
When I put shard and Replica Set together, I am confused.
Why does the reference say that the shards are replica sets?
Do replica sets contains shards?
Can someone give me a conceptual explanation?
Replica Set is a cluster of MongoDB servers which implements Master - slave implementation. So, basically same data is shared between multiple replica i.e Master and Slave(s). Master is also termed as primary node and Slave(s) is/are considered as Secondary nodes.
It replicates your data on multiple mongo instances to solve/avoid fail overs. MongoDB also perform election of Primary node between secondary nodes automatically whenever Primary node goes down.
Sharding is used to store large data set between multiple machines. So basically, if you simply wants to compare Sharded nodes doesnt/may not contain same data where as Relicated nodes contains same data.
Sharding has different purpose,large data set is spread accross multiple machines.
Now, this large data set's subset can also be replicated to multiple nodes as primary and secondary to overcome failovers. So basically a shard can have multiple replica-set. These replica set of a shard contains subset of data for a large data set.
So, multiple shards can complete the whole large data set which are separated in the form of chunks. These chunks can be replicated within a Shard using Replica set.
You can also get more details related to this in MongoDB manual.
Sharding happens one level above replication.
When you use both sharding and replication, your cluster consists of many replica-sets and one replica-set consists of many mongod instances.
However, it is also possible to create a cluster of stand-alone mongod instances which are not replicated or have only some shards implemented as replica-sets and some shards implemented as stand-alone mongod instances.
Each shard is a replica set, not the shards are replica sets.
This is a language barrier, in English to say such a thing really means the same as "each shard is a replica set" in this context.
So to explain, say you have a collection of names a-z. Shard 1 holds a-b. This shard is also a replica set which means it has automated failover and replication of that range as well. So sharding in this sense is a top level term that comes above replica sets.
Shards are used to break a collection and store parts of it in different places. It is not necessary that a shard be a replica set, it can be a single server, but to achieve reliability and avoid loss of data, a replica set can be used as a shard instead of a single server. So, if one of the servers in the replica set goes down, the others will still hold the data.
I got a mongodb running online, which contains about 200 million object, and the file size is about 20GB. And, I found that the insert speed become very slow (about 2000 per sec, and this value is more than 10000 in the beginning). So I decide to divide the data to optimize the insert speed.
I would like the know if i can divide the mongodb data without stopping service, and how?
You just described "sharding". Luckily for you, MongoDB has nice sharding features out of the box.
Your migration will consist of:
Create 3 mongo config servers
Create more than 1 mongos router
Add your current replica set as a shard
Point your application to connect to your mongos
Configure shard keys your current collections
Then, you are set to add shards as needed
For detailed instructions, see 10gen's sharding overview
At MongoHQ, we convert replica sets to shards all the time. So, should be quick, painless, and without downtime.