I have a collection with around 30 thousands documents. My security rules are configured so that only authenticated users can read such documents from this collection. I cannot add more constraints regarding read operations for this specific collection.
When using my app, up to 50 documents are averagely returned, depending on the query.
Is there any way to prevent a malicious user to download the entire collection in Firestore?
To limit the number of documents a user can read at once, you can include a limit in your security rules as shown in the documentation on securely querying data:
allow list: if request.query.limit <= 50;
Keep in mind that rules are not filters, so the application code will also need to include this (or a lower) limit in its code.
Related
I using Firestore for my flutter's app. I have two collections with with several hundred documents (up to 600). I need to notify the application user when a document changes. first I used this code:
_collectionRef.snapshots(includeMetadataChanges: true).listen((event) {
event.docChanges.forEach((change) {
if (change.type == DocumentChangeType.modified) {
print(change.doc.id);
}
});
but then I thought how optimal it is to keep track of hundreds of documents in two collections.
I got familiar with the best practices for Firestore. Found it:
Limit snapshot listeners per client 100
Keep the number of snapshot listeners per client under 100.
does this mean no more than 100 snapshots().listen in one application? did I understand it correctly? Or is it a limitation on the number of documents that the listener will listen to?
Is it normal practice to change hundreds of documents? Or is it better to change the data structure and limit the number of listened documents?
The best practice you quote is to limit the number of listeners per application to at most 100. So that is the number of times you have an active call to .listen, onSnapshot or similar APIs. This recommendation is not related to the number of collections or documents that you listen to, although there may be separate recommendations for that.
If you want to notify the user when there is a change in a collection, I typically add a lastUpdated timestamp to each document in that collection that you update on every write to that document. You can then use that field in queries, for example to get just the last modified document for the use-case you describe.
is there a way to view the reads/writes in mongodb on a per collection-basis? I would like to see how many documents have been read and written on a specific collection.
We are currently researching the costs of some specific queries and try to find out more about if they are heavy read- and/or write-tasks.
Thank you :-)
Yes you can. You can perform db.collection.stats()
this will return the size and count of documents, index information and a lot of other useful information. But you want to count the number of reads and writes performed on a specific collection. For that you can use mongostat. It captures and returns the counts of database operations by type (e.g. insert, query, update, delete, etc.). These counts report on the load distribution on the server. Read more about mongostat on their documentation. Here's the link https://docs.mongodb.com/manual/reference/program/mongostat/#bin.mongostat
In my application user gets to pick specific documents out of the list, for example: 1,5,8 from the list containing documents 1,2,3,4,5,6,7,8,9. When logged into the application next time, I want to first fetch all of the chosen documents (considering pagination, because the number of documents user picked could be very high), and then start fetching the remaining documents as the user finishes viewing picked documents by scrolling down the list.
As it turns out, available Firestore querying methods are not capable of skipping the specific documents.
My current idea:
Make single document references for the user-specific documents and fetch them.
Make single document references for the documents between the range of user-specific documents (From the example that would be documents number: 2,3,4,6,7).
After that start making 'big queries' for the remaining documents.
This looks like a working solution, but I'm sure that there is a better way to accomplish the goal, since what I've done is not asynchronous and very slow. Help is appreciated!
Firestore doesn't have any way to exclude specific documents from queries. You may only include them using some existing field values. If you already know the documents to fetch, you can just get() them individually.
It sounds like you are already able to work around these requirements. I don't believe you have any alternatives.
I got a collection Users, that is name, password, email etc.
Also i got a collection Groups, every group has it's members - array of users.
How should i design my database? I clearly see 2 ways of doing so:
Way 1 (MySQL-like): every user has an _id, so i just put it into the members array and so be it.
Way 2: copy a whole user document inside plus add some fields.
On the MongoDB site they are telling that duplicate data is nothing to worry bcs of the low price of storages. Also they say that we should avoid JOINs on data read.
duplicate data is nothing to worry about
This is something to worry about when it comes to updating. Suppose you have user details nested and duplicated in every document. What happens when a user changes their name? You'll have to update every instance of that user in every document.
Be careful to differentiate between data and entities. A user is an entity, think carefully before duplicating entities as fixing it later could be hard work.
Personally, I'd split them unless you find yourself in a situation where performance is too slow to do the joining in real time. Then, and only then, consider merging.
Actually answer to this question depends on what kind of screens you are designing and what kind of queries you are going to make to fetch data. Lets go through pros and cons of each option which will help you in weighing each option.
Way 1 :- Putting array of user_ids in group collection
Pros
1) If you have a screen which shows group details of a particular group and list of all members (users_ids) belonging to that group, then one query can fetch all the details needed for this screen and it would be faster too.
Cons
1) If in group detail screen, you have to show details of users along with group details, then since mongodb does not provide any joins, you would be fetching user details in a separate query and would be joining both on the client side. This can lead to a impact on performance.
2) If you have a screen which shows user details and all the groups he/she belongs, then you will be searching user_id in user array in group collection. If you are expecting number of members in a group to be very high(millions), then searching inside the array can lead to a huge performance impact.
Way 2 :- Copy user document inside inside group collection
Duplicating data is not a problem in Mongodb, but you should have a really good reason for that. Thumb rule should be duplicate data when relationship is 1:few and not 1:many.
Pros
1) This approach will save you from joining group and user collection at client side as one query can fetch all the details of group along with its users.
Cons
1) Suppose you have a million groups and user_id_1 belongs to 100,000 groups, then whenever you have an update on user_id_1, you will have to update 100,000 documents. This can again lead to huge performance impact.
2) Also if a large number of users subscribe to 1 group, then document size of this group keeps on increasing. In Mongodb The maximum BSON document size is 16 megabytes that means you cannot have a document greater than 16MB, so you cannot add users to a group infinitely. This will limit your functionality.
Way 3 :- Embed group details in user collection
Pros
1) One query can fetch user details along with all the details of all the groups this user belongs to.
2) If you have are expecting few users in a group, then you will have few group arrays in a user document. This will not exceed 16MB limit.
Cons
1) If you are expecting that a user can subscribe to a lot many groups(millions), then user document may exceed 16MB limit.
2) Also if you have very frequent updates in group details then you will have to update the same in many user documents.
You can also go through the following link to get more details about data model design :-
https://docs.mongodb.org/manual/core/data-model-design/
It depends on how you will use data in your application.
If you have more than 2 groups and you will have to search a user in all of the groups, embed the user document within the group (way 2) is not a good idea. So in this case I sugest to use the way 1.
If you have only 2 groups or the user group will be known before your application when doing the query, then use the way 2.
I guess that separating the data is the way to go, since it will be better to direct update, get and delete user data directly.
I have a basic question about where I should embed a collection of followers/following in a mongo db. It makes sense to have an embedded collection of following in a user object, but does it also make sense to also embed the converse followers collection as well? That would mean I would have to update and embed in the profile record of both the:
following embedded list in the follower
And the followers embedded list of the followee
I can't ensure atomicity on that unless I also somehow keep a transaction or update status somewhere. Is it worth it embedding in both entities or should I just update #1, embed following in the follower's profile and, put an index on it so that I can query for the converse- followers across all profiles? Is the performance hit on that too much?
Is this a candidate for a collection that should not be embedded? Should I just have a collection of edges where I store following in its own collection with followerid and followedbyId ?
Now if I also have to update a feed to both users when they are followed or following, how should I organize that?
As for the use case, the user will see the people they are following when viewing their feeds, which happens quite often, and also see the followers of a profile when they view the profile detail of anyone, which also happens often but not quite as much as the 1st case. In both cases, the total numbers of following and followers shows up on every profile page.
In general, it's a bad idea to embed following/followed-by relationships into user documents, for several reasons:
(1) there is a maximum document size limit of 16MB, and it's plausible that a popular user of a well-subscribed site might end up with hundreds of thousands of followers, which will approach the maximum document size,
(2) followership relationships change frequently, and so the case where a user gains a lot of followers translates into repeated document growth if you're embedding followers. Frequent document growth will significantly hinder MongoDB performance, and so should be avoided (occasional document growth, especially is documents tend to reach a stable final size, is less of a performance penalty).
So, yes, it is best to split out following/followed-by relationship into a separate collection of records each having two fields, e.g., { _id : , oid : }, with indexes on _id (for the "who am I following?" query) and oid (for the "who's following me?" query). Any individual state change is modeled by a single document addition or removal, though if you're also displaying things like follower counts, you should probably keep separate counters that you update after any edge insertion/deletion.
(Of course, this supposes your business requirements allow you some flexibility on the consistency details: in general, if your display code tells a user he's got 304 followers and then proceeds to enumerate them, only the most fussy user will check that the followers enumerated tally up to 304. If business requirements necessitate absolute consistency, you'll either need a database that isolates transactions for you, or else you'll have to do the counting yourself as part of displaying all user identities.)
You can embed them all but create a new document when you reach a certain limit. For example you can limit a document to an array of 500 elements then create a new one. Also, if it is about feed, when viewed, you dont have to keep the viewed publications you can replace by new ones so you don't have to create new document for additional publication storage.
To maintain your performance, I'd advice you to make a collection that can use graphlookup aggregation, where you store your following. Being followed can reach millions of followers, so you have to store what pwople follow instead of who follows them.
I hope it helps you.