Context
I have a database with a collection of documents using this schema (shortened schema because some data is irrelevant to my problem):
{
title: string;
order: number;
...
...
...
modificationsHistory: HistoryEntry[];
items: ListRow[];
finalItems: ListRow[];
...
...
...
}
These documents can easily reach 100 or 200 kB, depending on the amount of items and finalItems that they hold. It's also very important that they are updated as fast as possible, with the smallest bandwidth usage possible.
This is inside a web application context, using Angular 9 and #angular/fire 6.0.0.
Problems
When the end user edits one item inside the object's item array, like editing just a property, reflecting that inside the database requires me to send the entire object, because firestore's update method doesn't support array indexes inside the field path, the only operations that can be done on arrays are adding or deleting an element as described inside documentation.
However, updating an element of the items array by sending the entire document creates poor performances for anyone without a good connection, which is the case for a lot of my users.
Second issue is that having everything in realtime inside one document makes collaboration hard in my case, because some of these elements can be edited by multiple users at the same time, which creates two issues:
Some write operations may fail due to too much contention on the document if two updates are made in the same second.
The updates are not atomic as we're sending the entire document at once, as it doesn't use transactions to avoid using bandwidth even more.
Solutions I already tried
Subcollections
Description
This was a very simple solution: create a subcollection for items, finalItems and modificationsHistory arrays, making them easy to edit as they now have their own ID so it's easy to reach them to update them.
Why it didn't work
Having a list with 10 finalItems, 30 items and 50 entries inside modificationsHistory means that I need to have a total of 4 listeners opened for one element to be listened entirely. Considering the fact that a user can have many of these elements opened at once, having several dozens of documents being listened creates an equally bad performance situation, probably even worse in a full user case.
It also means that if I want to update a big element with 100 items and I want to update half of them, it'll cost me one write operation per item, not to mention the amount of read operations needed to check permissions, etc, probably 3 per write so 150 read + 50 write just to update 50 items in an array.
Cloud Function to update the document
const {
applyPatch
} = require('fast-json-patch');
function applyOffsets(data, entries) {
entries.forEach(customEntry => {
const explodedPath = customEntry.path.split('/');
explodedPath.shift();
let pointer = data;
for (let fragment of explodedPath.slice(0, -1)) {
pointer = pointer[fragment];
}
pointer[explodedPath[explodedPath.length - 1]] += customEntry.offset;
});
return data;
}
exports.updateList = functions.runWith(runtimeOpts).https.onCall((data, context) => {
const listRef = firestore.collection('lists').doc(data.uid);
return firestore.runTransaction(transaction => {
return transaction.get(listRef).then(listDoc => {
const list = listDoc.data();
try {
const [standard, custom] = JSON.parse(data.diff).reduce((acc, entry) => {
if (entry.custom) {
acc[1].push(entry);
} else {
acc[0].push(entry);
}
return acc;
}, [
[],
[]
]);
applyPatch(list, standard);
applyOffsets(list, custom);
transaction.set(listRef, list);
} catch (e) {
console.log(data.diff);
}
});
});
});
Description
Using a diff library, I was making a diff between previous document and the new updated one, and sending this diff to a GCF that was operating the update using the transaction API.
Benefits of this approach being that since transaction happens inside GCF, it's super fast and doesn't consume too much bandwidth, plus the update only requires a diff to be sent, not the entire document anymore.
Why it didn't work
In reality, the cloud function was really slow and some updates were taking over 2 seconds to be made, they could also fail due to contention, without firestore connector knowing it, so no possibility to ensure data integrity in this case.
I will be edited accordingly to add more solutions if I find other stuff to try
Question
I feel like I'm missing something, like if firestore had something I just didn't know at all that could solve my use case, but I can't figure out what it is, maybe my previously tested solutions were badly implemented or I missed something important. What did I miss? Is it even possible to achieve what I want to do? I am open to data remodeling, query changes, anything, as it's mostly for learning purpose.
You should be able to reduce the bandwidth required to update your documents by using Maps instead of Arrays to store your data. This would allow you to send only the item that is being updated using its key.
I don't know how involved this would be for you to change, but it sounds like less work than the other options.
You said that it's not impossible for your documents to reach 200kb individually. It would be good to keep in mind that Firestore limits document size to 1mb. If you plan on supporting documents beyond that, you will need to find a way to fragment the data.
Regarding your contention issues... You might consider a system that "locks" the document and prevents it from receiving updates while another user is attempting to save. You could use a simple message system built with websockets or Firebase FCM to do this. A client would subscribe to the document's channel, and publish when they are attempting an update. Other clients would then receive a notice that the document is being updated and have to wait before they can save their own changes.
Also, I don't know what the contents of modificationsHistory look like, but that sounds to me like the type of data that you might keep in a subcollection instead.
Of the solutions you tried, the subcollection seems like the most scalable to me. You could look into the possibility of not using onSnapshot listeners and instead create your own event system to notify clients of changes. I suppose it could work similar to the "locking" system I mentioned above. A client sends an event when it updates an item belonging to a document. Other clients subscribed to that document's channel will know to check the database for the newest version.
Your diff-approach appeared mostly sensible, details aside.
You should store items inline, but defer modificationsHistory into a sub collection. For the entire root document, record which elements of modificationsHistory have been merged yet (by timestamp should suffice), and all elements not merged yet, you have to re-apply individually on each client, querying with aforementioned timestamp.
Each entry in modificationsHistory should not describe a single diff, but whenever possible a set of diffs.
Apply changes from modificationsHistory collections onto items in batch, deferred via GCF. You may defer this arbitrarily far, and you may want to exclude modifications performed only in the last few seconds, to account for not established consistency in Firestore. There is no risk of contention, that way.
Cleanup from the modificationsHistory collection has to be deferred even further, until you can be sure that no client has still access to an older revision of the root document. Especially if you consider that the client is not strictly required to update the root document when the listener is triggered.
You may need to reconstruct the patch stack on the client side if modificationsHistory changes in unexpected ways due to eventual consistency constraints. E.g. if you have a total order in the set of patches, you need to re-apply the patch stack from base image if the collection unexpectedly suddenly contains "older" patches unknown to the client before.
All in all, you should be able avoid frequent updates all together, and limit this solely to inserts into to modificationsHistory sub-collection. With bandwidth requirements not exceeding the cost of fetching the entire document once, plus streaming the collection of not-yet-applied patches. No contention expected.
You can tweak for how long clients may ignore hard updates to the root document, and how many changes they may batch client-side before submitting a new diff. Latter is also a tradeof with regard to how many documents another client has to fetch initially, with regard to max-documents-per-query limits.
If you require other information which are likely to suffer from contention, like list of users currently having a specific document open, that should go into sub-collections as well.
Should the latency for seeing changes by other users eventually turn out to be unacceptable, you may opt for an additional, real-time capable data channel for distribution of patches on a specific document. ActiveMQ or some other message broker operated on dedicated resources, running independently from FireStore.
I am new to NoSQL and I'm trying to figure out a good way to represent my data. I have a series of workers that need to request vacations via mobile app.
When I try to write a Firebase query with Flutter, I can do this:
Firestore.instance
.collection("ferie_permessi")
.document("worker1#test.com")
.snapshot();
It works but there are two main errors:
If I try to create another collection called "Woker info" I cannot use worker1#test.com as document ID as it already esists;
I have to sort data client side because firestore doesn't give me the possibility (with this setup I've made).
I'm quite sure that this structure isn't good at all. Each worker needs to have 2 lists: vacations and other. What is wrong?
My guess is that I should move worker1#test.com together with vacations and other so that I can make a query of this kind:
Firestore.instance
.collection("ferie_permessi")
.where("user", "==", "worker1#test.com)
.snapshot();
But now the id? Is an automatic one good?
I had a chance to recently explore creating an app using a firebase-firestore. A couple of things will help here:
Yes, the autogenerated id is good since it is unique, for example, you can have a collections vacation_requests, users you can then use that user_id as a document in vaccation_requests -> user_id -> vacations, instead of using email as a document key.
Or
You can do it like this collections users, vacation_requests, and requests.
store user details in users.
store requests in requests with from and to dates.
store reference of User and Request in vaccation_requests.
Hope this helps!
I have 2 collections. One of them is named "USERS", and the other one "MATCHES". USERS, can join in the MATCHES, appearing the avatar of the user who has joined in the match. The problem is that when the user changes their avatar image after joining in the match, the match avatar doesn't changed, because the match has the old avatar.
The avatar is saved as Base64 in Firestore, but I will change it to "Storage" in the near future.
I have been trying to set the reference, but that only gives me the path.
If I have to do a Database Api Call for each match which is joined the user, maybe I have to do 20 Api calls updating the matches. It can be a solution, but not the best.
Maybe the solution is in the Google Functions?
I'm out of ideas.
Maybe the solution is in the Google Functions?
Cloud Functions also access Firestore through an SDK, so they can't magically do things that the SDK doesn't allow.
If you're duplicating data and you update one of the duplicates, you'll have to consider updating the others. If they all need to be updated, that indeed requires a separate call for each duplicate.
If you don't want to have to do this, don't store duplicate data.
For more on the strategies for updating duplicated data, see How to write denormalized data in Firebase
I have a set of static and pre-defined to-do's that each user in my app needs to be able to set as completed on their account.
At the moment, I use a map on the todo item that specifies which users has completed the task. My data model at the moment looks like this:
- todos (collection)
- todoA (document)
- title, description etc
- completedBy {
uid1: true,
uid2: true,
uid3: false
}
This allows me to easily set todos as completed/not completed for each user and I can easily filter/query. It does have two drawbacks though:
A single Firestore document can "only" have 20 000 properties. If my app would grow large, this would be an issue.
Document size
I was thinking of maybe creating a similar map on my user document instead, setting todo ID's as true/false. This would get rid of the two drawbacks above but I'd need two database queries whenever I'm getting my todo items, one for the todo and one to check if it's completed.
Is there a better way to achieve the desired functionality in Firestore?
If you are running into either the maximum number of fields or the maximum document size, then typically that means that you should be using a separate collection for "the thing that makes your documents so big".
In your case that'd mean that you store the "user has completed a task" in a separate collection. This can be a subcollection of the user document, a subcollection of the task document, and/or a separate top-level collection. Which one is correct depends on your use-case.
There is no single best data model in NoSQL databases. It all depends on your use-cases, trade-offs, and some personal preferences. For a great introduction read NoSQL data modeling and watch Get to Know Cloud Firestore.
I am developing an app in Meteor for the first time. As seen in the documentation, I am loading my collections this way:
Items = new Mongo.Collection("items")
The items collection has more than a million documents and makes page loading very slow. How can I avoid this overhead?
First remove the autopublish package from the console:
$ meteor remove autopublish
Otherwise all records will be published to all clients and 1M records will be very slow.
Second, create a publication that filters the collection to only publish those documents that are actually relevant to the current user in the current application context:
Server:
Meteor.publish('myItems',function(){
if ( this.userId ){
return Items.find({ some query relevant to the user },
{fields: { key1: 1, key2: 1, ... only relevant fields }});
}
this.ready();
});
Client:
Meteor.subscribe('myItems');
Your query and list of relevant fields might vary by class of user. You can also have multiple publications on the same collection for different use cases.
If your collection has millions of document, it very bad to have entire data loaded on front-end may it be any circumstances.
you should wisely use PAGINATION, as there are packages out there for pagination in meteor. But if you want to use pagination with simple Session variable handling (the variable keeps the "skip" value which is accessible at both client and server) with aslagle:reactive-table for better presentation, you can watch this video https://www.youtube.com/watch?v=UivnTM1YA-I
After implementing this feature, you yourself will feel that loading entire data on UI is not feasible solution and pagination works without page refresh, asynchronously and reactively.