Firestore - recommended way to write to a document and tell its size? [duplicate] - google-cloud-firestore

Question

From Firestore docs, we get that the maximum size for a Firestore document is:
Maximum size for a document 1 MiB (1,048,576 bytes)
QUESTION
How can I know the current size of a single doc, to check if I'm approaching
that 1mb limit?
Example:
var docRef = db.collection("cities").doc("SF");
docRef.get().then(function(doc) {
if (doc.exists) {
console.log("Document data:", doc.data());
// IS THERE A PROPERTY THAT CAN DISPLAY THE DOCUMENT FILE SIZE?
} else {
// doc.data() will be undefined in this case
console.log("No such document!");
}
}).catch(function(error) {
console.log("Error getting document:", error);
});

Answer 1

The calculations used to compute the size of a document is fully documented here. There is a lot of text there, so please navigate there to read it. It's not worthwhile to copy all that text here.
If you're having to manually compute the size of a document as it grows, my opinion is that you're probably not modeling your data scalably. If you have lists of data that can grow unbounded, you probably shouldn't be using a list field, and instead put that data in documents in a new collection or subcollection. There are some exceptions to this rule, but generally speaking, you should not have to worry about computing the size of a document in your client code.

Answer 2

I've published a npm package that calculates the size of a Firestore document.
Other packages like sizeof or object-sizeof that calculate the size of JS object will not give you a precise result because some primitives in Firestore have different byte value. For example boolean in Js is stored in 4 bytes, in a Firestore document it's 1 byte. Null is 0 bytes, in Firestore it's 1 byte.
Additionally to that Firestore has own unique types with fixed byte size: Geo point, Date, Reference.
Reference is a large object. Packages like sizeof will traverse through all the methods/properties of Reference, instead just doing the right thing here. Which is to sum String value of a document name + path to it + 16 bytes. Also, if Reference points to a parent doc , sizeof or object-sizeof will not detect circular reference here which might spell even bigger trouble than incorrect size.

Answer 3

For Android users who want to check the size of a document against the maximum of 1 MiB (1,048,576 bytes) quota, there is a library I have made and that can help you calculate that:
https://github.com/alexmamo/FirestoreDocument-Android/tree/master/firestore-document
In this way, you'll be able to always stay below the limit. The algorithm behind this library is the one that is explained in the official documentation regarding the Storage Size.

Answer 4

I was looking in the Firebase reference expecting the metadata would have an attribute, but it doesn't. You can check it here.
So my next approach would be to figure the weight of the object as an approximation. The sizeOf library seems to have a reasonable API for it.
So it would be something like:
sizeof.sizeof(doc.data());
I wouldn't use the document snapshot, because it contains metadata, like if there are pending saves. On another hand overestimating could be better in some cases.
[UPDATE] Thanks to Doug Stevenson for the wonderful insight
So I was curious how much the difference would actually be, so with my clunky js I made a dirty comparison, you can see the demo here
Considering this object:
{
"boolean": true,
"number": 1,
"text": "example"
}
And discounting the id this is the result:
| Method | Bytes |
|---------|-------|
| FireDoc | 37 |
| sizeOf | 64 |
So sizeOf library could be a good predictor if we want to overestimate (assuming calculations are fine and will behave more or less equal for more complex entities). But as explained in the comment, it is a rough estimation.

Answer 5

For Swift users,
If you want to estimate the document size then I use the following. Returns the estimated size of document in Bytes. It's not 100% accurate but gives a solid estimate. Basically just converts each key, value in the data map to a string and returns total bytes of String + 1. You can see the following link for details on how Firebase determines doc size: https://firebase.google.com/docs/firestore/storage-size.
func getDocumentSize(data: [String : Any]) -> Int{
var size = 0
for (k, v) in data {
size += k.count + 1
if let map = v as? [String : Any]{
size += getDocumentSize(data: map)
} else if let array = v as? [String]{
for a in array {
size += a.count + 1
}
} else if let s = v as? String{
size += s.count + 1
}
}
return size
}

Answer 6

You can use this calculator (code snipped), i write by myself.
source : https://firebase.google.com/docs/firestore/storage-size
<!DOCTYPE html>
<html>
<head>
<title>Calculte Firestore Size</title>
</head>
<body>
<h1>Firestore Document Size Calculator</h1>
<h2 id="response" style="color:red">This is a Heading</h2>
<textarea id="id" style="width: 100%" placeholder="Firestore Doc Ref"></textarea>
<textarea id="json" style="width: 100%; min-height: 200px" placeholder="Firestore Doc Value JSON STRING"></textarea>
<textarea id="quantity" style="width: 100%;" placeholder="How Many repeat this value?"></textarea>
<script>
document.getElementById("json").value='{"type": "Personal","done": false , "priority": 1 , "description": "Learn Cloud Firestore"}'
document.getElementById("id").value = 'users/jeff/tasks/my_task_id'
calculate()
function yuzdeBul(total,number) {
if (number == 0) {
return 0;
}
const sonuc = Math.ceil(parseInt(number) / (parseInt(total) / 100));
return sonuc;
}
function calculate(){
var quantity = parseInt(document.getElementById("quantity").value || 1);
var firestoreId = document.getElementById("id").value;
var refTotal = firestoreId
.split("/")
.map((v) => v.length + 1)
.reduce((a, b) => a + b, 0) + 16;
var idTotal = 0
//console.log(idTotal);
var parseJson = JSON.parse(document.getElementById("json").value);
idTotal += calculateObj(parseJson);
idTotal+=32;
idTotal*=quantity;
idTotal+=refTotal;
document.getElementById("response").innerHTML = idTotal + "/" + 1048576 + " %"+yuzdeBul(1048576,idTotal);
}
function calculateObj(myObj) {
var total = Object.keys(myObj).map((key) => {
var keySize = key.toString().length + 1;
var findType = typeof myObj[key];
//console.log(key,findType)
if (findType == "string") {
keySize += myObj[key].length + 1;
} else if (findType == "boolean") {
keySize += 1;
}
if (findType == "number") {
keySize += 8;
}
if (findType == "object") {
keySize += calculateObj(myObj[key]);
}
return keySize;
});
return total.reduce((a, b) => a + b, 0);
}
document.getElementById("json").addEventListener("change", calculate);
document.getElementById("id").addEventListener("change", calculate);
document.getElementById("quantity").addEventListener("change", calculate);
</script>
</body>
</html>

Answer 7

So I was looking for a way to reduce unnecessary document reads by accumulating data in arrays and go worried about the size.
Turns out I wasn't even close to the limit.
Here's what you can do,
Create a new collection and add a document with the worst-case scenario for live data and using cloud console export that collection, you will see the document size.
Here is a screenshot of my export
Assuming all the documents are equal in size, each is 0.0003MB
You can also see if the documents exceed the 1024byte limit
document exceeding limit from the console
Note: you can only export when you have enabled billing.!