Flutter query random snapshot from firebase [duplicate] - flutter

It is crucial for my application to be able to select multiple documents at random from a collection in firebase.
Since there is no native function built in to Firebase (that I know of) to achieve a query that does just this, my first thought was to use query cursors to select a random start and end index provided that I have the number of documents in the collection.
This approach would work but only in a limited fashion since every document would be served up in sequence with its neighboring documents every time; however, if I was able to select a document by its index in its parent collection I could achieve a random document query but the problem is I can't find any documentation that describes how you can do this or even if you can do this.
Here's what I'd like to be able to do, consider the following firestore schema:
root/
posts/
docA
docB
docC
docD
Then in my client (I'm in a Swift environment) I'd like to write a query that can do this:
db.collection("posts")[0, 1, 3] // would return: docA, docB, docD
Is there anyway I can do something along the lines of this? Or, is there a different way I can select random documents in a similar fashion?
Please help.

Using randomly generated indexes and simple queries, you can randomly select documents from a collection or collection group in Cloud Firestore.
This answer is broken into 4 sections with different options in each section:
How to generate the random indexes
How to query the random indexes
Selecting multiple random documents
Reseeding for ongoing randomness
How to generate the random indexes
The basis of this answer is creating an indexed field that when ordered ascending or descending, results in all the document being randomly ordered. There are different ways to create this, so let's look at 2, starting with the most readily available.
Auto-Id version
If you are using the randomly generated automatic ids provided in our client libraries, you can use this same system to randomly select a document. In this case, the randomly ordered index is the document id.
Later in our query section, the random value you generate is a new auto-id (iOS, Android, Web) and the field you query is the __name__ field, and the 'low value' mentioned later is an empty string. This is by far the easiest method to generate the random index and works regardless of the language and platform.
By default, the document name (__name__) is only indexed ascending, and you also cannot rename an existing document short of deleting and recreating. If you need either of these, you can still use this method and just store an auto-id as an actual field called random rather than overloading the document name for this purpose.
Random Integer version
When you write a document, first generate a random integer in a bounded range and set it as a field called random. Depending on the number of documents you expect, you can use a different bounded range to save space or reduce the risk of collisions (which reduce the effectiveness of this technique).
You should consider which languages you need as there will be different considerations. While Swift is easy, JavaScript notably can have a gotcha:
32-bit integer: Great for small (~10K unlikely to have a collision) datasets
64-bit integer: Large datasets (note: JavaScript doesn't natively support, yet)
This will create an index with your documents randomly sorted. Later in our query section, the random value you generate will be another one of these values, and the 'low value' mentioned later will be -1.
How to query the random indexes
Now that you have a random index, you'll want to query it. Below we look at some simple variants to select a 1 random document, as well as options to select more than 1.
For all these options, you'll want to generate a new random value in the same form as the indexed values you created when writing the document, denoted by the variable random below. We'll use this value to find a random spot on the index.
Wrap-around
Now that you have a random value, you can query for a single document:
let postsRef = db.collection("posts")
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: random)
.order(by: "random")
.limit(to: 1)
Check that this has returned a document. If it doesn't, query again but use the 'low value' for your random index. For example, if you did Random Integers then lowValue is 0:
let postsRef = db.collection("posts")
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: lowValue)
.order(by: "random")
.limit(to: 1)
As long as you have a single document, you'll be guaranteed to return at least 1 document.
Bi-directional
The wrap-around method is simple to implement and allows you to optimize storage with only an ascending index enabled. One downside is the possibility of values being unfairly shielded. E.g if the first 3 documents (A,B,C) out of 10K have random index values of A:409496, B:436496, C:818992, then A and C have just less than 1/10K chance of being selected, whereas B is effectively shielded by the proximity of A and only roughly a 1/160K chance.
Rather than querying in a single direction and wrapping around if a value is not found, you can instead randomly select between >= and <=, which reduces the probability of unfairly shielded values by half, at the cost of double the index storage.
If one direction returns no results, switch to the other direction:
queryRef = postsRef.whereField("random", isLessThanOrEqualTo: random)
.order(by: "random", descending: true)
.limit(to: 1)
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: random)
.order(by: "random")
.limit(to: 1)
Selecting multiple random documents
Often, you'll want to select more than 1 random document at a time. There are 2 different ways to adjust the above techniques depending on what trade offs you want.
Rinse & Repeat
This method is straight forward. Simply repeat the process, including selecting a new random integer each time.
This method will give you random sequences of documents without worrying about seeing the same patterns repeatedly.
The trade-off is it will be slower than the next method since it requires a separate round trip to the service for each document.
Keep it coming
In this approach, simply increase the number in the limit to the desired documents. It's a little more complex as you might return 0..limit documents in the call. You'll then need to get the missing documents in the same manner, but with the limit reduced to only the difference. If you know there are more documents in total than the number you are asking for, you can optimize by ignoring the edge case of never getting back enough documents on the second call (but not the first).
The trade-off with this solution is in repeated sequences. While the documents are randomly ordered, if you ever end up overlapping ranges you'll see the same pattern you saw before. There are ways to mitigate this concern discussed in the next section on reseeding.
This approach is faster than 'Rinse & Repeat' as you'll be requesting all the documents in the best case a single call or worst case 2 calls.
Reseeding for ongoing randomness
While this method gives you documents randomly if the document set is static the probability of each document being returned will be static as well. This is a problem as some values might have unfairly low or high probabilities based on the initial random values they got. In many use cases, this is fine but in some, you may want to increase the long term randomness to have a more uniform chance of returning any 1 document.
Note that inserted documents will end up weaved in-between, gradually changing the probabilities, as will deleting documents. If the insert/delete rate is too small given the number of documents, there are a few strategies addressing this.
Multi-Random
Rather than worrying out reseeding, you can always create multiple random indexes per document, then randomly select one of those indexes each time. For example, have the field random be a map with subfields 1 to 3:
{'random': {'1': 32456, '2':3904515723, '3': 766958445}}
Now you'll be querying against random.1, random.2, random.3 randomly, creating a greater spread of randomness. This essentially trades increased storage to save increased compute (document writes) of having to reseed.
Reseed on writes
Any time you update a document, re-generate the random value(s) of the random field. This will move the document around in the random index.
Reseed on reads
If the random values generated are not uniformly distributed (they're random, so this is expected), then the same document might be picked a dispropriate amount of the time. This is easily counteracted by updating the randomly selected document with new random values after it is read.
Since writes are more expensive and can hotspot, you can elect to only update on read a subset of the time (e.g, if random(0,100) === 0) update;).

Posting this to help anyone that has this problem in the future.
If you are using Auto IDs you can generate a new Auto ID and query for the closest Auto ID as mentioned in Dan McGrath's Answer.
I recently created a random quote api and needed to get random quotes from a firestore collection.
This is how I solved that problem:
var db = admin.firestore();
var quotes = db.collection("quotes");
var key = quotes.doc().id;
quotes.where(admin.firestore.FieldPath.documentId(), '>=', key).limit(1).get()
.then(snapshot => {
if(snapshot.size > 0) {
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
}
else {
var quote = quotes.where(admin.firestore.FieldPath.documentId(), '<', key).limit(1).get()
.then(snapshot => {
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
})
.catch(err => {
console.log('Error getting documents', err);
});
}
})
.catch(err => {
console.log('Error getting documents', err);
});
The key to the query is this:
.where(admin.firestore.FieldPath.documentId(), '>', key)
And calling it again with the operation reversed if no documents are found.
I hope this helps!

Just made this work in Angular 7 + RxJS, so sharing here with people who want an example.
I used #Dan McGrath 's answer, and I chose these options: Random Integer version + Rinse & Repeat for multiple numbers. I also used the stuff explained in this article: RxJS, where is the If-Else Operator? to make if/else statements on stream level (just if any of you need a primer on that).
Also note I used angularfire2 for easy Firebase integration in Angular.
Here is the code:
import { Component, OnInit } from '#angular/core';
import { Observable, merge, pipe } from 'rxjs';
import { map, switchMap, filter, take } from 'rxjs/operators';
import { AngularFirestore, QuerySnapshot } from '#angular/fire/firestore';
#Component({
selector: 'pp-random',
templateUrl: './random.component.html',
styleUrls: ['./random.component.scss']
})
export class RandomComponent implements OnInit {
constructor(
public afs: AngularFirestore,
) { }
ngOnInit() {
}
public buttonClicked(): void {
this.getRandom().pipe(take(1)).subscribe();
}
public getRandom(): Observable<any[]> {
const randomNumber = this.getRandomNumber();
const request$ = this.afs.collection('your-collection', ref => ref.where('random', '>=', randomNumber).orderBy('random').limit(1)).get();
const retryRequest$ = this.afs.collection('your-collection', ref => ref.where('random', '<=', randomNumber).orderBy('random', 'desc').limit(1)).get();
const docMap = pipe(
map((docs: QuerySnapshot<any>) => {
return docs.docs.map(e => {
return {
id: e.id,
...e.data()
} as any;
});
})
);
const random$ = request$.pipe(docMap).pipe(filter(x => x !== undefined && x[0] !== undefined));
const retry$ = request$.pipe(docMap).pipe(
filter(x => x === undefined || x[0] === undefined),
switchMap(() => retryRequest$),
docMap
);
return merge(random$, retry$);
}
public getRandomNumber(): number {
const min = Math.ceil(Number.MIN_VALUE);
const max = Math.ceil(Number.MAX_VALUE);
return Math.floor(Math.random() * (max - min + 1)) + min;
}
}

The other solutions are better but seems hard for me to understand, so I came up with another method
Use incremental number as ID like 1,2,3,4,5,6,7,8,9, watch out for delete documents else we
have an I'd that is missing
Get total number of documents in the collection, something like this, I don't know of a better solution than this
let totalDoc = db.collection("stat").get().then(snap=>snap.size)
Now that we have these, create an empty array to store random list of number, let's say we want 20 random documents.
let randomID = [ ]
while(randomID.length < 20) {
const randNo = Math.floor(Math.random() * totalDoc) + 1;
if(randomID.indexOf(randNo) === -1) randomID.push(randNo);
}
now we have our 20 random documents id
finally we fetch our data from fire store, and save to randomDocs array by mapping through the randomID array
const randomDocs = randomID.map(id => {
db.collection("posts").doc(id).get()
.then(doc => {
if (doc.exists) return doc.data()
})
.catch(error => {
console.log("Error getting document:", error);
});
})
I'm new to firebase, but I think with this answers we can get something better or a built-in query from firebase soon

After intense argument with my friend, we finally found some solution
If you don't need to set document's id to be RandomID, just name documents as size of collection's size.
For example, first document of collection is named '0'.
second document name should be '1'.
Then, we just read the size of collection, for example N, and we can get random number A in range of [0~N).
And then, we can query the document named A.
This way can give same probability of randomness to every documents in collection.

undoubtedly Above accepted Answer is SuperUseful but There is one case like If we had a collection of some Documents(about 100-1000) and we want some 20-30 random Documents Provided that Document must not be repeated. (case In Random Problems App etc...).
Problem with the Above Solution:
For a small number of documents in the Collection(say 50) Probability of repetition is high. To avoid it If I store Fetched Docs Id and Add-in Query like this:
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: lowValue).where("__name__", isNotEqualTo:"PreviousId")
.order(by: "random")
.limit(to: 1)
here PreviousId is Id of all Elements that were fetched Already means A loop of n previous Ids.
But in this case, network Call would be high.
My Solution:
Maintain one Special Document and Keep a Record of Ids of this Collection only, and fetched this document First Time and Then Do all Randomness Stuff and check for previously not fetched on App site. So in this case network call would be only the same as the number of documents requires (n+1).
Disadvantage of My solution:
Have to maintain A document so Write on Addition and Deletion. But it is good If reads are very often then Writes which occurs in most cases.

You can use listDocuments() property for get only Query list of documents id. Then generate random id using the following way and get DocumentSnapshot with get() property.
var restaurantQueryReference = admin.firestore().collection("Restaurant"); //have +500 docs
var restaurantQueryList = await restaurantQueryReference.listDocuments(); //get all docs id;
for (var i = restaurantQueryList.length - 1; i > 0; i--) {
var j = Math.floor(Math.random() * (i + 1));
var temp = restaurantQueryList[i];
restaurantQueryList[i] = restaurantQueryList[j];
restaurantQueryList[j] = temp;
}
var restaurantId = restaurantQueryList[Math.floor(Math.random()*restaurantQueryList.length)].id; //this is random documentId

Unlike rtdb, firestore ids are not ordered chronologically. So using Auto-Id version described by Dan McGrath is easily implemented if you use the auto-generated id by the firestore client.
new Promise<Timeline | undefined>(async (resolve, reject) => {
try {
let randomTimeline: Timeline | undefined;
let maxCounter = 5;
do {
const randomId = this.afs.createId(); // AngularFirestore
const direction = getRandomIntInclusive(1, 10) <= 5;
// The firestore id is saved with your model as an "id" property.
let list = await this.list(ref => ref
.where('id', direction ? '>=' : '<=', randomId)
.orderBy('id', direction ? 'asc' : 'desc')
.limit(10)
).pipe(take(1)).toPromise();
// app specific filtering
list = list.filter(x => notThisId !== x.id && x.mediaCounter > 5);
if (list.length) {
randomTimeline = list[getRandomIntInclusive(0, list.length - 1)];
}
} while (!randomTimeline && maxCounter-- >= 0);
resolve(randomTimeline);
} catch (err) {
reject(err);
}
})

I have one way to get random a list document in Firebase Firestore, it really easy. When i upload data on Firestore i creat a field name "position" with random value from 1 to 1 milions. When i get data from Fire store i will set Order by field "Position" and update value for it, a lot of user load data and data always update and it's will be random value.

For those using Angular + Firestore, building on #Dan McGrath techniques, here is the code snippet.
Below code snippet returns 1 document.
getDocumentRandomlyParent(): Observable<any> {
return this.getDocumentRandomlyChild()
.pipe(
expand((document: any) => document === null ? this.getDocumentRandomlyChild() : EMPTY),
);
}
getDocumentRandomlyChild(): Observable<any> {
const random = this.afs.createId();
return this.afs
.collection('my_collection', ref =>
ref
.where('random_identifier', '>', random)
.limit(1))
.valueChanges()
.pipe(
map((documentArray: any[]) => {
if (documentArray && documentArray.length) {
return documentArray[0];
} else {
return null;
}
}),
);
}
1) .expand() is a rxjs operation for recursion to ensure we definitely get a document from the random selection.
2) For recursion to work as expected we need to have 2 separate functions.
3) We use EMPTY to terminate .expand() operator.
import { Observable, EMPTY } from 'rxjs';

Ok I will post answer to this question even thou I am doing this for Android. Whenever i create a new document i initiate random number and set it to random field, so my document looks like
"field1" : "value1"
"field2" : "value2"
...
"random" : 13442 //this is the random number i generated upon creating document
When I query for random document I generate random number in same range that I used when creating document.
private val firestore: FirebaseFirestore = FirebaseFirestore.getInstance()
private var usersReference = firestore.collection("users")
val rnds = (0..20001).random()
usersReference.whereGreaterThanOrEqualTo("random",rnds).limit(1).get().addOnSuccessListener {
if (it.size() > 0) {
for (doc in it) {
Log.d("found", doc.toString())
}
} else {
usersReference.whereLessThan("random", rnds).limit(1).get().addOnSuccessListener {
for (doc in it) {
Log.d("found", doc.toString())
}
}
}
}

Based on #ajzbc answer I wrote this for Unity3D and its working for me.
FirebaseFirestore db;
void Start()
{
db = FirebaseFirestore.DefaultInstance;
}
public void GetRandomDocument()
{
Query query1 = db.Collection("Sports").WhereGreaterThanOrEqualTo(FieldPath.DocumentId, db.Collection("Sports").Document().Id).Limit(1);
Query query2 = db.Collection("Sports").WhereLessThan(FieldPath.DocumentId, db.Collection("Sports").Document().Id).Limit(1);
query1.GetSnapshotAsync().ContinueWithOnMainThread((querySnapshotTask1) =>
{
if(querySnapshotTask1.Result.Count > 0)
{
foreach (DocumentSnapshot documentSnapshot in querySnapshotTask1.Result.Documents)
{
Debug.Log("Random ID: "+documentSnapshot.Id);
}
} else
{
query2.GetSnapshotAsync().ContinueWithOnMainThread((querySnapshotTask2) =>
{
foreach (DocumentSnapshot documentSnapshot in querySnapshotTask2.Result.Documents)
{
Debug.Log("Random ID: " + documentSnapshot.Id);
}
});
}
});
}

If you are using autoID this may also work for you...
let collectionRef = admin.firestore().collection('your-collection');
const documentSnapshotArray = await collectionRef.get();
const records = documentSnapshotArray.docs;
const index = documentSnapshotArray.size;
let result = '';
console.log(`TOTAL SIZE=====${index}`);
var randomDocId = Math.floor(Math.random() * index);
const docRef = records[randomDocId].ref;
result = records[randomDocId].data();
console.log('----------- Random Result --------------------');
console.log(result);
console.log('----------- Random Result --------------------');

Easy (2022). You need something like:
export const getAtRandom = async (me) => {
const collection = admin.firestore().collection('...').where(...);
const { count } = (await collection.count().get()).data();
const numberAtRandom = Math.floor(Math.random() * count);
const snap = await accountCollection.limit(1).offset(numberAtRandom).get()
if (accountSnap.empty) return null;
const doc = { id: snap.docs[0].id, ...snap.docs[0].data(), ref: snap.docs[0].ref };
return doc;
}

The next code (Flutter) will return one or up to ten random documents from a Firebase collection.
None of the documents will be repeated
Max 10 documents can be retrieved
If you pass a greater numberOfDocuments than existing documents in the collection, the loop will never end.
Future<Iterable<QueryDocumentSnapshot>> getRandomDocuments(int numberOfDocuments) async {
// Queried documents
final docs = <QueryDocumentSnapshot>[];
// Queried documents id's. We will use later to avoid querying same documents
final currentIds = <String>[];
do {
// Generate random id explained by #Dan McGrath's answer (autoId)
final randomId = FirebaseFirestore.instance.collection('random').doc().id;
var query = FirebaseFirestore.instance
.collection('myCollection') // Change this for you collection name
.where(FieldPath.documentId, isGreaterThanOrEqualTo: randomId)
.limit(1);
if (currentIds.isNotEmpty) {
// If previously we fetched a document we avoid fetching the same
query = query.where(FieldPath.documentId, whereNotIn: currentIds);
}
final querySnap = await query.get();
for (var element in querySnap.docs) {
currentIds.add(element.id);
docs.add(element);
}
} while (docs.length < numberOfDocuments); // <- Run until we have all documents we want
return docs;
}

Related

MongoDB bulkwrite with upsert option is taking more than a minute for 100k+ records. Is it possible to improve?

I have read the following two questions but did not see a satisfactory solution or suggestion:
Performance degrade with Mongo when using bulkwrite with upsert
Mongodb/Mongoose bulkwrite(upsert) performance issues
I have an API that will be receiving ~80k objects every 5 - 6 seconds.
These objects will need to be inserted if they do not exist and updated if they do exist.
The collection has a unique index (of type asc 1), which I use in the filter expression below.
Upserting 100k documents using the code below takes more than a minute, which is too long for my requirement (instantiating the objects takes less than a second).
MongoClient _mongoClient = new MongoClient("connection string");
IMongoDatabase _hotsauceOdds = _mongoClient.GetDatabase("DatabaseName");
var collection = _hotsauceOdds.GetCollection<Person>("TestMongo");
List<Person> peeps = new List<Person>();
for (int i = 0; i < 100000; i++)
{
peeps.Add(new Person { Age = i, Name = Guid.NewGuid().ToString() });
}
var models = peeps.Select(item => new ReplaceOneModel<Person>(new ExpressionFilterDefinition<Person>(doc => doc.Name == item.Name), item) { IsUpsert = true });
await collection.BulkWriteAsync(models);
I am testing this on the free tier of mongo atlas.
Is it possible to make this operation run more quickly, or is this as good as it gets for mongodb?

Iterate through all documents and sum a field - Firebase

I have iterated through all documents of specific collection and made the sum of specific property/field to get the total amount, I able to did it but the problem is that this method is not effective in terms of performance and cost performance because firebase charges according to the usage (read/write document)
So what could be the better approach to do it?
here is my code
basically there is an sub collection called 'userEntries' and it has lots of document and each document has a field called 'amount'
double totalAmount = 0;
void getTotalAmount() async {
QuerySnapshot customerEntries = await FirebaseFirestore.instance.collection('entries').doc(widget.firebaseUser!.uid).collection('userEntries').where('customerId', isEqualTo: widget.customerMap['customerId']).get();
double total = 0;
for (var element in customerEntries.docs) {
total += element['amount'];
}
setState(() {
totalAmount = total;
});
}
What I can think of is maintain a separate sub collection where I would write/update totalAmount field whenever a user would make a new entry, is it right way to do it?
What I can think of is maintain a separate sub collection where I
would write/update totalAmount field whenever a user would make a new
entry, is it right way to do it?
Yes, this is the right way to do that. There is a specific page in the Firestore documentation which explains how to do that with several documents in order to solve the maximum limit of one write to a document per second.
The code examples in this page do not show the Dart version but it is easy to implement it based on the JS example.

DynamoDB - How to upsert nested objects with updateItem

Hi I am newbie to dynamoDB. Below is the schema of the dynamo table
{
"user_id":1, // partition key
"dob":"1991-09-12", // sort key
"movies_watched":{
"1":{
"movie_name":"twilight",
"movie_released_year":"1990",
"movie_genre":"action"
},
"2":{
"movie_name":"harry potter",
"movie_released_year":"1996",
"movie_genre":"action"
},
"3":{
"movie_name":"lalaland",
"movie_released_year":"1998",
"movie_genre":"action"
},
"4":{
"movie_name":"serendipity",
"movie_released_year":"1999",
"movie_genre":"action"
}
}
..... 6 more attributes
}
I want to insert a new item if the item(that user id with dob) did not exist, otherwise add the movies to existing movies_watched map by checking if the movie is not already available the movies_watched map .
Currently, I am trying to use update(params) method.
Below is my approach:
function getInsertQuery (item) {
const exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(item).forEach(([key, item]) => {
if (key !== 'user_id' && key !== 'dob' && key !== 'movies_watched') {
exp.UpdateExpression += ` #${key} = :${key},`
exp.ExpressionAttributeNames[`#${key}`] = key
exp.ExpressionAttributeValues[`:${key}`] = item
}
})
let i = 0
Object.entries(item. movies_watched).forEach(([key, item]) => {
exp.UpdateExpression += ` movies_watched.#uniqueID${i} = :uniqueID${i},`
exp.ExpressionAttributeNames[`#uniqueID${i}`] = key
exp.ExpressionAttributeValues[`:uniqueID${i}`] = item
i++
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1)
return exp
}
The above method just creates update expression with expression names and values for all top level attributes as well as nested attributes (with document path).
It works well if the item is already available by updating movies_watched map. But throws exception if the item is not available and while inserting. Below is exception:
The document path provided in the update expression is invalid for update
However, I am still not sure how to check for duplicate movies in movies_watched map
Could someone guide me in right direction, any help is highly appreciated!
Thanks in advance
There is no way to do this, given your model, without reading an item from DDB before an update (at that point the process is trivial). If you don't want to impose this additional read capacity on your table for update, then you would need to re-design your data model:
You can change movies_watched to be a Set and hold references to movies. Caveat is that Set can contain only Numbers or Strings, thus you would have movie id or name or keep the data but as JSON Strings in your Set and then parse it back into JSON on read. With SET you can perform ADD operation on the movies_watched attribute. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html#Expressions.UpdateExpressions.ADD
You can go with single table design approach and have these movies watched as separate items with (PK:userId and SK:movie_id). To get a user you would perform a query and specify only PK=userId -> you will get a collection where one item is your user record and others are movies_watched. If you are new to DynamoDB and are learning the ropes, then I would suggest go with this approach. https://www.alexdebrie.com/posts/dynamodb-single-table/

Firestore: How to get random documents in a collection

It is crucial for my application to be able to select multiple documents at random from a collection in firebase.
Since there is no native function built in to Firebase (that I know of) to achieve a query that does just this, my first thought was to use query cursors to select a random start and end index provided that I have the number of documents in the collection.
This approach would work but only in a limited fashion since every document would be served up in sequence with its neighboring documents every time; however, if I was able to select a document by its index in its parent collection I could achieve a random document query but the problem is I can't find any documentation that describes how you can do this or even if you can do this.
Here's what I'd like to be able to do, consider the following firestore schema:
root/
posts/
docA
docB
docC
docD
Then in my client (I'm in a Swift environment) I'd like to write a query that can do this:
db.collection("posts")[0, 1, 3] // would return: docA, docB, docD
Is there anyway I can do something along the lines of this? Or, is there a different way I can select random documents in a similar fashion?
Please help.
Using randomly generated indexes and simple queries, you can randomly select documents from a collection or collection group in Cloud Firestore.
This answer is broken into 4 sections with different options in each section:
How to generate the random indexes
How to query the random indexes
Selecting multiple random documents
Reseeding for ongoing randomness
How to generate the random indexes
The basis of this answer is creating an indexed field that when ordered ascending or descending, results in all the document being randomly ordered. There are different ways to create this, so let's look at 2, starting with the most readily available.
Auto-Id version
If you are using the randomly generated automatic ids provided in our client libraries, you can use this same system to randomly select a document. In this case, the randomly ordered index is the document id.
Later in our query section, the random value you generate is a new auto-id (iOS, Android, Web) and the field you query is the __name__ field, and the 'low value' mentioned later is an empty string. This is by far the easiest method to generate the random index and works regardless of the language and platform.
By default, the document name (__name__) is only indexed ascending, and you also cannot rename an existing document short of deleting and recreating. If you need either of these, you can still use this method and just store an auto-id as an actual field called random rather than overloading the document name for this purpose.
Random Integer version
When you write a document, first generate a random integer in a bounded range and set it as a field called random. Depending on the number of documents you expect, you can use a different bounded range to save space or reduce the risk of collisions (which reduce the effectiveness of this technique).
You should consider which languages you need as there will be different considerations. While Swift is easy, JavaScript notably can have a gotcha:
32-bit integer: Great for small (~10K unlikely to have a collision) datasets
64-bit integer: Large datasets (note: JavaScript doesn't natively support, yet)
This will create an index with your documents randomly sorted. Later in our query section, the random value you generate will be another one of these values, and the 'low value' mentioned later will be -1.
How to query the random indexes
Now that you have a random index, you'll want to query it. Below we look at some simple variants to select a 1 random document, as well as options to select more than 1.
For all these options, you'll want to generate a new random value in the same form as the indexed values you created when writing the document, denoted by the variable random below. We'll use this value to find a random spot on the index.
Wrap-around
Now that you have a random value, you can query for a single document:
let postsRef = db.collection("posts")
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: random)
.order(by: "random")
.limit(to: 1)
Check that this has returned a document. If it doesn't, query again but use the 'low value' for your random index. For example, if you did Random Integers then lowValue is 0:
let postsRef = db.collection("posts")
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: lowValue)
.order(by: "random")
.limit(to: 1)
As long as you have a single document, you'll be guaranteed to return at least 1 document.
Bi-directional
The wrap-around method is simple to implement and allows you to optimize storage with only an ascending index enabled. One downside is the possibility of values being unfairly shielded. E.g if the first 3 documents (A,B,C) out of 10K have random index values of A:409496, B:436496, C:818992, then A and C have just less than 1/10K chance of being selected, whereas B is effectively shielded by the proximity of A and only roughly a 1/160K chance.
Rather than querying in a single direction and wrapping around if a value is not found, you can instead randomly select between >= and <=, which reduces the probability of unfairly shielded values by half, at the cost of double the index storage.
If one direction returns no results, switch to the other direction:
queryRef = postsRef.whereField("random", isLessThanOrEqualTo: random)
.order(by: "random", descending: true)
.limit(to: 1)
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: random)
.order(by: "random")
.limit(to: 1)
Selecting multiple random documents
Often, you'll want to select more than 1 random document at a time. There are 2 different ways to adjust the above techniques depending on what trade offs you want.
Rinse & Repeat
This method is straight forward. Simply repeat the process, including selecting a new random integer each time.
This method will give you random sequences of documents without worrying about seeing the same patterns repeatedly.
The trade-off is it will be slower than the next method since it requires a separate round trip to the service for each document.
Keep it coming
In this approach, simply increase the number in the limit to the desired documents. It's a little more complex as you might return 0..limit documents in the call. You'll then need to get the missing documents in the same manner, but with the limit reduced to only the difference. If you know there are more documents in total than the number you are asking for, you can optimize by ignoring the edge case of never getting back enough documents on the second call (but not the first).
The trade-off with this solution is in repeated sequences. While the documents are randomly ordered, if you ever end up overlapping ranges you'll see the same pattern you saw before. There are ways to mitigate this concern discussed in the next section on reseeding.
This approach is faster than 'Rinse & Repeat' as you'll be requesting all the documents in the best case a single call or worst case 2 calls.
Reseeding for ongoing randomness
While this method gives you documents randomly if the document set is static the probability of each document being returned will be static as well. This is a problem as some values might have unfairly low or high probabilities based on the initial random values they got. In many use cases, this is fine but in some, you may want to increase the long term randomness to have a more uniform chance of returning any 1 document.
Note that inserted documents will end up weaved in-between, gradually changing the probabilities, as will deleting documents. If the insert/delete rate is too small given the number of documents, there are a few strategies addressing this.
Multi-Random
Rather than worrying out reseeding, you can always create multiple random indexes per document, then randomly select one of those indexes each time. For example, have the field random be a map with subfields 1 to 3:
{'random': {'1': 32456, '2':3904515723, '3': 766958445}}
Now you'll be querying against random.1, random.2, random.3 randomly, creating a greater spread of randomness. This essentially trades increased storage to save increased compute (document writes) of having to reseed.
Reseed on writes
Any time you update a document, re-generate the random value(s) of the random field. This will move the document around in the random index.
Reseed on reads
If the random values generated are not uniformly distributed (they're random, so this is expected), then the same document might be picked a dispropriate amount of the time. This is easily counteracted by updating the randomly selected document with new random values after it is read.
Since writes are more expensive and can hotspot, you can elect to only update on read a subset of the time (e.g, if random(0,100) === 0) update;).
Posting this to help anyone that has this problem in the future.
If you are using Auto IDs you can generate a new Auto ID and query for the closest Auto ID as mentioned in Dan McGrath's Answer.
I recently created a random quote api and needed to get random quotes from a firestore collection.
This is how I solved that problem:
var db = admin.firestore();
var quotes = db.collection("quotes");
var key = quotes.doc().id;
quotes.where(admin.firestore.FieldPath.documentId(), '>=', key).limit(1).get()
.then(snapshot => {
if(snapshot.size > 0) {
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
}
else {
var quote = quotes.where(admin.firestore.FieldPath.documentId(), '<', key).limit(1).get()
.then(snapshot => {
snapshot.forEach(doc => {
console.log(doc.id, '=>', doc.data());
});
})
.catch(err => {
console.log('Error getting documents', err);
});
}
})
.catch(err => {
console.log('Error getting documents', err);
});
The key to the query is this:
.where(admin.firestore.FieldPath.documentId(), '>', key)
And calling it again with the operation reversed if no documents are found.
I hope this helps!
Just made this work in Angular 7 + RxJS, so sharing here with people who want an example.
I used #Dan McGrath 's answer, and I chose these options: Random Integer version + Rinse & Repeat for multiple numbers. I also used the stuff explained in this article: RxJS, where is the If-Else Operator? to make if/else statements on stream level (just if any of you need a primer on that).
Also note I used angularfire2 for easy Firebase integration in Angular.
Here is the code:
import { Component, OnInit } from '#angular/core';
import { Observable, merge, pipe } from 'rxjs';
import { map, switchMap, filter, take } from 'rxjs/operators';
import { AngularFirestore, QuerySnapshot } from '#angular/fire/firestore';
#Component({
selector: 'pp-random',
templateUrl: './random.component.html',
styleUrls: ['./random.component.scss']
})
export class RandomComponent implements OnInit {
constructor(
public afs: AngularFirestore,
) { }
ngOnInit() {
}
public buttonClicked(): void {
this.getRandom().pipe(take(1)).subscribe();
}
public getRandom(): Observable<any[]> {
const randomNumber = this.getRandomNumber();
const request$ = this.afs.collection('your-collection', ref => ref.where('random', '>=', randomNumber).orderBy('random').limit(1)).get();
const retryRequest$ = this.afs.collection('your-collection', ref => ref.where('random', '<=', randomNumber).orderBy('random', 'desc').limit(1)).get();
const docMap = pipe(
map((docs: QuerySnapshot<any>) => {
return docs.docs.map(e => {
return {
id: e.id,
...e.data()
} as any;
});
})
);
const random$ = request$.pipe(docMap).pipe(filter(x => x !== undefined && x[0] !== undefined));
const retry$ = request$.pipe(docMap).pipe(
filter(x => x === undefined || x[0] === undefined),
switchMap(() => retryRequest$),
docMap
);
return merge(random$, retry$);
}
public getRandomNumber(): number {
const min = Math.ceil(Number.MIN_VALUE);
const max = Math.ceil(Number.MAX_VALUE);
return Math.floor(Math.random() * (max - min + 1)) + min;
}
}
The other solutions are better but seems hard for me to understand, so I came up with another method
Use incremental number as ID like 1,2,3,4,5,6,7,8,9, watch out for delete documents else we
have an I'd that is missing
Get total number of documents in the collection, something like this, I don't know of a better solution than this
let totalDoc = db.collection("stat").get().then(snap=>snap.size)
Now that we have these, create an empty array to store random list of number, let's say we want 20 random documents.
let randomID = [ ]
while(randomID.length < 20) {
const randNo = Math.floor(Math.random() * totalDoc) + 1;
if(randomID.indexOf(randNo) === -1) randomID.push(randNo);
}
now we have our 20 random documents id
finally we fetch our data from fire store, and save to randomDocs array by mapping through the randomID array
const randomDocs = randomID.map(id => {
db.collection("posts").doc(id).get()
.then(doc => {
if (doc.exists) return doc.data()
})
.catch(error => {
console.log("Error getting document:", error);
});
})
I'm new to firebase, but I think with this answers we can get something better or a built-in query from firebase soon
After intense argument with my friend, we finally found some solution
If you don't need to set document's id to be RandomID, just name documents as size of collection's size.
For example, first document of collection is named '0'.
second document name should be '1'.
Then, we just read the size of collection, for example N, and we can get random number A in range of [0~N).
And then, we can query the document named A.
This way can give same probability of randomness to every documents in collection.
undoubtedly Above accepted Answer is SuperUseful but There is one case like If we had a collection of some Documents(about 100-1000) and we want some 20-30 random Documents Provided that Document must not be repeated. (case In Random Problems App etc...).
Problem with the Above Solution:
For a small number of documents in the Collection(say 50) Probability of repetition is high. To avoid it If I store Fetched Docs Id and Add-in Query like this:
queryRef = postsRef.whereField("random", isGreaterThanOrEqualTo: lowValue).where("__name__", isNotEqualTo:"PreviousId")
.order(by: "random")
.limit(to: 1)
here PreviousId is Id of all Elements that were fetched Already means A loop of n previous Ids.
But in this case, network Call would be high.
My Solution:
Maintain one Special Document and Keep a Record of Ids of this Collection only, and fetched this document First Time and Then Do all Randomness Stuff and check for previously not fetched on App site. So in this case network call would be only the same as the number of documents requires (n+1).
Disadvantage of My solution:
Have to maintain A document so Write on Addition and Deletion. But it is good If reads are very often then Writes which occurs in most cases.
You can use listDocuments() property for get only Query list of documents id. Then generate random id using the following way and get DocumentSnapshot with get() property.
var restaurantQueryReference = admin.firestore().collection("Restaurant"); //have +500 docs
var restaurantQueryList = await restaurantQueryReference.listDocuments(); //get all docs id;
for (var i = restaurantQueryList.length - 1; i > 0; i--) {
var j = Math.floor(Math.random() * (i + 1));
var temp = restaurantQueryList[i];
restaurantQueryList[i] = restaurantQueryList[j];
restaurantQueryList[j] = temp;
}
var restaurantId = restaurantQueryList[Math.floor(Math.random()*restaurantQueryList.length)].id; //this is random documentId
Unlike rtdb, firestore ids are not ordered chronologically. So using Auto-Id version described by Dan McGrath is easily implemented if you use the auto-generated id by the firestore client.
new Promise<Timeline | undefined>(async (resolve, reject) => {
try {
let randomTimeline: Timeline | undefined;
let maxCounter = 5;
do {
const randomId = this.afs.createId(); // AngularFirestore
const direction = getRandomIntInclusive(1, 10) <= 5;
// The firestore id is saved with your model as an "id" property.
let list = await this.list(ref => ref
.where('id', direction ? '>=' : '<=', randomId)
.orderBy('id', direction ? 'asc' : 'desc')
.limit(10)
).pipe(take(1)).toPromise();
// app specific filtering
list = list.filter(x => notThisId !== x.id && x.mediaCounter > 5);
if (list.length) {
randomTimeline = list[getRandomIntInclusive(0, list.length - 1)];
}
} while (!randomTimeline && maxCounter-- >= 0);
resolve(randomTimeline);
} catch (err) {
reject(err);
}
})
I have one way to get random a list document in Firebase Firestore, it really easy. When i upload data on Firestore i creat a field name "position" with random value from 1 to 1 milions. When i get data from Fire store i will set Order by field "Position" and update value for it, a lot of user load data and data always update and it's will be random value.
For those using Angular + Firestore, building on #Dan McGrath techniques, here is the code snippet.
Below code snippet returns 1 document.
getDocumentRandomlyParent(): Observable<any> {
return this.getDocumentRandomlyChild()
.pipe(
expand((document: any) => document === null ? this.getDocumentRandomlyChild() : EMPTY),
);
}
getDocumentRandomlyChild(): Observable<any> {
const random = this.afs.createId();
return this.afs
.collection('my_collection', ref =>
ref
.where('random_identifier', '>', random)
.limit(1))
.valueChanges()
.pipe(
map((documentArray: any[]) => {
if (documentArray && documentArray.length) {
return documentArray[0];
} else {
return null;
}
}),
);
}
1) .expand() is a rxjs operation for recursion to ensure we definitely get a document from the random selection.
2) For recursion to work as expected we need to have 2 separate functions.
3) We use EMPTY to terminate .expand() operator.
import { Observable, EMPTY } from 'rxjs';
Ok I will post answer to this question even thou I am doing this for Android. Whenever i create a new document i initiate random number and set it to random field, so my document looks like
"field1" : "value1"
"field2" : "value2"
...
"random" : 13442 //this is the random number i generated upon creating document
When I query for random document I generate random number in same range that I used when creating document.
private val firestore: FirebaseFirestore = FirebaseFirestore.getInstance()
private var usersReference = firestore.collection("users")
val rnds = (0..20001).random()
usersReference.whereGreaterThanOrEqualTo("random",rnds).limit(1).get().addOnSuccessListener {
if (it.size() > 0) {
for (doc in it) {
Log.d("found", doc.toString())
}
} else {
usersReference.whereLessThan("random", rnds).limit(1).get().addOnSuccessListener {
for (doc in it) {
Log.d("found", doc.toString())
}
}
}
}
Based on #ajzbc answer I wrote this for Unity3D and its working for me.
FirebaseFirestore db;
void Start()
{
db = FirebaseFirestore.DefaultInstance;
}
public void GetRandomDocument()
{
Query query1 = db.Collection("Sports").WhereGreaterThanOrEqualTo(FieldPath.DocumentId, db.Collection("Sports").Document().Id).Limit(1);
Query query2 = db.Collection("Sports").WhereLessThan(FieldPath.DocumentId, db.Collection("Sports").Document().Id).Limit(1);
query1.GetSnapshotAsync().ContinueWithOnMainThread((querySnapshotTask1) =>
{
if(querySnapshotTask1.Result.Count > 0)
{
foreach (DocumentSnapshot documentSnapshot in querySnapshotTask1.Result.Documents)
{
Debug.Log("Random ID: "+documentSnapshot.Id);
}
} else
{
query2.GetSnapshotAsync().ContinueWithOnMainThread((querySnapshotTask2) =>
{
foreach (DocumentSnapshot documentSnapshot in querySnapshotTask2.Result.Documents)
{
Debug.Log("Random ID: " + documentSnapshot.Id);
}
});
}
});
}
If you are using autoID this may also work for you...
let collectionRef = admin.firestore().collection('your-collection');
const documentSnapshotArray = await collectionRef.get();
const records = documentSnapshotArray.docs;
const index = documentSnapshotArray.size;
let result = '';
console.log(`TOTAL SIZE=====${index}`);
var randomDocId = Math.floor(Math.random() * index);
const docRef = records[randomDocId].ref;
result = records[randomDocId].data();
console.log('----------- Random Result --------------------');
console.log(result);
console.log('----------- Random Result --------------------');
Easy (2022). You need something like:
export const getAtRandom = async (me) => {
const collection = admin.firestore().collection('...').where(...);
const { count } = (await collection.count().get()).data();
const numberAtRandom = Math.floor(Math.random() * count);
const snap = await accountCollection.limit(1).offset(numberAtRandom).get()
if (accountSnap.empty) return null;
const doc = { id: snap.docs[0].id, ...snap.docs[0].data(), ref: snap.docs[0].ref };
return doc;
}
The next code (Flutter) will return one or up to ten random documents from a Firebase collection.
None of the documents will be repeated
Max 10 documents can be retrieved
If you pass a greater numberOfDocuments than existing documents in the collection, the loop will never end.
Future<Iterable<QueryDocumentSnapshot>> getRandomDocuments(int numberOfDocuments) async {
// Queried documents
final docs = <QueryDocumentSnapshot>[];
// Queried documents id's. We will use later to avoid querying same documents
final currentIds = <String>[];
do {
// Generate random id explained by #Dan McGrath's answer (autoId)
final randomId = FirebaseFirestore.instance.collection('random').doc().id;
var query = FirebaseFirestore.instance
.collection('myCollection') // Change this for you collection name
.where(FieldPath.documentId, isGreaterThanOrEqualTo: randomId)
.limit(1);
if (currentIds.isNotEmpty) {
// If previously we fetched a document we avoid fetching the same
query = query.where(FieldPath.documentId, whereNotIn: currentIds);
}
final querySnap = await query.get();
for (var element in querySnap.docs) {
currentIds.add(element.id);
docs.add(element);
}
} while (docs.length < numberOfDocuments); // <- Run until we have all documents we want
return docs;
}

How to batch update documents on the server in MongoDB using calculated values?

I have a collection of things, each thing has some user reviews and each review has a numeric (1-5) rating. I would like to periodically run a batch job that will calculate the average rating for each "thing" based on individual ratings in that thing's user reviews and will then save the "thing" again with that new rating. I would like that to happen on the server without the need to download and upload each document.
In other words, I am looking for a solution that will iterate over all things in my collection, sum the ratings from the user reviews then divide that sum by the count of ratings (thus getting the average) and store that average in a property of each document.
I tried approaching this with a map/reduce at first, something like:
function () {
var rating = 0;
for (var i = 0; i < this.Reviews.length; i++) {
rating += this.Reviews[i].Rating;
}
rating = rating / this.Reviews.length;
emit(this._id, rating);
}
function(key, values) {
return null;
}
The ratings seem to be calculated properly, but I was unable to call db.things.update() inside either the first or the second function. I was able to update the ratings by using the resulting table like so:
db.reduced.find().forEach(function(thing) {
db.things.update({_id: thing._id},{$set:{Rating:thing.value}});
});
But this is a client side solution which is not satisfactory... It doesn't have to be map/reduce, but probably that is the way to go?
Use db.eval( your_JS_code_here ).