Firebase Combining Query and Pagination - swift

Query can be used to filter a large items down to a smaller number suitable for synchronizing to the client.
Pagination is a also serve the same purpose, that is to limit the items to a smaller numbers suitable to be be fetched by the client.
Consider the following database schema:
"users": {
"-KRyXWjI0X6UvffIB_Gc": {
"active": true,
"name": "John Doe",
"occupation": "Looking for firebase answer"
},
"-KRyXBWwaK112OWGw5fa": {
"active": false,
"name": "Jane Doe",
"occupation": "Still stuck on combining query and pagination"
},
"-KRyWfOg7Nj59qtoCG30": {
"active": true,
"name": "Johnnie Doe",
"occupation": "There is no greater sorrow than to recall in misery the time when we were stuck"
}
}
If I were to get all the active users, it will be like this: (Code in Swift)
let usersRef = ref.child("users");
let query = usersRef.queryOrderedByChild("active")
.queryEqualToValue(true)
After that filtering, it left me with 10,000 users. Fetching all of those users at the same time is out of question. It must be paginated.
To do the pagination, I have to do the query on the unique sorted value, which is none other than the key itself. This is how it looks now:
let usersRef = ref.child("users");
let query = usersRef.queryOrderedByChild("active")
.queryEqualToValue(true)
let usersPerPage = 10;
query.queryOrderedByKey()
.queryStartingAtValue(lastKey)
.queryLimitedToFirst(usersPerPage)
This wouldn't work because:
You can only use one order-by method at a time. Calling an order-by
method multiple times in the same query throws an error.
After I spent 2 days on thinking how am I supposed to solve this situation, I can only came up with this "anti best practice" solution.
I modified the database schema. I convert the active boolean value to the string and append it after the key to give the order importance control to the key. This is how it looks now:
"users": {
"-KRyXWjI0X6UvffIB_Gc": {
"key_active": "-KRyXWjI0X6UvffIB_Gc true"
"active": true,
"name": "John Doe",
"occupation": "Looking for firebase answer"
}
}
Now I could do both the pagination and the query using the single orderBy:
let usersPerPage = 10;
query.queryOrderedByChild("key_active")
.queryStartingAtValue("\(lastKey) true")
.queryLimitedToFirst(usersPerPage)
Somehow my brain reject the idea of having the key inside the key because it's the worst dirty solution it can be. I want to know the right solution for this particular situation, any solution would be greatly appreciated.

Just add this to your JSON tree:-
active_Users :{
uid1 : {
index : 1,
name : "John Doe"
},
uid2 : {
index : 2,
name : "Johnnie Doe"
}
}
After this just follow this answer :- Retrieving 5 user's at a time, modify it according to your requirements.
Note:- Total no of users/posts in that answers is being retrieved by doing children count. Given your heavy database, you might wanna store totalNoOfUser in a separate node and increment it every time a new user is added.

Related

What has better performance, double parallel query or query and then array.reduce? MongoDB and JavaScript Performance

I want to know, in the long run, what option has better performance, assuming that DB is full of documents (may be thousands).
1.Does two queries in parallel:
The first query gets all rooms properties that user is in.
The second query gets a list of different rooms where user is in and each room has sharing property set to true.
const roomsInfo = await Promise.all([
db.collection('rooms').find({'users.userId': user.userId}).toArray()
db.collection('rooms').distinct('users.id', {'users.id': "myuserId", share: true}),
])
Example output:
roomsInfo[0] = [{
"id": "room1",
"name": "room1name",
"data": "XXXX",
"users" [{
"id": "User1",
"data": "XXXX",
},{
"id": "User2",
"data": "XXXX",
}],
share: true
}, {
"id": "room2",
...
"share": true
},{
"id": "room3",
...
"share": false
},{
"id": "room4",
...
"share": false
}]
roomsInfo[1] = ["room1", "room2"]
2.Does one query and then reduces:
const roomInfo = await db.collection('rooms').find({'users.id': "myuserId"}).toArray()
const roomFiltered = roomInfo.reduce((a, o) => (o.share && a.push(o.id), a), [])
Second option has same result as first one
(2nd option) roomInfo = roomInfo[0] (1st option)
(2nd option) roomFiltered = roomInfo[1] (1st option)
Thank you very much in advance.
I think to use the first solution (two queries in parallel)
for example, you have 1000000 records in your DB , speed of MongoDB for distinct query is faster than of your local for loop
Tested it with 100.000, 500.000 and 1M documents using mgodatagen and my conclusions are:
Queries in parallel are better than query and filter, map, reduce or any other array.method for big data.
When you've big data in your database you just query by index. If not you won't have any result in a reasonable time.
Divide your queries as many as you can: for example, if you query all rooms documents with a given array of rooms Id's, and rooms are divided in 2 types (single and group), is better to do 2 parallel queries first finding all single rooms that correspond to given array and second, all group rooms that correspond to given array.

Firebase observe key contains

I'd like to fetch snapshot which contains typed text. For example node look like this
"Team": {
"Yankees": {
"uid1": "name",
"uid2": "name"
},
"Angels": {
"uid1": "name"
"uid3": "name"
}
and if user typed yan in search bar then I want to fetch "Yankees" snapshot. I saw some document and stack over flow post and tried like so
ref.child("Team").queryStarting(atValue: "yan").queryEnding(atValue: "yan\u{f8ff}").observe
but it doesn't work. How can I do this? Thank you!
Firebase searches are case sensitive. Since your key starts with an uppercase Y, the query only matches if it also does that:
ref.child("Team")
.queryOrderedByKey()
.queryStarting(atValue: "Yan")
.queryEnding(atValue: "Yan\u{f8ff}").observe
I also queryOrderedByKey() to be explicit about what you want to order/filter on.
If you want to allow case-insensitive filtering, the typical approach is to add a property with the all-lowercase value to each team:
"Team": {
"Yankees": {
"teamNameForSearch": "yankees",
"uid1": "name",
"uid2": "name"
},
"Angels": {
"teamNameForSearch": "angels",
"uid1": "name"
"uid3": "name"
}
Now you can search with:
ref.child("Team")
.queryOrdered(byChild: "teamNameForSearch")
.queryStarting(atValue: "yan")
.queryEnding(atValue: "yan\u{f8ff}").observe
A final note is that both approaches only do so-called prefix matches: they find teams whose name starts with what the user typed. If you want a contains operation (as the title of your question suggests), you will have to look beyond Firebase for a solution. For more on that, see Kato's answer here: Firebase query - Find item with child that contains string
You need to change the db to this:
"Team": {
"randomid": {
"team":"Yankees",
"uid1": "name",
"uid2": "name"
},
"randomid": {
"team":"Angels"
"uid1": "name"
"uid3": "name"
}
and now you can do this:
ref.child("Team").queryOrdered(byChild: "team").queryStarting(atValue: "Yan").queryEnding(atValue: "Yan\u{f8ff}").observe
First in your query above, you need to use queryOrdered() to know which child you want to order.
Also in your database the node Team is not equal to anything, it is a parent node that has child nodes.
So to fix this, the node Team needs to be equal to a value(like in the database in this answer) so you will be able to order it and use queryStarting and queryEnding on it.

MongoDB - Manual Referencing Without Application

I understand Manual referencing being, one document containing the reference to another.
As in the MongoDB's documentation:
original_id = ObjectId()
db.places.insert({
"_id": original_id,
"name": "Broadway Center",
"url": "bc.example.net"
})
db.people.insert({
"name": "Erin",
"places_id": original_id,
"url": "bc.example.net/Erin"
})
I am able to use a find on the places to get a certain name.
db.places.find({name : "Broadway Center"});
This will give the id. And then my application can use this ID to query the people to see who live here?
But, If i dont have a application. How would I go about doing this solely in a mongo shell? I was thinking about using find, and then iterating over the cursor using a forEach? But this seems a bit hacky.
Any suggestions?
You don't need to loop through using the cursor from find(), just use findOne() because a single document is returned with this method and you have access to the document directly (no need to apply cursor methods to the result). In mongo shell you can do this:
var result = db.places.findOne({name : "Broadway Center"});
if (result) {
var place_id = result._id;
var peopleArray = db.people.find({ "places_id": place_id }).toArray();
if (peopleArray.length > 0) { printjson (peopleArray[0]); }
}

mongodb-php: "key" side value for nested querying of find() function doesnot work

I want to retrive record which are matching to booking's client id & want to show it to client. I am doing the following:
$mongoDb = $mongoDb->selectCollection('booking');
$bookingInfo = $mongoDb->find(array("client.id" => $_SESSION['client_id']));
My mongo database record looks like this:
"paymentDue": "",
"client": {
"contacts": [
{
"name": "loy furison",
"email": "loy#hotmail.com"
}
],
"id": "5492abba64363df013000029",
"name": "Birdfire"
},
want to fire the query with key value as client.id in find function. But this query doesnt work..whats the issue
I got a little logic that is different by key name only. If i find it with client.name then i shows me records & there i need to insert these in json object & then through foreach loop each record if i retrive & compare then it works...got it but the expected doesnt work why?????...didnt get:-!

How do I manage a sublist in Mongodb?

I have different types of data that would be difficult to model and scale with a relational database (e.g., a product type)
I'm interested in using Mongodb to solve this problem.
I am referencing the documentation at mongodb's website:
http://docs.mongodb.org/manual/tutorial/model-referenced-one-to-many-relationships-between-documents/
For the data type that I am storing, I need to also maintain a relational list of id's where this particular product is available (e.g., store location id's).
In their example regarding "one-to-many relationships with embedded documents", they have the following:
{
name: "O'Reilly Media",
founded: 1980,
location: "CA",
books: [12346789, 234567890, ...]
}
I am currently importing the data with a spreadsheet, and want to use a batchInsert.
To avoid duplicates, I assume that:
1) I need to do an ensure index on the ID, and ignore errors on the insert?
2) Do I then need to loop through all the ID's to insert a new related ID to the books?
Your question could possibly be defined a little better, but let's consider the case that you have rows in a spreadsheet or other source that are all de-normalized in some way. So in a JSON representation the rows would be something like this:
{
"publisher": "O'Reilly Media",
"founded": 1980,
"location": "CA",
"book": 12346789
},
{
"publisher": "O'Reilly Media",
"founded": 1980,
"location": "CA",
"book": 234567890
}
So in order to get those sort of row results into the structure you wanted, one way to do this would be using the "upsert" functionality of the .update() method:
So assuming you have some way of looping the input values and they are identified with some structure then an analog to this would be something like:
books.forEach(function(book) {
db.publishers.update(
{
"name": book.publisher
},
{
"$setOnInsert": {
"founded": book.founded,
"location": book.location,
},
"$addToSet": { "books": book.book }
},
{ "upsert": true }
);
})
This essentially simplified the code so that MongoDB is doing all of the data collection work for you. So where the "name" of the publisher is considered to be unique, what the statement does is first search for a document in the collection that matches the query condition given, as the "name".
In the case where that document is not found, then a new document is inserted. So either the database or driver will take care of creating the new _id value for this document and your "condition" is also automatically inserted to the new document since it was an implied value that should exist.
The usage of the $setOnInsert operator is to say that those fields will only be set when a new document is created. The final part uses $addToSet in order to "push" the book values that have not already been found into the "books" array (or set).
The reason for the separation is for when a document is actually found to exist with the specified "publisher" name. In this case, all of the fields under the $setOnInsert will be ignored as they should already be in the document. So only the $addToSet operation is processed and sent to the server in order to add the new entry to the "books" array (set) and where it does not already exist.
So that would be simplified logic compared to aggregating the new records in code before sending a new insert operation. However it is not very "batch" like as you are still performing some operation to the server for each row.
This is fixed in MongoDB version 2.6 and above as there is now the ability to do "batch" updates. So with a similar analog:
var batch = [];
books.forEach(function(book) {
batch.push({
"q": { "name": book.publisher },
"u": {
"$setOnInsert": {
"founded": book.founded,
"location": book.location,
},
"$addToSet": { "books": book.book }
},
"upsert": true
});
if ( ( batch.length % 500 ) == 0 ) {
db.runCommand( "update", "updates": batch );
batch = [];
}
});
db.runCommand( "update", "updates": batch );
So what is doing in setting up all of the constructed update statements into a single call to the server with a sensible size of operations sent in the batch, in this case once every 500 items processed. The actual limit is the BSON document maximum of 16MB so this can be altered appropriate to your data.
If your MongoDB version is lower than 2.6 then you either use the first form or do something similar to the second form using the existing batch insert functionality. But if you choose to insert then you need to do all the pre-aggregation work within your code.
All of the methods are of course supported with the PHP driver, so it is just a matter of adapting this to your actual code and which course you want to take.