In Firebase Cloud Firestore, I have "user_goals" in collections and goals may be a predefined goal (master_id: "XXXX") or a custom goal (no "master_id" key)
In JavaScript, I need to write two functions, one to get all predefined goals and other to get all custom goals.
I have got some workaround to get custom goals by setting "master_id" as "" empty string and able to get as below:
db.collection('user_goals')
.where('challenge_id', '==', '') // workaround works
.get()
Still this is not the correct way, I continued to use this for predefined goals where it has a "master_id" as below
db.collection('user_goals')
.where('challenge_id', '<', '') // this workaround
.where('challenge_id', '>', '') // is not working
.get()
Since Firestore has no "!=" operator, I need to use "<" and ">" operator but still no success.
Question: Ignoring these workarounds, what is the preferred way to get docs by checking whether a specific field exists or does not exists?
As #Emile Moureau solution. I prefer
.orderBy(`field`)
To query documents with the field exists. Since it will work with any type of data with any value even for null.
But as #Doug Stevenson said:
You can't query for something that doesn't exist in Firestore. A field needs to exist in order for a Firestore index to be aware of it.
You can't query for documents without the field. At least for now.
The preferred way to get docs where a specified field exists is to use the:
.orderBy(fieldPath)
As specified in the Firebase documentation:
Thus the answer provided by #hisoft is valid. I just decided to provide the official source, as the question was for the preferred way.
Firestore is an indexed database. For each field in a document, that document is inserted into that field's index as appropriate based on your configuration. If a document doesn't contain a particular field (like challenge_id) it will not appear in that field's index and will be omitted from queries on that field. Normally, because of the way Firestore is designed, queries should read an index in one continuous sweep. Prior to the introduction of the != and not-in operators, this meant you couldn't exclude particular values as this would require jumping over sections of an index. This limitation is still encountered when trying to use exclusive ranges (v<2 || v>4) in a single query.
Field values are sorted according to the Realtime Database sort order except that the results can be sorted by multiple fields when duplicates are encountered instead of just the document's ID.
Firestore Value Sort Order
Priority
Sorted Values
Priority
Sorted Values
1
null
6
strings
2
false
7
DocumentReference
3
true
8
GeoPoint
4
numbers
9
arrays
5
Timestamp
10
maps
Inequality !=/<>
This section documents how inequalities worked prior to the release of the != and not-in operators in Sep 2020. See the documentation on how to use these operators. The following section will be left for historical purposes.
To perform an inequality query on Firestore, you must rework your query so that it can be read by reading from Firestore's indexes. For an inequality, this is done by using two queries - one for values less than the equality and another for values greater than the equality.
As a trivial example, let's say I wanted the numbers that aren't equal to 3.
const someNumbersThatAreNotThree = someNumbers.filter(n => n !== 3)
can be written as
const someNumbersThatAreNotThree = [
...someNumbers.filter(n => n < 3),
...someNumbers.filter(n => n > 3)
];
Applying this to Firestore, you can convert this (formerly) incorrect query:
const docsWithChallengeID = await colRef
.where('challenge_id', '!=', '')
.get()
.then(querySnapshot => querySnapshot.docs);
into these two queries and merge their results:
const docsWithChallengeID = await Promise.all([
colRef
.orderBy('challenge_id')
.endBefore('')
.get()
.then(querySnapshot => querySnapshot.docs),
colRef
.orderBy('challenge_id')
.startAfter('')
.get()
.then(querySnapshot => querySnapshot.docs),
]).then(results => results.flat());
Important Note: The requesting user must be able to read all the documents that would match the queries to not get a permissions error.
Missing/Undefined Fields
Simply put, in Firestore, if a field doesn't appear in a document, that document won't appear in that field's index. This is in contrast to the Realtime Database where omitted fields had a value of null.
Because of the nature of NoSQL databases where the schema you are working with might change leaving your older documents with missing fields, you might need a solution to "patch your database". To do this, you would iterate over your collection and add the new field to the documents where it is missing.
To avoid permissions errors, it is best to make these adjustments using the Admin SDK with a service account, but you can do this using a regular SDK using a user with the appropriate read/write access to your database.
This function is recursive, and is intended to be executed once.
async function addDefaultValueForField(queryRef, fieldName, defaultFieldValue, pageSize = 100) {
let checkedCount = 0, pageCount = 1;
const initFieldPromises = [], newData = { [fieldName]: defaultFieldValue };
// get first page of results
console.log(`Fetching page ${pageCount}...`);
let querySnapshot = await queryRef
.limit(pageSize)
.get();
// while page has data, parse documents
while (!querySnapshot.empty) {
// for fetching the next page
let lastSnapshot = undefined;
// for each document in this page, add the field as needed
querySnapshot.forEach(doc => {
if (doc.get(fieldName) === undefined) {
const addFieldPromise = doc.ref.update(newData)
.then(
() => ({ success: true, ref: doc.ref }),
(error) => ({ success: false, ref: doc.ref, error }) // trap errors for later analysis
);
initFieldPromises.push(addFieldPromise);
}
lastSnapshot = doc;
});
checkedCount += querySnapshot.size;
pageCount++;
// fetch next page of results
console.log(`Fetching page ${pageCount}... (${checkedCount} documents checked so far, ${initFieldPromises.length} need initialization)`);
querySnapshot = await queryRef
.limit(pageSize)
.startAfter(lastSnapshot)
.get();
}
console.log(`Finished searching documents. Waiting for writes to complete...`);
// wait for all writes to resolve
const initFieldResults = await Promise.all(initFieldPromises);
console.log(`Finished`);
// count & sort results
let initializedCount = 0, errored = [];
initFieldResults.forEach((res) => {
if (res.success) {
initializedCount++;
} else {
errored.push(res);
}
});
const results = {
attemptedCount: initFieldResults.length,
checkedCount,
errored,
erroredCount: errored.length,
initializedCount
};
console.log([
`From ${results.checkedCount} documents, ${results.attemptedCount} needed the "${fieldName}" field added.`,
results.attemptedCount == 0
? ""
: ` ${results.initializedCount} were successfully updated and ${results.erroredCount} failed.`
].join(""));
const errorCountByCode = errored.reduce((counters, result) => {
const code = result.error.code || "unknown";
counters[code] = (counters[code] || 0) + 1;
return counters;
}, {});
console.log("Errors by reported code:", errorCountByCode);
return results;
}
You would then apply changes using:
const goalsQuery = firebase.firestore()
.collection("user_goals");
addDefaultValueForField(goalsQuery, "challenge_id", "")
.catch((err) => console.error("failed to patch collection with new default value", err));
The above function could also be tweaked to allow the default value to be calculated based on the document's other fields:
let getUpdateData;
if (typeof defaultFieldValue === "function") {
getUpdateData = (doc) => ({ [fieldName]: defaultFieldValue(doc) });
} else {
const updateData = { [fieldName]: defaultFieldValue };
getUpdateData = () => updateData;
}
/* ... later ... */
const addFieldPromise = doc.ref.update(getUpdateData(doc))
The solution I use is:
Use: .where('field', '>', ''),
Where "field" is the field we are looking for!
As you correctly state, it is not possible to filter based on !=. If possible, I would add an extra field to define the goal type. It is possible to use != in security rules, along with various string comparison methods, so you can enforce the correct goal type, based on your challenge_id format.
Specify the goal type
Create a type field and filter based on this field.
type: master or type: custom and search .where('type', '==', 'master') or search for custom.
Flag custom goals
Create a customGoal field which can be true or false.
customGoal: true and search .where('customGoal', '==', true) or false (as required).
Update
It is now possible to perform a != query in Cloud Firestore
Firestore does pick up on boolean, which is a thing! and can be orderBy'd.
So often, like now, for this, I add this into the array-pushing from onSnapshot or get, use .get().then( for dev...
if (this.props.auth !== undefined) {
if (community && community.place_name) {
const sc =
community.place_name && community.place_name.split(",")[1];
const splitComma = sc ? sc : false
if (community.splitComma !== splitComma) {
firebase
.firestore()
.collection("communities")
.doc(community.id)
.update({ splitComma });
}
const sc2 =
community.place_name && community.place_name.split(",")[2];
const splitComma2 =sc2 ? sc2 : false
console.log(splitComma2);
if (community.splitComma2 !== splitComma2) {
firebase
.firestore()
.collection("communities")
.doc(community.id)
.update({
splitComma2
});
}
}
This way, I can query with orderBy instead of where
browseCommunities = (paginate, cities) => {
const collection = firebase.firestore().collection("communities");
const query =
cities === 1 //countries
? collection.where("splitComma2", "==", false) //without a second comma
: cities //cities
? collection
.where("splitComma2", ">", "")
.orderBy("splitComma2", "desc") //has at least two
: collection.orderBy("members", "desc");
var shot = null;
if (!paginate) {
shot = query.limit(10);
} else if (paginate === "undo") {
shot = query.startAfter(this.state.undoCommunity).limit(10);
} else if (paginate === "last") {
shot = query.endBefore(this.state.lastCommunity).limitToLast(10);
}
shot &&
shot.onSnapshot(
(querySnapshot) => {
let p = 0;
let browsedCommunities = [];
if (querySnapshot.empty) {
this.setState({
[nuller]: null
});
}
querySnapshot.docs.forEach((doc) => {
p++;
if (doc.exists) {
var community = doc.data();
community.id = doc.id;
It is not an ideal solution, but here is my workaround when a field does not exist:
let user_goals = await db.collection('user_goals').get()
user_goals.forEach(goal => {
let data = goal.data()
if(!Object.keys(data).includes(challenge_id)){
//Perform your task here
}
})
Note that it would impact your read counts a lot so only use this if you have small collection or can afford the reads.
I have three fields to be searched inside a mongodb database. Sometimes I have to search based on only one field, sometimes two and sometimes three as well. It is something like-if there is only one field value present, then search on the basis of only one field, if two are there then find using like $and and if three all of the three,find using like $and. Is there any single query to make the search working?
the below is the image where i should make selection from
You need to make a query string like below,
let queryString = {};
if (field1 !== undefined) {
queryString.field1 = field1;
}
if (field2 !== undefined) {
queryString.field2 = field2;
}
if (field3 !== undefined) {
queryString.field3 = field3;
}
let documents = await db.find({ $and: [queryString] }).exec();
I want to compare two fields in my Waterline query in Sails.js application, e.g.: SELECT * FROM entity E WHERE E.current < E.max.
I've tried the following code, but it's expecting integer value to be passed to it instead of column name:
Entity.find({
where: {
current: {'<': 'max'}
}
});
So, how do I compare two columns?
I have ran some tests and at the same time read the Waterline documentation. There is no indication of anything that could possibly do comparison of two fields/columns via .find() or .where() methods. Reference: http://sailsjs.org/documentation/concepts/models-and-orm/query-language
Instead, I have used .query() method to compare two fields via SQL string such as :
Entity.query("SELECT * FROM `Entity` E WHERE E.current < E.max", function( err, res) {
if(err) {
//exception
} else {
console.log('response',res);
}
});
The other way would be to use one query to get the max before putting it in the criteria.
EntityOne.find({
sort: 'column DESC'
}).limit(1)
.exec(function(err,found){
var max = found[0]
EntityTwo.find({
where: {
current: {'<': found}
}
}).exec((err,found) {
// Do stuff here
});
});
The query method is ultimately going to be faster however
Setup:
I got a large collection with the following entries
Name - String
Begin - time stamp
End - time stamp
Problem:
I want to get the gaps between documents, Using the map-reduce paradigm.
Approach:
I'm trying to set a new collection of pairs mid, after that I can compute differences from it using $unwind and Pair[1].Begin - Pair[0].End
function map(){
emit(0, this)
}
function reduce(){
var i = 0;
var pairs = [];
while ( i < values.length -1){
pairs.push([values[i], values[i+1]]);
i = i + 1;
}
return {"pairs":pairs};
}
db.collection.mapReduce(map, reduce, sort:{begin:1}, out:{replace:"mid"})
This works with limited number of document because of the 16MB document cap. I'm not sure if I need to get the collection into memory and doing it there, How else can I approach this problem?
The mapReduce function of MongoDB has a different way of handling what you propose than the method you are using to solve it. The key factor here is "keeping" the "previous" document in order to make the comparison to the next.
The actual mechanism that supports this is the "scope" functionality, which allows a sort of "global" variable approach to use in the overall code. As you will see, what you are asking when that is considered takes no "reduction" at all as there is no "grouping", just emission of document "pair" data:
db.collection.mapReduce(
function() {
if ( last == null ) {
last = this;
} else {
emit(
{
"start_id": last._id,
"end_id": this._id
},
this.Begin - last.End
);
last = this;
}
},
function() {}, // no reduction required
{
"out": { "inline": 1 },
"scope": { "last": null }
}
)
Out with a collection as the output as required to your size.
But this way by using a "global" to keep the last document then the code is both simple and efficient.
I have a link tracking table that has (amongst other fields) track_redirect and track_userid. I would like to output both the total count for a given link, and also the unique count - counting duplicates by the user id. So we can differentiate if someone has clicked the same link 5 times.
I've tried emitting this.track_userid in both the key and values parts but can't get to grips with how to correctly access them in the reduce function.
So if I roll back to when it actually worked, I have the very simple code below - just like it would be in a 'my first mapreduce function' example
map
function() {
if(this.track_redirect) {
emit(this.track_redirect,1);
}
}
reduce
function(k, vals) {
var sum = 0;
for (var i in vals) {
sum += vals[i];
}
return sum;
}
I'd like to know the correct way to emit the additional userid information and access it in the mapreduce please. or am i thinking about it in the wrong way?
in case it's not clear, I don't want to calculate the total clicks a userid has made, but to count the unique clicks of each url + userid - not counting any duplicate clicks a userid made on each link
can someone point me in the right direction please? thanks!
You can actually pass arbitrary object on the second parameter of the emit call. That means you can take advantage of this and store the userid in it. For example, your map function can look like this:
var mapFunc = function() {
if (this.track_redirect) {
var tempDoc = {};
tempDoc[this.track_userid] = 1;
emit(this.track_redirect, {
users_clicked: tempDoc,
total_clicks: 1
});
}
};
And your reduce function might look like this:
var reduceFunc = function(key, values) {
var summary = {
users_clicked: {},
total_clicks: 0
};
values.forEach(function (doc) {
summary.total_clicks += doc.total_clicks;
// Merge the properties of 2 objects together
// (and these are actually the userids)
Object.extend(summary.users_clicked, doc.users_clicked);
});
return summary;
};
The users_clicked property of the summary object basically stores the id of every user as a property (since you can't have duplicate properties, you can guarantee that it will store unique users). Also note that you have to be careful of the fact that some of the values passed to the reduce function can be result of a previous reduce and the sample code above takes that into account. You can find more about the said behavior in the docs here.
In order to get the unique count, you can pass in the finalizer function that gets called when the reduce phase is completed:
var finalFunc = function(key, value) {
// Counts the keys of an object. Taken from:
// http://stackoverflow.com/questions/18912/how-to-find-keys-of-a-hash
var countKeys = function(obj) {
var count = 0;
for(var i in obj) {
if (obj.hasOwnProperty(i))
{
count++;
}
}
return count;
};
return {
redirect: key,
total_clicks: value.total_clicks,
unique_clicks: countKeys(value.users_clicked)
};
};
Finally, you can execute the map reduce job like this (modify the out attribute to fit your needs):
db.users.mapReduce(mapFunc, reduceFunc, { finalize: finalFunc, out: { inline: 1 }});