I'd like to fetch snapshot which contains typed text. For example node look like this
"Team": {
"Yankees": {
"uid1": "name",
"uid2": "name"
},
"Angels": {
"uid1": "name"
"uid3": "name"
}
and if user typed yan in search bar then I want to fetch "Yankees" snapshot. I saw some document and stack over flow post and tried like so
ref.child("Team").queryStarting(atValue: "yan").queryEnding(atValue: "yan\u{f8ff}").observe
but it doesn't work. How can I do this? Thank you!
Firebase searches are case sensitive. Since your key starts with an uppercase Y, the query only matches if it also does that:
ref.child("Team")
.queryOrderedByKey()
.queryStarting(atValue: "Yan")
.queryEnding(atValue: "Yan\u{f8ff}").observe
I also queryOrderedByKey() to be explicit about what you want to order/filter on.
If you want to allow case-insensitive filtering, the typical approach is to add a property with the all-lowercase value to each team:
"Team": {
"Yankees": {
"teamNameForSearch": "yankees",
"uid1": "name",
"uid2": "name"
},
"Angels": {
"teamNameForSearch": "angels",
"uid1": "name"
"uid3": "name"
}
Now you can search with:
ref.child("Team")
.queryOrdered(byChild: "teamNameForSearch")
.queryStarting(atValue: "yan")
.queryEnding(atValue: "yan\u{f8ff}").observe
A final note is that both approaches only do so-called prefix matches: they find teams whose name starts with what the user typed. If you want a contains operation (as the title of your question suggests), you will have to look beyond Firebase for a solution. For more on that, see Kato's answer here: Firebase query - Find item with child that contains string
You need to change the db to this:
"Team": {
"randomid": {
"team":"Yankees",
"uid1": "name",
"uid2": "name"
},
"randomid": {
"team":"Angels"
"uid1": "name"
"uid3": "name"
}
and now you can do this:
ref.child("Team").queryOrdered(byChild: "team").queryStarting(atValue: "Yan").queryEnding(atValue: "Yan\u{f8ff}").observe
First in your query above, you need to use queryOrdered() to know which child you want to order.
Also in your database the node Team is not equal to anything, it is a parent node that has child nodes.
So to fix this, the node Team needs to be equal to a value(like in the database in this answer) so you will be able to order it and use queryStarting and queryEnding on it.
Related
I understand that index has a cost in firestore. Most of the time we simply store objects without really caring about index and even if we don’t want most of the fields to be indexed.
If I understand correctly, any field at any level are indexed. I.e. for the following document in pseudo json
{
"root_field1": "abc" (indexed)
"root_field2": "def" (indexed)
"root_field3": {
"Sub_field1: "ghi" (indexed)
"sub_field2: "jkl" (indexed)
"sub_field3: {
"Inner_field1: "mno" (indexed)
"Inner_field2: "pqr" (indexed)
}
}
Let’s assume that I have the following record
{
"name": "abc"
"birthdate": "2000-01-01"
"gender": "m"
}
Let’s assume that I just want the field "name" to be indexed. One solution (A), without having to specify every field is to define it this way (i.e. move the root fields to a sub level unindexed), and exclude unindexed from being indexed
{
"name": "abc"
"unindexed" {
"birthdate": "2000-01-01"
"gender": "m"
}
Ideally I would like to just specify a prefix such as _ to prevent each field to be indexed but there is no global solution for that.
{
"name": "abc"
"_birthdate": "2000-01-01"
"_gender": "m"
}
Is my solution (A) correct and is there a more elegant generic solution?
Thanks!
Accordinig to the documentation
https://cloud.google.com/firestore/docs/query-data/indexing
Add a single-field index exemption
Single-field index exemptions allow you to override automatic index
settings for specific fields in a collection. You can add a
single-field exemptions from the console:
Go to the Single Field Indexes section.
Click Add Exemption.
Enter a Collection ID and Field path.
Select new indexing settings for this field. Enable or disable
automatically updated ascending, descending, and array-contains
single-field indexes for this field.
Click Save Exemption.
I have documents that I want to index into Elasticsearch with an existing unique "id" field.
I get an array of documents from a REST api endpoint ( eg.: http://some.url/api/products) in no particular order and if a document with the _id already exists in Elasticsearch it should update and reindex the document.
I want to create a new document if no document with the _id in Elasticsearch exists and then update a document, if it matches with an existing document in Elasticsearch.
This could be done with:
PUT products/product/un1qu3-1d-b718-105973677e95
{
"id": "un1qu3-1d-b718-105973677e95",
"state": "packaged"
}
The basic idea is to use the provided "id" field to create or update a document. Extraction of _id from document fields seems deprecated (link). But the indexing/ reindexing of documents with the "id" field can be done manually very easy with the kibana dev tools, with postman or a cURL request.
I want to achieve this (re-)indexing of documents that I receive over this api endpoint programmatically.
Is it possible to achieve this with logstash or a simple cronjob? Does Elasticsearch provide any functionality for this? Or do I need to write some custom backend to achieve this?
I thought of either:
1) index the document into Elasticsearch with the "id" field of my document or
2) find an Elasticsearch query that first searches for the document with the specific "id" field and then updates the document.
I was unable to find a solution for either way and have no clue how a good approach would look like.
Can anyone point me into the right direction on how to achieve this, suggest a better approach or provide a solution?
Any help much appreciated!
Update
I solved the problem with the help of the accepted answer. I used Logstash, the Http_poller input plugin, this article: https://www.elastic.co/blog/new-way-to-ingest-part-1 and this elastic.co question: https://discuss.elastic.co/t/upsert-with-logstash/59116
My output of logstash looks like this at the moment:
output {
elasticsearch {
index => "products"
document_type => "product"
pipeline => "rename_id"
document_id => "%{id}"
doc_as_upsert => true
action => "update"
}
Update 2
just for the sake of completeness I added the "rename_id" pipeline
{
"rename_id": {
"description": "_description",
"processors": [
{
"set": {
"field": "_id",
"value": "{{id}}"
}
}
]
}
}
It works this way!
Thanks alot!
Peter,
If I understand correctly, you want to ingest your documents into elastic search and will have some updates in future for these documents ?
If that's the case,
- Use your documents primary key as id for elastic documents.
- You can ingest entire document with updated values, elastic will replace the previous document with new one. given the primary key is same. Old document with same id will be deleted.
We use this approach for our search data.
you can use ingest pipelines to extract the id from the body and the _create endpoint to only create a document if it does not exist. Minor note: If you could specify the id on the client side indexing would be faster, as adding a pipeline adds a certain overhead.
PUT _ingest/pipeline/my_pipeline
{
"description": "_description",
"processors": [
{
"set": {
"field": "_id",
"value": "{{id}}"
}
}
]
}
PUT twitter/tweet/1?op_type=create&pipeline=my_pipeline
{
"foo" : "bar",
"id" : "123"
}
GET twitter/tweet/123
# this call will fail
PUT twitter/tweet/1?op_type=create&pipeline=my_pipeline
{
"foo" : "bar",
"id" : "123"
}
You can use script to UPSERT (update or insert) your document
PUT /products/product/un1qu3-1d-b718-105973677e95/_update
{
"script": {
"inline": "ctx._source.state = \"packaged\"",
"lang": "painless"
},
"upsert": {
"id": "un1qu3-1d-b718-105973677e95",
"state": "packaged"
}
}
Above query find the document with _id = "un1qu3-1d-b718-105973677e95"
if it is able to find any document then it will update state to "packaged" otherwise create a new document with field "id" and "state" (you can insert as many fields as you want).
Let's say I have a blog dataset which looks like ths
{
"_id" : "abcde",
"author" : "xyz",
"body" : "this is test body",
"date" : "xyz",
"tags" : ["tag1","tag2"],
"comments":[
{ "body": "comment1",
"email": "email1",
"author" :"author1"
},
{ "body": "comment2",
"email": "email2",
"author" :"author2"
}
]
}
Here each document is representing a blog post. Each blog post can have multiple comments, but one my one user. Let's say I have to find all comments made by a particular author. What will be the query for that?
One approach would be to unwind comments, then group by comments.author and push comments.body.
However, Can it be done without aggregation pipeline, just by using find? Because I think I am supposed to do it just by find. Can anyone help me with this?
You can use dot notation to query against an array of subdocuments.
Your query would look something like this:
db.blog_posts.find({"comments.author": "author1"});
This will return all documents which contain subdocuments that have an author value of author1. The results will be the entire post document, so you might want to specify the fields that you want returned. This is called query projection.
db.blog_posts.find({"comments.author": "author1"});
To specify what fields you want, add an extra object as the second parameter to your .find() function:
db.blog_posts.find({"comments.author": "author1"}, {"comments": 1} );
Now the resulting documents will only contain the _id field and the comments field.
Keep in mind that you are actually querying the blog posts collection so your returned results will still be the blog posts - but only the ones where the specified author has commented on.
A simple way to take the next step and extract only the comment objects might look something like this:
var author = 'author1';
var comments = [];
db.blog_posts.find(...).forEach(function(doc){
var post_comments = doc.comments;
for (var i=0; i<post_comments.length; i++){
if (post_comments[i].author == author){
comments.push(post_comments[i]);
}
}
});
Here is the relevant documentation page:
http://docs.mongodb.org/manual/tutorial/query-documents/#array-of-embedded-documents
Match a Field Without Specifying Array Index
If you do not know the index position of the document in the array, concatenate the name of the field that contains the array, with a dot (.) and the name of the field in the embedded document.
I need to search inside a map element with a certain value in mongodb.
I have this element in data base:
{
"_id": ObjectId("52950e93c4aad399cff0d9f9"),
"_class": "com.company.model.customer.DbCustomer",
"version": NumberLong(0),
"channels": {
"adea3d4e-2a73-4f3e-8a89-a336d6132909": {
"value": "dominik.czech.aal#gmail.com",
"alias": "email1",
"deliveryChannel": "EMAIL",
"status": "GOOD",
"_class": "com.company.model.customer.CustomerEmail"
}
}
}
Where "adea3d4e-2a73-4f3e-8a89-a336d6132909" is a key of a map of channels.
What I want to search is a channel with certain value.
If "channels" were an array the query would be this way:
{ "channels" :
{ "$elemMatch" : { "value" : "dominik.czech.aal#gmail.com" } }
}
But, as channels is a map, I can't use this approach.
Is it possible to search inside a map the same way you search inside an array?
Notice that I want to use a single query, for security reasons I cannot use the map reduce functionality in my database.
Thanks in advance.
AFAIK it's not possible with the current MongoDB operators anyway, without scripting or map/reduce or knowing the keys you want to query in advance.
As a side note, you should think your data structure against how you want to query it - i.e. you should probably consider transform the channels document into an array.
i am not seeing any array set in above sample code, array set must be look like myarray[1,2,3]
so with this sample code if you want to search sub-field value you can try like following.
>db.Collection.find({"channels.value" : "dominik.czech.aal#gmail.com"})
Hope this will help.....
Part 1:
I have (student) collection:
{
sname : "",
studentId: "123"
age: "",
gpa: "",
}
im trying to get only two keys from it :
{
sname : "",
studentId: "123"
}
so i need to eliminate age and gpa to have only name and studentId , how could i do that ?
Part2:
Then I have 'subject' collection :
{
subjectName : "Math"
studentId : "123"
teacherName: ""
}
I need to match/combine the previous keys (in part1) with the correct studentId so I will end up with something like this :
{
sname : "",
studentId: "123",
subjectName : "Math"
}
How can i do this and is that the right way to think to get the result? i tried to read about group and mapReduce but i didnt find a clear example.
To answer your first question, you can do this:
db.student.find({}, {"sname":1, "studentId":1});
The first {} in that is the limiting query, which in this case includes the entire collection. The second half specifies keys with a 1 or 0 depending on whether or not you want them back. Don't mix include and excludes in a single query though. Except for a couple special cases, mongo won't accept it.
Your second question is more difficult. What you're asking for is a join and mongo doesn't support that. There is no way to connect the two collections on studentId. You'll need to find all the students that you want, then use those studentIds to find all the matching subjects. Then you'll need to merge the two results in your own code. You can do this through whatever driver you're using, or you can do this in javascript in the shell itself, but either way, you'll have to merge them with your own code.
Edit:
Here's an example of how you could do this in the shell with the output going to a collection called "out".
db.student.find({}, {"sname":1, "studentId":1}).forEach(
function (st) {
db.subject.find({"studentId":st.studentId}, {"subjectName":1}).forEach(
function (sub) {
db.out.insert({"sname":st.sname, "studentId":st.studentId, "subjectName":sub.subjectName});
}
);
}
);
If this isn't data that changes all that often, you could just drop the "out" collection and repopulate it periodically with this shell script. Then your code could query directly from "out". If the data does change frequently, you'll want to do this merging in your code on the fly.
Another, and possibly better, option is to include the "subject" data in the "student" collection or vice versa. This will result in a more mongodb friendly structure. If you run into this joining problem frequently, mongo may not be the way to go and a relational database may be better suited to your needs.
Mongo's find() operator lets you include or exclude certain fields from the results
Check out Field Selection in the docs for more info. You could do either:
db.users.find({}, { 'sname': 1, 'studentId': 1 });
db.users.find({}, { 'age': 0, 'gpa': 0 });
For relating your student and subject together, you could either lookup which subjects a student has separately, like this:
db.subjects.find({ studentId: 123 });
Or embed subject data with each student, and retrieve it together with the student document:
{
sname : "Roland Browning",
studentId: "123"
age: 14,
gpa: "B",
subjects: [ { name : "French", teacher: "Mr Bronson" }, ... ]
}