For reference here is the code. I am trying to make a hubot plugin that logs to elasticsearch and then uses hubot commands to search those logs.
https://gist.github.com/4050748
I am trying to retrieve records that match two queries.
{
query: {
match: {
user: "SomeUsername"
},
range: {
date: {
from: (Date.now() - 3600)
}
}
},
size: 50
}
I was expecting:
Up to 50 records
records that had the given user
records in the last hour
I got:
up to 10 records
records that had the given user
from any time
How do I get all the records with some username in the last hour? Do I need to use match_all with filters? Is what I am attempting unsupported?
In SQL it would be something like:
Select (*) from messages where user_name = ? and time > ?
For anyone who stumbles on this question and wonders what it looks like to combine a match and range query in ElasticSearch, this example would look like
curl 'localhost:9200/<index>/_search?pretty=true' -d '{
"query" : {
"bool": {
"must": [
{
"match": {
"user": "SomeUsername"
}
},
{
"range" : {
"date": {
"gt": "now-1h"
}
}
}
]
}
}
}'
You need to use the bool query to combine different queries together. You can then choose whether each single query must match, should match (optional), or must not match.
Related
The core collection (other collections in the DB refer back to this one) in my DB contains 3 fields with date information which at this point is formatted as strings like MM/DD/YYYY. Further, there are a range of documents for which this field contains missing data, i.e. "". I populated this collection by running the mongoimport command on a JSON file.
My goal is to convert these date-fields into actual ISODate data types, so as to allow filtering the collection by dates. Further, I want MongoDB to know that empty strings indicate missing values. I have read quite widely on this, leading me to try a bunch of things:
Trying a forEach statement - This worked, but only for the very first document.
db.collection.find().forEach(function(element){
element.startDate = ISODate(element.startDate);
db.collection.save(element);
})
Using kind of a for-loop: this worked well, but stopped once it encountered a missing value (so it transformed about 11 values):
db.collection.update(
{
"startDate":{
"$type":"string"
}
},
[
{
"$set":{
"startDate":{
"$dateFromString":{
"dateString":"$startDate",
"format":"%m/%d/%Y"
}
}
}
}
]
)
So, both of these approaches kind of worked - but I don't know how to apply them to the entire collection. Further, I'd be interested in performing this task in the most efficient way possible. However, I only want to do this once - data that will be added in the future should hopefully be correctly formatted at the import stage.
db.collection.updateMany(
{
"$and": [
{ "startDate": { "$type": "string" } },
{ "startDate": { "$ne": "" } }
]
},
[
{
"$set": {
"startDate": {
"$dateFromString": {
"dateString": "$startDate",
"format": "%m/%d/%Y"
}
}
}
}
]
)
Filtering out empty string than doing the transformation will ignore documents that have empty string in date field.
I'm not quite understanding how fuzzy works in the $searchBeta stage of aggregation. I'm not getting the desired result that I want when I'm trying to implement full-text search on my backend. Full text search for MongoDB was released last year (2019), so there really aren't many tutorials and/or references to go by besides the documentation. I've read the documentation, but I'm still confused, so I would like some clarification.
Let's say I have these 5 documents in my db:
{
"name": "Lightning Bolt",
"set_name": "Masters 25"
},
{
"name": "Snapcaster Mage",
"set_name": "Modern Masters 2017"
},
{
"name": "Verdant Catacombs",
"set_name": "Modern Masters 2017"
},
{
"name": "Chain Lightning",
"set_name": "Battlebond"
},
{
"name": "Battle of Wits",
"set_name": "Magic 2013"
}
And this is my aggregation in MongoDB Compass:
db.cards.aggregate([
{
$searchBeta: {
search: { //search has been deprecated, but it works in MongoDB Compass; replace with 'text'
query: 'lightn',
path: ["name", "set_name"],
fuzzy: {
maxEdits: 1,
prefixLength: 2,
maxExpansion: 100
}
}
}
}
]);
What I'm expecting my result to be:
[
{
"name": "Lightning Bolt", //lightn is in 'Lightning'
"set_name": "Masters 25"
},
{
"name": "Chain Lightning", //lightn is in 'Lightning'
"set_name": "Battlebond"
}
]
What I actually get:
[] //empty array
I don't really understand why my result is empty, so it would be much appreciated if someone explained what I'm doing wrong.
What I think is happening:
db.cards.aggregate... is looking for documents in the "name" and "set_name" fields for words that have a max edit of one character variation from the "lightn" query. The documents that are in the cards collection contain edits that are greater than 2, and therefor your expected result is an empty array. "Fuzzy is used to find strings which are similar to the search term or terms"; used with maxEdits and prefixLength.
Have you tried the term operator with the wildcard option? I think the below aggregation would get you the results you were actually expecting.
e.g.
db.cards.aggregate([
{$searchBeta:
{"term":
{"path":
["name","set_name"],
"query": "l*h*",
"wildcard":true}
}}]).pretty()
You need to provide an index to use with your search query.
The index is basically the analyzer that your query will use to process your results regarding if you want to a full match of the text, or you want a partial match etc.
You can read more about Analyzers from here
In your case, an index based on STANDARD analyzer will help.
After you create your index your code, modified below, will work:
db.cards.aggregate([
{
$search:{
text: { //search has been deprecated, but it works in MongoDB Compass; replace with 'text'
index: 'index_name_for_analyzer (STANDARD in your case)'
query: 'lightn',
path: ["name"] //since you only want to search in one field
fuzzy: {
maxEdits: 1,
prefixLength: 2,
maxExpansion: 100
}
}
}
}
]);
I have a data
[{ "name":"BS",
"keyword":"key1",
"city":"xyz"
},
{ "name":"AGS",
"keyword":"Key2",
"city":"xyz1"
},
{ "name":"QQQ",
"keyword":"key3",
"city":"xyz"
},
{ "name":"BS",
"keyword":"Keyword",
"city":"city"
}]
and i need to search records which have name= "BS" OR keyword="key2" with the help of query
db.collection.find({"$OR" : [{"name":"BS"}, {"keyword":"Key2"}]});
These records i need in the sequence
[{ "name":"BS",
"keyword":"key1",
"city":"xyz"
},
{ "name":"BS",
"keyword":"Keyword",
"city":"city"
},
{ "name":"AGS",
"keyword":"Key2",
"city":"xyz1"
}]
but i am getting in following sequences:
[{ "name":"BS",
"keyword":"key1",
"city":"xyz"
},
{ "name":"AGS",
"keyword":"Key2",
"city":"xyz1"
},
{ "name":"BS",
"keyword":"Keyword",
"city":"city"
}]
Please provide some suggestion i am stuck with this problem since 2 days.
Thanks
The order of results returned by MongoDB is not guaranteed unless you explicitly sort your data using the sort function. For smaller datasets you maybe "lucky" in the sense that the results are always returned in the same order, however, for bigger datasets and in particular when you have sharded Mongo clusters this is very unlikely. As proposed by Yathish you need to explicitly order your results using the sort function. Based on the suggested output, it seems you want to sort by name in descending order so I have set the sorting flag to -1 for the field name.
db.collection.find({"$or" : [{"name":"BS"}, {"keyword":"Key2"}]}).sort({"name" : -1});
If you need a more complex sorting algorithm as specified in your comment, you can convert your results to a Javascript array and create a custom sort function. This sort function will first list documents with a name equal to "BS" and then documents containing the keyword "Key2"
db.data.find({
"$or": [{
"name": "BS"
}, {
"keyword": "Key2"
}]
}).toArray().sort(function(doc1, doc2) {
if (doc1.name == "BS" && doc2.keyword == "Key2") {
return -1
} else if (doc2.name == "BS" && doc1.keyword == "Key2") {
return 1
} else {
return doc1.name < doc2.name
}
});
I have an array of objects containing dates of when a hotel is available to book within Mongo. It looks something like this, using ISO Date formats as said here.
Here's what document looks like, trying to keep it short for the example.
available: [
{
"start":"2014-04-07T00:00:00.000000",
"end":"2014-04-08T00:00:00.000000"
},
{
"start":"2014-04-12T00:00:00.000000",
"end":"2014-04-15T00:00:00.000000"
},
{
"start":"2014-04-17T00:00:00.000000",
"end":"2014-04-22T00:00:00.000000"
},
]
Now, I need query two dates, check in date and check out date. If the dates are available, Mongo should return the document, otherwise it won't. Here are a few test cases:
2014-04-06 TO 2014-04-08 should NOT return.
2014-04-13 TO 2014-04-16 should NOT return.
2014-04-17 TO 2014-04-21 should return.
How would I go about forming this in to a Mongo query? Using $elemMatch looked like it would be a good start, but I don't know where to take it after that so all three examples I posted above work with the same query. Any help is appreciated.
db.collection.find({
"available": {
"$elemMatch": {
"start": { "$lte": new Date("2014-04-17") },
"end": { "$gte": new Date("2014-04-21") }
}
}
})
How about this command?
Well I actually hope your documents have real ISODates rather than what appears to be strings. When they do then the following query form matches as expected:
db.collection.find({
"available": {
"$elemMatch": {
"start": { "$gte": new Date("2014-04-17") },
"end": { "$gte": new Date("2014-04-21") }
}
}
})
I would like to perform a query for a given nested value on multiple subdocuments.
In the example below, I would like to perform the search across multiple "product_types" objects.
{
"product_types": {
"type_1": [
{
name: "something",
price: 100
},
{
name: "something else",
price: 50
}
],
"type_2": [
{
name: "another one",
price: 20
},
{
name: "and a last one",
price: 30
}
]
}
}
I understood the dollar sign matches any subdocument. Here is what I came up with to get all the product with a "price" value of 100. But it doesn't work. Any idea?
db.inventory.find( { product_types.$.price : 100 } )
PS: I anticipate on some answers saying that such a db design to store products would be very bad and I agree; this is just an example to illustrate the kind of query I want to perform.
MongoDB doesn't support any sort of wildcard property like you're trying to do here with $. However, you can search multiple properties using an $or operator:
db.inventory.find({ $or: [
{ product_types.type_1.price: 100 },
{ product_types.type_2.price: 100 }
]})
But this is going to return the matching documents in full rather than just the matched array elements, so you'll also have to post-process the docs in your code to pull those out.
Searching with embedded documents can be done as
db.inventory.find({"product_types.type_1.price":100})
Field name should be inside " "! Otherwise it will throw syntax error.