I am using Scala 2.12 to query the ElasticSearch (6.5).
I am able to use querybuilders for a single field search like below:
val searchSourceBuilder = new SearchSourceBuilder()
val qb = new BoolQueryBuilder()
.must(QueryBuilders.regexpQuery("header.fieldname", "01_.+_20190711_data"))
searchSourceBuilder.query(qb)
Using the above (I need regex search) I can search the relevant documents.
However, I have more complex requirement, where I have to match the documents on more than one field-value pair.
i.e.
header.fieldname should match pattern "01_.+data"
AND
header.fieldname2 should match pattern "type.+_2019-07-11"
Basically, it is like SQL where clause on 2 or more columns (and value string).
I was checking https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html
But this is like searching the same string (value) in multiple fields. This is NOT what I want.
I basically want something like SQL AND in where clause (better if it is with regex too).
UPDATE:
Please note the below answer by #Meet Rathod works and accepted.
However, to take it forward, so if I need one more condition which is SQL OR, is my below code correct.
Required:
header.fieldname: 01_.+data AND header.fieldname2: type.+_2019-07-11 AND (header.fieldname3: some_thing OR header.fieldname3: some_other_thing)
Code:
val qb = new BoolQueryBuilder()
.must(QueryBuilders.regexpQuery("header.fieldname", "01_.+_20190711_data"))
.must(QueryBuilders.regexpQuery("header.fieldname2", "type.+_2019-07-11"))
.should(QueryBuilders.regexpQuery("header.fieldname3", "some_thing"))
.should(QueryBuilders.regexpQuery("header.fieldname3", "some_other_thing"))
Is this correct or I am missing something?
As far as I understand, you want only those document which satisfies all your conditions should list out in the result. And if that's the case I believe adding another must clause in your query should get your expected result. The raw query will look something like this.
{
"query": {
"bool": {
"must": [
{
"regexp": {
"header.fieldname": "01_.+data"
}
},
{
"regexp": {
"header.fieldname2": "type.+_2019-07-11"
}
}
]
}
}
}
I'm not sure but, your Scala code should look something like this.
val qb = new BoolQueryBuilder()
.must(QueryBuilders.regexpQuery("header.fieldname", "01_.+_20190711_data"))
.must(QueryBuilders.regexpQuery("header.fieldname2", "type.+_2019-07-11"))
Related
I'm not sure this is possible, but i'd like to create a single view or at least a single query that looks in different collections based on what's being queried.
for example, if the first character is an "A" look in the "Aresults" collection, if it's a "B" look in the "Bresults" collection, etc.
I could potentially create a "A-Z" collection with just those letters, and do a $lookup from there based on a condition, but i'm not sure how to do that either.
I am aware that i could create a view with a $unionWith having all the "*results" collections, but that seems very inefficient.
Any other ideas? Is there perhaps some type of dynamic query structure within mongodb like in MySQL (couldn't find any)?
Thanks
Something like this?
const prefix = db.meta_data.findOne({field: condition}).prefix ;
db.createView('view_name', prefix + 'results', [<your aggregation pipeline>]);
or this?
const pipeline = [];
db.meta_data.find({ field: condition }).forEach(x => {
pipeline.push({ $unionWith: { coll: prefix + 'results' } });
});
db.collection.aggregate([pipeline]);
A mongo db has documents that look like:
{
"_id": : ObjectId("55cb43e8c78b04f43f2eb503"),
<some fields>
"topics": {
"test/23/result": 149823788,
"test/27/result": 147862733,
"input/misc/test": 14672882
}
}
I need to find all documents that have a topics field that contains a particular key. i.e. find all documents that have a topics.key = "test/27/result"
I've tried a number of things but none work yet, neither attempt below work,
they return no records event though some should match:
db.collName.find({"topics.test/27/result": {$exists:true}});
db.collName.find({"topics.test\/27\/result": {$exists:true}});
How can I make the query work?
The slash characters are inserted by another process. They are mqtt topic names.
I found the solution to my problem:
I was building the query wrong in my code. In the example below, evtData.source contains the key name to search for, i.e. 'test/27/result'
The query methodology that works for me is:
var query = {};
query['topics.' + evtData.source] = {$exists: true};
db.collName.find(query)
I'm trying to put this into morphia query :
db.woot.find({
"bar.tables": {
$elemMatch: {
"tableId": {
$in: [3,
4]
},
"tab": {
$gte: 20000
}
}
}
})
So I have :
Query<Table> q
q.field("bar.table").hasThisElement()
And after this I don't know how to finish the query and still using FieldEnd which supports in(), gte() methods without writing whole query myself with BasicDBObjects.
Please help me transform above query to the nicest possible Morphia equivalent.
EDIT: bar.tables is an array so matching must be done with elemMatch or else it can match first condition from some element and the second condition from the other element, but only elements matching both conditions are valid.
Morhpia now has elemMatch so you can execute your original query.
Query<Example> tableQuery = mongoDatastore.createQuery(Example.class)
.disableValidation()
.field("tableId").hasAnyOf(Arrays.asList(3, 4))
.field("tab").greaterThanOrEq(20000);
Query<Table> query = getCDB().getDatastore()
.createQuery(Table.class)
.field("bar.table").elemMatch(tableQuery);
Something along these lines should work.
I would try something like this:
Query<Table> query = mongoDataStore.find(Table.class)
.field("bar.table.tableId").hasAnyOf(tableIdArrayList)
.field("bar.table.tab").greaterThan(20000);
You'll probably need some custom query builder, where you can set some conditions and it will then put together the right query for you — at least that's our approach.
The mongo docs specify that you can specify a query hint for count queries using the following syntax:
db.orders.find(
{ ord_dt: { $gt: new Date('01/01/2012') }, status: "D" }
).hint( { status: 1 } ).count()
Can you do this using the mongo template? I have a Query object and am calling the withHint method. I then call mongoTemplate.count(query); However, I'm pretty sure it's not using the hint, though I'm not positive.
Sure, there are a few forms of this including going down to the basic driver, but assuming using your defined classes you can do:
Date date = new DateTime(2012,1,1,0,0).toDate();
Query query = new Query();
query.addCriteria(Criteria.where("ord_dt").gte(date));
query.addCriteria(Criteria.where("status").is("D"));
query.withHint("status_1");
long count = mongoOperation.count(query, Class);
So you basically build up a Query object and use that object passed to your operation, which is .count() in this case.
The "hint" here is the name of the index as a "string" name of the index to use on the collection. Probably something like "status_1" by default, but whatever the actual name is given.
I am trying to make a simple map/reduce function on one of my MongoDB database collections.
I get data but it looks wrong. I am unsure about the Map part. Can I use IF/ELSE in this way?
UPDATE
I want to get the amount of authors that ownes the files. In other words how many of the authors own the uploaded files and thus, how many authors has no files.
The objects in the collection looks like this:
{
"_id": {
"$id": "4fa8efe33a34a40e52800083d"
},
"file": {
"author": "john",
"type": "mobile",
"status": "ready"
}
}
The map / reduce looks like this:
$map = new MongoCode ("function() {
if (this.file.type != 'mobile' && this.file.status == 'ready') {
if (!this.file.author) {
return;
}
emit (this.file.author, 1);
}
}");
$reduce = new MongoCode ("function( key , values) {
var count = 0;
for (index in values) {
count += values[index];
}
return count;
}");
$this->cimongo->command (array (
"mapreduce" => "files",
"map" => $map,
"reduce" => $reduce,
"out" => "statistics.photographer_count"
)
);
The map part looks ok to me. I would slightly change the reduce part.
values.forEach(function(v) {
count += v;
}
You should not use for in loop to iterate an array, it was not meant to do this. It is for enumerating object's properties. Here is more detailed explanation.
Why do you think your data is wrong? What's your source data? What do you get? What do you expect to get?
I just tried your map and reduce in mongo shell and got correct (reasonable looking) results.
The other way you can do what you are doing is get rid of the inner "if" condition in the map but call your mapreduce function with appropriate query clause, for example:
db.files.mapreduce(map,reduce,{out:'outcollection', query:{"file.author":{$exists:true}}})
or if you happen to have indexes to make the query efficient, just get rid of all ifs and run mapreduce with query:{"file.author":{$exists:true},"file.type":"mobile","file.status":"ready"} clause. Change the conditions to match the actual cases you want to sum up over.
In 2.2 (upcoming version available today as rc0) you can use the aggregation framework for this type of query rather than writing map/reduce functions, hopefully that will simplify things somewhat.