I tried this but with no success (I got an empty result):
{
queryString: {
defaultPath: 'anyfield',
query: 'createdAt:["2000-01-30T20:19:53.123Z" TO *]'
}
}
I tried with no quotes too.
And I tried the range operator directly and it works, but my query comes as queryString, so I need it working in queryString.
if you have a date object handy great. If not, something like this (pseudocode/JS):
date = new Date;
date = date.toISOString();
then, here is the query from the docs:
{
"$search": {
"index": "default",
"range": {
"path": "anyField",
"gte": "2000-01-30T20:19:53.123Z",
"lte": date,
}
}
}
According to mongodb engineers, the queryString operators only accept AND and OR. It doesn't accept any other operator like TO, used in range filters.
Related
I have a collection where the same column has Date and String format dates for different records, with the same ISO pattern like 2021-04-22T14:10:48.751779Z.
And I need to build a unified query for search, with such =, >, < options for search.
The second problem is an input search query has a different pattern ("2021-04-22")
So the question is:
can I write some kind of transformation of existing table data before the search, for example from Date to String, or in case I have a string value this transformation would not cause any exception.
In this case, I would be able to perform a search by $eq, $lt, $gt, and $regex query options.
I`m not considering casting strings to dates even if this is possible because my input did not have "14:10:48.751779Z" this part of the date and query would not find anything because of dates would not match. For this case, $regex looks like the only solution.
Or your any other suggestions would be considered.
Current query which not satisfies case when DB column is String type:
{
"aggregate":"collection_name",
"pipeline":[
{
"$match":{
"$and":[
{
"some_column":{
"$eq":"some_value"
}
},
{
"date_column_with_string_or_date_type":{
"$gt":{
"$date":"1980-01-01T00:00:00Z"
}
}
}
]
}
},
{
"$project":{
"_id":1
}
}
]
}
Solved by using $expr and $toString
This works with Date And String types of "date_column"
$match: {
$expr: {
$gt : [
{ $toString:"$date_column" },
"2021-04-22"
]
}
}
operator:
I've made a database of sensor data, which in essence stores Double values paired with Date timestamps.
I've tried querying them with what seems to be the right syntax, but it returns just [].
The API is made in MongoDB Stitch, and I'm calling it through Postman.
If I make my API return all documents, it works fine.
This is an example of a Document in my database:
_id:5caf026c8bc97c06677967d6
time:"2019-04-11T09:01:32Z"
val:102.85
topic:"pressure/stjernelaks/sensor0"
isotime:2019-04-11T09:01:32.000+00:00
Edit:
This is how the document looks when returned as Array through the API:
"_id": {
"$oid": "5caf18078bc97c066779724a"
},
"time": "2019-04-11T10:33:43Z",
"val": {
"$numberDouble": "102.88"
},
"topic": "pressure/stjernelaks/sensor0",
"isotime": {
"$date": {
"$numberLong": "1554978823000"
}
}
},
And this is the query I'm making:
https://eu-west-1.aws.webhooks.mongodb-stitch.com/api/client/v2.0/app/semapres-charts-dsioa/service/get-chart-data/incoming_webhook/get-all-between?arg2=1554970464&arg1=1554980464
Alternatively I've used the arguments 2019-04-11T07:01:32.000+00:00 and 2019-04-11T09:01:32.000+00:00. It's the exact same format as in the database.
It's created by LocalDateTime with DateTimeFormatter.ISO_DATE_TIME in Java, and my MongoDB says the data field type is Date, not String.
This is my webhook in all it's simplicity:
exports = function(payload) {
const {arg1, arg2} = payload.query;
const contentTypes = payload.headers["application/json"];
const body = payload.body;
const mongodb = context.services.get("mongodb-atlas");
const mycollection = mongodb.db("pressure").collection("stjernelaks");
return mycollection.find({
isotime: {
'$gte': new Date(arg1),
'$lt': new Date(arg2)
}
}).toArray();
};
I can't figure out why it doesn't return any documents, so any help would be great.
Update:
I tried modifying my query to return documents with "val" inside a range. Also there was the same problem. Just as with the time and date string, I can use my webhook and hardcode the arguments, and it works. And I can make my webhook return the arguments, it reads them back to me without incident. But it just won't return any documents when it's making a query based on my arguments! It's bizarre!
Use $lte instead of $lt
mycollection.find({
isotime: {
'$gte': new Date("2019-04-11T07:01:32.000+00:00"),
'$lte': new Date("2019-04-11T09:01:32.000+00:00")
}
})
I am try to search product listing with Price range filter in Elastic search
but result showing 0.
Query :
GET magento2651_default_catalog_product/_search
{
"query": {
"range": {
"price.price": {
"gte": 1,
"lte": 100
}
}
}
}
Data :
Let me know if anyone have idea.
Thanks,
You need to check your index mapping
This can be because field price.price is mapped as text.
This can be a reason double queries are not working.
Range queries also work on string field but by comparing characters
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
I'm trying to use expression languge to generate ISO string in Nifi getMongo Query field using following query,
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
But i'm getting invalid JSON error error as double quotes are not escaped. When we try to escape it using \ operator, nifi is not evaluating the expression language. Is there any method or workaround to get this working ?
Thanks in advance
GetMongo processor of nifi requires your query to be in extended json format of mongo.So you can use query of below format to query mongo based on datetime:
{"bday":{"$gt":{"$date":"2014-01-01T05:00:00.000Z"}, "$lt" :{"$date":"2019-01-
01T05:00:00.000Z"}}}
I used your not changed expression in UpdateAttribute processor to evaluate new flowFile attribute.
your expression:
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
the result:
{
"remindmeDate": {
"$gte": "2017-06-16T07:38:04.811Z",
"$lte": "2017-06-16T07:44:04.810Z"
}
}
and this is a correct json object.
Finally I found that GetMongo.Query property does not support nifi expression language (nifi 1.2.0 and 1.3.0). Just hover the question mark near parameter.
It means no way to build dynamic query (
Seems need to register an issue... https://issues.apache.org/jira/browse/NIFI-4082
But it's possible to specify current and relative date in mongo query language. something like this:
{
"remindmeDate": {
"$gte": new Date(),
"$lte": new Date(ISODate().getTime() + 359999)
}
}
Nifi's getMongo Query field doesnt support EL. So i created a stored function in MongoDB for my dynamic query and called it from Nifi.
{
"_id" : "reminderDateGMT",
"value" : function (reminderDateGMT) {
var reminder = new Date(reminderDateGMT)
var fromDate = new Date();
var toDate = new Date(new Date().getTime()+(1000 * 60 * 60));
if ((reminder >= fromDate) && (reminder <=toDate )) {
return true;
} else {
return false;
}
}
}
In nifi GetMongo Query,
{
"$where": "reminderDateGMT(this.reminderDateGMT)"
}
I think you may be able to use the unescapeJson expression language function to handle this. You have to provide valid JSON (escaped quotes) for the field level (PropertyDescriptor in NiFi parlance) validation, but the expression language string expects unescaped JSON during expression parsing, so the unescapeJson function removes the escapes first and then format receives a properly quoted string.
{
"remindmeDate": {
"$gte": "${now():format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}",
"$lte": "${now():toNumber():plus(359999):format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}"
}
}
I had a similar discussion on the mailing list, and here is the solution I found that works:
Mongo console:
db.system.js.save({
"_id": "lastFiveMinutes",
"value": function() {
return new Date(ISODate().getTime() - (1000 * 60 * 5));
}
});
db.loadServerScripts();
Query field:
{
"$where": "obj.ts >= lastFiveMinutes()"
}
Note: you probably want to set this on a timer in the scheduling property.
I know this is pretty old post, but I spent lot many hours and found a solution which worked for me.
Use UpdateAttribute Processor and created two attributes to calculate the date range, I need to fetch the mongo documents :
startDate: "${now():format('yyyy-MM-dd')}"
endDate : "${now():toNumber():plus(86400000):format('yyyy-MM-dd')}"
enter image description here
After that pass these attributes to GetMongo processor:
Query : {"createdDate":{"$gte":ISODate(${startDate}), "$lt":ISODate(${endDate})}}
I'm working on a Meteor application and I have data for a weekly timetable of the format -
events:
event:
day: "Monday"
time: "9am"
location: "A"
event:
day: "Monday"
time: "10am"
location: "B"
There are numerous entries for each day. Can I run a query that will return an object of the format -
day: "Monday"
events:
event:
time: "9am"
location: "A"
event:
time: "10am"
location: "B"
I could store the object in the second format but prefer the first for ease of deleting and updating individual events.
I also want to return them ordered by day of week if there's a nice way to do that.
Several options:
You can use an aggregation command but be warned, you will loose reactivity: it means that except when you reload your template, you will not get external updates. You would also need to use a package to add the aggregation command to Meteor in order to achieve that.
My personal favorite: you don't need to aggregate (and loose reactivity) to achieve your data transformation. You can use a simple Collection.find() query and extend/reduce/modify it using a clever mix of cursor.Observe() and conditional modifications. Have a look at this answer, it did the trick for me (I needed a sum with black listing of some fields, but you can easily adapt it to your group/sorting case) : https://stackoverflow.com/a/30813050/3793161. Comment if you need more details on this
If you plan to have several servers, be warned that each server will have to observe so it may lead to an unnecessary load. So my third solution is either use collection hooks or methods to update an additional field containing every event for each day/user (whatever you need).See #David Weldon answer about that here: https://stackoverflow.com/a/31190896/3793161. In you case, it would probably mean to re-think your database structure to fit your needs (i.e. adding more fields so you ca update them on insert.
EDIT Here are some thoughts on your comment:
If you stick to what you described in the question, you would need seven documents, one per day, with an events field where you put all the events. My second solution is good if you need to rework a collection structure before sending it. However, in your case, you just need an object week with 7 sub-objects matching the days of the week.
I advise you to possible approaches:
use the aggregation in a method, as described by #chridam. Be warned that you will not be able to directly get a sorted array, as stated in mongo $group documentation
$group does not order its output documents
So you need to sort them (by day and by hour within each day) using, for example _.sortBy() before you return your method result. By the way, if you want to know what is going on in your method call, clientside, here is how you should write the call:
Meteor.call("getGroupedDailyEvents", userId, function(error, result){
if(error){
console.log("error", error);
}
if(result){
//do whatever you need
}
});
Make the data sorting client-side. You are looking for an overkill solution because, afaik, you don't need to filter any data to keep it from the user, and you are going to send the data anyway (just with another structure). This is much easier to make a simple helper in your template like this:
Template.displaySchedule.helpers({
"monday_events": function() {
return _.sortBy (events.find({day:"Monday"}).fetch(), "time")
},
//add other days
);
It assumes the format of your time field is sortable this way. If not, you just need to create a function to sort them accordingly to their formats or change the original format into something better suited.
The rest (HTML) would just be to iterate on Monday events using a {{#each monday_events}}
To achieve the desired result, use the aggregation framework where the $group pipeline operator groups all the input documents and apply the accumulator expression $push to the group to get the events array.
Your pipeline would look like this:
var Events = new Mongo.Collection('events');
var pipeline = [
{
"$group": {
"_id": "$day",
"events": {
"$push": {
"time": "$time"
"location": "$location"
}
}
}
},
{
"$project": {
"_id": 0, "day": "$_id", "events": 1
}
}
];
var result = Events.aggregate(pipeline);
console.log(result)
You can add the meteorhacks:aggregate package to implement the aggregation in Meteor:
Add to your app with
meteor add meteorhacks:aggregate
Since this package exposes .aggregate method on Mongo.Collection instances, you can define a method that gets the aggregated result array. For example
if (Meteor.isServer) {
var Events = new Mongo.Collection('events');
Meteor.methods({
getGroupedDailyEvents: function () {
var pipeline = [
{
"$group": {
"_id": "$day",
"events": {
"$push": {
"time": "$time"
"location": "$location"
}
}
}
},
{
"$project": {
"_id": 0, "day": "$_id", "events": 1
}
}
];
var result = Events.aggregate(pipeline);
return result;
}
});
}
if (Meteor.isClient) {
Meteor.call('getGroupedDailyEvents', logResult);
function logResult(err, res) {
console.log("Result: ", res)
}
}