I need to convert a unix_timestamp field to date by MVEL operators.
I have a field which is filled by a mysql imported timestamp. Now I am to get the day date (not datetime) from it and use it in an aggregation.
My Aggregation is like this:
"aggregations" : {
"grouped_item" : {
"terms" : {
"script" : "doc['time_stamp'].value",
"size" : 50
}
}
}
The result of above aggregate is grouped 'by second', but I need 'by date'.
Thanks in advance.
Instead of trying to value script this and use the terms aggregation, use the date histogram aggregation that exists for exactly this purpose.
Related
I have documents in the database with a dateTime value like so:
{
"_id" : ObjectId("5a66fa22d29dbd0001521023"),
"exportSuccessful" : true,
"month" : 0,
"week" : 4,
"weekDay" : "Mon",
"dateTime" : ISODate("2018-01-22T09:02:26.525Z"),
"__v" : 0
}
I'd like to:
query the database for a given date and have it return the document that contains the dateTime if the date matches (I don't care about the time). This is mainly to test before inserting a document that there isn't already one for this date. In the above example, if my given date is 2018-01-22 I'd like the document to be returned.
retrieve all documents with a distinct date from the database (again, I don't care about the time portion). If there are two documents with the same date (but different times), just return the first one.
From what I understand Mongo's ISODate type does not allow me to store only a date, it will always have to be a dateTime value. And on my side, I don't have control over what goes in the database.
Try range query with start date time from start of the day to end date time to end of the day. So basically create dates a day apart.
Something like
var start = moment().utc().startOf('day');
var end = moment().utc().endOf('day');
db.collection.find({
dateTime: {
$gte: start,
$lte: end
}
})
Get all distinct dates documents:
db.collection.aggregate(
{"$group":{
"_id":{
"$dateToString":{"format":"%Y-%m-%d","date":"$dateTime"}
},
"first":{
"$first":"$$ROOT"
}
}}])
assuming I have the following nested document structure, where my document contains nested routes with an array of date time values.
{
property_1: ...,
routes: [
{
start_id: 1,
end_id: 2,
execution_times: ['2016-08-28T11:11:47+02:00', ...]
}
]
}
Now I could filter my documents that match certain execution_times with something like this.
query: {
filtered: {
query: {
match_all: { }
},
filter: {
nested: {
path: 'routes',
filter: {
bool: {
must: [
{
terms: {
'routes.execution_times': ['2016-08-28T11:11:47+02:00', ...]
}
},
...
]
}
}
}
}
}
}
But what if I would like to filter my documents based on execution dates. What's the best way achieving this?
Should I use a range filter to map my dates to time ranges?
Or is it better to use a script query and do a conversion of the execution_times to dates there?
Or is the best way to change the document structure to contain both, the execution_date and execution_time?
Update
"The dates are not a range but individual dates like [today, day after tomorrow, 4 days from now, 10 days from now]"
Well, this is still a range as a day means 24 hours. So if you store your field as date time, you can use leverage range query : from 20-Nov-2010 00:00:00 TO 20-Nov-2010 23:59:59 with appropriate time zone for a specific day.
If you store it as a String then you will lose all the flexibility of date maths and you would be able to do only exact String matches. You will then have to do all the date manipulations at the client side to find exact matches and ranges.
I suggest play with range queries using Sense plugin and I am sure it will satisfy almost all your requirements.
-----------------------
You should make sure that you use appropriate date-time mapping for your field and use range filter over that field. You don't need to split into 2 separate fields. Date maths will allow you to query just based on date.
This will make your life much easier if you want to do aggregations over date time field.
Reference:
Date Maths:
https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#date-math
Date Mapping : https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html
Date Range Queries:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
In My mongodb there is a video collection in which there is a creationTime list like following and i want to fetch the record between any two dates by using creationTime.
"creationTime" : {
"logtime" : ISODate("2013-08-12T10:54:44.914Z"),
"logtimeStr" : "12-08-2013 04:24:44",
"day" : 12,
"month" : 8,
"year" : 2013,
"hour" : 16,
"min" : 24,
"second" : 44
}
A little bit nicer way to write a range query, and also just mentioning that $and is implicit within MongoDB and only needs to be used in specific cases that actually require it:
db.collection.find({
"creationTime.logtime": {
"$gt": new Date("2014-08-01"), "$lt" new Date("2014-08-13")
}
})
At least that is the theory for a range query. But currently you have a really big problem and that is your dates are actually being stored as strings.
This is a huge problem as basically when these are strings ( and especially as they are currently formatted ) then you are stuck with a lexical comparison of strings, and the strings you have to not lexically compare properly in the sense of one being "greater" in value than the other.
Strings would need to be presented in the order of "year" then "month" then "day", and with two digit representations in order to match a lexical comparison. What you have currently does not.
But do not change the strings, change them to proper date types and then the query as shown will actually work correctly.
Use Mongodb comparison operators $lt and $gt ( or $lte & $gte for inclusive limits )
How you create START_DATE and END_DATE depends on language you are using .e.g. for javascript you would use new Date()
db.myVideoCollection.find({
'$and':[
{'$creationTime.logtime':{'$gt':START_DATE },
{'$creationTime.logtime':{'$lt':END_DATE }
]
})
if i want to find a document created on a specific Day, until now i used a range
from the first minute of the day, to the last minute of the day in seconds , sth like :
query":{"dtCreated":{"$gte":{"sec":1381356782,"usec":0},"$lt":{"sec":1389356782,"usec":0}}}
is is possible to to somehow find all documents where only the Day, Month and year equals "dtCreated" ?
in pseudocode like :
query:{"dtCreated":ISODate("2014-01-23")} <- i know that may not be a valid iso date
but what i want is to find all documents for one day without using lt and gt ?
Sry for bad english and for any hints thanks in advance!
You can do it with the aggregation framework using the date aggregation operators.
Assuming dtCreated is an ISODate field, you could try something like this:
query = [
{
'$project': {
'year': {'$year':'$dtCreated'},
'month': {'$month':'$dtCreated'},
'day':{'$dayOfMonth':'$dtCreated'}
}
},
{
'$match' : {'year':'2014', 'month':'1', day:'1'}
}
]
db.mycollection.aggregate(query)
Edit: as orid rightly remarks, though this is an answer to your question (query for date without using date range), it's not a good way to solve your problem. I would probably do it this way: a range greater than or equal to today, but less than tomorrow
db.foo.find({'dtCreated':{'$gte':ISODate("2014-01-23"), '$lt':ISODate("2014-01-24")}})
I am attempting to perform a calculation in an aggregation based on the day of the current month. For example, I want to divide the total number of transactions by the day of the month to get transactions per day.
The problem is I can't figure out how to get the current date. I see plenty of examples where it's hardcoded, but what I need is more like the MySQL NOW() function.
I've tried something like this:
> db.statistics.aggregate([{$project: {dayofmonth: {$dayOfMonth: Date()}}}])
{
"errmsg" : "exception: can't convert from BSON type String to Date",
"code" : 16006,
"ok" : 0
}
But that produces the error you see.
How can I get the current day of the current month for use in an aggregation calculation?
You almost did it. You have to write new Date()
db.statistics.aggregate([
{$project: {dayofmonth: {$dayOfMonth: new Date()}}}
])
it will produce results like this:
{
"result" : [
{
"_id" : "statisctiId",
"dayofmonth" : 9
}
}