Because the large number of docs, someone decided to create everyday a new collection automatically in our Mongodb with the date as sufix with this format XXXXX_YYYY_MM_DD.
I want to use this DB as datasource in Grafana (with the JSON datasource). I have found aggregation queries that union several collections, but the problem is to dinamycally get the dates to get the collections to union.
The only dates I can get from Grafana is __from and __to dates, so i can lookup the collections of these dates, but not the whole range of dates between from and to.
Any idea how to perform it?
you can have some JS/shell script which can execute in mongo shell to get collection names:
var startDate= new Date("2021-09-01");
var endDate = new Date("2021-09-09");
var collections=[];
var startTime = startDate.getTime(), endTime = endDate.getTime();
//Then you could loop from one to another incrementing loopTime by 86400000 (1000*60*60*24) - number of milliseconds in one day:
for(let loopTime = startTime; loopTime <= endTime; loopTime += 86400000)
{
var loopDay=new Date(loopTime)
//generate collection for each day
collections.push("COLLECTIONXXXX_"+loopDay.toISOString().slice(0, 10).replace(/-/g, '_'))
}
print(JSON.stringify(collections))
Output:
["COLLECTIONXXXX_2021_09_01","COLLECTIONXXXX_2021_09_02","COLLECTIONXXXX_2021_09_03","COLLECTIONXXXX_2021_09_04","COLLECTIONXXXX_2021_09_05","COLLECTIONXXXX_2021_09_06","COLLECTIONXXXX_2021_09_07","COLLECTIONXXXX_2021_09_08","COLLECTIONXXXX_2021_09_09"]
In above example we had start date(2021_09_01) and end date (2021_09_09)and we have generated all collection names and stored in array, now you can use collection names from this collections array and run your aggregation union.
Related
How can i filter database entries that have a datetime less than 60min in the past?
I tried some date operations as follows in mongodb with two fields timestamp and marketstartime that are of type date in all my documents:
{"$subtract": ["$timestamp", "$marketstartime"]}
but it returns always null for that operation. Why?
My timestamp and marketstartime entries in the db are in date type and look as follows, this should be correct:
2017-12-23 12:00:00.000Z
The actual question I’m trying to solve: How can I get all entries that have a timestamp less than 60 min in the past from now?
A query can composed for documents with timestamp value set less than 60 minutes ago.
from datetime import datetime, timedelta
query = {
'$timestamp': {
'$lt': datetime.now() + timedelta(minutes=-60)
}
}
cursor = db.collection.find(query)
I am creating a script that would run after every x minutes and needs to gather data from MongoDB by timestamps.
For example, how would I match the documents with aggregation that have a timestamp in the following timeframe:
start_time = current_time - 60 min
end_time = start_time + 30 min
And I would need to get all the documents that stay within that time frame.
The MongoDB objects have ISODate timestamps on them.
Thanks!
You can create date objects in mongo shell like so:
db.getCollection('order').aggregate([
{
$match : {
start_time : {$gte : new Date(ISODate().getTime() - 1000 * 60 * 60)},
end_time : {$lte : new Date(ISODate().getTime() + 1000 * 60 * 30)}
}
}
...
])
You can use this in aggregate but also in normal find queries.
Note I wrote this without testing, so it might have syntax errors..
db.collection.find({"createdAt": { $gt: new Date('2017-04-25')},"updatedAt":{$lt:new Date('2017-06-25')}})
updatedAt and createdAt are the feilds I have taken at the time of designing the schema by timestamp. You could give feilds according to you design.
the find query would be little better than aggregate in this case as no complex functions have to be performed
I m using Moloquent for mongoDB in my project, my dates is stored in 2015-09-28 03:30:00 format in mongoDB and want to write a query like
select * from table where date(storeddate)='2015-09-28'
and
select * from table where year(storeddate)='2015'
in mongoDB with Moloquent.
You could use the query builder to take advantage of native MongoDate objects in constructing a date range that you can then use in the query. For example, the following query
select * from users where date(storeddate)='2015-09-28'
can be written as
$start = new MongoDate(strtotime("2015-09-28 00:00:00"));
$end = new MongoDate(strtotime("2015-09-29 00:00:00"));
$users = DB::collection('users')->whereBetween('storeddate', array($start, $end))->get();
Eloquent also supports Carbon or DateTime objects instead of MongoDate objects which will be converted internally to MongoDate objects when saved to the database.
Using the same example as above, if you want to query for records from Users data where a mongodb datetime field storeddate year part is 2015
select * from users where year(storeddate)='2015'
use Carbon's createFromDate() property to create the date range that spans a year:
$users = User::whereBetween(
'storeddate', array(
Carbon::createFromDate(2015, 1, 1),
Carbon::createFromDate(2015, 12, 31)
)
)->get();
I'm trying to get a specific range of documents, based on when they were created. What I'm trying to do is something like:
/getclaims/2015-01
/getclaims/2015-02
...
that way a user can browse through all records based on the selected month.
In my database I'm not storing a created_at date, but I know mongodb stores this in the objectid.
I found that I can get records like this:
db.claims.find({
$where: function () { return Date.now() - this._id.getTimestamp() < (365 * 24 * 60 * 60 * 1000) }
})
of course that doesn't filter based on a specific month, but only within a certain time limit.
What would be a possible way of limited a query based on a specific month, using the Timestamp from the objectid's?
I'm using mongoose, but it's probably a good idea to start in mongo shell itself.
Based on the function borrowed from the answer to this question - https://stackoverflow.com/a/8753670/131809
function objectIdWithTimestamp(timestamp) {
// Convert date object to hex seconds since Unix epoch
var hexSeconds = Math.floor(timestamp/1000).toString(16);
// Create an ObjectId with that hex timestamp
return ObjectId(hexSeconds + "0000000000000000");
}
Create a start and an end date for the month you're looking for:
var start = objectIdWithTimestamp(new Date(2015, 01, 01));
var end = objectIdWithTimestamp(new Date(2015, 01, 31));
Then, run the query with $gte and $lt:
db.claims.find({_id: {$gte: start, $lt: end}});
I have a table that records activities. All activities have an activitydate. I want to count how many activities for a given period of time (day, month, qtr, etc.). I want to include all dates even those that have zero activities. I could do this in the Data Tier with a DateDimension table where the date table has a single column called day containing one row for each calendar day and a outer join, group by query:
DateDimension Table
| Day |
|1/1/2013 00:00:00 |
|1/1/2013 00:00:00 |
|1/1/2013 00:00:00 |
Query
SELECT CAST(Day AS DATE), COUNT() AS CountOfActivities
FROM DateDimension dd LEFT OUTER JOIN Activities a
ON CAST(dd.Day AS DATE) = CAST(a.ActivityDate AS DATE)
WHERE Day BETWEEN MyStartDate AND MyEndDate
GROUP BY CAST(Day AS DATE)
ORDER BY CAST(Day AS DATE)
I'm using EntityFramework so I'd like to execute this query using Linq. The DateDimension table has no business value residing in the database. It exists only to support these aggregate queries by providing a list of dates so I can ensure a row is returned if no activities exist for a given day.
I have the idea that I could manufacture a list of days in memory and weave them in to the results of a much simpler database query at runtime. By perhaps Concatenating the results from 2 IEnumerables - 1 from the in memory enemurable of dates and the other from the database results. How could I do that? Should I do that?
How about something like this:
Example date range:
var from = DateTime.Today.AddDays(-30);
var to = DateTime.Today;
Dictionary to hold your tally of activities per day:
var activityCounts = new Dictionary<DateTime, int>();
Seed with a zero count for each day in the range (this is equivalent to setting up your date dimensions table):
Enumerable.Range(0, (to - from).Days + 1)
.ToList()
.ForEach(x => activityCounts[from.AddDays(x)] = 0);
Add in the real activity counts for each day in the range:
context.Activities.Where(a => a.DateTime >= from && a.DateTime <= to)
.GroupBy(a => a.DateTime)
.ToList()
.ForEach(x => activityCounts[x.Key] = x.Count());
In this way, you only hit the database for the aggregation of activities for dates with activities. The padding out of the resultset with contiguous dates within the date range is then performed on the app server.
Just need to be careful how your dates are stored in the database. This code example is expecting to be able to match keys in the activity dictionary based on the the format of the calls to DateTime.Today. You will need to shape your dates in your database query accordingly.