I have a MongoDB collection with documents that look like this
{
"_id" : ObjectId("5aab91b2caa256021558f3d2"),
"Timestamp" : "2017-11-16T14:43:07.5357785+01:00",
"status" : 1,
"created_at" : 1521193394,
"updated_at" : 1521193394,
"deleted_at" : ""
}
Data gets entered into the collection every 15 minutes. Using the created_at field, which is in epoch time, I would like to find a way to fetch data at the top of every hour. So for example, data is entered at 12.00 12.15 12.30 12.45 13.00 13.15 13.30 13.45 14.00.
I would like to fetch entries from the collection that were entered at 12.00 13.00 and 14.00.
I am also open to suggestions as to whether or not using epoch time is the best way to go about it.
Using epoch time is really a good way to go.
Since you are stoing in seconds, every round hour can be divisible by 3600(seconds in hours) without remainder. You can make use of this property to find your documents.
db.collection.find({created_at: {$mod: [ 3600, 0 ]}});
According to $mod documentation, it will,
Select documents where the value of a field divided by a divisor has
the specified remainder
We provided divisor as 3600 and remainder as 0. This should give what you expect.
To ignore seconds:
For this condition, mod(epoch, 3600) should be less than 59. This query can be formed using $expr of mongo 3.6
db.collection.find({$expr: {$lte: [{ $mod: [ '$created_at', 3600 ] }, 59]}});
Hope this helps!
Related
I am trying to store the price of stocks price per minute, so I can easily return results based on the minute date time per minute interval and store historical data, so i can query like last 24 hours, last 30 days etc (please also let me know if this is wrong approach)
for example if i check current time with fmt.Println("time now: ", time.Now()) i get the following date time 2022-01-29 11:47:02.398118591 +0000 UTC m=+499755.770119738
so what i want is to only get up to minute level, so i can store per minute
so i will liek to use this date time 2022-01-29 11:47:00 +0000 UTC
I will like to UTC, so i can stick to that universal time zone to store and retrive data
Each row will be a list of multiple stock price data
Do i need to have the _id field? Am not sure, so just looking for best practice as help.
database name: "stock-price-db"
collection name: "stock-price"
Thinking of something like this, just for example
[
{
"_id" : ObjectId("5458b6ee09d76eb7326df3a4"),
"2022-01-29 11:48:00 +0000 UTC":
[
{
"stock": "TSLA",
"price": "859.83",
"marketcap": "8938289305",
},
{
"stock": "AAPL",
"price": "175.50",
"marketcap": "3648289305",
},
]
},
{
"_id" : ObjectId("5458b6ee09d76eb7326df3a4"),
"2022-01-29 11:47:00 +0000 UTC":
[
{
"stock": "TSLA",
"price": "855.50",
"marketcap": "8848289305",
},
{
"stock": "AAPL",
"price": "172.96",
"marketcap": "3638289305",
},
]
},
]
First, is this the right way to do store this type of data in mongodb and how do I structure the model to store the data this way so I can store the data per minute interval, so I can query per minute interval?
There are few drawbacks in your design.
Do not use dynamic keys - you will end up using few extra aggregation pipelines.
Store the date in a static-key field i.e time:ISODate()
Better store all the available time units, till milliseconds, it will be helpful to handle the future requirement changes
If there are too many stocks changes, it is not a scalable design.
If you want to find out historical data for a stock, provided design may have performance issues.
You will end up with issues in sharding.
What other alternatives:
Not all the use-cases can be solved by one design.
If this use case is purely for time series use case, I would recommend you to use a time series design/ time series database i.e influx, tsdb.
If you need to cover all the use-cases, normalise and use GQL.
Let there be a MongoDB collection Data that holds the history of temperatures of some items. So the items are like this:
{
"itemId" : "ABCD",
"timePoint" : NumberLong("1618922410288"),
"temperature" : 15.15
}
meaning, that the item "ABCD" had a temperature 15.15 at 1618922410288ms since the epoch.
What is the query that will result in the history of some span-average temperatures of the given item starting from the given time point?
So e.g. for itemId="ABCD", span=60*60*1000, startingTimePoint=0 it has to be a query that will result to the hourly-avarage temperatures of "ABCD" starting throm 1970-01-01 00:00:00.000.
I would also like the query not to be "welded" with the average, but rather accept an arbitrary aggregation function, so it will be easy to use the same technique for avg, min, max, or something else. But this is the second question.
what i want is to get the users who have created there account in between the range of dates and also want to get the customer with order count greater then 1 for those new customers, so what i am doing is trying to get the data greater then the last month i.e 1st of march 2020 but the output is giving me users till 1 april, why not till today i.e 11 april, the data is in following format
"_id" : ObjectId("1dv47sd1a10048521sa1234d"),
"updatedAt" : ISODate("2020-04-01T16:19:26.460+05:30"),
"createdAt" : ISODate("2020-04-01T16:18:46.066+05:30"),
"email" : "edx#gmail.com",
"phone" : "xxxxxxxxxx",
"password" : "$awdad$2b$10$4YaO6AEZqXA9ba0iz14ALi",
"dob" : "00/20/1990",
"stripeID" : "xxxxxxxxxxxxxxxx",
"__t" : "Customer",
"banned" : false,
"picture" : "link to image",
"name" : {
"first" : "ababab",
"last" : "Saaa"
},
"orderCount" : 2,
"creditCards" : [ ],
"addresses" : [ ],
"__v" : 0,
"isEmailVerified" : true
i have written a query for extracting data from the date greater then last month but it is giving me data till the 1 of april, my query is as follows
db.users.find({
"createdAt" : { "$gte" :new Date("2020-03-1") }
})
so i want to get data timm today from 1 march 2020 also order count is greter then 1,thanks in advance i am preety new with mongo
MongoDB only stores absolute times (aka timestamps or datetimes), not dates. As such the queries should generally be specified using times, not dates, also.
Furthermore, MongoDB always stores times in UTC. Different systems can operate in different timezones. For example, the shell may be operating in a different timezone from your application.
To query using dates:
Determine what timezone the dates are in.
Convert dates to times in that timezone. For example, 2020-03-01 00:00:00 UTC - 2020-03-31 23:59:59 UTC and 2020-03-01 00:00:00 -0400 - 2020-03-31 23:59:59 -0400 are both "March" and would match different sets of documents.
Convert the times to UTC. This may be something that a framework or library you are using handles for you, for example Mongoid in Ruby does this.
Use the UTC times to construct MongoDB queries.
This is stated in the documentation here.
I have documents in the database with a dateTime value like so:
{
"_id" : ObjectId("5a66fa22d29dbd0001521023"),
"exportSuccessful" : true,
"month" : 0,
"week" : 4,
"weekDay" : "Mon",
"dateTime" : ISODate("2018-01-22T09:02:26.525Z"),
"__v" : 0
}
I'd like to:
query the database for a given date and have it return the document that contains the dateTime if the date matches (I don't care about the time). This is mainly to test before inserting a document that there isn't already one for this date. In the above example, if my given date is 2018-01-22 I'd like the document to be returned.
retrieve all documents with a distinct date from the database (again, I don't care about the time portion). If there are two documents with the same date (but different times), just return the first one.
From what I understand Mongo's ISODate type does not allow me to store only a date, it will always have to be a dateTime value. And on my side, I don't have control over what goes in the database.
Try range query with start date time from start of the day to end date time to end of the day. So basically create dates a day apart.
Something like
var start = moment().utc().startOf('day');
var end = moment().utc().endOf('day');
db.collection.find({
dateTime: {
$gte: start,
$lte: end
}
})
Get all distinct dates documents:
db.collection.aggregate(
{"$group":{
"_id":{
"$dateToString":{"format":"%Y-%m-%d","date":"$dateTime"}
},
"first":{
"$first":"$$ROOT"
}
}}])
I am attempting to perform a calculation in an aggregation based on the day of the current month. For example, I want to divide the total number of transactions by the day of the month to get transactions per day.
The problem is I can't figure out how to get the current date. I see plenty of examples where it's hardcoded, but what I need is more like the MySQL NOW() function.
I've tried something like this:
> db.statistics.aggregate([{$project: {dayofmonth: {$dayOfMonth: Date()}}}])
{
"errmsg" : "exception: can't convert from BSON type String to Date",
"code" : 16006,
"ok" : 0
}
But that produces the error you see.
How can I get the current day of the current month for use in an aggregation calculation?
You almost did it. You have to write new Date()
db.statistics.aggregate([
{$project: {dayofmonth: {$dayOfMonth: new Date()}}}
])
it will produce results like this:
{
"result" : [
{
"_id" : "statisctiId",
"dayofmonth" : 9
}
}