I'm using $where to query objects by month and date using code like the following to get UserInfo collections with a Birthdate of May 7:
db.UserInfo.find( function() {
var d = new Date(this.Birthdate);
return d.getDate() === 7 && d.getMonth() === 4;
});
This works perfectly locally, returning UserInfo objects with birthdates set to May 7th. However, this breaks remotely (Heroku+Mongolab) because I get back objects with Birthdate set to 1210222800000 for example, which is May 8th. Why is this happening and how can I get mongo to return the correct objects?
It looks like a time zone issue. I assume your dates are all supposed to be "midnight" on the day in question. This one is 8 hours off.
# TZ=UTC date -d #1210222800000
Tue Jun 8 08:00:00 UTC 40320
Since JSON doesn't really have a Date type, you have to be really clear where the conversion happens. Best practice is that it's the app's responsibility to always convert to UTC before sending to the database. (And to strip off the time if you're trying to just store a date. Otherwise your date comparisons will be wrong.)
It's also best practice to run your database servers and application servers in the UTC time zone. (The app should convert to local time if desired. Usually per-user since users are often in different time zones.)
Related
I will begin using MongoDB in a project and made the following observation:
A ‘Date’, e.g.: ISODate("2000-07-24T00:00:00.000Z") is stored in as UTC 0 (or Zulu).
Let’s assume that an end user submits a plain date, such as “2000-07-24” and his or her local timezone is -5.
Because of the ‘Z’ character in the time (indicates UTC 0), storing the following would be incorrect because the user’s 5 hour difference:
ISODate("2000-07-24T00:00:00.000Z")
I am assuming that it would be more proper to consider the end user’s local time zone into account and store the date as follows:
ISODate(“2000-07-23T19:00:00.000Z”)
By doing this, I can see that any queries would need a timezone adjustment for 'Date'.
By storing all ‘Date’ fields as UTC 0, we can guarantee that exact points in time from people posting from different continents is properly handled.
Does this seem like a good way to handle timezones in MongoDB?
I'm writing a web app using Firestore that needs to be able to show "Today's Popular Posts" and I'm having trouble getting the queries right when considering users in different timezones.
The dates are stored in the DB as UTC 0, then adjusted to the current user's UTC offset in the client via Moment.js. This works correctly.
When adding a new post I use firebase.firestore.FieldValue.serverTimestamp() to store the current server timestamp in a field called timestamp, like so:
const collectionRef = db.collection('posts');
collectionRef.add({
name: "Test Post",
content: "Blah blah blah",
timestamp: firebase.firestore.FieldValue.serverTimestamp(),
likeCount: 0
});
Then on the server I have a Cloud Function that runs on create and adds another field to the document called datestamp which is the the UTC 0 timestamp, but adjusted so that the time is the beginning of the day. The function looks like this:
exports.updatePostDate = functions.firestore
.document('posts/{postID}')
.onCreate((event) => {
const db = admin.firestore();
const postRef = db.doc('post/'+event.params.postID);
const postData = event.data.data();
const startOfDay = moment(postData.timestamp).startOf('day').toDate();
return postRef.update({
datestamp: startOfDay
});
});
Storing a timestamp where the time is always the beginning of the day enables me to write a query like this for finding all posts and ordering by popularity on a given day:
const startOfDayUTC = moment.utc().startOf('day').toDate();
const postQuery = db.collection('posts')
.orderBy('likeCount', 'desc')
.orderBy('timestamp', 'desc')
.where('datestamp', '==', startOfDayUTC)
.limit(25);
The problem is, depending on the user's UTC offset, this can display posts with two different dates when parsing the post's timestamp field. So even though the query is correctly fetching all the posts where the datestamp is say, 2018-01-30T00:00:00Z, the timestamp's date might not be the same once parsed. Here's an example of two posts:
Post 2:
likeCount: 1
timestamp (UTC 0): 2018-01-30T06:41:58Z
timestamp (parsed to UTC-8): 2018-01-29T22:41:58-08:00
datestamp (UTC 0): 2018-01-30T00:00:00Z
Post 1:
likeCount: 0
timestamp (UTC 0): 2018-01-30T10:44:35Z
timestamp (parsed to UTC-8): 2018-01-30T02:44:35-08:00
datestamp (UTC 0): 2018-01-30T00:00:00Z
So you can see, while the posts have the same datestamp, after adjusting the timestamp to the local UTC, the timestamp fields can end up being on two different days.
If anyone has a solution to this I would be very grateful.
I think it is better to avoid functions in this case as you can perform compound queries now. You can simply use
query.where(date > lastMidnight).where(data < now).get().then(...)
so to speak to limit data which only belongs to one day and try to keep all your time variables in UTC 0 and just find the start point and the current time both client side and convert them to UTC0.
//get local time from midnight to now (local)
const now = new Date();
const lastMidnight = now.setHours(0,0,0,0);
//then convert those to UTC0 to pass on in your query to firestore
const lastMidNightUTC = new Date(lastMidnight + now.getTimezoneOffset() * 60000).toString();
const nowInUTC = new Date(now + now.getTimezoneOffset() * 60000).toString();
and you can get your data (remember you need to make an index or just run the query once and firebase SDK will generate a link to create the index in dev tools -> console , for you)
query.where(date > lastMidNightUTC).where(data < now).get().then(...)
I came up with a solution that I'm really not happy with... But it works!
The problem is fundamentally one post can be on more than one date, depending on the user's location. And since for this case we also want to order by a field other than timestamp we can't use a range query to select posts on a given date, because your first .orderBy must be on the field you're using a range query on (see Firestore docs).
My solution is to map localized datestamps to their corresponding UTC offset. The object contains every UTC offset as a key, and the post's datestamp in that offset's time.
An example post looks like this:
posts/{somepostid}
{
name: "Test Post",
content: "Blah blah blah",
timestamp: Mon Jan 29 2018 21:37:21 GMT-0800 (PST),
likeCount: 0,
utcDatemap: {
0: "2018-01-30,
...
-480: "2018-01-29",
...
}
}
The field utcDatemap is the the one we use in our queries now:
const now = moment();
const datestamp = now.format("YYYY-MM-DD");
const utcOffset = now.utcOffset();
const utcDatemapField = 'utcDatemap.'+utcOffset;
const postQuery = db.collection('posts')
.orderBy('likeCount', 'desc')
.orderBy('timestamp', 'desc')
.where(utcDatemapField, '==', datestamp)
.limit(25);
Now posts can show up on two different days, depending on where the user is querying from. And we can still convert the regular old timestamp to the user's local time on the client.
This is definitely not an ideal solution. For the above query I needed to create composite indexes for every single key in utcDatemap. I'm not sure what the rules of thumb are with composite indexes, but I'm thinking having 39 indexes for something simple like this is probably not great.
Additionally I checked it out using the roughSizeOfObject function from thomas-peter's answer on this post and the utcDatemap object, with all it's string datestamps clocked in at roughly 780 bytes, and it's not like 0.78kb is a lot, but you do need to be mindful of how much data you're transferring with a service like Firestore (0.78kb is a lot for a date).
I'm learning/reading up on Firestore and have Google'd to see how it deals with times, so discount my answer appropriately.
It looks as though Firestore converts times to UTC and stores them as its own Timestamp datatype. If so, then it's critical to know that this is a destructive conversion.
Though UTC is useful for comparing instants in time, it means that the wall-clock time as observed by the app user is lost forever. Some countries like the UK are in one of two timezones during the year, Daylight Savings Time and British Summer Time.
You can convert back to the user's observed time, but the problem is that the rules change over the years. You'd have to keep a record of all the different rule changes for all the timezones and countries of the world.
What was the time at the time?
The question is, what time did the user think an event happened ...at the time. This can have legal ramifications. You may need to go back through historic data to prove a person acted at a certain time as they observed it.
The solution is to capture the user's observed offset in an additional field. That way, you can always use this to convert.
Regarding the OPs problem, this seems somewhat philosophical for a web app. Does "today" mean the event, such as a post, must have happened within the user's Monday? Or just posts on today's date? Or posts within the last 24h?
An important thing to remember is that dates are the same all around the world, even when they begin and end at different instants.
What's Elvis got to do with all this?
Christmas Day is 25th everywhere. If I say something happened on Christmas Day and I'm in the USA, and then someone in Australia wants to see all the world's posts made on Christmas Day, then they need to query for posts where observedDate == 25th Dec.
Think about it. Such posts were all made on Christmas Day, even though it might have been Boxing Day for me in England at the instant that they posted.
Elvis died on 16th August. In the UK our radio stations don't all wait until it's the 16th in the timezone of the place of his death to start playing his records.
Another interesting one is whether something happened in Q1 or Q2 of a company's reporting year. Is a sale recognised as on the date at the point-of-sale in the store in New York, or in the database in LA?
The observed date is interesting.
My point is, think deeply about this and store both a normalised instant like UTC, but also the user's observed date, time and offset. Then you have all the information you'll need for the future.
Finally, consider adding observed year or week numbers, or day ordinals, H1/H2 for financial data, since it can be super useful for composing rapid queries, depending on your use-cases.
Here is my problem:
I want to calculate how long ago a record was updated in a DB.
The DB is in PostgreSQL, the update_time field is populated by a trigger that uses CURRENT_TIMESTAMP(2). The field is inflated to a DateTime object by DBIx::Class. I get the current time in my code using DateTime->now()
My problem is that when I retrieve the field value, it's off by 1 h (ie it's 1h ahead of DateTime->now()). I am in the CET time zone, so 1h ahead of UTC currently.
The right way to solve the problem is likely at the DB level. I have tried to replace CURRENT_TIMESTAMP with LOCALTIMESTAMP, to no avail.
I think actually a more robust solution (ie one that doesn't rely on getting the DB right) would be to get the current time stamp from the DB itself. I really just need the epoch, since that's what I use to compute the difference.
So the question is: is there a simple way to do this: get the current time from the DB using DBIx::Class?
A different way to get the DB and DateTime to agree on what the current time is would also be OK!
You can use dbh_do from your DBIx::Class::Storage to run arbitrary queries. With that, just SELECT the CURRENT_TIMESTAMP.
my ( $timestamp ) = $schema->storage->dbh_do(
sub {
my ($storage, $dbh) = #_;
$dbh->selectrow_array("SELECT CURRENT_TIMESTAMP");
},
);
I always recommend to do all date/time related things on the app server and not rely on the database server(s). Essentially that means to not use a trigger but pass the datetime on insert/update and make it mandatory (NOT NULL).
Besides that you should store datetimes in UTC and convert to your local or other required timezone in your code.
Your issue likely happens because of an incorrect or missing timezone configuartion in which case DateTime defaults to its floating timezone.
So I am trying to make a timesheeting app in meteor, creating projects and adding time entries. Why? it was all I could think of as a test app.
But, I'm more used to dealing with PHP, in PHP I would just store a date field with a time length. Right now, I'm wondering what's the best wat to deal with dates in Meteor.
Do… I do the same thing where I store a parsed string of the date, or is it a date time object? How would you deal with dates? (I'm only 3 hours into Meteor)
Meteor also includes the momentjs library which makes dealing with dates and times very easy. You get function to format and parse.
The best way to store your time is in a Date object. This is because in Mongo you will get the timestamp and its GMT deviation. Making the time TimeZone secure.
In order to manipulate and display times, use momentjs.
This community hackpad with recommended methods and packages for storing and using dates is pretty useful:
https://meteor.hackpad.com/Meteor-Cookbook-Using-Dates-and-Times-qSQCGFc06gH
The best way to represent dates on your collection documents is by directly using the Date object type. You can store Date objects directly into collection documents. If we are creating a document, we can generate a Date object as one of the properties supplied to the collection's insert() method.
I would suggest that you store the time in epoch. It will make it a lot easier to sort and search. Normally getTime() gets the time since the epoch in miliseconds but you can divide by 1000 to get the time in seconds.1
var d = new Date();
var seconds = d.getTime() / 1000;
To convert to the local date if you need it you can just
var d = new Date(0); // The 0 there is the key, which sets the date to the epoch
d.setUTCSeconds(seconds);
I have various timestamps stored in Mongo collections, some as floats and some as ints.
They are all stored in BST and the server will be switched to UTC soon. How do I convert them within Mongo to be UTC timestamps?
In MySQL I can do this:
UPDATE `table` SET `field` = CONVERT_TZ(`field`, 'Europe/London', 'UTC');
Is there a Mongo equivalent?
You'll have to use your language of choice and update them one at a time. It should be as simple as a for loop that loads the data and rewrites it.
Just double-check how your language of choice handles timestamps across timezones. Making such a data change can have all kinds of unexpected effects on production code.
Timestamps are generally in UTC and not in a specific timezone. All date/time libraries I've worked with return timestamps that are the number of seconds (or milliseconds) since Jan 1st 1970 UTC. Check the documentation for the library you used to create the timestamp to be sure.
This means that you should be ok, unless you have used a date/time library that does not follow this convention, or somehow calculated the timestamps yourself and accounted for the timezone.
For example in JavaScript if you store the value returned from new Date().getTime() and later pass that value to new Date(...) on a different system you will end up with the same absolute date/time, regardless of the timezones of the two systems. The same goes for Ruby, do Time.new.to_i on one machine, and run Time.at(...) on another and you will get the same absolute date/time. When I say "absolute date/time" I mean that it's UTC time will be the same, most likely the system will display it in the local time zone, but that is what you want.
Some points to consider about date in Mongo:
All dates are stored in UTC in MongoDB
MongoDB stores dates internally as a 64 bit integer representing
milliseconds since 1970-01-01T00:00:00Z
If the date value you provide is not already in UTC it
will be converted to UTC before being stored in MongoDB by the driver
The recommendation is not to use DateTime.Parse.
You will get into all sorts timezone issues regarding how DateTimes are formatted.
Instead just use one of the DateTime constructors with UTC flavor.
Example:
var dateTime = new DateTime(2011, 5, 5, 0, 0, 0, DateTimeKind.Utc);
Hope you find it useful.