I have django project and one separate service.
In django timezone is UTC+3 and in separate service I have written pure sql codes
like below
INSERT INTO Table(something,something,datetime) VALUES(%s,%s,%s)
as a datetime field if I send 2021-04-28 01:00:00, django will save it 2021-04-28 04:00:00,
3 hours more. How can I implement db to store exactly what I send there.
Related
I use MSK and I manually build aggregate tables of my streams in my application code (e.g. TypeScript in a node.js webservice). I have lots of data (approaching 1M events per day), and I want to be able to productionise different real-time 'views' on the incoming stream. E.g. for some sales data, I might want to create these views:
sales per customer (table schema: customer, sum_of_sales)
sales per day (table schema: date, sum_of_sales)
sale per customer per day (table schema: date, customer, sum_of_sales)
Today if I wanted to achieve this I would scaffold 3 tables up (could be RDMS or something like DynamoDB), and then in my application code, I would insert/upsert into the table for every sales event that arrived. The scaffolding around that feels a little tedious, I was wondering if there is a better way without having to write a bunch of code in my webservice to actually pull from the consumer, upsert the data into a table.
All I would expect my code in my web service to do is provide APIs (e.g. REST APIs) to fetch data from these views. E.g. a client makes a REST request to get all sales in the last 7 days for customers X, Y and Z.
There seems like a lot of technologies out there, but my use case is fairly trivial and from the not-so-brief look I took nothing does this.
Thanks
If it's noteworthy, I currently keep my data indefinitely.
We save all our datetime data on database in UTC (a timestamp with time zone column in postgresql).
Assuming "America/Sao_Paulo" timezone, if a user saves an event "A" to the database at 2021-08-24 22:00:00 (local time) this will be converted to UTC and saved as 2021-08-25 01:00:00.
So, we are wondering what would be the best way (here "the best way" refers to the developer experience) to consume an API where is possible to filter events by start and end date.
Imagine the following situation: the user is on the website and needs to generate a report with all events that happened on 2021-08-24 (local time America/Sao_Paulo). For this, the user fills start and end date both with 2021-08-24.
If the website forwards this request directly to the API, the server will receive the same date provided by the user and some outcomes can happen:
If the server does not apply any transformation at all, the data returned will not contain the event "A" — by the user perspective, this is wrong.
The server can assume that the date is in UTC and transform start date to 2021-08-24 00:00:00 and end date to 2021-08-24 23:59:59. Then, apply the timezone of the user, generating: 2021-08-24 03:00:00 and 2021-08-25 02:59:59. Filtering the database now would bring the expected event "A".
The API itself could expected a start and end datetime in UTC. This way, the developer can apply the user timezone on client side and then forward to server (2021-08-24T03:00:00Z and 2021-08-25T02:59:59Z).
The API itself could expected a start and end datetime either in UTC or in with the supplied offset (2021-08-24T00:00:00-03:00 and 2021-08-24T23:59:59-03:00). Github does it this way.
What got us thinking was that a lot of APIs accept only a date part on a range filter (like the github API). So, are those APIs filtering the data in the client timezone or they assume the client knows the equivalent UTC date that they should filter by (we could not find any documentation that explains how github deals with an incoming date only filter)?
For us, makes more sense the date filter consider the timezone of the client and not leave to them the burden to know the equivalent UTC datetime of the saved event. But this complicates a bit the filtering logic.
To facilitate the filter logic, we thought that maybe have another column on database to also save the local datetime of the event (or only the local date) would be interesting. Is this a valid approach? Do you know any drawbacks?
*We know that on a database perspective, it is recommended to save datetime in UTC (not always, as showed here) but in our case this seems to only make things more difficult when handling API consumption.
*It is importante to know that, when saving an event, the user cannot provide when it happens, we always assume the event happens in the moment it is being saved.
I'm receiving a date from a node.js backend in flutter. The date is converted to the users timezone in node.js.
For some reason, flutter automatically converts it to utc. How can I receive the dateTime as is and not have flutter convert it?
Background: The way it works is flutter first sends a dateTime to the
backend already converted to utc so that it can be stored in the db in
utc. then anytime the backend sends back a date, it converts it to the
timezone the user was in when signing up. (not sure if this is a good
way of doing time conversion...)
UPDATE:
Seems like DateTime.parse always converts to utc... Any way to stop that behavior?
If you're using DateTime.parse, it says in the documentation that the way to handle the changing of time-zone is by providing an optional time-zone offset part in your date.
DateTime.parse("2020-09-21T14:00:00") // 2020-09-21 14:00:00.000
DateTime.parse("2020-09-21T14:00:00-1230") // 2020-09-22 02:30:00.000Z
Check out the documentation here https://api.flutter.dev/flutter/dart-core/DateTime/parse.html
I am trying to write a Rest API client where ServiceNow Database will be polled every 10 minutes to get the Data.
Below is the Url that I built:
"https://servicenowinstance.com/api/now/table/employee_table?sysparm_limit=200&sysparm_offset=0&sysparm_query=sys_created_onBETWEENjavascript:gs.dateGenerate('2018-02-28','14:23:40')#javascript:gs.dateGenerate('2018-02-28','15:17:04')^ORDERBYsys_created_on".
After implementing Pagination I am starting the Incremental Load. Where I poll every 10 minutes to get the New Data. So in the above URL I get the BETWEEN . So I will get the Data which satisfies the Between Condition.
My Question is that the VM machine I use maintains UTC time. And I am not sure which Timezone does the ServiceNow Tables use to store the Data.
In short my question is what Timezone does ServiceNow use to store its Sys_created Field. Is it same as UTC or is it different?
The database stores dates and times as UTC (KB0534905), but depending on how you pull data via REST, it may return in the timezone of the user account being used for authentication.
Take a look at Table API GET, in particular the sysparm_display_value field.
Data retrieval operation for reference and choice fields. Based on
this value, retrieves the display value and/or the actual value from
the database.
true returns display values for all fields.
false returns actual values from the database. If a value is not specified, this parameter defaults to false.
all returns both actual and display values.
In your case since you're not setting that parameter, it should be in UTC.
I am trying to save a date into meteor mongodb my challenge is as follows:
1) if i use new Date() it creates a date object in mongo DB however it saves the time as local time as javascript Date() this always comes with a timezone +0x:hours based on browser local timezone. When i retrieve this it causes havoc as i am assuming everything in my db is UTC.
2) I want to use moment js library which is great because it can represent dates in UTC properly but my challenge is how do i get mongo db to accept a moment time? The minute i use moment.format() it saves it as a string!
So how can i send a date to a mongodb insert command with a date object that is in UTC? string just dont work :(
Any help would be appreciated.
Thanks
I think everything you need to know about both of these questions can be found here and here.
TLDR:
If you directly insert/update from the client you will store a timestamp based on the user's clock. It will still be stored as UTC, but you may or may not want to trust that the time is correct. I strongly suggest using a method for any db modifications which involve time so that the server's version of time will always be used.
Moment objects are not serializable to a format compatible with mongodb. Use a date object and format it on the client.
The problem with saving dates on the client is that each client can have a different time zone, or even wrong time set. Thus the only solution is to have the date set on the server. Using a method for each insert / update is not an elegant solution.
A common practice is to modify the document inside allow or deny callback:
Messages.allow({
insert: function(userId, doc) {
...
doc.timestamp = new Date();
return true;
},
});
That way you ensure all documents have a compatible timestamp, and you can use usual db methods on the client.
The Meteor community recently started an extensive document about how to use dates and times. You'll find a lot of useful information there, in addition to David Weldon's links:
https://meteor.hackpad.com/Meteor-Cookbook-Using-Dates-and-Times-qSQCGFc06gH
However, in particular I recommend using https://github.com/mizzao/meteor-timesync when security is not a concern. It allows you to client-locally obtain an accurate server time even if the client's clock is way off, without a round-trip to the server. This can be useful for all kinds of reasons - in my apps, I universally just use server-relative time and don't care about what the client's time is at all.