in my DB2 database, I have the UTC data in the format 1160307144500000 and a field which indicates the time difference respect to local date, for example 3600 in seconds.
I need sum to UTC date the 3600 secs so I can get 1160307154500000.
Here is some DB2 code snippet
TIMESTAMP('1900-01-01 00:00:00')
+ INT(SUBSTR(utc_data,1,3)) YEAR
+ INT(SUBSTR(utc_data,4,2)) + 1 MONTH
+ INT(SUBSTR(utc_data,6,2)) DAY
+ INT(SUBSTR(utc_data,8,2)) HOUR
+ INT(SUBSTR(utc_data,10,2)) MINUTE
+ INT(SUBSTR(utc_data,12,2)) SECOND
+ 1000*INT(SUBSTR(utc_data,15,3)) MICROSECOND
+ diff_data SECOND
I'll leave the code to transform the timestamp back to the original format up to you :)
Related
In one of my use case i am trying to create a pipeline
whenever i sent the message from custom partition, i sent the timestamp in milliseconds with LONG data type because in the schema, the timestamp column has been defined as long.
Code that i had earlier in custom partition:
Date date = new Date();
long timeMilli = date.getTime();
System.out.println("date = " + date.toString() + " , time in millis = " + timeMilli);
Display result before i sent the record:
date = Tue Mar 26 22:02:04 EDT 2019 , time in millis = 1553652124063
value inserted in timestamp column in table2:
3/27/2019 2:02:04.063000 AM
Since its taking UK timezone (i believe), i put temporary fix for time being to subtract 4 hours from the current timestamp so that i can match with USA EST timestamp.
Date date = new Date();
Date adj_date = DateUtils.addHours(date,-4);
long timeMilli = adj_date.getTime();
System.out.println("date = " + date.toString() + " , time in millis = " + timeMilli);
Display result:
date = Tue Mar 26 22:04:43 EDT 2019 , time in millis = 1553637883826
value inserted in timestamp column in table2:
3/26/2019 10:04:43.826000 PM
Please let me know if i am missing anything as i am not sure why this is happening when i sent message from custom partition.
Under the hood Jdbc Source Connector use following query:
SELECT * FROM someTable
WHERE
someTimestampColumn < $endTimetampValue
AND (
(someTimestampColumn = $beginTimetampValue AND someIncrementalColumn > $lastIncrementedValue)
OR someTimestampColumn > $beginTimetampValue)
ORDER BY someTimestampColumn, someIncrementalColumn ASC
Summarizing: The query retrieve rows if their timestamp column's value is earlier the current timestamp and is later than last checked.
Above parameters are:
beginTimetampValue - value of timestamp column of last imported record
endTimetampValue - current timestamp according to the Database
lastIncrementedValue - value of incremental column of last imported record
I think in your case Producer put to the Tables records with higher timestamp, than you later insert manually (using the query).
When Jdbc Connector checks for new records to import to Kafka it skips them (because they don't fullfil someTimestampColumn < $endTimetampValue timestamp condition)
You can also change log level to DEBUG and see what is going on in logs
I have a table with records, where each record has a date column, then a start time column and end time column.
I am trying to do a datediff to get the duration in hours from start to end date with DateDiff('s',[Start Date[,[End Date])/3600.
This works perfectly for End dates that are on same day as date column, but sometimes the end date would be the next day like 12:45 AM. The date diff will give me a large negative number, how do I let it know its next day?
I dont own the data, so not much I can do with the table
Thanks!
Try something like this:
DateDiff('s',[Start Date],DateAdd('d',IIF([End date]<[Start Date],1,0),[End Date]))/3600
It can be done with pure math:
TotalHours = TimeValue(CDate([End Date] - [Start Date] + 1)) * 24
I try to understand why
print(pd.Timestamp("2015-01-01") - pd.DateOffset(day=1))
does not result in
pd.Timestamp("2014-12-31")
I am using Pandas 0.18. I run within the CET timezone.
You can check pandas.tseries.offsets.DateOffset:
*kwds
Temporal parameter that add to or replace the offset value.
Parameters that add to the offset (like Timedelta):
years
months
weeks
days
hours
minutes
seconds
microseconds
nanoseconds
Parameters that replace the offset value:
year
month
day
weekday
hour
minute
second
microsecond
nanosecond
print(pd.Timestamp("2015-01-01") - pd.DateOffset(days=1))
2014-12-31 00:00:00
Another solution:
print(pd.Timestamp("2015-01-01") - pd.offsets.Day(1))
2014-12-31 00:00:00
Also it is possible to subtract Timedelta:
print(pd.Timestamp("2015-01-01") - pd.Timedelta(1, unit='d'))
pd.DateOffset(day=1) works (ie no error is raised) because "day" is a valid parameter, as is "days".
Look at the below one: "day" resets the actual day, "days" adds to the original day.
pd.Timestamp("2019-12-25") + pd.DateOffset(day=1)
Timestamp('2019-12-01 00:00:00')
pd.Timestamp("2019-12-25") + pd.DateOffset(days=1)
Timestamp('2019-12-26 00:00:00')
Day(d) and DateOffset(days=d) do not behave exactly the same when used on timestamps with timezone information (at least on pandas 0.18.0). It looks like DateOffset add 1 day while keeping the hour information while Day adds just 24 hours of elapsed time.
>>> # 30/10/2016 02:00+02:00 is the hour before the DST change
>>> print(pd.Timestamp("2016-10-30 02:00+02:00", tz="Europe/Brussels") + pd.offsets.Day(1))
2016-10-31 01:00:00+01:00
>>> print(pd.Timestamp("2016-10-30 02:00+02:00", tz="Europe/Brussels") + pd.DateOffset(days=1))
2016-10-31 02:00:00+01:00
I have a column which is of bigint datatype(in seconds) which should be added to a date, so i need to convert this column into dateformat.
The arithmetic must be done against a timestamp data type in Teradata. The date data type does not have a time element associated with it. The following SQL should help point you in the right direction:
SELECT CAST(CAST(1234 AS BIGINT) AS INTERVAL SECOND(4)) AS Seconds_
, CURRENT_TIMESTAMP(0) AS CurrentTimestamp_
, CURRENT_TIMESTAMP + Seconds_ AS NewTimeStamp
If the number of seconds is less than 864000000 you can simply use interval arithmetic:
CAST(col AS TIMESTAMP) + (bigintcol * INTERVAL '0000 00:00:01' DAY TO SECOND)
Based on your other question your input is a Unixtime, those are two functions for converting them from/to Teradata timestamps:
/**********
Converting Unix/POSIX time to a Timestamp
Unix time: Number of seconds since 1970-01-01 00:00:00 UTC not counting leap seconds (currently 24 in 2011)
Also working for negative numbers.
The maximum range of Timestamps is based on the range of INTEGERs:
1901-12-13 20:45:52 (-2147483648) to 2038-01-19 03:14:07 (2147483647)
Can be changed to use BIGINT instead of INTEGER
20101211 initial version - Dieter Noeth
**********/
REPLACE FUNCTION UnixTime_to_TimeStamp (UnixTime INT)
RETURNS TimeStamp(0)
LANGUAGE SQL
CONTAINS SQL
DETERMINISTIC
SQL SECURITY DEFINER
COLLATION INVOKER
INLINE TYPE 1
RETURN
CAST(DATE '1970-01-01' + (UnixTime / 86400) AS TIMESTAMP(0))
+ ((UnixTime MOD 86400) * INTERVAL '00:00:01' HOUR TO SECOND)
;
SELECT
UnixTime_to_TimeStamp(-2147483648)
,UnixTime_to_TimeStamp(0)
,UnixTime_to_TimeStamp(2147483647)
;
/**********
Converting a Timestamp to Unix/POSIX time
Unix time: Number of seconds since 1970-01-01 00:00:00 UTC not counting leap seconds (currently 24 in 2011)
The maximum range of Timestamps is based on the range of INTEGERs:
1901-12-13 20:45:52 (-2147483648) to 2038-01-19 03:14:07 (2147483647)
Can be changed to use BIGINT instead of INTEGER
20101211 initial version - Dieter Noeth
**********/
REPLACE FUNCTION TimeStamp_to_UnixTime (ts TimeStamp(6))
RETURNS INTEGER
LANGUAGE SQL
CONTAINS SQL
DETERMINISTIC
SQL SECURITY DEFINER
COLLATION INVOKER
INLINE TYPE 1
RETURN
(CAST(ts AS DATE) - DATE '1970-01-01') * 86400
+ (EXTRACT(HOUR FROM ts) * 3600)
+ (EXTRACT(MINUTE FROM ts) * 60)
+ (EXTRACT(SECOND FROM ts))
;
SELECT
TimeStamp_to_UnixTime(TIMESTAMP '1901-12-13 20:45:52')
,TimeStamp_to_UnixTime(CURRENT_TIMESTAMP)
,TimeStamp_to_UnixTime(TIMESTAMP '2038-01-19 03:14:07')
;
I have a timestamp field on a mysql table , i want to know if that timestamp is 24hrs old or more. What would be the best way to do that in Perl?
SQL that would return the timestamp 24 hours ago.
SELECT UNIX_TIMESTAMP(NOW() - INTERVAL 24 HOUR)
Now if your timestamp is < the timestamp returned by the above SQL , 24 hours have passed.
The best thing is to let the database do the work. See SQL statement to select all rows from previous day for an example.
By timestamp I am assuming it's Unix timestamp, i.e. seconds since epoch.
my $ts = 1393662619;
my $day_24hr = 24 * 60 * 60; ## seconds in 24 hrs
my $prev_time = time() - $day_24hr; ## 24hours ago
if ( $ts < $prev_time ) {
print "timestamp is 24 hour old";
}
You can use localtime for that.
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($unix_timestamp);