In one of my use case i am trying to create a pipeline
whenever i sent the message from custom partition, i sent the timestamp in milliseconds with LONG data type because in the schema, the timestamp column has been defined as long.
Code that i had earlier in custom partition:
Date date = new Date();
long timeMilli = date.getTime();
System.out.println("date = " + date.toString() + " , time in millis = " + timeMilli);
Display result before i sent the record:
date = Tue Mar 26 22:02:04 EDT 2019 , time in millis = 1553652124063
value inserted in timestamp column in table2:
3/27/2019 2:02:04.063000 AM
Since its taking UK timezone (i believe), i put temporary fix for time being to subtract 4 hours from the current timestamp so that i can match with USA EST timestamp.
Date date = new Date();
Date adj_date = DateUtils.addHours(date,-4);
long timeMilli = adj_date.getTime();
System.out.println("date = " + date.toString() + " , time in millis = " + timeMilli);
Display result:
date = Tue Mar 26 22:04:43 EDT 2019 , time in millis = 1553637883826
value inserted in timestamp column in table2:
3/26/2019 10:04:43.826000 PM
Please let me know if i am missing anything as i am not sure why this is happening when i sent message from custom partition.
Under the hood Jdbc Source Connector use following query:
SELECT * FROM someTable
WHERE
someTimestampColumn < $endTimetampValue
AND (
(someTimestampColumn = $beginTimetampValue AND someIncrementalColumn > $lastIncrementedValue)
OR someTimestampColumn > $beginTimetampValue)
ORDER BY someTimestampColumn, someIncrementalColumn ASC
Summarizing: The query retrieve rows if their timestamp column's value is earlier the current timestamp and is later than last checked.
Above parameters are:
beginTimetampValue - value of timestamp column of last imported record
endTimetampValue - current timestamp according to the Database
lastIncrementedValue - value of incremental column of last imported record
I think in your case Producer put to the Tables records with higher timestamp, than you later insert manually (using the query).
When Jdbc Connector checks for new records to import to Kafka it skips them (because they don't fullfil someTimestampColumn < $endTimetampValue timestamp condition)
You can also change log level to DEBUG and see what is going on in logs
Related
very new to the backend as well as all things postgresql, at the moment all ive been able to do is
SELECT * FROM nutrition WHERE timestamp >= (extract(epoch from now())::bigint * 1000) - 86400000 AND timestamp <= (extract(epoch from now())::bigint * 1000) + 86400000
in the frontend using js, im using Date.now() to store the timestamp in the DB.
timestamp is a column in my db thats logging the unix time in bigint format in which the food was logged. I want to get all the data from the current day from the hours beteen 12 AM midnight, and 11:59 PM. thanks.
for example, the last item i logged was last night at 10pm (1663995295337 UNIX time) so the data shouldnt include it.
show timezone returns;
America/Los_Angeles
Solution below --------------------------------------------------------------------
const today = new Date();
const beginningOfDay = today.setUTCHours(7, 0, 0, 0);
const endOfDay = today.setUTCHours(30, 59, 59, 99);
switch (method) {
case "GET":
try {
const text = `
select
*
from
nutrition
where
timestamp
between ${beginningOfDay} and ${endOfDay}`
this was the solution i was looking for, thanks for the help. sorry if i wasnt descriptive enough.
Assuming that by Unix time you mean epoch.
select extract(epoch from now());
extract
-------------------
1664038032.392004
select to_timestamp(1664038032.392004);
to_timestamp
--------------------------------
09/24/2022 09:47:12.392004 PDT
select
*
from
some_table
where
to_timestamp(1664038032.392004)
between
current_date + '00:00:00'::time AND current_date + '23:59:59'::time
UPDATE
Using timestamptz field in Postgres and an ISO datetime string from Javascript in order to properly deal with time zone.
create table tsz_test(id integer, tsz_fld timestamptz);
--In Web frontend today = new Date().toISOString(); "2022-09-24T20:57:05.830Z"
insert into
tsz_test
values (1, '2022-09-24T20:57:05.830Z'), (2, '2022-09-25T08:57:05.830Z');
select * from tsz_test ;
id | tsz_fld
----+----------------------------
1 | 09/24/2022 13:57:05.83 PDT
2 | 09/25/2022 01:57:05.83 PDT
--Borrowing from #a_horse_with_no_name answer
select * from tsz_test where tsz_fld::date = '09/24/2022'::date;
id | tsz_fld
----+----------------------------
1 | 09/24/2022 13:57:05.83 PDT
You can convert your "unix timestamp" to a date, then compare it with "today":
where to_timestamp("timestamp")::date = current_date
This assumes that your column named "timestamp" is not really a timestamp
If the column doesn't actually store seconds (which would be a unix epoch), but milliseconds you need to_timestamp("timestamp"/1000)::date instead (another source of problems that wouldn't exist if you had used a proper timestamptz or at least timestamp data type).
I want to compare the date in database with current dateTime in JPA query :
captureLimitDate < currentDateTime
my requirement is as follows :
database.captureLimitDate : 04/07/2012 19:03:00
currentDateTime : 04/07/2012 20:03:00
My JPAQuery is this :
SELECT o FROM Operation o"
+ " WHERE ( o.merchantId =:merchantId ) AND "
+ "(o.captureLimitDate < currentDateTime ) ";
And Operation class has captureLimitDate as java.util.Date
#Generated(value = "XA", comments = "0,_8BedAMXZEeGHf_Dj4YaPyg")
private Date captureLimitDate;
I want to compare both current date and time . will the above query works. ??
CURRENT_TIMESTAMP must be used to refere to the current date and time in a JPQL query:
select o from Operation o
where o.merchantId = :merchantId
and o.captureLimitDate < CURRENT_TIMESTAMP
If the current date and time is in fact a date coming from user input (and which is thus not the current date and time, then you do it like you do for the merchantId:
select o from Operation o
where o.merchantId = :merchantId
and o.captureLimitDate < :maxDateTime
And you set the parameter using
query.setParameter("maxDateTime", maxDateTime, TemporalType.TIMESTAMP);
JPA defines special JPQL expressions that are evaluated to the date and time on the database server when the query is executed:
CURRENT_DATE - is evaluated to the current date (a java.sql.Date instance).
CURRENT_TIME - is evaluated to the current time (a java.sql.Time instance).
CURRENT_TIMESTAMP - is evaluated to the current timestamp, i.e. date and time
(a java.sql.Timestamp instance).
More info: Click here
I want to count records that are not marked as completed with a Received date less than the Received date of the row (group by Received date "Details") This will be the Start of Day column showing how many records are in queue.
I have a tablix in VS 2017 SSDT.
Tablix is grouped by Received Date
COLUMNS
Received Date (group by Details) another column same field (Textbox5)
Start of Day
New Tasks
Completed
I'm having an issue with the code logic for Start of Day column field.
I want to count records that are not marked as completed with a Received date less than the Received date of the row (group by Received date "Details")
This code works for New Tasks column.
=COUNT(IIF(Fields!Received.Value < Fields!Received.Value AND
Fields!Completed.Value = "NO", 1,0),"Details")
When I attempt the Start of Day expression I get errors. Textbox5 is the same dataset field used in group by field (Received). I added it to test different approach.
=COUNT(IIF(Fields!Received.Value < ReportItems!Textbox5.Value AND
Fields!Completed.Value = "NO", 1,0),"Details")
Error: rsAggregateReportItemlnBody aggregate functions can be used
only on report items contained in page headers and footers.
Sample data and expected output for Start of Day Column: it should count records in the group by row if they were in queue prior to start of day (yesterday).
You can use running value to calculate the total. To get the prior date total you subtract the group total value
Start Of Day
= RunningValue( 1, SUM, "Tablix1")- SUM( 1)
New Task
= SUM(1) or COUNT(1)
Complete
= RunningValue( Iif(Fields!CompYN.Value = "YES",1,0), SUM, "Tablix1")- SUM( Iif(Fields!CompYN.Value = "YES",1,0))
New Tasks column: =COUNT(IIF(Fields!Received.Value = Fields!Received.Value AND Fields!Completed.Value = "NO", 1,0),"Details")
Start of day column: =COUNT(IIF(Fields!Received.Value < Fields!Received.Value AND Fields!Completed.Value = "NO", 1,0),"Details")
I am trying to convert a long utc value into "yyyy-MM-dd HH:mm:ss" formatted pattern. I am expecting my data to be converted on 24 hours range scale and in GMT. My code passes all the test cases, I push the data into database using the jar that is newly built with this code -
dbRecord("order_dt_utc") = if (orderTs.isDefined) Some(new DateTime(orderTs.get, DateTimeZone.UTC).toString("yyyy-MM-dd HH:mm:ss")) else None
and now, when I query my database, I find that the data is still converting on 12 hours range. The query -
SELECT order_id, order_dt, order_dt_utc, order_ts_utc, from_unixtime(order_ts_utc/1000) FROM order_items where order_dt >= '2018-08-01' AND order_dt <= '2018-08-02' ORDER BY order_dt_utc LIMIT 1000;
And you can see the the values are not matching in the columns from_unixtime(order_ts_utc/1000) and order_dt_utc -
I am not able to figure the reason for this behaviour.
To convert Time Zone use the function first:
CONVERT_TZ (dateobj, oldtz, newtz)
After that use the date_format function:
date_format(from_unixtime(order_ts_utc), '%Y-%m-%d %H:%i:%s');
to format your time to 00-23 format.
I have a column in a table in which we are storing date in DATETIME format. (DD-MON-RRRR HH24:MI:SS) - Database Oracle 11g.
Data Type of a column is DATE, and storing date in 01-01-2012 01:00 PM (i.e. jan 1, 2012) format.
entity
#NotNull
#Column(name = "dateColumnName")
#Temporal(TemporalType.TIMESTAMP)
private Date sampleDate;
I am fetching all data by passing date
SAMPLE_QUERY = "select * from TableA tab where tab.dateWithTime = :sampleDate order by tab.dateWithTime ASC "
singleDate is "Tue Jan 24 00:00:00 IST 2012" , fasttime :
1327343400000
The problem is I am passing only date in the query, though Date through which records are being fetched is in DATE TIME format i.e 01-01-2012 01:00 PM.
How can i change my query so that it fetches all the records in ascending order of DateTime.
If you want to fetch all times for that day, then change your query to be more like
SELECT ... WHERE dateField >= :lowerParam AND dateField < :upperParam
Oracle has no DATE TIME datatype. The DATE datatype contains both a date and a time component, down to the second. TIMESTAMPS get a bit more complicated.
If your dateWithTime column is indeed a DATE datatype, the ORDER BY dateWithTime ASC clause should order your results in ascending order.
You may not be displaying the time component of your date. You can convert a date to a string in that format with TO_CHAR( dateWithTime, 'dd/mm/yyyy hh24/mm/ss' ) or whatever format you want.
Edit:
Oh, do you mean you want to find the cases where the date component of the DATE matches, but you don't care about the time component? That can be handled in the where clause with something like:
WHERE TRUNC( tab.dateWithTime ) = TRUNC( :sampledate )
TRUNC by default truncates a date to the beginning of the day.