UnixTime saved as int, how to query as Date. Cassandra CQLSH - apache-kafka

Im building the pipeline: sensorData - MQTT broker - Kafka - Cassandra. Payload is transferred as JSON and when saving in Cassandra, date was save as int. I can't get the readable date when query CQLSH.
CREATE TABLE sensordata.mqttsensordata (
sensor text,
temperature float,
humidity int,
timestamp int,
battery int,
calibratedhumidity int,
datetime timestamp,
receiver text,
rssi float,
voltage float,
PRIMARY KEY (
(sensor, temperature, humidity),
timestamp
)
How do I get the readable timestamp when query the database like the picture below?
Query table

The only time function that can operate on a UNIX timestamp is [min|max]timeuuid (mintimeuuid() or maxtimeuuid()). You can use either of those on the timestamp column, and nest it inside the toTimestamp() function.
For example, if I have a table of sample times:
> SELECT * FROM sample_times WHERE a=1;
a | b | c | d
---+---------------------------------+--------------------------------------+---------------
1 | 2021-08-08 21:42:54.131000+0000 | 96594031-f891-11eb-b7bc-e12958c8479f | 1628458974131
Column d is a bigint where I have stored the UNIX timestamp. I can show that as a timestamp like this:
> SELECT totimestamp(mintimeuuid(d)) FROM sample_times WHERE a=1;
system.totimestamp(system.mintimeuuid(d))
-------------------------------------------
2021-08-08 21:42:54.131000+0000
(1 rows)

Related

How to convert from decimal to date in scala select?

I have a column datetime object declare as decimal (38,0) not timestamp or date and the data input is 'yyyMMdd'. How do I select data with that column convert as date format as 'yyyy-MM-dd' in spark sql (scala) within a day or two days old?
I have tried:
select count(*) from table_name where to_date('column_name','yyyy-MM-dd') = date_sub(current_date(),1));
this gives me 0 count when a data have quiet more than 500000 records
I tried:
select count(*) from table_name where from_unixtime(cast(load_dt_id as string), 'yyyy-MM-dd') = date_sub(current_date(), 1));
I got data in year 1970-01-31 which those year data are not in the table, even when I select that column where it's like '1970%', I got "OK" with bulk sign that accelerate query with Delta. The data select in order of that column started with 20140320
The format argument for to_date is the format of the input, not the desired output. Assuming you have yyyymmdd:
Seq(("20200208")).toDF("RawDate").select(col("RawDate"),to_date(col("RawDate"),"yyyyMMdd").as("formatted_date")).show()
+--------+--------------+
| RawDate|formatted_date|
+--------+--------------+
|20200208| 2020-02-08|
+--------+--------------+
Expanding this to filter by the derived date column:
val raw = Seq(("20200208"),("20200209"),("20200210")).toDF("RawDate")
raw: org.apache.spark.sql.DataFrame = [RawDate: string]
raw.select(col("RawDate"),to_date(col("RawDate"),"yyyyMMdd").as("formatted_date")).filter($"formatted_date".geq(date_add(current_date,-1))).show
+--------+--------------+
| RawDate|formatted_date|
+--------+--------------+
|20200209| 2020-02-09|
|20200210| 2020-02-10|
+--------+--------------+

Strange date format in database

I have some very strange looking 18 character alphanumeric datetimes in a SQL database, they seem to be using Hexadecimal?
I can find out what the dates are through the application which uses them, but I was looking for a way to convert them via a query. Do you know how I would convert these with TSQL?
000B3E4Bh01F2D947h - 29/05/2018 09:04:52
000B3E0Dh03A16C1Eh - 23/05/2018 10:22:26
000B3E4Eh0248C3D8h - 01/06/2018 10:38:43
000B3E4Eh0249B449h - 01/06/2018 10:39:44
I assume the date and time are separated as below, but I don't know to convert the individual parts if anyone can help with this? Thanks!!
000B3E4Eh (date) - 0249B449h (time)
(The dates are in dd/mm/yyyy format)
Your hex values are separated as you have assumed (with the h used as a delimiter) and represent integer values to add to a baseline date and time value.
Using your 000B3E54h0221CBFEh - 07/06/2018 09:56:09 value this translates as:
Date portion: 000B3E54
Integer Value: 736852
Time portion: 0221CBFE
Integer Value: 35769342
These integer values are then added as days to the date 0001/01/00 (which SQL Server can't handle, hence the +/-1 below) and milliseconds to 00:00:00 respectively, which you can see working in this script:
select convert(int, 0x000B3E54) as DateIntValue
,dateadd(day,convert(int, 0x000B3E54)-1,cast('00010101' as datetime2)) as DateValue
,convert(int, 0x0221CBFE) as TimeIntValue
,cast(dateadd(millisecond,convert(int, 0x0221CBFE),cast('19000101' as datetime2)) as time) as TimeValue
,cast(datediff(day,cast('00010101' as datetime2),'20180607')+1 as binary(4)) as DateHexValue
,cast(datediff(millisecond,cast('20180607' as date),cast('2018-06-07 09:56:09.342' as datetime2)) as binary(4)) as TimeHexValue
Which outputs:
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| DateIntValue | DateValue | TimeIntValue | TimeValue | DateHexValue | TimeHexValue |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| 736852 | 2018-06-07 00:00:00.0000000 | 35769342 | 09:56:09.3420000 | 0x000B3E54 | 0x0221CBFE |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
Note the judicious use of datetime2 values to ensure the right amount of milliseconds are output/returned, as SQL Server datetime is only accurate to the nearest 3 milliseconds.

Why does DATE in DB2 database have a time component in it?

How can I make the column data type be DATE like YYYY-MM-DD;
when I create a table with the Data type DATE, it will become TIMESTAMP(0)
When I ALTER SET DATA TYPE DATE, it still is TIMESTAMP(0)
and SELECT CHAR(CURRENT DATE, ISO) FROM SYSIBM.SYSDUMMY1;
it will be error with SQLCODE=-171, CURRENT DATE is 2017-02-28 19:19:09.0
it's too long.
database info:DB2 linux x64 10.5
CREATE TABLE "XCRSUSR"."TIMP_TASK_SERIAL" (
"SERIAL_NO" DECIMAL(16 , 0),
"TASK_NAME" VARCHAR(10),
"TASK_TYPE" DOUBLE,
"TASK_XML" CLOB(10) INLINE LENGTH 164,
"SEND_TIME" DATE,
"FINISH_TIME" DATE,
"TASK_STATUS" DOUBLE DEFAULT 0,
"RUN_TYPE" DOUBLE,
"FLAG" DOUBLE,
"TASK_ID" VARCHAR(10)
)
ORGANIZE BY ROW
DATA CAPTURE NONE
IN "CREDIT_U_16" INDEX IN "CREDIT_INDEX_16"
COMPRESS NO;
ALTER TABLE TIMP_TASK_SERIAL ALTER COLUMN SEND_TIME SET DATA TYPE DATE;
select CURRENT DATE from SYSIBM.SYSDUMMY1;
1
---------------------
2017-02-28 19:19:09.0
Check out the settings of the
Oracle_Compatibility
vector under
https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.apdv.porting.doc/doc/r0052867.html
Bit Position 7 in table 1 is wjat you are looking for.

How do I go from Varchar string into Unix time type? Greenplum

I'm fairly inexperienced with SQL let alone PostgreSQL so appreciate your help.
Let's say we have a varchar column called ID that is a combination of IP address and a unix epoch time string.
So when I run this query:
SELECT substring(b.id from char_length (r.IP_address)+2 for 10), b.id, r.ip_address
FROM bq b
INNER JOIN event r ON r.visitor_id::TEXT = b.id::TEXT
LIMIT 3;
Output
substring id ip_address
1460854333 97.128.39.256.1460854333288493 97.128.39.256
So the output of the substring is a unix time stamp but it is in a varchar format. How do I convert the varchar into a unix timestamp? Ultimately I am going to convert that time stamp into date I just don't think I can directly go from the varchar of a unix string to a date?
As Terra said, use the to_timestamp() function:
# select to_timestamp('1460854333') ;
to_timestamp
------------------------
2016-04-16 17:52:13-07
(1 row)
If a date is the type you're after, just add the ::date cast:
# select to_timestamp('1460854333')::date ;
to_timestamp
------------------------
2016-04-16
(1 row)
The function is basically executing this:
select ('epoch'::timestamptz + '1460854333'::float * '1 second'::interval) ;

Avoiding default date value to be used when only time is provided to a datetime field on Redshift

I created a table with a datetime field "dt". Using COPY command to load data. The corresponding value for the field from the file is just the hour information, i.e., say, 14:50:00. So, the value being stored is 1900-01-01 14:50:00. I don't need the date part. How to do that.
Or may be an alternate datatype which can store only time.
Amazon Redshift supports only date(year month day) and timestamp(year month day hour minute second) format, and it doesn't support time(hour minute second) format of Postgresql.
In my idea, there are two ways to work around.
As #Damien_The_Unbeliever mentioned, ignore the date part of the timestamp format.
create table date_test(id int, timestamp timestamp);
insert into date_test2 values (1, '1900-01-01 14:50:00');
insert into date_test2 values (2, '1900-01-01 17:20:00');
select * from date_test2 where timestamp > '1900-01-01 14:50:00';
select * from date_test where date_test.timestamp > '1900-01-01 14:50:00';
id | timestamp
----+---------------------
2 | 1900-01-01 17:20:00
(1 row)
Use char or varchar type to store the time value.
create table date_test2(id int, timestamp char(8));
insert into date_test2 values (1, '14:50:00');
insert into date_test2 values (2, '17:20:00');
select * from date_test2 where timestamp > '14:50:00';
id | timestamp
----+-----------
2 | 17:20:00
(1 row)
The second solution looks easier, but it is worse performance as Redshift doc says. If you store a large amount of data, you should consider of the first one.
Here are the related links to the document about date/time column.
http://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-timestamp-date-columns.html
http://docs.aws.amazon.com/redshift/latest/dg/r_Datetime_types.html