Can anyone tell me why I'm not getting counts for each f0, MONTH, DAY, HOUR, MINUTE group in my result set?
Result set:
Query:
SELECT t.f0, MONTH(TO_DATE(Hex2Dec(t.f2))), DAY(TO_DATE(Hex2Dec(t.f2))), HOUR(TO_DATE(Hex2Dec(t.f2))), MINUTE(TO_DATE(Hex2Dec(t.f2))), COUNT(DISTINCT t.f1)
FROM table t
WHERE (t.f0 = 1 OR t.f0 = 2)
AND (t.f3 >= '2013-02-06' AND t.f3 < '2013-02-15')
AND (Hex2Dec(t.f2) >= 1360195200 AND Hex2Dec(t.f2) < 1360800000)
AND *EXTRA CONDITIONS*
GROUP BY t.f0, MONTH(TO_DATE(Hex2Dec(t.f2))), DAY(TO_DATE(Hex2Dec(t.f2))), HOUR(TO_DATE(Hex2Dec(t.f2))), MINUTE(TO_DATE(Hex2Dec(t.f2)))
Schema:
f0 INT (Partition Column)
f1 INT
f2 STRING
f3 STRING (Partition Column)
f4 STRING
f5 STRING
f6 STRING
f7 MAP<STRING,STRING>
*f2 is a unix timestamp in Hexadecimal format
This might be because to_date returns null when it's applied on a unix time.
According to the Hive manual:
to_date(string timestamp): Returns the date part of a timestamp
string: to_date("1970-01-01 00:00:00") = "1970-01-01"
Use from_unixtime instead to get back the correct date parts.
Note:
I assume Hex2Dec UDF is taken from the core library of HIVE-1545
Related
I have a 'timestamp without time zone' value in a column example:
PunchIn = "2019-06-10 09:10:00"
How can I extract "2019-06-10 00:00:00" from this column?
demo:db<>fiddle
SELECT my_date::date::timestamp
This converts the date column my_date to date (cuts the time part) and if you cast it back into timestamp it gets the 0 time.
Alternatively you can use the date_trunc function:
SELECT date_trunc('day', my_date)
SELECT SUBSTRING("PunchIn = '2019-06-10 09:10:00'" FROM 11 FOR 21);
should return the string you're looking for (if I count the index number right:)
You can use the DATE function to extract date:
SELECT DATE(SUBSTRING("PunchIn = '2019-06-10 09:10:00'" FROM 11 FOR 21));
I'm trying to get records inserted after a certain date given to me by the client.
2018-06-06
Here's how I'm writing the query:
{:ok, date} = NaiveDateTime.from_iso8601(date_string)
from(
m in query,
where: m.inserted_at > ^date
)
(MatchError) no match of right hand side value: {:error, :invalid_format}
And when I try to use a simple Date object:
** (Ecto.Query.CastError) lib/messages/search.ex:77: value ~D[2018-06-06] in where cannot be cast to type :naive_datetime in query
How can I find all messages inserted after that dummy string date the client is passing me?
You have an ISO 8601 date there, not a datetime. You can convert it into a NaiveDateTime (with hour, minute, second all set to 0) like this:
iex(1)> date_string = "2018-06-06"
"2018-06-06"
iex(2)> ndt = NaiveDateTime.from_iso8601!(date_string <> " 00:00:00")
~N[2018-06-06 00:00:00]
Now you can use ndt in your query and it will work.
I'm trying to select records from a DB2 Iseries system where the date field is greater than the first of this year.
However, the date fields I'm selecting from are actually PACKED fields, not true dates.
I'm trying to convert them to YYYY-MM-DD format and get everything greater than '2018-01-01' but no matter what I try it says it's invalid.
Currently trying this:
SELECT *
FROM table1
WHERE val = 145
AND to_date(char(dateShp), 'YYYY-MM-DD') >= '2018-01-01';
it says expression not valid using format string specified.
Any ideas?
char(dateshp) is going to return a string like '20180319'
So your format string should not include the dashes.. 'YYYYMMDD'
example:
select to_date(char(20180101), 'YYYYMMDD')
from sysibm.sysdummy1;
So your code should be
SELECT *
FROM table1
WHERE val = 145
AND to_date(char(dateShp), 'YYYYMMDD') >= '2018-01-01';
Charles gave you a solution that converts the Packed date to a date field, and if you are comparing to another date field, this is a good solution. But if you are comparing to a constant value or another numeric field, you could just use something like this:
select *
from table1
where val = 145
and dateShp >= 20180101;
I've the following code:
Data = load '/user/cloudera/' using PigStorage('\t')
as
( ID:chararray,
Time_Interval:chararray,
Code:chararray);
transf = foreach Source_Data generate (int) ID,
ToString( ToDate((long) Time_Interval), 'yyyy-MM-dd hh:ss:mm') as TimeStamp,
(int) Code;
SPLIT transf INTO Src25 IF (ToString(TimeStamp, 'yyyy-MM-dd')=='2016-07-25'),
Src26 IF (ToString(TimeStamp, 'yyyy-MM-dd')=='2016-07-26');
STORE Src25 INTO '/user/cloudera/2016-07-25' using PigStorage('\t');
STORE Src26 INTO '/user/cloudera/2016-07-26' using PigStorage('\t');
I want to split the files by date and the rules that I'm putting in Split statement it gives me error...
How can I transform TimeStamp (used on transf statement) in Date to make the comparasion?
Many thanks!
After you get the datetime object from ToDate, use GetYear(),GetMonth(),GetDay() on the datetime object and use CONCAT to construct only the date.
transf = foreach Source_Data generate
(int) ID,
ToString( ToDate((long) Time_Interval), 'yyyy-MM-dd hh:ss:mm') as TimeStamp,
(int) Code;
transf_new = foreach transf generate
ID,
TimeStamp,
CONCAT(CONCAT(CONCAT(GetYear(TimeStamp),'-')),(CONCAT(GetMonth(TimeStamp),'-')),GetDay(TimeStamp)) AS Day,-- Note:Brackets might be slightly off but it should be like 'yyyy-MM-dd' format
Code;
-- Now use the new Day column to split the data
SPLIT transf_new INTO Src25 IF (Day =='2016-07-25'),
Src26 IF (Day =='2016-07-26');
Let's say that I have a range of SQL tables that are named name_YYYY_WW where YYYY = year and WW = week number. If I call upon a function that guides a user defined date to the right table.
If the date entered is "20110101":
SELECT EXTRACT (WEEK FROM DATE '20110101') returns 52 and
SELECT EXTRACT (YEAR FROM DATE '20110101') returns 2011.
While is nothing wrong with these results I want "20110101" to either point to table name_2010_52 or name_2011_01, not name_2011_52 as it does now when I concanate the results to form the query for the table.
Any elegant solutions to this problem?
The function to_char() will allow you to format a date or timestamp to output correct the iso week and iso year.
SELECT to_char('2011-01-01'::date, 'IYYY_IW') as iso_year_week;
will produce:
iso_year_week
---------------
2010_52
(1 row)
You could use a CASE:
WITH sub(field) AS (
SELECT CAST('20110101' AS date) -- just to test
)
SELECT
CASE
WHEN EXTRACT (WEEK FROM field ) > 1 AND EXTRACT (MONTH FROM field) = 1 AND EXTRACT (DAY FROM field) < 3 THEN 1
ELSE
EXTRACT (WEEK FROM field)
END
FROM
sub;