I'm trying to use Date functions built into pig.
The input file contains date in yyyy-mm-dd hh:mi:ss format.
I'm trying to use the following code:
row = foreach log generate FLATTEN(REGEX_EXTRACT_ALL (pattern) AS (date_time : datetime, other columns);
final_data = foreach row {
yyyy = (chararray)GetYear(date_time);
mm = (chararray)GetMonth(date_time);
dd = (chararray)GetDay(date_time);
hh = (chararray)GetHour(date_time);
mi = (chararray)GetMinute(date_time);
ss = (chararray)GetSecond(date_time);
generate CONCAT(CONCAT(CONCAT(yyyy, '-'), CONCAT(mm, '-')),dd) as myDate;
}
but I get an error:
ERROR 1066: Unable to open iterator for alias final_data. Backend error : java.lang.String cannot be cast to org.joda.time.DateTime
I'm trying to use workaround from: Formatting Date in Generate Statement
What format is expected?
REGEX_EXTRACT_ALL takes a string and returns a tuple with the extracted strings, i.e you can't extract into to DateTime fields.
You may use the ToDate UDF on the date string you loaded:
E.g:
cat data.txt
2014-03-11 13:44:11
2014-02-22 10:44:11
A = load 'data.txt' as (in:chararray);
B = foreach A generate ToDate(in,'yyyy-MM-dd HH:mm:ss') as (dt:DateTime);
C = foreach B {
year = (chararray)GetYear(dt);
month = (chararray)GetMonth(dt);
day = (chararray)GetDay(dt);
generate CONCAT(CONCAT(CONCAT(year, ''), CONCAT(month, '-')),day) as myDate;
};
dump M;
(2014-3-11)
(2014-2-22)
Related
I want to compare from_date and to_date with date_order using search orm in odoo.
I just want to extract date alone because in date_order it is given date with date time. how to work with it ?
here is my code :
from_date = fields.Date(string="From", default="Today")
to_date = fields.Date(string="To")
def update_commission(self):
sale_br = self.env['sale.order']
sale_sr = sale_br.search([('date_order', '=', self.from_date)])
Modify your function like this:
def update_commission(self):
sale_br = self.env['sale.order']
sale_sr = sale_br.search([]).filtered(lambda sale: sale.date_order.date < self.from_date)
If you are working in odoo's past version like 10,11 than you need to convert this datetime into datetime object because when you call this field it will return date in string format so you need to do fields.Datetime.from_string(sale.date_order.date)
Hello I would like to convert string date to date format:
for example from 190424 to 2019-01-24
I try with this code :
tx_wd_df = tx_wd_df.select(
'dateTransmission',
from_unixtime(unix_timestamp('dateTransmission', 'yymmdd')).alias('dateTransmissionDATE')
)
But I got this format : 2019-01-24 00:04:00
I would like only 2019-01-24
Any idea please?
Thanks
tx_wd_df.show(truncate=False)
You can simply use to_date(). This will discard the rest of the date, and pick up only the format that matches the input date format string.
import pyspark.sql.functions as F
date_column = "dateTransmission"
# MM because mm in Java Simple Date Format is minutes, and MM is months
date_format = "yyMMdd"
df = df.withColumn(date_column, F.to_date(F.col(date_column), date_format))
I have two dates in my SAS dataset that I would like to compare i.e., does date1 = date2. I attempted to do this using the ifn and ifc functions but have a suspicion they are not working correctly. I took this code directly from a sugi, but have worked with these functions successfully comparing character/numeric variables. I also successfully attempted the comparison using proc sql, but I want to learn to do this in a data step.
My code is below:
data not_in_HDD_3;
set not_in_HDD_2;
start=STMT_PERIOD_FROM;
if start=. then start=ADMIT_START_OF_CARE;
start_2=input(start, ANYDTDTE8.);
format start_2 MMDDYY10.;
drop start;
rename start_2=start;
dob=input(birth_date, ANYDTDTE8.);
format dob MMDDYY10.;
Birth_record = ifn (start eq dob, 0 , 1);
ifc_result = ifc(start=dob,"True","False","Missing");
ifn_result = ifn(start=dob,1,0,.);
ifn_result_fmt = put(ifn_result,ifn_label.);
fuzz_result = ifc(fuzz(start-dob),"True","False","Missing");
drop ifn_result;
run;
proc sql;
create table not_in_HDD_4 as
select *,
case
when (start=dob) then "True"
else "False"
end as sql_case_var length=8
from not_in_HDD_3;
quit;
Any insight is appreciated!
SAS dates are simply numbers relative to Jan 1st, 1960 (i.e., day 0). You can compare them using any standard type of comparison. Here are a few ways to do it.
data want;
set have;
dob = input(birth_date, anydtdte8.);
/* 1 and 0 indicator if they are equal/not equal */
result1 = (start = dob);
/* 1, 0, missing indicator: we are exploiting the fact
that adding a missing value produces a missing value.
Note that this is not the case with the sum() function */
if(start + dob = .) then call missing(result2);
else if(start = dob) then result2 = 1;
else result2 = 0;
/* select function */
select;
when(start = dob) result3 = 1;
when(start + dob = .) result3 = .;
otherwise result3 = 0;
end;
run;
I've the following code:
Data = load '/user/cloudera/' using PigStorage('\t')
as
( ID:chararray,
Time_Interval:chararray,
Code:chararray);
transf = foreach Source_Data generate (int) ID,
ToString( ToDate((long) Time_Interval), 'yyyy-MM-dd hh:ss:mm') as TimeStamp,
(int) Code;
SPLIT transf INTO Src25 IF (ToString(TimeStamp, 'yyyy-MM-dd')=='2016-07-25'),
Src26 IF (ToString(TimeStamp, 'yyyy-MM-dd')=='2016-07-26');
STORE Src25 INTO '/user/cloudera/2016-07-25' using PigStorage('\t');
STORE Src26 INTO '/user/cloudera/2016-07-26' using PigStorage('\t');
I want to split the files by date and the rules that I'm putting in Split statement it gives me error...
How can I transform TimeStamp (used on transf statement) in Date to make the comparasion?
Many thanks!
After you get the datetime object from ToDate, use GetYear(),GetMonth(),GetDay() on the datetime object and use CONCAT to construct only the date.
transf = foreach Source_Data generate
(int) ID,
ToString( ToDate((long) Time_Interval), 'yyyy-MM-dd hh:ss:mm') as TimeStamp,
(int) Code;
transf_new = foreach transf generate
ID,
TimeStamp,
CONCAT(CONCAT(CONCAT(GetYear(TimeStamp),'-')),(CONCAT(GetMonth(TimeStamp),'-')),GetDay(TimeStamp)) AS Day,-- Note:Brackets might be slightly off but it should be like 'yyyy-MM-dd' format
Code;
-- Now use the new Day column to split the data
SPLIT transf_new INTO Src25 IF (Day =='2016-07-25'),
Src26 IF (Day =='2016-07-26');
I generate an ISO datetime string (without time zone) like this:
val dateTime: LocalDateTime = LocalDateTime.now
val dateTimeStr: String = ISODateTimeFormat.dateTime.withZone(DateTimeZone.UTC).print(dateTime)
The code above produces the following string:
2014-04-10T06:13:19.283
Now I need to convert this string back to a LocalDateTime...
val dateTime = LocalDateTime.parse(dateTimeStr, ISODateTimeFormat.dateTime().withZone(DateTimeZone.UTC))
... and compare it with the current time:
val isBefore = dateTime.isBefore(LocalDateTime.now)
The code above doesn't work and produces the following error:
Invalid format: \"2014-04-27T17:51:06.780\" is too short
To fix the problem, I need to append a Z to dateTimeStr:
val dateTime = LocalDateTime.parse(s"${dateTimeStr}Z", ISODateTimeFormat.dateTime().withZone(DateTimeZone.UTC))
Is there a way to generate the ISO datetime string with the Z at the end?
A LocalDateTime has NO timezone. So you cannot associate a timezone-aware format (Z stands for UTC timezone) with this zoneless data type.
If you insist on having Z-formats then you have to work with a global type like DateTime. So you have two different steps. One step is object conversion between local and global type:
LocalDateTime ldt = ...;
DateTime dt = ldt.toDateTime(DateTimeZone.UTC); // or another timezone
// reverse
DateTime dt = ...;
LocalDateTime ldt = dt.toLocalDateTime();
Second step is conversion between formatted string form and global timestamp:
LocalDateTime ldt = ...;
DateTime dt = ldt.toDateTime(DateTimeZone.UTC); // or another timezone
String iso = ISODateTimeFormat.dateTime().print(dt);
// reverse
String iso = ...; // with trailing Z or another timezone information
DateTime dt = IsoDateTimeFormat.parseDateTime(iso);
LocalDateTime ldt = dt.toLocalDateTime();
Finally you can combine these two steps for conversion between ISO-format with timezone information and a LocalDateTime as shown in the second step.
If you only need formatted strings without any timezone or offset information then you can stick the global type DateTime completely and just use localDateOptionalTimeParser() as #Sebastian has correctly mentioned.
Try to use
val dateTime: DateTime = DateTime.now
instead of
val dateTime: LocalDateTime = LocalDateTime.now
Or if you want to stick to LocalDateTime change the parsing:
val dateTime = LocalDateTime.parse(dateTimeStr) // this uses the default localDateOptionalTimeParser