I want the output of dates from particular range in a specific format
date1: 2019-06-18
date2: 2019-06-22
Output will be:
2019-06-19|2019-06-19,2019-06-20|2019-06-20,2019-06-21|2019-06-21
I want to do this via Datastage Component only, no unix, no perl script.
From one input record you can generate multiple rows witha transformer loop.
Date1 is frim the inout and date2 could be built with currentdate() in the transformer.
In the loop #IERATION is the the loop counter so you could use DateOffsetByDays
with your input date and #ITERATION as second parameter. Run the loop while it is <= currentDate()
In a second transformer you could then concat whatever date format you need - ot could be helpful to add a artificial numer to all rows produced from one input row.
Related
I have a spark data frame having two columns (SEQ - Integer, MAIN_DATE - Date) as:
Now I want to add a column based on the condition that if the format of MAIN_DATE is "MMM-YYYY" then it should be converted to Last day of the month and new data frame should look like this:
Any suggestion will be much appreciated.
You can use Spark's when/otherwise methods in order to operate differently for each different date format of the MAIN_DATE column.
More specifically, you can simply match the MMM-yyyy date format values of the column based on the field's String length (since we know that those values we always have 8 characters) as a condition in when and then:
use to_date to convert the String value to a valid date based on a format we give as an argument, and
use last_date to get the last day of the month each curry date in MAIN_DATE is referring to.
As for the "regular" rows with the dd-MMM-yyyy date format, just a to_date conversion would be sufficient within the otherwise method.
After that, all there's left to do is to convert the dates back to the desired dd-MMM-yyyy format (because to_date converts a given date to the yyyy-MM-dd format).
This is the solution in Scala (split in into two withColumns to make it more readable, instead of an one-liner):
df.withColumn("END_DATE",
when(length(col("MAIN_DATE")).equalTo(8), last_day(to_date(col("MAIN_DATE"), "MMM-yyyy")))
.otherwise(to_date(col("MAIN_DATE"), "dd-MMM-yyyy")))
.withColumn("END_DATE", date_format(col("END_DATE"), "dd-MMM-yyyy"))
This is what the resulting df DataFrame will look like:
+---+-----------+-----------+
|SEQ| MAIN_DATE| END_DATE|
+---+-----------+-----------+
| 1|16-JAN-2020|16-Jan-2020|
| 2| FEB-2017|28-Feb-2017|
+---+-----------+-----------+
I want to create a monthly package that executes a daily query at ODBC and writes an output file.
More specifically the query must be first executed for the first day of the previous month (e.g. '01/11/2018') then the next one ('02/11/2018') until the last day of the previous month ('30/11/2018').
The date variables are currently saved as Strings and I also want to have a string variable with Oracle date format to be inserted into the query. How should it be organised? Is there a way that I could use the string variables in the expressions?
Break it into parts as follows:
Declare variables to store previous month start and end date as follows:
start_date(datetime) = (DT_DATE)((DT_WSTR,4)YEAR(DATEADD("MM",-1,GETDATE()))+"-"+RIGHT("0"+(DT_WSTR,2)MONTH(DATEADD("MM",-1,GETDATE())),2)+"-01")
end_date(datetime) = DATEADD("D", -(DAY(GETDATE())),GETDATE())
Declare variable Counter(datetime)
Create a For loop container as follows :
Rest of the Data Flow Task should be there within For loop container, which will create output file. You can use the variable Counter in SQL to parameterize it
In fact, I figured out that all I wanted to use in my loop was the daypart of the date, so I created two extra int variables that contains:
1) the first day of the month (1)
2) the last day of the month (28,30,31)
I used those two variables at the For Loop Expressions and convert the index to string, so I could add it in the query. Possibly there will be a better way and it would be welcome.
How to get the difference in time by comparing with the previous value and getting the result .Say for example
There are
2017-01-01 13:00:00
2017-01-01 13:15:00
I need the difference as 15 minutes after finding the difference,How to do it?
Firstly, you'll have to use TalendDate.diffDate(column1,column2,"pattern") to get the time difference.
Then, if you want to compare current value with previous one (in the same column), you can set a sequence on your flow, it will help you identify which one is the previous value. Then, you'll just have to read twice your flow, and have an inner join between current sequence and current sequence -1 to get the currentDate and the previous Date.
First subjob :
YourFlow -> tMap -> tHashOutput
In tMap, add a new "sequence" column to your field and use Numeric.sequence("s1",1,1).
This way all lines will have an ID.
Then, read twice your Hash , and join flows on "sequence - 1"
tHashInput_1----|
|--tMap--->Output
tHashInput_2----|
Put the TalendDate.diffDate() method in the output, using the two Dates fields.
Here is an alternative :
Start defining starting talend job execution time, this way (here in a tJava, but you can also use tSetGlobalVar component) :
globalMap.put("startDate", TalendDate.getDate("CCYY/MM/DD hh:mm:ss"));
The following code is used later in the job inside a tJava :
String endDate = TalendDate.getDate("CCYY/MM/DD hh:mm:ss");
long executionTime = format.parse(endDate).getTime() - format.parse(((String)globalMap.get("startDate"))).getTime();
System.out.println("Execution Time : "+(executionTime/(60*60*1000))+" Hour(s) "+(executionTime/(60*1000)%60)+" Minute(s) "+(executionTime/1000%60)+" second(s).");
I have this list of date. i need to select only any future date from the list without csv. how can i do this?
Image of list of date
Option 1: HTML Link Parser - see Poll Example
Option 2: Combination of XPath Extractor and __Random() function
You can use ${__time(dd,)} to get current date. This date will be in String format. Lets store this to variable name date
Convert this date in number with java script function parseInt in BSF pre processor. Lets store this to variable name date_int
Now with random function ${__Random(${date_int},30,)}, you will get random number between current date and 30th of the month.
Use this random number for selecting index of your date list.
code is here
This is working here.
I have putted the myDay,myMonth,myYear variables as parameter.
I have a var of birth date in this format: 15APR1954
I need to set a new var that will present the current age - as if today's date is 01.01.2011
in order to use the var, how do I convert the date?
otherwise it gives me the following error :"The MDY function call does not have enough arguments".
data DAT2;set DAT1;
array BD{*} birth_date;
Curage=0;
do i=1 to dim(BD);
Curage+(MDY(01012011)-(birth_date));
end;
drop i;
run;
The best way to calculate age is to use the SAS built-in function yrdif().
data dat2;
set dat1;
curage = yrdif(birth_date, today(), 'AGE');
run;
The function today() returns today's date. If you want the age as of a certain date, e.g. 2011-01-01 like in your example, you can replace today() with '01JAN2011'd or with mdy(1, 1, 2011). (Note that your syntax for mdy() was incorrect.)
I'll also note that your array approach doesn't make a whole lot of sense; you're defining an array with only one element, so you might as well just perform operations on that value. Arrays are useful when you wish to perform identical operations to a group of 2 or more variables. For thorough information on array processing in SAS, see this section of the documentation.