I want to create a monthly package that executes a daily query at ODBC and writes an output file.
More specifically the query must be first executed for the first day of the previous month (e.g. '01/11/2018') then the next one ('02/11/2018') until the last day of the previous month ('30/11/2018').
The date variables are currently saved as Strings and I also want to have a string variable with Oracle date format to be inserted into the query. How should it be organised? Is there a way that I could use the string variables in the expressions?
Break it into parts as follows:
Declare variables to store previous month start and end date as follows:
start_date(datetime) = (DT_DATE)((DT_WSTR,4)YEAR(DATEADD("MM",-1,GETDATE()))+"-"+RIGHT("0"+(DT_WSTR,2)MONTH(DATEADD("MM",-1,GETDATE())),2)+"-01")
end_date(datetime) = DATEADD("D", -(DAY(GETDATE())),GETDATE())
Declare variable Counter(datetime)
Create a For loop container as follows :
Rest of the Data Flow Task should be there within For loop container, which will create output file. You can use the variable Counter in SQL to parameterize it
In fact, I figured out that all I wanted to use in my loop was the daypart of the date, so I created two extra int variables that contains:
1) the first day of the month (1)
2) the last day of the month (28,30,31)
I used those two variables at the For Loop Expressions and convert the index to string, so I could add it in the query. Possibly there will be a better way and it would be welcome.
Related
How can I find the date of the last Saturday from a current date in ADF (set as a variable) ?
For example in SQL I can run this query and it will return the last Saturday:
but I don't want to create a lookup activity to connect to the database and run that query. I want to define it as a variable.
Is it possible to create a variable that always returns the last Saturday?
Kind regards
You can use below expression to get the last saturday always in set variable activity.
#FormatDateTime(addDays(subtractFromTime(utcnow(),dayOfWeek(utcnow()),'Day'),-1),'yyyy/MM/dd')
Set variable:
Result:
I am very new to Azure Data Factory. I have created a simple Pipeline using the same source and target table. The pipeline is supposed to take the date column from the source table, apply an expression to the column date (datatype date as shown in the schema below) in the source table, and it is supposed to either load 1 if the date is within the last 7 days or 0 otherwise in the column last_7_days (as in schema).
The schema for both source and target tables look like this:
Now, I am facing a challenge to write an expression in the component DerivedColumn. I have managed to find out the date which is 7 days ago with the expression: .
In summary, the idea is to load last_7_days column in target Table with value '1' if date >= current date - interval 7 day and date <= current date like in SQL.I would be very grateful, if anyone could help me with any tips and suggestions. If you require further information, please let me know.
Just for more information: source/target table column date is static with 10 years of date from 2020 till 2030 in yyyy-mm-dd format. ETL should run everyday and only put value 1 to the last 7 days: column last_7_days looking back from current date. Other entries must recieve value 0.
You currently use the expression bellow:
case ( date == currentDate(),1, date >= subDays(currentDate(),7),1, date <subDays(currentDate(),7,0, date > currentDate(),0)
If we were you, we will also choose case() function to build the expression.
About you question in comment, I'm afraid no, there isn't an another elegant way for. To achieve our request, Data Flow expression can be complex. It may be comprised with many functions. case() function is the best one for you.
It's very clear and easy to understand.
I'm attempting to make the parameters for a Crystal Report more user-friendly for a client, who has requested that they be able to have the default values for a Start and End date parameter be the first and last day of the previous month.
I know how to use either a formula in CR or a stored procedure to produce these values, but I want to know if a variable can be used in the 'Default Value' setting for a parameter, or if it only allows for static entries. Does anyone know? Right now the user can set the date parameters to null and the stored procedure generates the data for the previous month on its own, but I thought it'd be nice if the date parameters actually displayed the dates that were being used as defaults. Thanks in advance!
You can do it, Try below process:
Create a parameter ?date with String datatype and take static and write two default strings as below:
First day of previous month
Last day of Previous month
Now go to record selection formula and write below code:
if ({?date}="First day of previous month") then
table.date=DateSerial(year(currentdate),Month(Currentdate)-1,1)
else if ({?date}="Last day of previous month")
then
table.date=Cdate(DateAdd("d",-1,DateSerial(year(currentdate),Month(Currentdate),1)))
I have this list of date. i need to select only any future date from the list without csv. how can i do this?
Image of list of date
Option 1: HTML Link Parser - see Poll Example
Option 2: Combination of XPath Extractor and __Random() function
You can use ${__time(dd,)} to get current date. This date will be in String format. Lets store this to variable name date
Convert this date in number with java script function parseInt in BSF pre processor. Lets store this to variable name date_int
Now with random function ${__Random(${date_int},30,)}, you will get random number between current date and 30th of the month.
Use this random number for selecting index of your date list.
code is here
This is working here.
I have putted the myDay,myMonth,myYear variables as parameter.
I have a data set in which month and year are in one variable and come in the form 200801 which equates to 2008, January. How can I create a SAS date from this integer?
I would like something in the form of Jan 2008 - anything so that SAS recognizes it as a date, as I then need to subtract this value from service date to find out how much time has elapsed since enrollment into the dataset until date of service.
Please also keep in mind that this is a variable, and I have thousands of observations. So I also need the data step/ function to do this for the entire variable.
Any help is appreciated!
You need to put it to a character variable, then input back to numeric. You can do that pretty easily.
date_var = input(put(date_var_orig,6.)||'01',yymmdd8.);
You can also do it this way:
date_var = mdy(mod(date_var_orig,100),1,floor(date_var_orig/100));
Both assume you want the day to equal 1; make a choice there if you want something else (like end of month or middle of month).