I am very new to Azure Data Factory. I have created a simple Pipeline using the same source and target table. The pipeline is supposed to take the date column from the source table, apply an expression to the column date (datatype date as shown in the schema below) in the source table, and it is supposed to either load 1 if the date is within the last 7 days or 0 otherwise in the column last_7_days (as in schema).
The schema for both source and target tables look like this:
Now, I am facing a challenge to write an expression in the component DerivedColumn. I have managed to find out the date which is 7 days ago with the expression: .
In summary, the idea is to load last_7_days column in target Table with value '1' if date >= current date - interval 7 day and date <= current date like in SQL.I would be very grateful, if anyone could help me with any tips and suggestions. If you require further information, please let me know.
Just for more information: source/target table column date is static with 10 years of date from 2020 till 2030 in yyyy-mm-dd format. ETL should run everyday and only put value 1 to the last 7 days: column last_7_days looking back from current date. Other entries must recieve value 0.
You currently use the expression bellow:
case ( date == currentDate(),1, date >= subDays(currentDate(),7),1, date <subDays(currentDate(),7,0, date > currentDate(),0)
If we were you, we will also choose case() function to build the expression.
About you question in comment, I'm afraid no, there isn't an another elegant way for. To achieve our request, Data Flow expression can be complex. It may be comprised with many functions. case() function is the best one for you.
It's very clear and easy to understand.
Related
Power BI novice here. I have multiple reports which require date filtering by week. I can sometimes get the data to display with my Week column using dates from a column in the same table.
I thought building a Week column based on the date column would result in an easy to use visual. The week column is calculated by:
WeekYear = IF(
FORMAT(WEEKNUM(START.[Date],1)-1,"00"="00",
"Wk53-" & YEAR(START.[Date])-1,
"Wk" & FORMAT(WEEKNUM(START.[Date],1)-1,"00") & "-" & YEAR(START.[Date]))
This results in an x-axis displaying weeks in this format: Wk52-2019. If the underlying data of column STARTis in the proper datetime format, what could be the issue?
I noticed data on the visual which is not filtered for a date range display without issue. Trying to filter with DATESINPERIOD or other DAX date filters caused calculated measures to not display or break the model. I know a lot of references state having a separate calendar table is critical and I suppose I don't fully understand. Thanks in advance.
If you are trying to create the week in date format, then you can use the following calculation:
Week = Table[Start] - WEEKDAY(Table[Start],2)+1
This returns the Monday date of the week, if you want other days you can adjust the calculation accordingly.
If this is not what you are looking for, then you might have to clarify your requirements a bit more.
I'm searching for a way to have month selection and that will serve as startdate and enddate for the date filtering.
I'm building a monthly report on quicksight, I try to use the last 31 days but that give information of multiple months
I already create date picker for those parameters but didn't find any way to limit the value to be the complete month only.
Example : if select the 12 september I desire to get the September values only (from the 1er to the 31th)
Any advice is welcome
Thanks for your help
First, add a new date filter (if you want, e.g., today to be the default you'll need to add a dynamic default against a data set that returns today)
Add a new control (I found naming this to be difficult, perhaps you're better at picking names than I am)
Add a new calculated field that returns 1 if the truncDate of your date field and the truncDate of your parameter are equal, otherwise return 0
ifelse(
truncDate("MM", {date}) = truncDate("MM", ${InMonth}),
1,
0
)
Finally, add a filter that checks where your calculated field is 1 and apply it to all visuals
following 1 I started creating a Spreadsheet which reads data from BigQuery, but I'm having an issue handling parameters related to date values.
In the first sheet, I created 2 cells with 2 parameters, the start and the end of a date interval, with proper values. Both cells are formatted as "Date" value.
In the second sheet I configured BigQuery connector, for this example, I'm using a public dataset with dates. bigquery-public-data.utility_eu.date_greg
From the BigQuery connector wizard I added:
"STARTDATE" as "PARAMETERS!B1"
"ENDDATE" as "PARAMETERS!B2"
After this configuration, this is the resulting query:
SELECT
date,
date_str,
date_int
FROM `bigquery-public-data.utility_eu.date_greg`
WHERE date > DATE(#STARTDATE) AND date < DATE(#ENDDATE)
LIMIT 10
I'm getting an error directly from the editor with this message:
> Error BigQuery: No matching signature for function DATE for argument types: INT64. Supported signatures: DATE(TIMESTAMP, [STRING]); DATE(DATETIME); DATE(INT64, INT64, INT64) at [8:14]
As far as I can understand, the "date" cells are retrieved as a number, so the direct parse is not working. After a couple of tests, I understood the that given int value is the number I can obtain change cell format to "number".
If you convert cell value from DATE to NUMBER you get this value:
01/05/2019 -> 43.586
31/05/2019 -> 43.616
What is this number? It is not milliseconds, it increases by 1 every next day. In order to create the proper query that can parse this int, I need to understand what is this int (of course I can handle the cell as "text" and writing the timestamp value directly, but I would prefer to have the native date format so I can use the built-in calendar.
My consideration (with simple math) is that this number refers to a number of days since 30/12/1899, but it is very odd (also, every date BEFORE this days is always 0), so I'm asking you directly how to handle this value. Basing on my understanding of when the number counter starts (30/12/1899), I created this query which add the number retrieved from the cell:
SELECT *
FROM `bigquery-public-data.utility_eu.date_greg`
WHERE
date >= DATE_ADD(DATE("1899-12-30"), INTERVAL #DATAINIZIO DAY)
AND date <= DATE_ADD(DATE("1899-12-30"), INTERVAL #DATAFINE DAY)
It is working... but I think I'm doing a workaround that is not the proper way of doing this.
Also, is there any full documentation related to this BigQuery connection provided by Spreadsheet? Besides presentation in 1 I'm unable to find any specific documentation.
Spreadsheets (Google, Excel, ...) store the dates as days passed since a starting date with a fractional day representing time.
From here: "Excel stores dates and times as a number representing the number of days since 1900-Jan-0, plus a fractional portion of a 24 hour day: ddddd.tttttt . This is called a serial date, or serial date-time."
Now, you have to ways to filter by date on your Query:
In the query, you can use DATE_ADD to add your number of days (cell value) to the base date. (Carefull, DATE_ADD takes INT, and the date value is float so needs prior casting).
(preferred) on your spreadsheet you use TEXT(cell, "yyyy-mm-dd") so you can then use DATE() in the BigQuery query.
I use the second method as, though you need that extra cell (unless you directly store the date as YYYY-MM-DD; keeps the query cleaner than having a cast and date_add in there. Also would save you from the "1904 problem" explained in the link above.
What is this number? It is not milliseconds, it increases by 1 every next day.
This is so called serial number which represent number of days since "very beginning"
Google's Spreadsheet date calendar starts from 1900-01-01 - which is treated as a "very beginning"
In order to create the proper query that can parse this int, I need to understand what is this int
Armed with above info you can adjust you dates calculation to be in sync with what BigQuery expects
You mentioned that your fields are already in Date format, maybe you are doing an extra parsing in your query.
Try to do it without the DATE functions.
Also, I found this other doc, not merely related to connection, but might be helpful: Getting info from Spreadsheets with BigQuery.
I am new to Cognos and I am trying to add a filter to a column that only allows rows that are in between Yesterday at 4 AM and today at 3 AM. I have a working query in db2 but when I try to add it to the filter in Cognos I get a parsing error. Also, I found in the properties that the data type for the column I am trying to filter to be Unknown (Unsupported) type. I started off by creating two Data Item Expressions for each time frame I am trying to limit the data by. But I got a parsing error on the first one:
[Presentation Layer].[Cr dtime]=timestamp(current date) - 1 day + 4 hour
This works in my db2 local test database but doesn't even compile in Cognos. I also tried casting the column into a timestamp but that isn't working either. Any help is appreciated. I also tried using the _add_days function but I still get a parsing error. Also sampling the column I get values that appear to be timestamps as this string: 2016-01-02T11:11:45.000000000
Eventually if I get the two filters working I expect the original filter to be close to this syntax:
[Presentation Layer].[Cr dtime] is between [Yesterday 4AM] AND [Today 3AM]
Here is your filter:
[Presentation Layer].[Cr dtime] between
cast(_add_hours(_add_days(current_date,-1),4),timestamp)
and
cast(_add_hours(current_date,3),timestamp)
This works because current_date in Cognos does not have a time component. If you were to cast it directly to a timestamp type you would see the time part of the date as 12:00:00.000 AM, or midnight. Knowing this we can then simply add how much time after midnight we want, cast as a timestamp type and use this in the filter.
I have a decimal date field (TDDATR) that is in the YYYYMMDD format.
I would like to create a field that is TDDATR + 30 days but I am unable to.
Using 'Define Results Field' I have tried a few things;
Simply doing this;
TDDATR + 30 DAYS
But it returned this error: Labeled duration not used correctly.
I tried using the DIGITS and SUBSTR commands to create a field in the DDMMYYYY format and then +30 days but got the same error.
Same as above but in the DD/MM/YYYY format - same error.
Using DATE(TDDATR) but all I see is +'s in the field.
Using DATE( ) on the fields created in step 2 and 3 - still get +'s
I've ran out of ideas - any help would be greatly appreciated.
Query/400 lacks a lot of the features that an SQL based interface has.
I'd urge you to consider switching to Query Manager (STRQM) which is a fully SQL based product. You can even convert Query/400 queries to Query Manager queries with the RTVQMQRY command by having the ALWQRYDFN parm set to *YES.
The other option that IBM is pushing is Web Query. Again, fully SQL based and you can convert Query/400 queries into it.
Having said that, the problem is that FLD + 30 DAYS only works when FLD is a DATE data type. Query/400 includes a DATE() function to convert non-date types into date. But it's very limited in that it only works with character fields formatted according to your job defaults. Assuming you're in the US, it'd only work with a character value of '07/01/15'.
You could do a lot of manipulation in Query/400 and end up with a result field that meets DATE()'s requirements. But a better solution would be to create an SQL view over your table and have your numeric date converted into a date data type in the view.
You can find code examples that show how to convert a numeric YYYYMMDD to a actual date data type in the view. However, I'd recommend create a user defined function (UDF) that will do the conversion for you. That will make it much easier to use in the view and to reuse in other places.
If you'd like, there's an open source package called iDate, that includes all the code required for convert to/from date data types.
Download that, install/compile it and your SQL view becomes
select ... idate(TDDATR,'*CCYMD') as TD_DATE
from myfile
The use of days is as follow
Field Expression
CURDATE_30 days(current(date)) + 30
The solution to your problem is: given the field A dec(8,0)
Field Expression
YYYYMMDD_ date(substr(digits(a),5,2)||'/'||
substr(digits(a),7,2)||'/'||
substr(digits(a),3,2))
NEXT_MONTH DAYS(YYYYMMDD_) + 30
Remember to check the date format in your job description. In the example the format is MDY or MM/DD/YY.
More info here
Based on the information here, I created the below 2 fields;
TDDIGI DIGITS(TDDATR)
TDDAT1 SUBSTR(TDDIGI,7,2)||'/'||
SUBSTR(TDDIGI,5,2)||'/'||
SUBSTR(TDDIGI,3,2)
From here I was able to create a date field;
TDDAT2 DATE(TDDAT1)
Which allowed me to perform the necessary calculations.
The format of TDDAT1 is based on your job description which can be found by;
WRKJOB
Option 2
Page down
Date format..: X
Mine was *DMY, so TDDAT1 was formatted based on this.