I have a pipeline as seen in the screenshot below, the Filter activity below looks for CSV files + starting with a pattern as I specified + filters out the file if it contains the current date.
Condition below:
#and(and(and(equals(item().type,'File'),not(equals(item().name,concat('BBN_TX1_',formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd'),'.csv')))),startswith(item().name,'BBN_TX1_')),endswith(item().name,'.csv'))
My requirement:
I want to Filter out/Ignore any files which are of the current date or greater. I was able to achieve my requirement for the current date but for files greater than the current date I need help, please can anyone advise me on what can be done?
Thanks for your input! Here is the condition I used and it works -
#and(and(and(equals(item().type,'File'),startswith(item().name,'BBN_PLN_')),endswith(item().name,'.xlsx')),less(formatDateTime(substring(item().name,8,10),'yyyy-MM-dd'),formatDateTime(convertTimeZone(utcNow(),'UTC','GMT Standard Time'),'yyyy-MM-dd')))
Related
I am creating a pipeline where the source is csv files and sink is SQL Server.
The date column in CSV file may have values like
12/31/2020
10162018
20201017
31/12/1982
1982/12/31
I do not find the function which checks the format of the date. How do I check the format and convert the above values to yyyy-MM-dd format.
The solution is given by HimanshuSinha-msft
Solved the issues using expression builder in Derived Column in Mapping Data Flow.
coalesce(toDate(Somedate,'MM/dd/yyyy'),toDate(Somedate,'yyyy/MM/dd'),toDate(Somedate,'dd/MM/yyyy'),toDate(Somedate,'MMddyyyy'),toDate(Somedate,'yyyyddMM'),toDate(Somedate,'MMddyyyy'),toDate(Somedate,'yyyyMMdd'))
This coalesce function answer will not actually solve the problem. It just gets rid of the errors. There are plenty of dates that are valid in multiple formats. For example: "2/1/2020" (mm/dd/yyyy) and "1/2/2020" (dd/mm/yyyy). The previous answer just gets rid of errors, but your analyses downstream will be very incorrect.
You need to do an aggregate analysis of which date format best fits the incoming stream, and the route the logic to the respective separate pipeline branches.
You can configure this in the Mapping tab of your copy activity. The datetime format can be specified, but it only supports one format type. If you have a mix of formats like in your example then it will not work.
One option would be to ingest the column into a staging table as a nvarchar. Then in another copy activity use a custom select statement to detect the column format and cast the date as needed. You should be able to do this using a CASE SQL statement in your SELECT from the staging table.
FYI: data type mapping
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping#data-type-mapping
I am a new to Tableau and having some issues with it.I am Using a parameter to filter month data across Sheets connected to different Data Sources.
I want to Filter Current Month Data Automatically when the dashboard starts.
Can anyone guide me on how to do it.
Thanks in Advance.
T
Introduced in version 10.3 is a feature called Latest date preset. The release notes say "Start with up-to-date data. Set filters to display latest date values automatically, as soon as your workbook opens." Create a filter that spans applies across the data sources and use this new setting. The setting is in the filter dialog.
See https://www.tableau.com/new-features/10.3#tab-analytics-2
The critical aspect of your question is that you are filtering multiple data sources using a parameter. Given that, the most straightforward approach is to define a boolean calculated field in each data source, called say Within_Date_Range.
Each of those calculated fields should compare the appropriate Date field in that data source to the selected parameter value and return true or false to indicate whether the current record should be included in the query. (Parameters are scoped to the workbook and visible from all data sources)
Place the [Within_Date_Range] field for data source X on the filter shelf for the worksheets that use that data source X. Check the true box of course.
Now when you change the parameter, all the worksheets filter accordingly.
Alternatively, you can abandon the use of a parameter. Define relationships between your data sources using the corresponding Date fields (under the data menu). Show a filter control for the date field for one of the worksheets and set the scope to "related data sources"
Hope this helps!
Create a calculated field Date_Filter as
lookup(min([Your_Date_Field]),0)
Drag this to 'Filter'; select 'relative date' option and choose last 1
month (because you wanted to show just the current month data by default).
Right click this filter and select "Apply to selected worksheet" and
choose sheets in which you want this filter to be applied on the
dashboard.Viola!
With this solution you can very well see current month's data by default and can go back to as many time periods you want.
1) I need to pass a date (billing) to the RFC, but I am not sure how to map using tmap. How to set it up (see screenshots).
2) I need to run this job daily (M-F) and I am not sure how to automate the date input
3) For the date input, I thought of using a joblet, but I can't find it in Talend. Most screenshots shows the Joblets in the same window as job designs and metadata, but I don't have it. Seen Joblet image.
As you might guessed, I am very new to Talend.
Use a tMap, and inside it use the function TalendDate.parseDate("yyyy-MM-dd", sap_data.date) in the expression field where you want the output. Also, note that the output type must be Date. The Date Pattern in the type definition (on the bottom of the tmap) is irrelevant.
Something like that:
I am relatively new to using Crystal Reports. I am pulling the correct information from my database however I need to only pull the information from the beginning of the month to the current date, i.e. 09/01/2013 - 09/23/2013. Could anyone give me a snipit of code on how that would look.
Much appreciated.
The best way is you need to filter the data before it is populated on reports i.e. filtering the query itself. The other option is to create new parameter for the selected field and passing the value on runtime.
http://www.codeproject.com/Articles/205398/Crystal-Report-Filtering-Using-Selection-Parameter
In below figure you can add the records selection on the required field and add the condition similarly
I'm trying to create a Crystal Report that reads data from an access table. I have a particular field in the access table called REPT_YYMM. This is basically a run data field, so each month when a program is run it populates the current year/month in the field before exporting to access, so there is a way to do month to month comparisons. For the report I'm working on I just want to display the data along with the current REPT_YYMM field when the data is refreshed on a monthly basis. Is there a way to do this in a formula. I'm not familiar enough with date type functions to know where to start.
Any guidance or suggestions are appreciated!
Your question isn't very clear but if i understood correct you just need to filter the data where field REPT_YYMM = YYMM.
Goto selection formulas, record and enter:
{table.REPT_YYMM} = cStr(currentdate,'yyMM');