I have setup a copy pipeline, which is working fine.
Now I have tried to add a filter with last modified, but it is not working.
my filter
enter image description here
a sample file it should detect
enter image description here
it doesn't detect any files
enter image description here
this is the pipeline without the filter
enter image description here
I got similar kind issue with last modified filter using copy activity.
Make sure your date is of type UTC as like below, if not convert it into UTC like below :
The above will only read the files if they are in that time period of UTC.
Add this dynamic content filter by modified as per your required date.
#convertToUtc('01/10/23', 'Pacific Standard Time')
Try this alterative approach using get metadata.Follow this SO thread by Rakesh Govindula .
Hi you can use Azure Dynamic Functions like Addmintues so if your file was written two minutes ago it can detect by searching in Minutes and adding utcnow and -2 represent last two minutes. You can also add Days and Hours as well.
Let me know if this help otherwise same can be done with Get metadata Activity in azure
Related
i want to pass dynamic content to a parameter in "Execute SSIS Package" to pass
If sysdate= monday then get last 3 days
Can someone help to build the content using functions or expression
Thanks
There are 2 ways you can achieve this.
Use an IF activity to first check if today is a Monday. You can use an expression like #equals(dayOfWeek(utcnow()),1) for a Monday check.
If true, get utcNow() - 3 days using a dynamic content of #adddays(utcnow('yyyy-MM-dd'),-3). You may need to change the format to what you need.
Easier and more efficient option, just schedule your pipeline to run every Monday. Then there is no need to check anything.
I've pulled in data in Qlik Sense (cloud) from the Google Analytics connector. I'm trying to convert the standard date [ga_dateHourMinute] field to a more eligible date format.
The current format is YearMonthDayhHourMinute, for instance: 201810250004, I would like to convert this to the standard date format 2018-10-25 00:04:00. How do I do this? Answers concerning methods in the data load script or master formula's or variables are all welcome.
Click to see screenshot: left = current date format and right = desired date format
This is one approach:
DateAlter:
LOAD * INLINE [
Date
201810250004
];
DateAlter2:
LOAD
Date(Date#([Date], 'YYYYMMDDhhmm') ,'YYYY-MM-DD hh:mm') AS [MyDateField]
Resident
DateAlter;
If you haven't played around with the data manager, it will allow you to edit fields and handle dates in a fairly easy manner:
Qlik Sense Data Manager
I ended up creating a text fields that combined the different digits from the one field into a new order, then pushed the data format into a date/time. Quite a bit of work, but fairly solid.
Using Tableau, I need to create a visualisation that shows me the total volume of started and ended work items per day.
Typical sql data consists of columns for each:
Reference (Unique Work Item Reference)
Start Date (Date Work Item Started)
End Date (Date Work Item Finished)
For example, if on the 6th of April 2018, 55 work items were started and 5 items were finished. Each represented date should show in its own line over time.
The issue I'm getting at the moment, when I have start date as a continuous column, it calculates the number of work items started for the end date and not the number of items finished for the end date.
Any help guidance would be greatly appreciated.
Tableau Public Link
This is a simple problem if you correct the shape of your data. You can achieve this by using the pivot functionality in Tableau or Custom SQL(If your data source does not support pivot).
Pivot the date columns
2. Rename the pivoted fields
Now build the view as below.(null values are due to tasks that are not yet closed, you can just filter it out)
I assume that the data is of this form Data structure
With this format, you could achieve something like this Gantt Chart
If you notice, I have created a calculated field called "Duration" which is in size shelf.
the calculated field reads
If [Status]="Finished" then
[End Date]-[Start Date] ELSE
TODAY()-[Start Date]
end
Hope this helps! Let me know if this is not clear enough
Even if the Status column is not there in the data source, it can be added using the following code
if ISNULL([End Date]) then
"WIP" else "Finished" end
Also, is this how you want to see the information?
Click here
Could you please have a look at it and let me know your thoughts?
https://us-east-1.online.tableau.com/t/tableaumanoraj/views/VolumeExample/Dashboard1?iframeSizedToWindow=true&:embed=y&:showAppBanner=false&:display_count=no&:showVizHome=no
Context: I am using Microsoft Dynamics (CRM) and Eloqua to send email campaigns. I have a date field in CRM that I want to check against in Eloqua for a specific campaign. This campaign needs to check to see if the date field is <= today's date + 90 days. I am using the campaign UI in Eloqua, not doing anything programmatically at this point.
I have tried using the Compare Custom Object Fields decision in Eloqua by finding the date field, setting the comparator to dynamically on or before, and I want to make the compared value Today + 90 days. I'm not sure how to accomplish this in this type of Decision object because the only options I have to compare the date field to are Yesterday, Today, or Tomorrow. See image below:
I have also tried to use the Compare Date Decision object, but there is no dynamic comparison, just hard-coded date options.
The last thing I tried was a Wait step, but that only waits a hard-coded number of days rather than checking dynamically.
Has anyone run into this issue or know of a solution to this problem?
We were able to find an Eloqua Date App to download that adds a Date Decision step to the program builder which allowed us more flexibility with comparing dates in a custom range.
I am using Tableau 9.3 to do a preliminary data analysis on one of my log file, the log file is like below:
"199.72.81.55",01/Jul/1995:00:00:01,/history/apollo/,200,6245,Sat
As you can see, there is a datetime for timestamp
In Tableau, initially it is recognized as a string like below:
That's fine, I want to make the field into datetime, and Tableau seems failed on it:
Why? How do I fix it?
Thank you very much.
UPDATED: after applying the formula suggested below, Tableau still cannot recognize the timestamp, here is the screenshot:
UPDATED AGAIN: after tested by nick, it is confirmed his first script is correct and working on his Tableau, why it fails on mine, I don't know, you are welcome to share any clue please, thank you.
Tableau implicit conversions are limited to more standard formats. You can still create a DATETIME field from your timestamp string using a calculated field with the following formula:
DATEPARSE('dd/MMM/yyyy:HH:mm:ss',[timestamp])
Using the above will transform a string like 01/Jul/1995:00:00:01 to a date and time of 7/1/1995 12:00:01 AM
Output using example data:
Sometimes the "date parse" function in Tableau doesn't quite do the job.
When this happens it is worth testing manual string manipulation with your timestamp field to put it into ISO-standard format and only then trying to convert it into a date. ISO format is yyyy-mm-dd hh:mm:ss (eg 2012-02-28 13:04:30). It is common to find that the original string has spurious characters or spaces that throw dateparse. But these are usually easy to manipulate away with suitable text manipulations. This can sometimes be longwinded, but it always works.
It turned out to be the region setting issue, it works after I switch it to USA