I have an AlterRow transformation that marks each row with the appropriate CRUD operation in an ADFv2 data flow. I don't see any output variables on this activity that will give me the total inserts, updates, etc. I do, however, see methods in the expression syntax to tell me if a particular row is an IsInsert(), IsUpdate(), etc.
Would the correct way to get counts be to
Add another output from the AlterRow transformation
Add derived column that uses the expression syntax IsInsert(), IsUpdate() to set operation type (I, U, D)
Add an aggregate to group by this column to get total counts for each operation
When creating the aggregate, I don't see any metadata that would allow me to group by the CRUD operation type so I assume I would have to create this myself, but it seems like it should already be there since that's the purpose of the AlterRow transformation. Am I working too hard to get these counts?
Add an aggregate after your AlterRow with no group-by and use these formulas:
Related
I'm using Grafana v9.3.2.2 on Azure Grafana
I have a line chart with labels of an ID. I also have an SQL table in which the IDs are mapped to simple strings. I want to alias the IDs in the label to the strings from the SQL
I am trying to look for a transformation to do the conversion.
There is a transformation called “rename by regex”, but that will require me to hardcode for each case. Is there something similar with which I don't have to hardcode for each case.
There is something similar for variables - https://grafana.com/blog/2019/07/17/ask-us-anything-how-to-alias-dashboard-variables-in-grafana-in-sql/. But I don't see anything for transformations.
Use 2 queries in the panel - one for data with IDs and seconds one for mapping ID to string. Then add transformation Outer join and use that field ID to join queries results into one result.
You may need to use also Organize fields transformation to rename, hide unwanted fields, so only right fields will be used in the label at the end.
I'm using Data Stage version 11.7 and encountered the error message below from the Lookup stage while compiling the job:
"The supplied expression was empty."
In the Lookup Stage, there are two links from two transformers and there is no common key column between the two datasets.
I googled how to merge or combine the two datasets from two transformers without a common key column. However, I couldn't find a proper way to solve this issue or the way implementing my job in DataStage.
Empty Expression
Is there anyone who knows how to solve this problem? If so, please let me know which stage is good for my job or how to solve the error. I would appreciate it.
If you need to join n:m, add a dummy column to each input link and fill it with a constant value like 1. Then join over that column. Decide if mutiple matches result in mutiple output rows or if the first match 'wins' - which would be like a random n:1 then, since every row matches when joining over a const value.
But if you need to join specific rows, it indicates that there actually is a common key but it's not obvious or visible. Either Transform the sources so that they get a common key or use an anchor table that provides the relations. Join that to the first source, then join the second source.
I have a Set Variable activity which uses the logic:
#range(int(pipeline().parameters.start),int(pipeline().parameters.end))
It is wierd that I cant find any logic in documents where I can mention a step so that I can generate few numbers as shown below
1,3,5,7,9,...
Is there work around to it, other than introducin a new parameter that is equal to step and generate next number using logic last = last+step.
It is possible to do this using the Filter activity and the range function. Use the range function to generate all numbers and then the Filter condition with mod to get odd numbers, ie
Property
Value
Items
#range(1,10)
Condition
#equals(mod(item(),2),1)
A screenprint of the results:
The other way to do it would be just use a Lookup activity and query a numbers table.
I agree with you that it's a shame range does not have a step argument, and that generally the ADF expression language isn't a bit more fully featured.
I would like to filter out all values that does not end with ":Q_TT".
I have tried with the following activities
My bronze data has a column named "pnt_name". One of the rows in the table ends with ":Q_TT" so I would expect that the Exists activity would pass that row through.
Custom expression in Exists1
endsWith(':Q_TT', pnt_name)
In the future I would like the SourceData dataset to hold the filter values.
Thanks very much
You should be using filter activity instead of exists activity for your case.
The pipeline(where you can see the 2 test cases of A and B:Q_TT:
Here is the preview of the filter activity, you should use an expression of endsWith(pnt_name, ":Q_TT") too. You can see A is removed and B:Q_TT is kept.
I am getting my data from denormalized table, where I keep names and actions (apart from other things). I want to create a calculated field that will return sum of workgroup names but only when there are more than five actions present in DB for given workgroup.
Here's how I have done it when I wanted to check if certain action has been registered for workgroup:
WINDOW_SUM(COUNTD(IF [action] = "ADD" THEN [workgroup_name] END))
When I try to do similar thing with count, I am getting "Cannot mix aggregate and non-aggregate arguments":
WINDOW_SUM(COUNTD(IF COUNT([Number of Records]) > 5 THEN [workgroup_name] END))
I know that there's problem with the IF clause, but don't know how to fix it.
How to change the IF to be valid? Maybe there's an easier way to do it, that I am missing?
EDIT:
(after Inox's response)
I know that my problem is mixing aggregate with non-aggregate fields. I can't use filter to do it, because I want to use it later as a part of more complicated view - filtering would destroy the whole idea.
No, the problem is to mix aggregated arguments (e.g., sum, count) with non aggregate ones (e.g., any field directly). And that's what you're doing mixing COUNT([Number of Records]) with [workgroup_name]
If your goal is to know how many workgroup_name (unique) has more than 5 records (seems like that by the idea of your code), I think it's easier to filter then count.
So first you drag workgroup_name to Filter, go to tab conditions, select By field, Number of Records, Count, >, 5
This way you'll filter only the workgroup_name that has more than 5 records.
Now you can go with a simple COUNTD(workgroup_name)
EDIT: After clarification
Okay, than you need to add a marker that is fixed in your database. So table calculations won't help you.
By definition table calculation depends on the fields that are on the worksheet (and how you decide to use those fields to partition or address), and it's only calculated AFTER being called in a sheet. That way, each time you call the function it will recalculate, and for some analysis you may want to do, the fields you need to make the table calculation correct won't be there.
Same thing applies to aggregations (counts, sums,...), the aggregation depends, well, on the level of aggregation you have.
In this case it's better that you manipulate your data prior to connecting it to Tableau. I don't see a direct way (a single calculated field that would solve your problem). What can be done is to generate a db from Tableau (with the aggregation of number of records for each workgroup_name) then export it to csv or mdb and then reconnect it to Tableau. But if you can manipulate your database outside Tableau, it's usually a better solution