I have a JSON file which contains a boolean and what I wanted to know was whether it's actually possible to convert these TRUE and FALSE's to 1's and 0's within the pipeline?
Any help or information will be greatly appreciated.
We can convert these TRUE and FALSE's to 1's and 0's via Direved Column in data flow.
Related
I have an imported CSV file with string values.
In this file there are amounts, of which several lines equal 0,00
I want to create a TotalCA column by adding several fields in my table and convert it to a numeric value.
I use the toDecimal function and the values are all returned NULL and the created column is grayed..
I have done a lot of research and I can't find a solution. Can you help me?
Thank you
Lea
I made an example csv data if I understand you correctly:
Like you said, some rows are enriched with values greater than 0, and others contain "0.00" when it is a zero value. Actually, the row data contains different data type, int and decimal.
For these reason and as I tested, no matter toDecimal(), toFloat() or toDouble(), all of the functions don't work. I use Derived column expression to do the data conversion.
We can't keep these data and only can choose one type of them. If you choose the decimal or float, other rows data would be converted to '11.0', I think that also doesn't you want.
Source Projection: I preset the column type to double:
(Decimal can't keep '0.00', it only returns '0')
In one word, the only way is that use String data type to keep the data. And also use String data type to receive the data in sink dataset.
HTH.
Thank you all for your answers.
Here is my CSV file
If I go to the Source Projection module and change the type of my column LFC1_UM01S to decimal this is what I get:
Why are some values considered as NULL?
To decimal column
Working info
I have two different sources of data set so I have created a dataflow in data factory in which for first data(A) set I am doing some transformation and loading into sink,in another data set(B) similarly am performing some transformation and loading into another sink.
Issue
Now I have some requirements in which there is date column DT_COLUMN_A(11-04-2020 01:17:40) in first data set(A)which needs to be compared with a date column DT_COLUMN_B(01-01-2020 16:32:00) in second data set (B) and store the compared output as a column in second dataset(B).
So I need the min and max(date range) of date column from dataset A ,apply it to min and max of date column to dataset B and find the dates which are matching in A and B and store it as YES if not matching NO.
Code approach thought
Logic needed:
if(min(DT_COLUMN_A) and max(DT_COLUMN_A) == min(DT_COLUMN_B) and max(DT_COLUMN_B) then YES else No.
I am trying to achieve this in ADF data flow but unable to do it.
To get MIN and MAX of a dataset in ADF, you will need the Aggregate transformation. Create new columns called MinA, MinB, MaxA, MaxB from each of the relative streams in your data flow using Aggregate. Set the aggregate function to MIN and MAX appropriately for each. Then, you'll be able to set an iif() expression afterward, or use a Filter or Conditional Split transformation that uses those stored min & max values.
I managed to get something similar to work using using a mapLoop() expression to first build an array of dates in a derived column transformation followed by a flatten transformation
https://stackoverflow.com/a/73453351/12592985
I am trying to filter a table visualization of all of my data by looking to see if a Study Number contains Activity A. If a Study Number contains Activity A then I want to filter for all rows containing those Study Numbers even if the Activity is not A. See mock data below. In my real data set I have ~55,000 rows.
I have created a calculated to return Study Numbers if Activity= A but I am not sure where to go from there. Thanks for any help.
If(UniqueConcatenate([Activity]) OVER ([Study Number])~="A","Y","N")
Will give you this resulting column that you can then filter on (or you can use the formula as a Data Limiting Expression:
I want to visualise the below excel table in Tableau.
When adding this table to Tableau it shows Salary values as String and thus under Dimension Tab and not under Measure, thus cannot make proper graph from it.
How to convert this Salary range values to Int ?
As #Alexandru Porumb suggested, the best solution is to have a min_salary column and a max_salary column — unless you really have the actual salary available which is even better.
If you don’t want to revise the incoming data, you can get the same effect using the Split() function in a calculated field from Tableau to derive two integer fields from the original string field.
For example, you could define a calculated field called min_salary as INT(SPLIT([Salary], ‘-‘, 1)). Split() extracts part of a string based on a separator string. Int() converts the string to an integer.
You could simplify the way it sees the data and separate the salary column into Min and Max, thus you wouldn't have the hyphen that makes Tableau consider the entry as a string.
Simplistic idea, I know but it may help until a better solution will be provided.
Hope it helps
I have a dataset containing a mixture of numerical and nominal attributes. I want to convert all the nominal attributes in the dataset to numeric so that I can apply the SVM classifier kernel(PolyKernel and RBFKernel) that only works with numeric attributes. Any help would be greatly appreciated. FYI I've already tried NominalToBinary Filter(Its not really what I want)
One thing you could do is convert all of the label names for the attribute using RenameNominalValues. Please note that all of these new labels would need to be numeric, so you might need to change them as below:
Once this is done, then you could save the .ARFF File, and change the entry in your attributes list from:
#attribute a0 {false,true,maybe}
to
#attribute a0 numeric
Once saved, reload the document and hopefully all will load okay.
Alternatively, you could try your favorite Spreadsheet Application if conversion of your data back to ARFF would not be an issue.
Hope this Helps!
There is no direct filter to convert nominal data to numeric data. If your nominal attribute has 2 values (SEX: MALE, FEMALE) you can easily apply the filter under unsupervised filters "nominal to binary".
But if you have more than 2 variations for the attribute, you cannot use "nominal to binary". So you need to use a filter called "Rename Nominal vales". There you can convert nominal value to numeric value.
Eg: if your dataset has an attribute called " region" and it has "INNER_CITY, TOWN, SUBURBAN, SUBURBAN" for values, you can easily convert those nominal values using the "Rename Nominal Value" filter.
There is a value replacement form, you have to do only add values like below. INNER_CITY:0, TOWN:1, SUBURBAN:2, SUBURBAN:3
you can see your results.
The NominalToNumeric filter (package: weka.filters.unsupervised.attribute) that is part of ADAMS allows you to do exactly that. You can either use the internal representation (i.e., order of labels starting at 0), or, if there is a numeric part in the label that can be turned into a number, use regular expressions to convert these sub-strings.
ADAMS also offers the Weka Investigator, a more powerful tool than the Weka Explorer. Just download the adams-ml-app-snapshot snapshot to get access to this filter.
Yes, you can convert nominal data to numeric in weka:
Example:
select: filters.unsupervised.attribute.OrdinalToNumeric