I am working with tableau which is connected to database. The one column in it is date information which is in sting format has missing values hence it is very difficult to convert it to DATE type. Is there any way I can convert blanks with null or 0s? Following code did not work.
If you go to "Data Source" tab you can set the column type as Date and everything that is not a date should be set to NULL.
For example, if you import this CSV
"C1";"C2";
"31/01/2021";"A";
"24/12/2001";"B";
" ";"C";
"";"D";
"1/1/2001";"E";
And you set Date on column C1 as in the following image.
alla blank values should be set to NULL.
Related
I have data from two data sources i.e SQL and PostgreSQL. For every table want to replace the column having 'Null values' with MEAN if column type is integer and by 'Unknown' if column type is string.
I have tried using derived column but i am not sure how to pass on dynamic column values.
I created a pipeline with the 'LookUp' activity and 'ForEach' activity and calling a dataflow.
The migration is happening from SQL to Postgres so need to validate tables as well null values.
you have 2 cases here, the first one is replacing a null values in a string column with 'unknown' and the second case is replacing null values in an integer column with the mean of the values in the same column.
Main idea:
add a derived column , replace the null values in a string with unknown
fix the null values in an integer column,replace null with zeros so we will replace these zeros with the mean value when we calculate it by using a window activity.
Here is a quick demo that i built in ADF.
First, i created a dataset with 3 columns (name,height,address), height type is integer and address is a string like so:
ADF:
Derived Column activity:
modified address and height column as mentioned above.
Window activity:
in window activity, the idea is to replace the zeros with the mean value, to see the difference, i added a new column named it 'newHeight' just we can see the difference but you can override the original height column
in window settings -> window columns :
added a new column newHeight with the value :
case(height == 0 ,divide(sum(height),count(height)),toLong(height))
Output:
please read more about window transformation here:
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-window
I am using Tableau Prep to load PDFs to SQL server. The PDF contains several tables. All the columns except the one with NULL values is not created. If there are 12 columns in the PDF, I'm seeing only 11 in the Output. There's a column that has NULL values. Is there a way to create a column to have NULL values? The column is blank initially(eg, Jan, and values are being populated from Feb onwards) and then is being populated with float values. The PDF's need to be loaded daily. I created the column but it gives an error 'Error adding Text[columnname]. Expected different text.' Prep Version is 20.3.3.
Do I continue loading the Jan files and then is it possible to add the column and map it Feb onwards?
Is there another way to accomplish this?
Thank you in advance.
A little late answering but, you can add a null column as follows:
After connecting to your pdf table, add a Clean activity
Select data grid view and right click on a column
Select Create Calculated Field then Custom Calculation
Name your calculation and set the calculation value as NULL
Reset the data type as Number (decimal)
Add an output activity, saving output to Database table
I have two tables with similar columns
Apply append, all details were ok except for the date of the new table.
Old data dates are available, but the new one is missing and specified as "null"
I check their format, both are the same
Anyone once knows what is the issue.
Below screenshot for reference
enter image description here
It's look like new table has different column name for date.
I'm trying to convert eight digit yyyymmdd to date format with DAX function.
column = DATE(LEFT(TABLE[COLUMN],4),MID(TABLE[COLUMN],5,2),RIGHT(TABLE[COLUMN],2))
However, I've got an error because of the original column has some records with "00000000", so how can I make a default value with IF statement or are there any better solution?
Best regards
What I typically do is just make 2 distinct Power Query steps and this is handled automatically.
just make the yyyymmdd column a text column
make the text column from step 1 a date column (when prompted, be sure to select 'Add New Step')
replace errors with null
That's it. You can even Ctrl-Click to select multiple columns and combine them into the 1,2, and 3 steps with multiple columns.
Please check out "ferror" function IFERROR(value, value_if_error) for more information please visit Microsoft MSDN with link below
https://msdn.microsoft.com/en-us/library/ee634765.aspx
column = IFERROR( DATE(LEFT(TABLE[COLUMN],4),MID(TABLE[COLUMN],5,2),RIGHT(TABLE[COLUMN],2)), DATE(yyyy,mm,dd))
I'm using hive to create and try to load file content into the table.
There's a column type "Date" and the date format in the file is dd/mm/yy, for example: 01/12/2013
But when I trie to load the data into table from the file, the column values corresponding to the "Date" is always NULL, as if failed to load the date content.
I put the column content into a txt file and upload to the hdfs, so, the column may be:
id, name, birthdate
and corresponding value are:
1, "Joan", 04/05/1989
But the "04/05/1989" seems can't be read into the table, always null.
Please tell me if the format in my txt file is wrong or I need some specific grammar when loading date type data into Hive table.
Thanks!
Date data type format is YYYY-MM-DD. You need to format field accordingly.
More details on
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-date