I have an imported CSV file with string values.
In this file there are amounts, of which several lines equal 0,00
I want to create a TotalCA column by adding several fields in my table and convert it to a numeric value.
I use the toDecimal function and the values are all returned NULL and the created column is grayed..
I have done a lot of research and I can't find a solution. Can you help me?
Thank you
Lea
I made an example csv data if I understand you correctly:
Like you said, some rows are enriched with values greater than 0, and others contain "0.00" when it is a zero value. Actually, the row data contains different data type, int and decimal.
For these reason and as I tested, no matter toDecimal(), toFloat() or toDouble(), all of the functions don't work. I use Derived column expression to do the data conversion.
We can't keep these data and only can choose one type of them. If you choose the decimal or float, other rows data would be converted to '11.0', I think that also doesn't you want.
Source Projection: I preset the column type to double:
(Decimal can't keep '0.00', it only returns '0')
In one word, the only way is that use String data type to keep the data. And also use String data type to receive the data in sink dataset.
HTH.
Thank you all for your answers.
Here is my CSV file
If I go to the Source Projection module and change the type of my column LFC1_UM01S to decimal this is what I get:
Why are some values considered as NULL?
To decimal column
Related
I have a pandas dataframe with a "year" column. However some rows have a np.NaN value due to an outer merge. The data type of the column in pandas is therefore converted to float64 instead of integer (integer cannot store NaNs?). Next, I want to store the dataframe on a postGreSQL database. For this I use:
df.to_sql()
Everything works fine but my postGreSQL column is now type "double precision" and the np.NaN values are now [null]. This all makes sense since the input column type was float64 and not integer type.
I was wondering if there is a way to store the results in an integer type column with [nans].
Example Notebook
Result of Ami's answer:
(integer cannot store NaNs?)
No, they cannot. If you look at the postgresql numeric documentation, you can see that the number of bytes, and ranges, are completely specified, and integers cannot store this.
A common solution in this case is to decide, by convention, that some number is logically a nan. In your case, if it is year, you might choose a negative value (or just -1) as that. Before writing, you could use
df.year = df.year.fillna(-1).astype(int)
Alternatively, you can define another column as year_is_none.
Alternatively, you can store them as floats.
These solutions range from most efficient, to least efficient in terms of memory.
You should use it;
df.year = df.year.fillna(-1) OR 0
I want to visualise the below excel table in Tableau.
When adding this table to Tableau it shows Salary values as String and thus under Dimension Tab and not under Measure, thus cannot make proper graph from it.
How to convert this Salary range values to Int ?
As #Alexandru Porumb suggested, the best solution is to have a min_salary column and a max_salary column — unless you really have the actual salary available which is even better.
If you don’t want to revise the incoming data, you can get the same effect using the Split() function in a calculated field from Tableau to derive two integer fields from the original string field.
For example, you could define a calculated field called min_salary as INT(SPLIT([Salary], ‘-‘, 1)). Split() extracts part of a string based on a separator string. Int() converts the string to an integer.
You could simplify the way it sees the data and separate the salary column into Min and Max, thus you wouldn't have the hyphen that makes Tableau consider the entry as a string.
Simplistic idea, I know but it may help until a better solution will be provided.
Hope it helps
I have imported CSV file in SQLite Studio. I know want to change column data types and I have problems with decimal numbers. In column X I have numbers like:
636000
432.25
4.49
8.96
269906.81
6.26
Then I want to convert them to decimal(10,2) so I could for example get right maximum of the column X. When converting, I get this numbers:
636000
432.25
4.4900000000000002
8.9600000000000009
269906.81
6.2599999999999998
Why is that so? Thanks for help.
I have a .csv file and I can't read it on Octave. On R I just use the command below and everything is read alright:
myData <- read.csv("myData.csv", stringsAsFactors = FALSE)
However, when I go to Octave it doesn't do it properly with the below command:
myData = csvread('myData.csv',1,0);
When I open the file with Notepad, the data looks something like the below. Note there isn't a comma separating the last column name (i.e. Column3) from the first value (i.e. Value1) and the same thing happens with the last value of the first row (i.e. Value3) and the first value of the second row (i.e Value4)
Column1,Column2,Column3Value1,Value2,Value3Value4,Value5,Value6
The Column1 is meant for date values (with format yyyy-mm-dd hh:mm:ss), I don't know if that has anything to do with the problem.
Alex's answers already explains why csvread does not work for your case. That function only reads numeric data and returns an array. Since your fields are all strings, you need something that reads a csv file into a cell array.
That function is named csv2cell and is part of the io package.
As a separate note, if you plan to make operation with those dates, you may want to convert those dates as strings, into serial date numbers. This will allow you to put your dates in a numeric array which will allow for faster operations and reduced memory usage. Also, the financial package has many functions to deal with dates.
csvread only reads numeric data, so a date does not qualify unfortunately.
In Octave you might want to check out the dataframe package. In Matlab you would do readtable.
Otherwise there are also more primitive functions you can use like textscan.
I have built a query which contains UNION ALL, but the two parts of it
have not the same data type. I mean, i have to display one column but the
format of the two columns, from where i get the data have differences.
So, if i get an example :
select a,b
from c
union all
select d,b
from e
a and d are numbers, but they have different format. It means that a's length is 15
and b's length is 13. There are no digits after the floating point.
Using digits, varchar, integer and decimal didn't work.
I always get the message : Data conversion or data mapping error.
How can i convert these fields in the same format?
I've no DB2 experience but can't you just cast 'a' & 'd' to the same types. That are large enough to handle both formats, obviously.
I have used the cast function to convert the columns type into the same type(varchar with a large length).So i used union without problems. When i needed their original type, back again, i used the same cast function(this time i converted the values into float), and i got the result i wanted.