The problem we have is reduced to the following two statements:
select convert(float, (convert(float,5741.61)/convert(float, 196.00)) * convert(float,14.00)) as unrounded, round(convert(float, (convert(float,5741.61)/convert(float, 196.00)) * convert(float,14.00)), 2) as roundedTo2dp
select convert(float, 410.115) as unrounded, ROUND( convert(float, 410.115), 2) as roundedTo2dp
The first statement uses floats to calculate a value of 410.115, and also that result with a round() to 2 decimal places. The rounded value comes out at 410.11.
The second statement uses the float value 410.115 and also rounds it to 2 decimal places. The rounded result comes out as 410.12.
Why is one rounding down and the other rounding up when the value being rounded it the same?
How can I get the first statement to round to 410.12?
EDIT: apologies for formatting -- stackoverflow isn't showing any formatting on this machine (very odd).
Decimals are better with precision than floats. If you changed up the float to be something like DECIMAL(18,2), you'll get what you are expecting and you don't need to call the round function anymore.
select convert(decimal(18,2), (convert(decimal(18,2),5741.61)/convert(decimal(18,2), 196.00)) * convert(decimal(18,2),14.00)) as unrounded, round(convert(decimal(18,2), (convert(decimal(18,2),5741.61)/convert(decimal(18,2), 196.00)) * convert(decimal(18,2),14.00)), 2) as roundedTo2dp
results in
unrounded roundedTo2dp
410.12 410.12
Link to the MSDN about decimals. http://msdn.microsoft.com/en-us/library/ms187746.aspx
Hope that helps...
The numbers are not equal:
SELECT CAST(convert(float, (convert(float,5741.61)/convert(float, 196.00)) * convert(float,14.00)) AS BINARY(8))
UNION ALL
SELECT CAST(convert(float, 410.115) AS BINARY(8)) as bin
----
0x4079A1D70A3D70A3
0x4079A1D70A3D70A4
'float' is an approximate number data type and hence not all values in the data type range can be represented exactly.
This is based on http://msdn.microsoft.com/en-us/library/ms173773.aspx.
I believe this is why there is rounding issue while using float values. You can never be 100% sure!
Ex.
Select round(convert(float, 1.5555), 2) --Gives 1.56
Select round(convert(float, 1.555), 2) --Gives 1.55!
With such a simple number there is difference in expected result while using float.
Related
I have performed ST_Area on a shapefile but the resulting numbers are VERY long. Need to reduce them to two decimals. This is the code so far:
SELECT mtn_name, ST_Area(geom) / 1000000 AS km2 FROM mountain ORDER BY 2 DESC;
This is what I get:
mtn_name KM2
character varying double precision
1 Monte del Pueblo de Jerez del Marquesado 6.9435657067528e-9
2 Monte de La Peza 6.113288075418532e-9
I tried ROUND() but it brings KM to 0.00
Since it is not simply possible to round a decimal value (Decimal Precision problem) you will not get a double value which is exactly 6.94e-9. It would be something like 6.9400000001e-9 after rounding.
You can do:
demos:db<>fiddle
If the exponent is always the same (in your example it is always e-9) you can round with a fixed value. With double values, this results in the problem described above.
SELECT
round(area * 10e8 * 100) / 100 / 10e8
FROM area_result
To avoid these precision problems, you can use numeric type
SELECT
round(area * 10e8 * 100)::numeric / 100 / 10e8
FROM area_result
If you have different exponents, you have to calculate the multiplicator first. According to this solution you can do:
For double output
SELECT
round(area / mul * 100) * mul / 100
FROM (
SELECT
area,
pow(10, floor(log10(area))) as mul
FROM area_result
) s
For numeric output
SELECT
round((area / mul) * 100)::numeric * mul / 100
FROM (
SELECT
area,
pow(10, floor(log10(area)))::numeric as mul
FROM area_result
) s
However, your exponential result is just a view of the values. This can vary from database tool to database tool. Internally they are not stored as the view. So, if you fetch these values, you will, in fact, get a value like 0.00000000694 and not 6.94e-9, which is just a textual representation.
If you want to ensure to get exactly this textual representation, you can use number formatting to_char() for this, which, of course, returns a type text, not a number anymore:
SELECT
to_char(area, '9.99EEEE')
FROM area_result
i need numeric mask like this:
100
101.1
102.123
Take maksimum given decimal places but if last digit is 0 trim it.
Something like:
#n-12_`2 but trim right 0 and .
Ex:
x = 102.1230057::double precision
select rtrim(rtrim(round(x::numeric, 2)::text, '0'), '.')::numeric
If you don't need right aligned numbers, you can use #S10 (or the length you want).
#P<#.##P style pictures might help.
I have a calculation in my t-sql code that I expect will show decimal result (with at least 2 digits after comma)
My fields that I am using are integer type, but the calculations result is decimal
I tried using CAST as float, but won't work
(COUNT(ct.[ClientFK]) / ehrprg.AnnualGoalClientsServed) AS [AnnualGoal]
I tried:
CAST((COUNT(ct.[ClientFK]) / ehrprg.AnnualGoalClientsServed) as float)
AS[AnnualGoal]
I expect to see at lest two digits after comma -
2/50 to be 0.04 while now I am getting 0
Any advice / help would be much appreciated
Try explicitly casting the denominator to float before the quotient is taken:
COUNT(ct.[ClientFK]) / CAST(ehrprg.AnnualGoalClientsServed AS float) AS [AnnualGoal]
In the above approach, because one of the two terms in the quotient is floating point, the other term (in this case, the count) should be promoted to float as well.
I am trying to reduce the decimal places of my number to two. Unfortunately is not possible. For this reason I added some of my code, maybe you will see the mistake...
Update [dbo].[company$Line] SET
Amount = ROUND((SELECT RAND(1) * Amount),2),
...
SELECT * FROM [dbo].[company$Line]
Amount in db which I want to change:
0.00000000000000000000
1914.65000000000010000000
376.81999999999999000000
289.23000000000002000000
Result I get after executing the code:
0.00000000000000000000
1366.28000000000000000000
268.89999999999998000000
206.38999999999999000000
Result I want to get (or something like this):
0.00000000000000000000 or 0.00
1366.30000000000000000000 or 1366.30
268.99000000000000000000 or 268.99
206.49000000000000000000 or 206.49
RAND() returns float.
According to data type precedence the result of multiplying decimal and float is float, try:
ROUND(CAST(RAND(1) as decimal(28,12)) * Amount, 2)
this should do the trick.
I am using Postgresql to generate a simple quotient of two fields " a / b " = -3 / 300 = -.01 or -1.00%. Instead it displays as zero. I've tried many variations of to_char with no success. How could I get "select -3/300 as quotient" to display the quotient in xxx.xxx% format? Thanks
To display with the percentage sign, you can use the following format. Don't forget to first multiply by 100.0 so 1) is it indeed a percentage and 2) as noted by #dhke the computation is done with floats, not integers.
select to_char(100.0*-3/300,'999D99%') a;
a
----------
-1.00%
(1 row)