I'm trying to do basic division and it always returns 0 as an answer.
let mathStuff = Double((stepCount / Level.expRequired())) * 100
print ("\(totalSteps) / \(Level.expRequired()) * 100 = \(mathStuff)")
My print returns
2117 / 2500 * 100 = 0.0
I've tried using NSDecimal instead of a Double and have also tried not using Double or NSDecimal and having it just do the math, which comes back as 0 instead of 0.0.
I'm really confused on what I'm doing wrong here, this seems like basic math and I'm not sure why I'm always given 0 as an answer.
Your problem probably lies here: 2117 /2500, both 2500 and 2117 are Ints.
If they were Double, then it would work: 2117.0 /2500.0 ==> produces non-zero division
Try casting those variables to double first, and you don't need to cast the result itself:
Double(stepCount) /Double(Level.expRequired()))*100
In fact, I believe only one needs to be cast:
Double(stepCount)/Level.expRequired())*100
I have a calculation in my t-sql code that I expect will show decimal result (with at least 2 digits after comma)
My fields that I am using are integer type, but the calculations result is decimal
I tried using CAST as float, but won't work
(COUNT(ct.[ClientFK]) / ehrprg.AnnualGoalClientsServed) AS [AnnualGoal]
I tried:
CAST((COUNT(ct.[ClientFK]) / ehrprg.AnnualGoalClientsServed) as float)
AS[AnnualGoal]
I expect to see at lest two digits after comma -
2/50 to be 0.04 while now I am getting 0
Any advice / help would be much appreciated
Try explicitly casting the denominator to float before the quotient is taken:
COUNT(ct.[ClientFK]) / CAST(ehrprg.AnnualGoalClientsServed AS float) AS [AnnualGoal]
In the above approach, because one of the two terms in the quotient is floating point, the other term (in this case, the count) should be promoted to float as well.
I want to select only two decimal places without rounding up.
$d = 123000.1264
'{0:f2}' -f $d
Result: 123000,13, but I need the result 123000,12
Any ideas to solve this problem?
Thank you in advance!
[Math]::Truncate(123000.1264 * 100) / 100
does it.
123000.1264 * 100 = 12300012.64
[Math]::Truncate(12300012.64) = 12300012
12300012 / 100 = 123000.12
You should use the [decimal] type for numbers when you need to preserve the accuracy of the fractional part, e.g.
$d = [decimal]123000.1264
and then [Math]::Truncate will use its decimal overload to give a decimal, and a decimal divided by an integer (or a double) will give a decimal result.
Of course, there is more than one way to interpret "up": it could mean increase in value (3 > -5) or increase in magnitude (|-5| > |3|). If you need the former, then use [Math]::Floor (which converts -1.1 -> -2.0) instead of [Math]::Truncate (which converts -1.1 -> 1.0).
The problem we have is reduced to the following two statements:
select convert(float, (convert(float,5741.61)/convert(float, 196.00)) * convert(float,14.00)) as unrounded, round(convert(float, (convert(float,5741.61)/convert(float, 196.00)) * convert(float,14.00)), 2) as roundedTo2dp
select convert(float, 410.115) as unrounded, ROUND( convert(float, 410.115), 2) as roundedTo2dp
The first statement uses floats to calculate a value of 410.115, and also that result with a round() to 2 decimal places. The rounded value comes out at 410.11.
The second statement uses the float value 410.115 and also rounds it to 2 decimal places. The rounded result comes out as 410.12.
Why is one rounding down and the other rounding up when the value being rounded it the same?
How can I get the first statement to round to 410.12?
EDIT: apologies for formatting -- stackoverflow isn't showing any formatting on this machine (very odd).
Decimals are better with precision than floats. If you changed up the float to be something like DECIMAL(18,2), you'll get what you are expecting and you don't need to call the round function anymore.
select convert(decimal(18,2), (convert(decimal(18,2),5741.61)/convert(decimal(18,2), 196.00)) * convert(decimal(18,2),14.00)) as unrounded, round(convert(decimal(18,2), (convert(decimal(18,2),5741.61)/convert(decimal(18,2), 196.00)) * convert(decimal(18,2),14.00)), 2) as roundedTo2dp
results in
unrounded roundedTo2dp
410.12 410.12
Link to the MSDN about decimals. http://msdn.microsoft.com/en-us/library/ms187746.aspx
Hope that helps...
The numbers are not equal:
SELECT CAST(convert(float, (convert(float,5741.61)/convert(float, 196.00)) * convert(float,14.00)) AS BINARY(8))
UNION ALL
SELECT CAST(convert(float, 410.115) AS BINARY(8)) as bin
----
0x4079A1D70A3D70A3
0x4079A1D70A3D70A4
'float' is an approximate number data type and hence not all values in the data type range can be represented exactly.
This is based on http://msdn.microsoft.com/en-us/library/ms173773.aspx.
I believe this is why there is rounding issue while using float values. You can never be 100% sure!
Ex.
Select round(convert(float, 1.5555), 2) --Gives 1.56
Select round(convert(float, 1.555), 2) --Gives 1.55!
With such a simple number there is difference in expected result while using float.
I am working in C#. I have decimal variable startFilter that contains value say 66.76. Now I want this decimal value to appear in the seach filter $0 to $100. But what I also want is, that the search filter starts from the first decimal value that comes in startFilter variable. So for instance in this case the search filter will start from $0 to $100 because the value in startFilter variable is 66.76, but in another case it can be $100 to $200 if the first value that comes in searchFilter is say $105.
Having said that, how should I round off the value in seachFilter to previous hundreds and the next hundreds. Like if the value is 66.76 it rounds off to 0 as floor and 100 as ceiling, so on and so forth.
Any idea how to do that in C#?
double value = ...
int rounded = ((int) Math.Round(value / 100.0)) * 100;
divide your original number by 100. get floor and celing values. Multiply each of them by 100.