Why does the select statement below return two different values ?
declare #tempDec decimal
set #tempDec = 1.0 / (1.0 + 1.0)
select #tempDec, 1.0 / (1.0 + 1.0)
That's fine for literals like 1.0, but if you're pulling the data from table columns, you need to cast/convert the first evaluated number in your equation:
convert(decimal, [col1]) / ([col2] + [col3])
-or-
convert(decimal(15, 2), [col1]) / ([col2] + [col3])
I found out from a coworker just as I posted this.
You need to specify the default precision and scale.
This works in this scenario:
declare #tempDec decimal(3,2)
From MSDN:
decimal[ (p[ , s] )] and numeric[ (p[ , s] )]
Fixed precision and scale numbers. When maximum precision is used, valid values are from - 10^38 +1 through 10^38 - 1. The SQL-92 synonyms for decimal are dec and dec(p, s). numeric is functionally equivalent to decimal.
p (precision)
The maximum total number of decimal digits that can be stored, both to the left and to the right of the decimal point. The precision must be a value from 1 through the maximum precision of 38. The default precision is 18.
s (scale)
The maximum number of decimal digits that can be stored to the right of the decimal point. Scale must be a value from 0 through p. Scale can be specified only if precision is specified. The default scale is 0; therefore, 0 <= s <= p. Maximum storage sizes vary, based on the precision.
Related
n = 215;
N = 215.01:0.1:250;
p = 0.52;
q = 0.48;
Gamblers = (1 - (q/p)^n)./(1 - (q/p).^N);
plot(Gamblers)
Matlab takes the numerators and denominators as simply 1, filling the array with nothing but that value. How can I correct this?
Your denominator and numerator are very close to 1 but not exactly 1. The plot(Gamblers) confirms this.
By default MATLAB will display numbers with four digits after the decimal place. Your numerator is 0.999999966414861, which with four digits rounds to 1. MATLAB uses double precision numbers so your calculation here is still accurate.
Try double clicking on the Gamblers variable to open the variables window and then double clicking on one of the results. You'll see it change from the default display precision to a much more accurate depiction of your variable.
Is there a simply way to convert the following Excel formula into TSQL:
ROUNDDOWN((table.OrderQty / table.PacksPerCase),0)
table.OrderQty = 9
table.PacksPerCase = 5
"0" indicate that the result will have zero decimal places
Result = 1.0
table.OrderQty = 9
table.PacksPerCase = 4
"0" indicate that the result will have zero decimal places
Result = 2.0
Try using FLOOR :
The FLOOR function returns the largest integer less than or equal to
the specified numeric expression.
Your expression should be:
FLOOR(table.OrderQty / table.PacksPerCase)
ROUNDDOWN() rounds towards zero for negative numbers.
Since FLOOR rounds towards negative infinity, the closest equivalent in TSQL I can think of is ROUND with function set to 1;
ROUND((table.OrderQty / table.PacksPerCase), 0, 1)
If you're only dealing with positive numbers, of course FLOOR will also work well.
how to filter out float values with only zeroes beyond decimal & others having some non-zero values beyond decimal too.
for example.
13.000000
13.120001
i want it like this:
13.0
13.120001
If your application stores the floating point values in something like C/C++'s float's or double's, you may have a problem here.
First of all, float/double in the majority of cases represents values using base-2 and when you do something like double x = 0.1;, x in fact will never be equal to 0.1, but it will be very close to 0.1. That is because not every decimal (base-10) fraction can be represented exactly in base-2.
When you print or convert that x to a string using one of the printf-like functions, the resultant string can vary from something like 0.099999999 to 0.1 to 0.10000000000000000555. The result will depend on what's in x and on how you convert it to a string (how many digits you allow for the integer part, fractional part or all).
Generally there's no one-size-fits-all solution here. In one case the exactness and rounding issues can be unimportant, while in another they can be the most important. When performance is more important than the exact representation of decimal fractions, you use base-2 types. Otherwise, for example, when you're counting money, you use special types (and you may first need to construct them) that can represent decimal fractions exactly and then you avoid some of the described issues at a cost of extra computation time and maybe storage.
You may sometimes use fixed-point types and arithmetic. For example, if your numbers should not have more than 3 fractional digits and aren't too big, you can use scaled integers:
long x = 123456; // x represents 123.456
You add, subtract and compare them just as regular integers, but you multiply and divide them differently, for example like this:
long x = 123456; // x represents 123.456
long y = 12345; // y represents 12.345
long p = (long)(((long long)x * y) / 1000); // p=1524064, representing x*y=1524.064
long q = (long)((long long)x * 1000) / y); // q=10000, representing x/y=10.000
And in this case it is relatively easy to figure out the digits of the fractional part, just look at (x%1000)/100, (x%100)/10 and x%10.
split 13.00000 or 13.20001 into 2 values like in 13.0000 as 13 and 0000 ...so compare 2nd value whether it is greater than 0 (ex:13.20001 where 2001>0) if its yes then set same value . else (ex 13.0000 where 0000=0) then set %.2f=> 13.00
I'm trying to get a string from a double like this:
double aDouble;
NSString* doubleString = [NSString stringWithFormat:#"%g", aDouble];
With most numbers I get the desired results, but for
10000.03
I get:
10000
as my string. Is there a way to prevent this behavior? I would like a result a string of
10000.03
%g can be tricky in the absence of a precision specifier. Note that %f can be used with double values, and you can limit the number of digits after the decimal point by using %.2f (replace 2 with whichever number of digits you want).
%g is a format specifier that chooses between %f or %e according to the following algorithm:
If a non-zero precision has been specified, let P be that precision
If a zero precision has been specified, let P be 1
If a precision hasn’t been specified, let P be 6
Let X be the exponent if the conversion were to use the %e format specifier
If P > X >= -4, the conversion is with style %f and precision P - (X + 1)
Otherwise, the conversion is with style %e and precision P - 1.
Unless the # flag is used, any trailing zeros are removed from the fractional portion of the result, and the decimal point character is removed if there is no fractional portion remaining.
In your case, %g doesn’t specify a precision, hence P = 6. When converting 10000.03 with %e, which gives 1.000003e+04, the exponent is 4, hence X = 4. Since P > X >= -4, the conversion is with style %f and precision P - (X + 1) = 6 - (4 + 1) = 1. But 10000.03 with precision 1 (one digit after the decimal point) yields 10000.0, which has no actual fractional portion, hence %g formats 10000.03 as 10000.
Try %.2f instead of %g
floats and double are base two representations of number that we like to see in base ten, just as there is no way to exactly represent the number 1/3 in base ten with a finite number of digits there are many base 10 number which can not be exactly represented in base 2. For example 0.1 (1/10) can not be represented exactly in base 2.
At some point in an algorithm I need to compare the float value of a property of a class to a float. So I do this:
if (self.scroller.currentValue <= 0.1) {
}
where currentValue is a float property.
However, when I have equality and self.scroller.currentValue = 0.1 the if statement is not fulfilled and the code not executed! I found out that I can fix this by casting 0.1 to float. Like this:
if (self.scroller.currentValue <= (float)0.1) {
}
This works fine.
Can anyone explain to my why this is happening? Is 0.1 defined as a double by default or something?
Thanks.
I believe, having not found the standard that says so, that when comparing a float to a double the float is cast to a double before comparing. Floating point numbers without a modifier are considered to be double in C.
However, in C there is no exact representation of 0.1 in floats and doubles. Now, using a float gives you a small error. Using a double gives you an even smaller error. The problem now is, that by casting the float to a double you carry over the bigger of error of the float. Of course they aren't gone compare equal now.
Instead of using (float)0.1 you could use 0.1f which is a bit nicer to read.
The problem is, as you have suggested in your question, that you are comparing a float with a double.
There is a more general problem with comparing floats, this happens because when you do a calculation on a floating point number the result from the calculation may not be exactly what you expect. It is fairly common that the last bit of the resulting float will be wrong (although the inaccuracy can be larger than just the last bit). If you use == to compare two floats then all the bits have to be the same for the floats to be equal. If your calculation gives a slightly inaccurate result then they won't compare equal when you expect them to. Instead of comparing the values like this, you can compare them to see if they are nearly equal. To do this you can take the positive difference between the floats and see if it is smaller than a given value (called an epsilon).
To choose a good epsilon you need to understand a bit about floating point numbers. Floating point numbers work similarly to representing a number to a given number of significant figures. If we work to 5 significant figures and your calculation results in the last digit of the result being wrong then 1.2345 will have an error of +-0.0001 whereas 1234500 will have an error of +-100. If you always base your margin of error on the value 1.2345 then your compare routine will be identical to == for all values great than 10 (when using decimal). This is worse in binary, it's all values greater than 2. This means that the epsilon we choose has to be relative to the size of the floats that we are comparing.
FLT_EPSILON is the gap between 1 and the next closest float. This means that it may be a good epsilon to choose if your number is between 1 and 2, but if your value is greater than 2 using this epsilon is pointless because the gap between 2 and the next nearest float is larger than epsilon. So we have to choose an epsilon relative to the size of our floats (as the error in the calculation is relative to the size of our floats).
A good(ish) floating point compare routine looks something like this:
bool compareNearlyEqual (float a, float b, unsigned epsilonMultiplier)
{
float epsilon;
/* May as well do the easy check first. */
if (a == b)
return true;
if (a > b) {
epsilon = scalbnf(1.0f, ilogb(a)) * FLT_EPSILON * epsilonMultiplier;
} else {
epsilon = scalbnf(1.0, ilogb(b)) * FLT_EPSILON * epsilonMultiplier;
}
return fabs (a - b) <= epsilon;
}
This comparison routine compares floats relative to the size of the largest float passed in. scalbnf(1.0f, ilogb(a)) * FLT_EPSILON finds the gap between a and the next nearest float. This is then multiplied by the epsilonMultiplier, so the size of the difference can be adjusted, depending on how inaccurate the result of the calculation is likely to be.
You can make a simple compareLessThan routine like this:
bool compareLessThan (float a, float b, unsigned epsilonMultiplier)
{
if (compareNearlyEqual (a, b, epsilonMultiplier)
return false;
return a < b;
}
You could also write a very similar compareGreaterThan function.
It's worth noting that comparing floats like this may not always be what you want. For instance this will never find that a float is close to 0 unless it is 0. To fix this you'd need to decide what value you thought was close to zero, and write an additional test for this.
Sometimes the inaccuracies you get won't depend on the size of the result of a calculation, but will depend on the values that you put into a calculation. For instance sin(1.0f + (float)(200 * M_PI)) will give a much less accurate result than sin(1.0f) (the results should be identical). In this case your compare routine would have to look at the number you put into the calculation to know the margin of error of the answer.
Doubles and floats have different values for the mantissa store in binary (float is 23 bits, double 54). These will almost never be equal.
The IEEE Float Point article on wikipedia may help you understand this distinction.
In C, a floating-point literal like 0.1 is a double, not a float. Since the types of the data items being compared are different, the comparison is done in the more precise type (double). In all implementations I know about, float has a shorter representation than double (usually expressed as something like 6 vs. 14 decimal places). Moreover, the arithmetic is in binary, and 1/10 does not have an exact representation in binary.
Therefore, you're taking a float 0.1, which loses accuracy, extending it to double, and expecting it to compare equal to a double 0.1, which loses less accuracy.
Suppose we were doing this in decimal, with float being three digits and double being six, and we were comparing to 1/3.
We have the stored float value being 0.333. We're comparing it to a double with value 0.333333. We convert the float 0.333 to double 0.333000, and find it different.
0.1 is actually a very dificult value to store binary. In base 2, 1/10 is the infinitely repeating fraction
0.0001100110011001100110011001100110011001100110011...
As several has pointed out, the comparison has to made with a constant of the exact same precision.
Generally, in any language, you can't really count on equality of float-like types. In your case since it looks like you have more control, it does appear that 0.1 is not float by default. You could probably find that out with sizeof(0.1) (vs. sizeof(self.scroller.currentValue).
Convert it to a string, then compare:
NSString* numberA = [NSString stringWithFormat:#"%.6f", a];
NSString* numberB = [NSString stringWithFormat:#"%.6f", b];
return [numberA isEqualToString: numberB];