I need to do some precision calcs with Big Numbers and I have been trying with Scala BigDecimal but I have noted loss of precision.
As an example:
2^63 == 9223372036854775808
2^64 == 18446744073709551616
However when I do
println(BigDecimal.decimal(scala.math.pow(2, 63)).toBigIntExact())
println(BigDecimal.decimal(scala.math.pow(2, 64)).toBigIntExact())
I get
9223372036854776000 != 9223372036854775808
18446744073709552000 != 18446744073709551616
I don't know if I can get the exact BigInt.
Maybe I have to take other approach.
Could anyone help me to fix this issue?
# scala.math.pow(2, 63)
res0: Double = 9.223372036854776E18
You get Double on math.pow, and then you pass the result to BigDecimal - it means that you lost precision even before you started using Big* classes.
If you put numbers into BigDecimal when they are still small and haven't yet lost precision (and if you use the constructors correctly) then you'll get the expected result:
# BigDecimal(2).pow(63).toBigInt
res4: BigInt = 9223372036854775808
# BigDecimal(2).pow(64).toBigInt
res5: BigInt = 18446744073709551616
# BigDecimal(2).pow(63).toBigIntExact
res6: Option[BigInt] = Some(9223372036854775808)
# BigDecimal(2).pow(64).toBigIntExact
res7: Option[BigInt] = Some(18446744073709551616)
Related
Given:
scala> val s =99999999999999947.89
s: Double = 9.9999999999999952E16
scala> f"$s%.2f"
res15: String = 99999999999999952.00
i want output should be = 99999999999999947.89
how to avoid round off and scientific notation both.
The Double datatype doesn't support the precision you're after. The closest number 99999999999999952 is the number closest to 99999999999999947.89 that can be represented as a Double.
That's not just the case for very big numbers. Double values can soon start to drift. You can see this for example in that (0.2 + 0.2 + 0.2) != 0.6
Use a BigDecimal instead.
I'm not able to do an add on long type.
scala or the processor doesn't manage correctly the sign
scala> var i="-1014570924054025346".toLong
i: Long = -1014570924054025346
scala> i=i+92233720368547758L
i: Long = -922337203685477588
scala> var i=9223372036854775807L
i: Long = 9223372036854775807
scala> i=i+5
i: Long = -9223372036854775804
The first test where a negative number doesn't pass to a positive one is a problem for me
I have not fully understood the question, but for the first example, you get the expected result. What happens in the second example, the Long number happens to be the maximum value for a Long (i.e Long.MaxValue) so essentially when you had another positive number, it's overflowing:
scala> Long.MaxValue
res4: Long = 9223372036854775807L
scala> Long.MaxValue + 1
res7: Long = -9223372036854775808L // which is Long.MinValue
scala> Long.MinValue + 4
res8: Long = -9223372036854775804L // which is the result that you get
In other words:
9223372036854775807L + 5
is equivalent to:
Long.MaxValue + 5
which is equivalent to:
Long.MinValue + 4 // because (Long.MaxValue + 1) = Long.MinValue
which is equals to -9223372036854775804L
If you really need to use such big numbers, you might try using BigInt
scala> val x = BigInt(Long.MaxValue)
x: scala.math.BigInt = 9223372036854775807
scala> x + 1
res6: scala.math.BigInt = 9223372036854775808
scala> x + 5
res11: scala.math.BigInt = 9223372036854775812
scala> x + 10
res8: scala.math.BigInt = 9223372036854775817
scala> x * 1000
res10: scala.math.BigInt = 9223372036854775807000
scala> x * x
res9: scala.math.BigInt = 85070591730234615847396907784232501249
scala> x * x * x * x
res13: scala.math.BigInt = 7237005577332262210834635695349653859421902880380109739573089701262786560001
scala>
The documentation on BigInt is rather, err, small. However, i believe that it is basically an infinite precision integer (can support as many digits as you need). Having said that, there will probably at some point be a limit. There is a comment on BigDecimal - which has more documentation - that at about 4,934 digits there might be some deviation between BigDecimal and BigInt.
I will leave it to someone else to work out whether or not x ^ 4 is the value shown above.
Oh, I almost forgot your negative number test, I aligned the sum with the initialisation, to make it easier to visualise that the result appears to be correct:
scala> val x = BigInt("-1014570924054025346")
x: scala.math.BigInt = -1014570924054025346
scala> x + 92233720368547758L
res15: scala.math.BigInt = -922337203685477588
scala>
As for Ints, Longs and similar data types, they are limited in their size due to the number of bits they are constrained to. Int's are typically 32 bit and longs are typically 64 bits.
It is easier to visualise when you look at them in hexadecimal. A signed Byte (at 8 bits) has a maximum positive value of 0x7F (127). When you add one to it, you get 0x80 (-128). This is because we use the "Most Significant Bit" as an indicator of whether the number is positive or negative.
If the same byte was interpreted as unsigned, then 0x7F (127) would still become 0x80 when 1 is added to it. However, since we are interpreting it as unsigned, this would be equivalent to 128. We can keep adding one until we get to 0xFF (255) at which point if we add another 1 we will end up at 0x00 again which is of course 0.
Here are some references that explain this in much more detail:
Wikipedia - Twos complement
Cornell University - what is twos complement
Stack Overflow - what is 2s complement
I want to multiply a financial amount with a quantity. I know Scala uses Java's BigDecimal under the hood but the syntax doesn't seem to be the same.
val price = BigDecimal("0.01") // £0.01
val qty = 10
I tried to do this
BigDecimal(price).*(BigDecimal(qty))
But it's a compile error. If I look at the Java SO posts you can pass integer into BigDecimal and then multiply it like this
BigDecimal(price).multiply(BigDecimal(qty))
So how do you do this in Scala? And are there any dangers in losing precision by multiplying a decimal and integer like this? I will need sum a lot of these together as well
You can actually multiply a BigDecimal with an Int using BigDecimal's multiplication operator:
def *(that: BigDecimal): BigDecimal
since the Int you will provide as its parameter will be implicitly converted to a BigDecimal based on:
implicit def int2bigDecimal(i: Int): BigDecimal
You can thus safely multiply your BigDecimal with an Int as if it was a BigDecimal and receive a BigDecimal with no loss in precision:
val price = BigDecimal("0.01")
// scala.math.BigDecimal = 0.01
val qty = 10
// Int = 10
price * qty // same as BigDecimal("0.01").*(10)
// scala.math.BigDecimal = 0.10
You can do this:
val a = 10
val b = BigDecimal(0.1000000000001)
a * b
res0: scala.math.BigDecimal = 1.0000000000010
As you can see you don´t lose precision
The problem is actually this:
BigDecimal(price)
price is already a BigDecimal so the compiler does't know what to do! If you fix this, the first version works. The second version fails because there is no multiply method on BigDecimal.
However, as others have pointed out, the simple solution is just
price*qty
Below is the sample test code and its output. I see that java bigDecimal stores all the digits where as scala BigDecimal is losing on precision and does some rounding off and the same is happening with spark. Is there a way to set the precision or say never round off. I do not want to truncate or round off in any case
val sc = sparkSession
import java.math.BigDecimal
import sc.implicits._
val bigNum : BigDecimal = new BigDecimal(0.02498934809987987982348902384928349)
val convertedNum: scala.math.BigDecimal = scala.math.BigDecimal(bigNum)
val scalaBigNum: scala.math.BigDecimal = scala.math.BigDecimal(0.02498934809987987982348902384928349)
println("Big num in java" + bigNum)
println("Converted " + convertedNum)
println("Big num in scala " + scalaBigNum)
val ds = List(scalaBigNum).toDS()
println(ds.head)
println(ds.toDF.head)
Output
Big num in java0.0249893480998798801773208566601169877685606479644775390625
Converted 0.0249893480998798801773208566601169877685606479644775390625
Big num in scala 0.02498934809987988
0.024989348099879880
[0.024989348099879880]
Based on spark.apache.org/docs
The precision can be up to 38, scale can also be up to 38 (less or equal to precision). The default precision and scale is (10, 0).
here: https://www.scala-lang.org/api/2.12.5/scala/math/BigDecimal.html
But if you want in a simple way then how about convert it to String before
converting to DF or DS in order to get the precise value. :)
Just try if you want :)
I can't find the ceil function in the places I've looked:
https://duckduckgo.com/?q=scala+bigdecimal+ceil+function&t=ffab&ia=qa
http://www.scala-lang.org/api/current/scala/math/BigDecimal.html
But I do find a ceil method on Double and Float.
I know MathContext can be made to use the ceiling rounding mode, but I can't seem to make it work:
scala> BigDecimal("1.1").round(new MathContext(0, RoundingMode.UP))
res2: scala.math.BigDecimal = 1.1
scala> BigDecimal("1.8").round(new MathContext(0, RoundingMode.UP))
res3: scala.math.BigDecimal = 1.8
scala> BigDecimal("1.8").round(new MathContext(0, RoundingMode.CEILING))
res5: scala.math.BigDecimal = 1.8
scala> BigDecimal("1.8", new MathContext(0, RoundingMode.CEILING))
res6: scala.math.BigDecimal = 1.8
scala> res6.rounded
res8: scala.math.BigDecimal = 1.8
I expected the result of all the operations above to be == BigDecimal(2).
I use a BigDecimal because these are money operations, hence I need exact values.
To answer the title question: there isn't. At least, not in the way that it exists for Double (and it doesn't really exist for Double, either, it's added implicitly).
You are specifying the precision of the MathContext as zero, which means you're saying you want unlimited precision--that is, no rounding will be done. If you want the true ceiling (rounding up to the nearest integer), then you don't want to use BigDecimal#round at all. The first parameter of the MathContext is precision, which is the number of significant digits you want to maintain. Rounding up a number like 1234567 to keep five significant digits would give you 1234600.
To find the ceiling of a number, you want to set the scale, which is the number of digits to maintain after the decimal point. In this case, you'd want a scale of zero.
import scala.math._, BigDecimal._
scala> BigDecimal("3134.1222").setScale(0, RoundingMode.CEILING)
res1: scala.math.BigDecimal = 3135
scala> BigDecimal("1.1").setScale(0, RoundingMode.CEILING)
res2: scala.math.BigDecimal = 2
scala> BigDecimal("1.8").setScale(0, RoundingMode.CEILING)
res3: scala.math.BigDecimal = 2
We can use the enrich my library pattern to add a ceil method to BigDecimal implicitly:
implicit class RichBigDecimal(bd: BigDecimal) {
def ceil: BigDecimal = bd.setScale(0, RoundingMode.CEILING)
}
scala> BigDecimal("1.1").ceil
res6: scala.math.BigDecimal = 2