What is the difference between Reactive programming and plain old closures? - scala

Example from scala.rx:
import rx._
val a = Var(1); val b = Var(2)
val c = Rx{ a() + b() }
println(c()) // 3
a() = 4
println(c()) // 6
How is the above version better than:
var a = 1; var b = 2
def c = a + b
println(c) // 3
a = 4
println(c) // 6
The only thing I can think of is that the first example is efficient in the sense that unless a or b changes, c is not recalculated but in my version, c is recomputed every time I invoke c() but that is just a special case of memoization with size=1 e.g. I can do this to prevent re-computations using a memoization macro:
var a = 1; var b = 2
#memoize(maxSize = 1) def c(x: Int = a, y: Int = z) = x + y
Is there anything that I am missing to grok about reactive programming that provides insight into why it might be a better paradigm (than memoized closures) in certain cases?

Problem: It's a bad example
The example on the web page doesn't illustrate the purpose of Scala.RX very well. In that sense it is a quite bad example.
What is Scala.RX for?
It's about notifications
The idea of Scala.Rs is that a piece of code can get notifications, when data changes. Usually the this notification is used to (re-)calculate a result that depends on the changed data.
Scala.RX automates the wiring
When the calculation goes over multiple stages, it becomes quite hard to track which intermediate result depends on which data and on which other intermediate results. Additionally on must recalculate the intermediate results in the correct order.
You can think of this just like a big excel sheet which must of formulas that depend of each other. When you change one of the input values, Excel has to figure out, which parts of the sheet must be recalculated in which order. When Excel has re-calculated all the changed cells, it can update the display.
Scala.RX can do a similar thing than Excel: It tracks how the formulas depend on each other on notifies the ones that need to update in the correct order.
Purpose: MVC
Scala.RX is a nice tool to implement the MVC-pattern, especially when you have business applications that you could also bring to excel.
There is also a variant that works with Scala.js, i.e. that runs in the browser as part of a HTML site. This can be quite useful if you want to dynamically update parts of a HTML page according to changes on the server or edits of the user.
Limitations
Scala.RX doe not scale when you have a huge amounts of input data, e.g. operations on huge matrices.
A better example
import rx._
import rx.ops._
val a = Var(1); val b = Var(2)
val c: Rx[Int] = Rx{ a() + b() }
val o = c.foreach{value =>
println(s"c has a new value: ${value}")
}
a()=4
b()=12
a()=35
Gives you the following output:
c has a new value: 3
c has a new value: 6
c has a new value: 16
c has a new value: 47
Now imagine instead of printing the value, you will refresh controls in a UI or parts of a HTML page.

Related

Strange behavior of Scala

Can any of you explain why this is happening?
val s = "abcdefg"
val slides = s.sliding(4)
val n1 = slides.length
val n2 = slides.dropWhile(foo).length
println(n1) // 4
println(n2) // 0
println(slides.length) // 0
But:
val s = "abcdefg"
println(s.sliding(4).length) // 4
println(s.sliding(4).dropWhile(foo).length) // 3
println(s.sliding(4).length) // 4
Don't pay attention to the function "foo", it's a simple method to check if a string doesn't contain the letter "c".
Unfortunately, I don't understand this programming language behavior. Maybe someone with more knowledge can answer why this is happening.
slides is an Iterator. It is a special kind of "collection", that you can only traverse once.
Once you ask for its length, it has to scan through (and discard) all of its elements, to count them, so, it becomes empty, and when you ask for its length again (dropWhile is inconsequential here), it is 0.
This is useful in cases when you need to process a huge collection without loading it into memory all at once (e.g., reading a huge file line by line, to see if it contains the word "google" somewhere).
sliding returns an iterator, because making it traversable more than once may be expensive, but is rarely needed.
If you need to traverse it more than once, do val slides = s.sliding(4).toSeq

What's the simplest way to get a Spark DataFrame from arbitrary Array Data in Scala?

I've been breaking my head about this one for a couple of days now. It feels like it should be intuitively easy... Really hope someone can help!
I've built an org.nd4j.linalg.api.ndarray.INDArray of word occurrence from some semi-structured data like this:
import org.nd4j.linalg.factory.Nd4j
import org.nd4s.Implicits._
val docMap = collection.mutable.Map[Int,Map[Int,Int]] //of the form Map(phrase -> Map(phrasePosition -> word)
val words = ArrayBuffer("word_1","word_2","word_3",..."word_n")
val windows = ArrayBuffer("$phrase,$phrasePosition_1","$phrase,$phrasePosition_2",..."$phrase,$phrasePosition_n")
var matrix = Nd4j.create(windows.length*words.length).reshape(windows.length,words.length)
for (row <- matrix.shape(0)){
for(column <- matrix.shape(1){
//+1 to (row,column) if word occurs at phrase, phrasePosition indicated by window_n.
}
}
val finalmatrix = matrix.T.dot(matrix) // to get co-occurrence matrix
So far so good...
Downstream of this point I need to integrate the data into an existing pipeline in Spark, and use that implementation of pca etc, so I need to create a DataFrame, or at least an RDD. If I knew the number of words and/or windows in advance I could do something like:
case class Row(window : String, word_1 : Double, word_2 : Double, ...etc)
val dfSeq = ArrayBuffer[Row]()
for (row <- matrix.shape(0)){
dfSeq += Row(windows(row),matrix.get(NDArrayIndex.point(row), NDArrayIndex.all()))
}
sc.parallelize(dfSeq).toDF("window","word_1","word_2",...etc)
but the number of windows and words is determined at runtime. I'm looking for a WindowsxWords org.apache.spark.sql.DataFrame as output, input is a WindowsxWords org.nd4j.linalg.api.ndarray.INDArray
Thanks in advance for any help you can offer.
Ok, so after several days work it looks like the simple answer is: there isn't one. In fact, it looks like trying to use Nd4j in this context at all is a bad idea for several reasons:
It's (really) hard to get data out of the native INDArray format once you've put it in.
Even using something like guava, the .data() method brings everything on heap which will quickly become expensive.
You've got the added hassle of having to compile an assembly jar or use hdfs etc to handle the library itself.
I did also consider using Breeze which may actually provide a viable solution but carries some of the same problems and can't be used on distributed data structures.
Unfortunately, using native Spark / Scala datatypes, although easier once you know how, is - for someone like me coming from Python + numpy + pandas heaven at least - painfully convoluted and ugly.
Nevertheless, I did implement this solution successfully:
import org.apache.spark.mllib.linalg.{Vectors,Vector,Matrix,DenseMatrix,DenseVector}
import org.apache.spark.mllib.linalg.distributed.RowMatrix
//first make a pseudo-matrix from Scala Array[Double]:
var rowSeq = Seq.fill(windows.length)(Array.fill(words.length)(0d))
//iterate through 'rows' and 'columns' to fill it:
for (row 0 until windows.length){
for (column 0 until words.length){
// rowSeq(row)(column) += 1 if word occurs at phrase, phrasePosition indicated by window_n.
}
}
//create Spark DenseMatrix
val rows : Array[Double] = rowSeq.transpose.flatten.toArray
val matrix = new DenseMatrix(windows.length,words.length,rows)
One of the main operations that I needed Nd4J for was matrix.T.dot(matrix) but it turns out that you can't multiply 2 matrices of Type org.apache.spark.mllib.linalg.DenseMatrix together, one of them (A) has to be a org.apache.spark.mllib.linalg.distributed.RowMatrix and - you guessed it - you can't call matrix.transpose() on a RowMatrix, only on a DenseMatrix! Since it's not really relevant to the question, I'll leave that part out, except to explain that what comes out of that step is a RowMatrix. Credit is also due here and here for the final part of the solution:
val rowMatrix : [RowMatrix] = transposeAndDotDenseMatrix(matrix)
// get DataFrame from RowMatrix via DenseMatrix
val newdense = new DenseMatrix(rowMatrix.numRows().toInt,rowMatrix.numCols().toInt,rowMatrix.rows.collect.flatMap(x => x.toArray)) // the call to collect() here is undesirable...
val matrixRows = newdense.rowIter.toSeq.map(_.toArray)
val df = spark.sparkContext.parallelize(matrixRows).toDF("Rows")
// then separate columns:
val df2 = (0 until words.length).foldLeft(df)((df, num) =>
df.withColumn(words(num), $"Rows".getItem(num)))
.drop("Rows")
Would love to hear improvements and suggestions on this, thanks.

What is the scala equivalent of Python's Numpy np.random.choice?(Random weighted selection in scala)

I was looking for Scala's equivalent code or underlying theory for pythons np.random.choice (Numpy as np). I have a similar implementation that uses Python's np.random.choice method to select the random moves from the probability distribution.
Python's code
Input list: ['pooh', 'rabbit', 'piglet', 'Christopher'] and probabilies: [0.5, 0.1, 0.1, 0.3]
I want to select one of the value from the input list given the associated probability of each input element.
The Scala standard library has no equivalent to np.random.choice but it shouldn't be too difficult to build your own, depending on which options/features you want to emulate.
Here, for example, is a way to get an infinite Stream of submitted items, with the probability of any one item weighted relative to the others.
def weightedSelect[T](input :(T,Int)*): Stream[T] = {
val items :Seq[T] = input.flatMap{x => Seq.fill(x._2)(x._1)}
def output :Stream[T] = util.Random.shuffle(items).toStream #::: output
output
}
With this each input item is given with a multiplier. So to get an infinite pseudorandom selection of the characters c and v, with c coming up 3/5ths of the time and v coming up 2/5ths of the time:
val cvs = weightedSelect(('c',3),('v',2))
Thus the rough equivalent of the np.random.choice(aa_milne_arr,5,p=[0.5,0.1,0.1,0.3]) example would be:
weightedSelect("pooh"-> 5
,"rabbit" -> 1
,"piglet" -> 1
,"Christopher" -> 3).take(5).toArray
Or perhaps you want a better (less pseudo) random distribution that might be heavily lopsided.
def weightedSelect[T](items :Seq[T], distribution :Seq[Double]) :Stream[T] = {
assert(items.length == distribution.length)
assert(math.abs(1.0 - distribution.sum) < 0.001) // must be at least close
val dsums :Seq[Double] = distribution.scanLeft(0.0)(_+_).tail
val distro :Seq[Double] = dsums.init :+ 1.1 // close a possible gap
Stream.continually(items(distro.indexWhere(_ > util.Random.nextDouble())))
}
The result is still an infinite Stream of the specified elements but the passed-in arguments are a bit different.
val choices :Stream[String] = weightedSelect( List("this" , "that")
, Array(4998/5000.0, 2/5000.0))
// let's test the distribution
val (choiceA, choiceB) = choices.take(10000).partition(_ == "this")
choiceA.length //res0: Int = 9995
choiceB.length //res1: Int = 5 (not bad)

Chisel compiler is very slow

I am working on a matrix summation kind of design. The compiler takes 4+hours to generate 1+million lines of codes. Every line is "assign....." I don't know if this is the inefficiency of the compiler or my coding style is bad. If someone could suggest some alternatives that will be great!
Here is the description of the code
The input will be AND with a random matrix element by element and summed up using .reduce, so the result matrix should be 140X6 vec, cat them together gives me a 840 bits output
(rndvec, which is supposed to be a 140x840x6 bits random matrix. since I don't know how to generate random value so I started with a fixed 140x6 to represent one row and feed it with input over and over again)
This following is my code
import Chisel._
import scala.collection.mutable.HashMap
import util.Random
class LBio(n: Int) extends Bundle {
var myinput = UInt(INPUT,840)
var myoutput = UInt (OUTPUT,840)
}
class Lbi(q: Int,n:Int,m :Int ) extends Module{
def mask(orig: Vec[UInt],maska:UInt,mi:Int)={
val result = Vec.fill(840){UInt(width =6)}
for (i<-0 until 840 ){
result(i) := orig(i)&Fill(6,maska(i)) //every bits of input AND with random vector
}
result
}
val io= new LBio(840)
val rndvec = Vec.fill(840){UInt("h13",6)} //random vector, for now its just replication of 0x13....
val resultvec = Vec.fill(140){UInt(width = 6)}
for (i<-0 until 140){
resultvec(i) := mask(rndvec,io.myinput,m).reduce(_+_) //add the entire row of 6 bits element together with reduce
}
io.myoutput := resultvec.toBits
}
The terminal report:
started inference
finished inference (4)
start width checking
finished width checking
started flattenning
finished flattening (941783)
resolving nodes to the components
finished resolving
started transforms
finished transforms
checking for combinational loops
NO COMBINATIONAL LOOP FOUND
COMPILING class TutorialExamples.Lbi 0 CHILDREN (0,0)
[success] Total time: 33453 s, completed Oct 16, 2013 10:32:10 PM
There's nothing obviously wrong with your Chisel code, but I should point out that if rndvec is 140x840x6 bits, that's ~689kB of state! And your reduce operation is on 5kB of state.
Chisel uses "assign" statements because your code is entirely combinational and Chisel produces a very structural form of Verilog.
I suspect the part that is killing the compile time (aside from the huge amount of state) is that you are generating and manipulating 140 Vecs with the mask() function.
I tried my hand at rewriting your code and got it down from 941,783 nodes to 202,723 (takes about 10-15 minutes to compile, but generates 11MB of Verilog code). I'm pretty sure this does what your code was doing:
class Hello(q: Int, dim_n:Int) extends Module
{
val io = new LBio(dim_n)
val rndvec = Vec.fill(dim_n){UInt("h13",6)}
val resultvec = Vec.fill(dim_n/6){UInt(width=6)}
// lift this work outside of the for loop
val padded_input = Vec.fill(dim_n){UInt(width=6)}
for (i <- 0 until dim_n)
{
padded_input(i) := Fill(6,io.myinput)
}
for (i <- 0 until dim_n/6)
{
val result = Bits(width=dim_n*6)
result := rndvec.toBits & padded_input.toBits
var sum = UInt(0) //advanced Chisel - be careful with the use of var!
for (j <- 0 until dim_n by 6)
{
sum = sum + result(j+6,j)
}
resultvec(i) := sum
}
io.myoutput := resultvec.toBits
}
What I did was avoid doing the same work over and over again - like padding out the myinput Vec inside of the for loop's mask() function. I also kept everything in Bits() instead of Vecs. Sadly it means I lose the awesome .reduce() function.
I think maybe the answer is "be cognizant of how much state you're creating" and "Vecs are awesome, but use carefully".
Do you have a Verilog version that's short and concise? It'd be interesting to see if there are areas where Chisel is losing out efficiency wise.

Adding immutable Vectors

I am trying to work more with scalas immutable collection since this is easy to parallelize, but i struggle with some newbie problems. I am looking for a way to create (efficiently) a new Vector from an operation. To be precise I want something like
val v : Vector[Double] = RandomVector(10000)
val w : Vector[Double] = RandomVector(10000)
val r = v + w
I tested the following:
// 1)
val r : Vector[Double] = (v.zip(w)).map{ t:(Double,Double) => t._1 + t._2 }
// 2)
val vb = new VectorBuilder[Double]()
var i=0
while(i<v.length){
vb += v(i) + w(i)
i = i + 1
}
val r = vb.result
}
Both take really long compared to the work with Array:
[Vector Zip/Map ] Elapsed time 0.409 msecs
[Vector While Loop] Elapsed time 0.374 msecs
[Array While Loop ] Elapsed time 0.056 msecs
// with warm-up (10000) and avg. over 10000 runs
Is there a better way to do it? I think the work with zip/map/reduce has the advantage that it can run in parallel as soon as the collections have support for this.
Thanks
Vector is not specialized for Double, so you're going to pay a sizable performance penalty for using it. If you are doing a simple operation, you're probably better off using an array on a single core than a Vector or other generic collection on the entire machine (unless you have 12+ cores). If you still need parallelization, there are other mechanisms you can use, such as using scala.actors.Futures.future to create instances that each do the work on part of the range:
val a = Array(1,2,3,4,5,6,7,8)
(0 to 4).map(_ * (a.length/4)).sliding(2).map(i => scala.actors.Futures.future {
var s = 0
var j = i(0)
while (j < i(1)) {
s += a(j)
j += 1
}
s
}).map(_()).sum // _() applies the future--blocks until it's done
Of course, you'd need to use this on a much longer array (and on a machine with four cores) for the parallelization to improve things.
You should use lazily built collections when you use more than one higher-order methods:
v1.view zip v2 map { case (a,b) => a+b }
If you don't use a view or an iterator each method will create a new immutable collection even when they are not needed.
Probably immutable code won't be as fast as mutable but the lazy collection will improve execution time of your code a lot.
Arrays are not type-erased, Vectors are. Basically, JVM gives Array an advantage over other collections when handling primitives that cannot be overcome. Scala's specialization might decrease that advantage, but, given their cost in code size, they can't be used everywhere.