Scala nested for loop - scala

I'm a new Scala fellow and using processing library in scala ,I have 2 questions here:
val grid: Array[Cell] = Array()
val w = 60
val rows = height/w
val cols = width /w
override def setup(): Unit = {
for(j <- 0 until rows;
i <- 0 until cols){
val cell = new Cell(i,j)
grid :+ cell
println(s"at row : $j, at col: $i") //it compiles only once (at row : 0,
} //at col: 0 )
}
override def draw(): Unit = {
background(0)
grid.foreach(cell => cell.display())//nothing happens
}
but if i replace the variables rows & cols by height/w & width/w in the nested loop as follows:
for(j <- 0 until height/w;
i <- 0 until width/w){
val cell = new Cell(i,j)
grid :+ cell
println(s"at row : $j, at col: $i") //it compiles ordinary as nested
} //for loop
the second question is in the class Cell here:
class Cell(i: Int,j:Int){
def display(): Unit = {
val x = this.i * w
val y = this.j * w
println("it works")//doesn't work
//creating a square
stroke(255)
line(x,y,x+w,y)//top
line(x+w,y,x+x,y+w)//right
line(x,y+w,x+w,y+w)//bottom
line(x,y,x,y+w)//left
}
}
the method display doesn't work when calling at function draw() but no errors show up

Use tabulate to initialise your Array:
val grid = Array.tabulate(rows * cols) { i => new Cell(i % cols, i / cols) }
If you still have a problem with the display function then please post it as a separate question.

Related

Shrink method in array stack (scala)

I am trying to implement a resize function that shrinks and grows an array stack depending on its number of elements and the array's size in Scala3. Here is my code below:
package adt
import scala.reflect.ClassTag
class ArrayStack[A: ClassTag]:
private var dataArray: Array[A] = Array.fill(10)(null.asInstanceOf[A])
private var top: Int = 0
private var sz: Int = 10
def push(elem: A): Unit =
if top == sz then resize(grow = true)
dataArray(top) = elem
top += 1
def pop(): A =
if top == (sz / 2) - 1 then resize(grow = false)
top -= 1
dataArray(top)
def peek(): A =
dataArray(top - 1)
def isEmpty(): Boolean =
top == 0
def resize(grow: Boolean): Unit =
val newSize = if grow then sz * 2 else sz / 2
val newArray: Array[A] = Array.fill(newSize)(null.asInstanceOf[A])
for (a <- 0 until dataArray.length) do newArray(a) = dataArray(a)
dataArray = newArray
sz = newSize
However, in my JUnit tests, I get an array index out of bounds exception.
#Test def pushMultiple(): Unit =
val stack = new ArrayStack[Int]
val pushArr = Array.tabulate(100)(i => Math.round(i * 100))
pushArr.foreach(stack.push(_))
for (n <- pushArr.reverse) do assertEquals(n, stack.pop())
#Test def popMultiple(): Unit =
val stack = new ArrayStack[Int]
val pushArr = Array.tabulate(100)(i => Math.round(i * 100))
pushArr.foreach(stack.push(_))
for (i <- 1 to 1000) do stack.pop()
assertTrue(stack.isEmpty())
Error message:
Test arraystack_test.pushMultiple failed: java.lang.ArrayIndexOutOfBoundsException: Index 80 out of bounds for length 80, took 0.032 sec
at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:75)
ScalaRunTime.scala:75
at adt.ArrayStack.resize$$anonfun$1(ArrayStack.scala:30)
ArrayStack.scala:30
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
at scala.collection.immutable.Range.foreach(Range.scala:190)
Range.scala:190
at adt.ArrayStack.resize(ArrayStack.scala:30)
ArrayStack.scala:30
at adt.ArrayStack.pop(ArrayStack.scala:17)
ArrayStack.scala:17
at arraystack_test.pushMultiple$$anonfun$2(arraystack_test.scala:26)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1324)
ArrayOps.scala:1324
at arraystack_test.pushMultiple(arraystack_test.scala:26)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
NativeMethodAccessorImpl.java:34
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
NativeMethodAccessorImpl.java:62
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
DelegatingMethodAccessorImpl.java:43
at java.lang.reflect.Method.invoke(Method.java:566)
Method.java:566
...
Test arraystack_test.popMultiple failed: java.lang.ArrayIndexOutOfBoundsException: Index 80 out of bounds for length 80, took 0.0 sec
at scala.runtime.ScalaRunTime$.array_update(ScalaRunTime.scala:75)
ScalaRunTime.scala:75
at adt.ArrayStack.resize$$anonfun$1(ArrayStack.scala:30)
ArrayStack.scala:30
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.scala:18)
at scala.collection.immutable.Range.foreach(Range.scala:190)
Range.scala:190
at adt.ArrayStack.resize(ArrayStack.scala:30)
ArrayStack.scala:30
at adt.ArrayStack.pop(ArrayStack.scala:17)
ArrayStack.scala:17
at arraystack_test.popMultiple$$anonfun$2(arraystack_test.scala:32)
at scala.runtime.java8.JFunction1$mcII$sp.apply(JFunction1$mcII$sp.scala:17)
at scala.collection.immutable.Range.foreach(Range.scala:190)
Range.scala:190
at arraystack_test.popMultiple(arraystack_test.scala:32)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
NativeMethodAccessorImpl.java:34
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
NativeMethodAccessorImpl.java:62
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
DelegatingMethodAccessorImpl.java:43
at java.lang.reflect.Method.invoke(Method.java:566)
Can anyone please help me figure out why my resize shrink function does not work?
Nevermind, I figured it out.
In the resize method, this causes an index out of bounds exception
for (a <- 0 until dataArray.length) do newArray(a) = dataArray(a)
When the method is called for grow, it works fine. But, when the method is called for shrink, newArray has a length that is half of dataArray, so it will cause an index out of bounds exception.
To fix this:
def resize(grow: Boolean): Unit =
val newSize = if grow then sz * 2 else sz / 2
val endRange: Int = if grow then dataArray.length else newSize
val newArray: Array[A] = Array.fill(newSize)(null.asInstanceOf[A])
for (a <- 0 until endRange) do newArray(a) = dataArray(a)
dataArray = newArray
sz = newSize

Is there any way I can rewrite this line of code in Scala?

I try to rewrite this line of Scala + Figaro using my function sum_ but I have some errors.
val sum = Container(vars:_*).reduce(_+_)
It uses the reduce() method to calculate the sum. I want to rewrite this line but I have errors because of the Chain return type [Double, Int]:
import com.cra.figaro.language._
import com.cra.figaro.library.atomic.continuous.Uniform
import com.cra.figaro.language.{Element, Chain, Apply}
import com.cra.figaro.library.collection.Container
object sum {
def sum_(arr: Int*) :Int={
var i=0
var sum: Int =0
while (i < arr.length) {
sum += arr(i)
i += 1
}
return sum
}
def fillarray(): Int = {
scala.util.Random.nextInt(10) match{
case 0 | 1 | 2 => 3
case 3 | 4 | 5 | 6 => 4
case _ => 5
}
}
def main(args: Array[String]) {
val par = Array.fill(18)(fillarray())
val skill = Uniform(0.0, 8.0/13.0)
val shots = Array.tabulate(18)((hole: Int) => Chain(skill, (s:Double) =>
Select(s/8.0 -> (par(hole)-2),
s/2.0 -> (par(hole)-1),
s -> par(hole),
(4.0/5.0) * (1.0 - (13.0 * s)/8.0)-> (par(hole)+1),
(1.0/5.0) * (1.0 - (13.0 * s)/8.0) -> (par(hole)+2))))
val vars = for { i <- 0 until 18} yield shots(i)
//this line I want to rewrite
val sum1 = Container(vars:_*).reduce(_+_)
//My idea was to implement in this way the line above
val sum2 = sum_(vars)
}
}
If you want use your function you can do so:
val sum2 = sum_(vars.map(chain => chain.generateValue()):_*)
or
val sum2 = sum_(vars.map(_.generateValue()):_*)
but I'd recommend to dive deeper into your library and functional paradigm.

Scala: Haar Wavelet Transform

I am trying to implement Haar Wavelet Transform in Scala. I am using this Python Code for reference Github Link to Python implementation of HWT
I am also giving here my Scala code version. I am new to Scala so forgive me for not-so-good-code.
/**
* Created by vipul vaibhaw on 1/11/2017.
*/
import scala.collection.mutable.{ListBuffer, MutableList,ArrayBuffer}
object HaarWavelet {
def main(args: Array[String]): Unit = {
var samples = ListBuffer(
ListBuffer(1,4),
ListBuffer(6,1),
ListBuffer(0,2,4,6,7,7,7,7),
ListBuffer(1,2,3,4),
ListBuffer(7,5,1,6,3,0,2,4),
ListBuffer(3,2,3,7,5,5,1,1,0,2,5,1,2,0,1,2,0,2,1,0,0,2,1,2,0,2,1,0,0,2,1,2)
)
for (i <- 0 to samples.length){
var ubound = samples(i).max+1
var length = samples(i).length
var deltas1 = encode(samples(i), ubound)
var deltas = deltas1._1
var avg = deltas1._2
println( "Input: %s, boundary = %s, length = %s" format(samples(i), ubound, length))
println( "Haar output:%s, average = %s" format(deltas, avg))
println("Decoded: %s" format(decode(deltas, avg, ubound)))
}
}
def wrap(value:Int, ubound:Int):Int = {
(value+ubound)%ubound
}
def encode(lst1:ListBuffer[Int], ubound:Int):(ListBuffer[Int],Int)={
//var lst = ListBuffer[Int]()
//lst1.foreach(x=>lst+=x)
var lst = lst1
var deltas = new ListBuffer[Int]()
var avg = 0
while (lst.length>=2) {
var avgs = new ListBuffer[Int]()
while (lst.nonEmpty) {
// getting first two element from the list and removing them
val a = lst.head
lst -= 1 // removing index 0 element from the list
val b = lst.head
lst -= 1 // removing index 0 element from the list
if (a<=b) {
avg = (a + b)/2
}
else{
avg = (a+b+ubound)/2
}
var delta = wrap(b-a,ubound)
avgs += avg
deltas += delta
}
lst = avgs
}
(deltas, avg%ubound)
}
def decode(deltas:ListBuffer[Int],avg:Int,ubound:Int):ListBuffer[Int]={
var avgs = ListBuffer[Int](avg)
var l = 1
while(deltas.nonEmpty){
for(i <- 0 to l ){
val delta = deltas.last
deltas -= -1
val avg = avgs.last
avgs -= -1
val a = wrap(math.ceil(avg-delta/2.0).toInt,ubound)
val b = wrap(math.ceil(avg+delta/2.0).toInt,ubound)
}
l*=2
}
avgs
}
def is_pow2(n:Int):Boolean={
(n & -n) == n
}
}
But Code gets stuck at "var deltas1 = encode(samples(i), ubound)" and doesn't give any output. How can I improve my implementation? Thanks in advance!
Your error is on this line:
lst -= 1 // removing index 0 element from the list.
This doesn't remove index 0 from the list. It removes the element 1 (if it exists). This means that the list never becomes empty. The while-loop while (lst.nonEmpty) will therefore never terminate.
To remove the first element of the list, simply use lst.remove(0).

Task not serializable Flink

I am trying to do the pagerank Basic example in flink with little bit of modification(only in reading the input file, everything else is the same) i am getting the error as Task not serializable and below is the part of the output error
atorg.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:179)
at org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:171)
Below is my code
object hpdb {
def main(args: Array[String]) {
val env = ExecutionEnvironment.getExecutionEnvironment
val maxIterations = 10000
val DAMPENING_FACTOR: Double = 0.85
val EPSILON: Double = 0.0001
val outpath = "/home/vinoth/bigdata/assign10/pagerank.csv"
val links = env.readCsvFile[Tuple2[Long,Long]]("/home/vinoth/bigdata/assign10/ppi.csv",
fieldDelimiter = "\t", includedFields = Array(1,4)).as('sourceId,'targetId).toDataSet[Link]//source and target
val pages = env.readCsvFile[Tuple1[Long]]("/home/vinoth/bigdata/assign10/ppi.csv",
fieldDelimiter = "\t", includedFields = Array(1)).as('pageId).toDataSet[Id]//Pageid
val noOfPages = pages.count()
val pagesWithRanks = pages.map(p => Page(p.pageId, 1.0 / noOfPages))
val adjacencyLists = links
// initialize lists ._1 is the source id and ._2 is the traget id
.map(e => AdjacencyList(e.sourceId, Array(e.targetId)))
// concatenate lists
.groupBy("sourceId").reduce {
(l1, l2) => AdjacencyList(l1.sourceId, l1.targetIds ++ l2.targetIds)
}
// start iteration
val finalRanks = pagesWithRanks.iterateWithTermination(maxIterations) {
// **//the output shows error here**
currentRanks =>
val newRanks = currentRanks
// distribute ranks to target pages
.join(adjacencyLists).where("pageId").equalTo("sourceId") {
(page, adjacent, out: Collector[Page]) =>
for (targetId <- adjacent.targetIds) {
out.collect(Page(targetId, page.rank / adjacent.targetIds.length))
}
}
// collect ranks and sum them up
.groupBy("pageId").aggregate(SUM, "rank")
// apply dampening factor
//**//the output shows error here**
.map { p =>
Page(p.pageId, (p.rank * DAMPENING_FACTOR) + ((1 - DAMPENING_FACTOR) / pages.count()))
}
// terminate if no rank update was significant
val termination = currentRanks.join(newRanks).where("pageId").equalTo("pageId") {
(current, next, out: Collector[Int]) =>
// check for significant update
if (math.abs(current.rank - next.rank) > EPSILON) out.collect(1)
}
(newRanks, termination)
}
val result = finalRanks
// emit result
result.writeAsCsv(outpath, "\n", " ")
env.execute()
}
}
Any help in the right direction is highly appreciated? Thank you.
The problem is that you reference the DataSet pages from within a MapFunction. This is not possible, since a DataSet is only the logical representation of a data flow and cannot be accessed at runtime.
What you have to do to solve this problem is to assign the val pagesCount = pages.count value to a variable pagesCount and refer to this variable in your MapFunction.
What pages.count actually does, is to trigger the execution of the data flow graph, so that the number of elements in pages can be counted. The result is then returned to your program.

use spark run KMeans cluster , program block?

when I use apache spark scala API run the KMeans cluster. my program as follow:
object KMeans {
def closestPoint(p: Vector, centers: Array[Vector]) = {
var index = 0
var bestIndex = 0
var closest = Double.PositiveInfinity
for(i <- 0 until centers.length) {
var tempDist = p.squaredDist(centers(i))
if(tempDist < closest) {
closest = tempDist
bestIndex = i
}
}
bestIndex
}
def parseVector(line: String): Vector = {
new Vector(line.split("\\s+").map(s => s.toDouble))
}
def main(args: Array[String]): Unit = {}
System.setProperty("hadoop.home.dir", "F:/OpenSoft/hadoop-2.2.0")
val sc = new SparkContext("local", "kmeans cluster",
"G:/spark-0.9.0-incubating-bin-hadoop2",
SparkContext.jarOfClass(this.getClass()))
val lines = sc.textFile("G:/testData/synthetic_control.data.txt") // RDD[String]
val count = lines.count
val data = lines.map(parseVector _) // RDD[Vector]
data.foreach(println)
val K = 6
val convergeDist = 0.1
val kPoint = data.takeSample(withReplacement = false, K, 42) // Array[Vector]
kPoint.foreach(println)
var tempDist = 1.0
while(tempDist > convergeDist) {
val closest = data.map(p => (closestPoint(p, kPoint), (p, 1)))
val pointStat = closest.reduceByKey{case ((x1, y1), (x2, y2)) =>
(x1+x2, y1+y2)}
val newKPoint = pointStat.map{pair => (
pair._1,pair._2._1/pair._2._2)}.collectAsMap()
tempDist = 0.0
for(i <- 0 until K) {
tempDist += kPoint(i).squaredDist(newKPoint(i))
}
for(newP <- newKPoint) {
kPoint(newP._1) = newP._2
}
println("Finish iteration (delta=" + tempDist + ")")
}
println("Finish centers: ")
kPoint.foreach(println)
System.exit(0)
}
when I apply run as local mode , the log info as follow:
..................
14/03/31 11:29:15 INFO HadoopRDD: Input split: hdfs://hadoop-01:9000/data/synthetic_control.data:0+288374
program begin block , no running continue........
Can anyone can help me???