How to claculate adjacent data with spark/scala - scala

I hava a RDD,the RDD type is Tuple2(value,timestamp),value is 1 or 0 , timestamp is a sequential, and a variable limitTime=4. When I map the RDD, if the value is 1 the output value from current timestamp to (timestamp +limitTime) is 1 ,else current value is 0, I call it is period. But there is a special case，when the value is 1 and its timestamp is in period，then it is ignored, the current value of output is 0
input : (0,0),(1,1),(0,3),(0,5),(0,7),(0,8),(0,10),(1,12),(0,14),(0,15)
expected output :(0,0),(1,1),(1,3),(1,5),(0,7),(0,8),(0,10),(1,12),(1,14),(1,15)
special input2: (0,0),(1,1),(0,3),(1,5),(0,7),(1,8),(0,10),(1,12),(0,14),(0,15)
expected output2:(0,0),(1,1),(1,3),(1,5),(0,7),(1,8),(1,10),(1,12),(0,14),(0,15)
this is my try:
var limitTime=4
var startTime= -limitTime
val rdd=sc.parallelize(List((0,0),(1,1),(0,3),(1,5),(0,7),(1,8),(0,10),(1,12),(0,14),(0,15)),4)
val results=rdd.mapPartitions(parIter => {
var resultIter = new ArrayBuffer[Tuple2[Int,Int]]()
while (parIter.hasNext) {
val iter = parIter.next()
if(iter._1==1){
if(iter._2<=startTime+limitTime&&iter._2!=0&&iter._2>=startTime){
resultIter.append(iter)
}else{
resultIter.append(iter)
startTime=iter._2
}
}else{
if(iter._2<=startTime+limitTime&&iter._2!=0&&iter._2>=startTime){
resultIter.append((1,iter._2))
}else{
resultIter.append(iter)
}
}
}
resultIter.toIterator
})
results.collect().foreach(println)
it's so inefficient，how can I get the same result without array?


Following code should work for both of your cases.
var limitTime=3
var first = true
var previousValue = 0
val rdd=sc.parallelize(List((0,0),(1,1),(0,3),(0,5),(0,7),(0,8),(0,10),(1,12),(0,14),(0,15)), 4)
val tempResult = rdd.collect.map(pair => {
if(first){
first = false
previousValue = pair._1
(pair._1, pair._2)
}
else {
if ((pair._1 == 1 || previousValue == 1) && limitTime > 0) {
limitTime -= 1
previousValue = 1
(1, pair._2)
}
else {
if (limitTime == 0) limitTime = 3
previousValue = pair._1
(pair._1, pair._2)
}
}
})
tempResult.foreach(print)
If it doesn't please let me know

Related

Continue in Scala for loops

How do i convert below Java code to scala and use continue in for loop, this program remove minimum number of extra closed parenthesis in a given string
input : lee(t(c)o)de)
output : leet(t(c)o)de
public String minRemoveToMakeValid(String s){
StringBuilder sb =new StringBuilder();
in open =0;
for (char c : s.toCharArray()){
if(c == '('){
open++
}
else if(c == ')'){
if(open == 0) continue;
open --;
}
sb.append(c)
}
return sb
}
https://leetcode.com/problems/minimum-remove-to-make-valid-parentheses/

import util.control.Breaks._
val searchMe = "peter piper picked a peck of pickled peppers"
var numPs = 0
for (i <- 0 until searchMe.length) {
breakable {
if (searchMe.charAt(i) != 'p') {
break // break out of the 'breakable', continue the outside loop
} else {
numPs += 1
}
}
}
println("Found " + numPs + " p's in the string.")
Try it: https://scastie.scala-lang.org/R9sr95WESLyiKamCHHUVdQ

Im able to get it worked using below code
def minRemoveToMakeValid(s: String): String = {
var open = 0
val sb = new StringBuilder
for (c <- s.toCharArray) {
breakable {
if (c == '(') open += 1
else if (c == ')') {
if (open == 0)break
open -= 1
}
sb.append(c)
}
}
var result = new StringBuilder()
for(i<-sb.length()-1 to 0 by -1)
{
breakable{
open-=1
if(sb.charAt(i) == '(' && open >0) break
result.append(sb.charAt(i))
}
}
result.reverse.toString()
}

Access ArrayList outside nested If else block in Scala

I have following code :-
var ArrayStop = new ArrayList[imeistoppage]()
val listings = rddStopCalculate.zipWithIndex().map(p => {
if (p._2 == 0) {
imei = p._1.imei
lat = p._1.latitude
long = p._1.longitude
gpsdt = p._1.gpsdt
} else if (p._2 > 0 && p._2 != lastindex) {
if (p._1.imei.equals(imei) && p._1.latitude == lat && p._1.longitude == long) {
flag += 1
newgpsdt = p._1.gpsdt
} else {
if (flag > 0) {
timeDiff = newgpsdt.getTime() - gpsdt.getTime()
if (timeDiff > 60000) {
ArrayStop.add(imeistoppage(p._1.imei, lat, long, timeDiff))
}
flag = 0
}
imei = p._1.imei
lat = p._1.latitude
long = p._1.longitude
gpsdt = p._1.gpsdt
}
} else {
if (p._1.imei.equals(imei) && p._1.latitude == lat && p._1.longitude == long) {
flag += 1
newgpsdt = p._1.gpsdt
}
if (flag > 0) {
timeDiff = newgpsdt.getTime() - gpsdt.getTime()
if (timeDiff > 60000) {
ArrayStop.add(imeistoppage(p._1.imei, lat, long, timeDiff))
}
flag = 0
}
}
ArrayStop
}).collect()
val returnList = listings(listings.length - 1)
val tempCollection = returnList.asScala
val tempRDD = sc.parallelize(tempCollection)
tempRDD.saveToCassandra("db", "table", SomeColumns("imei", "lat", "long", "duration"))
As we can see in above code I'm actually adding data to ArrayStop for specific IF condition only and I want this to be accessed outside this rdd loop but I was unable to do so So I created a variable "listings" to store data which is actually taking all the rows whereas I want only those entries which are added in ArrayStop. So what is the best way to bring any array outside nested If-else block. This is different from this issue
Scala spark, listbuffer is empty
Thanks,

How to define count without using var

I have this simple function that return some Status:
def getStatus : String =
{
//...
}
And i want to wait until this return specific status but still count this number of calls before exit:
def wait =
{
var count = 0
while (getStatus != "smeStatus" && count < 10) {
// some code here
count++
}
}
How can i avoid of using var ?

You can use a recursive method that takes the count-so-far and returns it + 1:
def waitUntilDone(countSoFar: Int): Int = {
if (getStatus != "smeStatus" && countSoFar < 10) {
// some code here
waitUntilDone(countSoFar + 1)
} else {
countSoFar
}
}
// invoke it starting with 0:
val count = waitUntilDone(0)

def status = ""
def waiting: Unit = {
def check(count: Int): Unit = {
if(count < 10 && status != "smeStatus") {
// do something
check(count + 1)
}
}
check(0)
}

Return list with the values on odd positions

Why does the following code return a empty list instead of a list with the values at odd positions ?
def f(arr:List[Int]) : List[Int] = {
def odd_concat(list_odd:List[Int], arr_index:Int) : List[Int] = {
if(arr_index == arr.size) {
list_odd
}
else if(arr_index % 2 == 0) {
odd_concat(list_odd, arr_index + 1)
}
else {
//println(arr(arr_index))
list_odd:+arr(arr_index)
odd_concat(list_odd, arr_index + 1)
}
}
odd_concat(List(), 0)
}

You are using immutable list, immutable means the object cannot be change.
Your code:
list_odd:+arr(arr_index)
It does not change the list_odd with the value of arr(arr_index) rather give a new instance of List with values added.
Try to insert that code inside the odd_concat() instead, like the following:
def f(arr:List[Int]) : List[Int] = {
def odd_concat(list_odd:List[Int], arr_index:Int) : List[Int] = {
if(arr_index == arr.size) {
list_odd
}
else if(arr_index % 2 == 0) {
odd_concat(list_odd, arr_index + 1)
}
else {
//println(arr(arr_index))
odd_concat(list_odd:+arr(arr_index), arr_index + 1)
}
}
odd_concat(List(), 0)
}

Connect-Four Game in Scala

I have to make a connect-four game using scala. I have attached the code but everytime the game runs and gets to row 3 it just continues to change the second rows entry instead of going to the next row. Any help would be appreciated. I found this code on another thread on here and couldn't figure out how to get it to work:
// makes the board
val table = Array.fill(9,8)('-')
var i = 0;
while(i < 8){
table(8)(i) = (i+'0').toChar
i = i+1;
}
// prints starting board
def printBoard(table: Array[Array[Char]]) {
table.foreach( x => println(x.mkString(" ")))
}
//player 1 moves
def playerMove1(){
val move = readInt
var currentRow1 = 7
while (currentRow1 >= 0)
if (table(currentRow1)(move) != ('-')) {
currentRow1 = (currentRow1-1)
table(currentRow1)(move) = ('X')
return (player2)}
} else {
table(currentRow1)(move) = ('X')
return (player2)
}
}
//player 2 moves
def playerMove2(){
val move = readInt
var currentRow2 = 7
while (currentRow2 >= 0)
if (table(currentRow2)(move) != ('-')) {
currentRow2 = (currentRow2-1)
table(currentRow2)(move) = ('O')
return (player1)}
} else {
table(currentRow2)(move) = ('O')
return (player1)
}
}
//player 1
def player1(){
printBoard(table)
println("Player 1 it is your turn. Choose a column 0-7")
playerMove1()
}
//player 2
def player2(){
printBoard(table)
println("Player 2 it is your turn. Choose a column 0-7")
playerMove2()
}
for (turn <- 1 to 32){
player1
player2
}

I've tried to make your code readable and compiling and also tried to fix some logic.
However, I've never worked with Scala so this is just a first sketch where you might want to continue ...
Some functions can be merged and the currentRow needed a fix. See here:
object ConnectFour{
val table = Array.fill(9,8)('-')
val currentRow = Array.fill(8)(8)
def main(args: Array[String]) {
var i = 0;
while(i < 8) {
table(8)(i) = (i+'0').toChar
i = i+1;
}
player(1)
}
def printBoard(table: Array[Array[Char]]) {
table.foreach( x => println(x.mkString(" ")))
}
def player(playerNr : Int){
printBoard(table)
println("Player " + playerNr + " it is your turn. Choose a column 0-7")
var column = readAndVerifyInt
var nextUser = 1 : Int
var symbol = 'O' : Char
if(playerNr == 1) {
symbol = 'X'
nextUser = 2
}
var curR = currentRow(column)
while (curR >= 0) {
if (table(curR)(column) != ('-')) {
curR = curR-1
currentRow(column) = curR
}
table(curR)(column) = symbol
player(nextUser)
}
}
def readAndVerifyInt() : Int = {
var column = readInt
if (column >= 0 && column <= 7) {
return column
} else {
println(" > Please try again")
return readAndVerifyInt
}
}
}