I am a beginner of Scala.
I immediately get a problem.
Let's say:
I have a Vector variable and 2 functions. The first one function is calling 2nd function. And there is a variable in 2nd function is what I need to get and then append it to Vector. (Without returning the variable in 2nd function.)
The structure looks like this:
def main(args: Array[String]): Unit = {
var vectorA = Vector().empty
}
def funcA(): sometype = {
...
...
...
funcB()
}
def funcB(): sometype = {
var error = 87
}
How can I add error variable in global Vector?
I tried to write vectorA :+ error but in vain.
You could do the following:
def main(args: Array[String]): Unit = {
val vectorA = funcA(Vector.empty)
}
def funcA(vec: Vector): sometype = {
...
...
...
funcB()
}
def funcB(vec: Vector): sometype = {
// Here you could append, which returns a new copy of the Vector
val error = 87
vec :+ error
}
Keep in mind that, it is advisable to use immutable variables. Though not always this might be true, but for most of the applications that just involve doing some CRUD type logic, it is better to use immutable variables.
Related
I have an object called Program which can only instantiate only one class, in my case val a = new A(), in it I have a local variable called localText, I want to convert it to global, I have created a variable outside the block called globalText and inside the block I assign it globalText = localText, when I do a println it shows it correctly "Hello World", I have another class called B that needs that value but I don't know how to access it because when I call it it appears with the null value and if I call the sum function what it returns is an Int. (The sum() def cannot be touched and the data comes like this from third party code) thank you very much for the help.
...
class A{
var globalText: String = _
def sum(): Int = {
val localText: String = "Hello World"
globalText = localText
println(globalText)
val b = new B()
3+4
}
println(globalText)
}
class B extends A{
var globalBText = globalText
println(globalBText)
}
object Program {
def main(args: Array[String]): Unit = {
val a = new A()
println(a.sum())
}
}
Output is
null
Hello World
null
null
7
...
This question has already been answered.
Please do not create duplicate posts with the same question. You can edit the previous question, if you feel like you missed something.
If you can't instantiate the class than this beats the whole purpose of being a class and you might as well model it as an object or create another object.
The idea is that a class must be instantiated in order for its fields to be initialized. There is no other way.
You could use an object so you don't have to instantiate any class at all, making sure that B.globalBText is initialized before you used it.
Here's what I mean:
object A {
var globalText: String = _
def sum(): Int = {
globalText = "Hello World"
3 + 4
}
}
object B {
var globalBText: String = {
A.sum()
A.globalText
}
println(globalBText)
}
object Program {
def main(args: Array[String]): Unit = {
println(B.globalBText)
}
}
Outputs:
Hello World
Hello World
Below here the code snippet, where I want to avoid using the 'var'. Not sure if there is a good way to do that
var randomInt = Random.nextInt(100)
private def getRandomInt(createNew:Boolean):Int = {
if(createNew){
randomInt = Random.nextInt(100)
}
randomInt
}
Create an "infinite" Iterator of random numbers. Advance to the next() only when needed.
val randomInt = Iterator.continually(Random.nextInt(100)).buffered
private def getRandomInt(createNew:Boolean):Int = {
if (createNew) randomInt.next()
randomInt.head
}
The class below holds the current random value and provides a method to return an instance holding the next random value.
It only uses immutable values, although the Random.nextInt(...) function isn't pure, because it doesn't return the same result for the same input.
The class is a direct translation of your 3 requirements:
to retrieve the previously generated number.
to generate a new number.
avoid using the 'var'.
This shows the basic technique of returning a new immutable instance instead of mutating a variable, although I find the infinite iterator answer by jwvh to be a more elegant solution.
import scala.util.Random
// A random number generator that remembers its current value.
case class RandomInt(size: Int) {
val value = Random.nextInt(size)
def nextRandomInt(): RandomInt = RandomInt(size)
}
// Test case
object RandomInt {
def main(args: Array[String]): Unit = {
val r0 = RandomInt(100)
(0 to 99).foldLeft(r0)((r, i) => {
println(s"[$i] ${r.value}")
r.nextRandomInt()
})
}
}
Not sure what changes you are open too, you can simply just add a parameter
val randomInt = Random.nextInt(100)
private def getRandomInt(createNew:Boolean,previous:Int):Int = {
if(createNew) Random.nextInt(100) else previous
}
In general, can you declare a val in scala before assigning it a value? If not, why not? An example where this might be useful (in my case at least) is that I want to declare a val which will be available in a larger scope than when I assign it. If I cannot do this, how can I achieve the desired behavior?
And I want this to be a val, not a var because after it is assigned, it should NEVER change, so a var isn't ideal.
For example:
object SomeObject {
val theValIWantToDeclare // I don't have enough info to assign it here
def main(): Unit = {
theValIWantToDeclare = "some value"
}
def someOtherFunc(): Unit {
val blah = someOperationWith(theValIWantToDeclare)
}
}
object SomeObject {
private var tviwtdPromise: Option[Int] = None
lazy val theValIWantToDeclare: Int = tviwtdPromise.get
private def declareTheVal(v: Int): Unit = {
tviwtdPromise = Some(v)
theValIWantToDeclare
}
def main(args: Array[String]): Unit = {
declareTheVal(42)
someOtherFunction()
}
def someOtherFunction(): Unit = {
println(theValIWantToDeclare)
}
}
It will crash with a NoSuchElementException if you try to use theValIWantToDeclare before fulfilling the "promise" with declareTheVal.
It sounds to me that you need a lazy val.
A lazy val is populated on demand and the result is cached for all subsequent calls.
https://blog.codecentric.de/en/2016/02/lazy-vals-scala-look-hood/
Why not define a SomeObjectPartial that is partially constructed, and class SomeObject(theVal) that takes the value as a parameter?
Then your program has two states, one with the partial object, and another with the completed object.
I have a singleton object Main, and defined private var
private var params: Broadcast[Parameters] = _
Then I am initializing it through the main method
def main(args: Array[String]) = {
...
params = spark.sparkContext.broadcast(new Parameters())
I have a method inside this object, then pass rdd argument and trying to get a value from broadcast variable in map transformation.
def testMethod(rdd: RDD[String]): Unit = {
rdd.map(elem -> {
val currentDate = params.value.CURRENT_DATE
...
Got an error on line where I try to get value grom params: NullPointerException. How to fix this code to make it work? I can't understand what is wrong with initialization.
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}