I am trying to replicate my problem on a smaller example.
I am getting compilation error at the shown location in the following code snippet.
class Outer {
class Inner
}
object OuterUtil {
val obj = new Outer
object xyz extends obj.Inner
//do something with xyz
}
//-------------------
object OuterUtil2 {
var m_obj: Outer = null
def createOuter() = {
m_obj = new Outer
}
def anotherMethod() {
//Compilation error here: stable identifier required,
//but OuterUtil2.this.m_obj found.
object xyz extends m_obj.Inner
}
}
object Test {
OuterUtil2.createOuter
OuterUtil2.anotherMethod
}
OuterUtil is working fine.
In OuterUtil2, I am splitting the functionality into two functions. I am storing the Outer instance m_obj as a member var. The createOuter method creates and stores the Outer instance in m_obj. In anotherMethod, I am getting compilation error. How to fix OuterUtil2?
The prefix of a type (ie, the m_obj in m_obj.Inner) must be a stable value; a var doesn't cut it. You could make that a val and move the initialization out of createOuter.
If you just want to solve the problem with your function, here is a solution (by fixing the var to a val in the function)
def anotherMethod = {
val obj = m_obj
new obj.Inner
}
Another solution would be to use some options, however to overpass the stable identifier method, you have to define a val out of m_obj value. This solution is more elegant because you don't have null pointer exceptions if m_obj is not defined.
object OuterUtil2 {
var m_obj: Option[Outer] = None
def createOuter {
m_obj = Some(new Outer)
}
def anotherMethod = {
m_obj match{
case None => None
case Some(_) => val obj = m_obj.get; Some(new obj.Inner)
}
}
}
Related
Primary goal
I want to use some static vals in a class so that I don't have to pass them as function parameters.
My approach
Since I want them to be static, I am declaring them in the companion object. But I cannot assign them values when I declare them, for some reasons. So I am following the below approach.
case class DemoParams(name: String)
class Demo {
def foo = {
println("Demo params name is: ", Demo.demoParams.name) // Works fine
anotherFoo(Demo.demoParams.name) // Throws NPE !
}
def anotherFoo(someName: String) = {
// some code
}
}
object Demo {
var demoParams: DemoParams = _ // Declare here
def apply() = new Demo()
def run = {
demoParams = DemoParams(name = "Salmon") // Define here
val demoObj = Demo()
demoObj.foo
}
def main() = {
run
}
}
Demo.main()
I am able to print Demo.demoParams but surprisingly, this throws a NullPointerException when I pass Demo.demoParams to another function, while running the Spark app on a cluster.
Questions
Firstly, is this the right way of declaring static values and defining them later? I would prefer to not use vars and use immutable vals. Is there a better alternative?
Second, could you think of any reason I would be getting a NPE while passing Demo.demoParams.name to another function?
Your code works fine and doesn't throw anything (after fixing a few compile errors).
But ... Don't do this, it's ... yucky :/
How about passing params to the class as ... well ... params instead?
case class Demo(params: DemoParams) {
def foo() = {
println("Demo params name is: " + params.name)
}
}
object Demo {
def run() = {
val demoParams = DemoParams(name = "Salmon")
val demoObj = Demo(demoParams)
demoObj.foo()
}
}
Not sure this is the best alternative, but consider using a trait, which still keeps you in the FP zone by avoiding the use of var:
case class DemoParams(name: String)
trait Demo {
val demoParams: DemoParams
}
Then just define it where you need it, and it's ready for use:
object MainApp extends App {
val demoObj = new Demo {
override val demoParams: DemoParams = DemoParams(name = "Salmon")
}
println("Demo params name is: ", demoObj.demoParams.name) // (Demo params name is: ,Salmon)
anotherFoo(demoObj.demoParams.name) // Salmon
def anotherFoo(name: String): Unit = println(name)
}
About the second question, without the actual code one can only guess (this sample code does not throw NPE). Probably somewhere you are using it without defining it previously, because var demoParams: DemoParams = _ just initializes demoParams to the default value of the reference type DemoParams, which is null in this case, and you get NPE when you try to access the name value of a null object. This is why using var is discouraged.
I am new to Scala, trying to understand the syntactic behaviors of Scala. I will be really appreciated if anybody help me. Thanks
With Anonymous Object: Here in this scenario if I want to print the value of resinside the main function body then what logic I need to apply?
package oops
object AnonymousObject
{
def main(args:Array[String]):Unit =
{
new student().detail(5,9) // Line 1
}
}
class student
{
def detail(x:Int, y:Int):Int =
{
val res = x*y
println(res)
}
}
Without Anonymous Object:
For more information, in this scenario given below, there has no problem to achieve it because of var s
class Student
{
var id:Int = 0; // All fields must be initialized
var name:String = null;
}
object MainObject
{
def main(args:Array[String])
{
var s = new Student() // Creating an object
println(s.id+" "+s.name);
}
}
An object which has no reference name. So simply you can print like this way inside the main
object AnonymousObject
{
def main(args:Array[String]):Unit =
{
val res = new student().detail(5,9)
println(res)
}
}
class student
{
def detail(x:Int, y:Int):Int =
{
x*y
}
}
Output: 45
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
I'm learning Scala and am making some Stack implementations as practice. I made this and there are some apparent issues.
class LinkedStack[T] extends Stack[T] {
var current: Node = null
var n: Int = 0
private class Node(e: T, prev: Node) {
val element: T = e
var previous: Node = prev
}
override def pop(): T = {
if (n == 0) {
throw new NoSuchElementException
}
val popNode: Node = current
current = current.previous
popNode.previous = null
n -= 1
popNode.element
}
override def peek(): T = {
if (n == 0) {
throw new NoSuchElementException
}
current.element
}
override def push(element: T): Unit = {
if (element == null) {
throw new NullPointerException
}
val newNode: Node = new Node(element, current)
current = newNode
n += 1
}
override def size(): Int = {
n
}
override def toString(): String = {
val builder = new StringBuilder("Stack top [")
var temp: Node = current
if (n == 0) {
builder.append("]")
return builder.toString()
}
while (temp.previous != null) {
builder.append(temp.element).append(", ")
temp = temp.previous
}
builder.append(temp.element).append("]")
builder.toString()
}
}
The trait includes all of the elements except toString. My main problem is that I'm using null pretty liberally. I know this shouldn't be done at all in Scala, and the line
var current: Node = null
in the constructor generates a compile error. How should I implement a constructor to create an empty stack? What's the best substitution for null?
Edit:
You may have noticed that the Node class should be rewritten as
private class Node(val element: T, var previous: Node) {}
I realized this after reading Rex Kerr's answer. I forgot that I was programming in Scala when I first wrote that.
There's nothing terribly wrong with using null as part of the internals for your class as long as those nulls never leak out and the logic is straightforward enough so you can be sure of that.
But if you want to not use null for practice, you have two choices. One is to use a built-in alternative: instead of Node use Option[Node] and use None for null. Given that your list is invariant, this is the easier way.
Second, you can replace Node with a hierarchy like so:
trait Node
class Elt(val element: T, val previous: Node) extends Node {}
object End extends Node
And then you use End wherever you use null now, and match on the Node any time you need to walk or do something, e.g.
def peek = current match {
case End => throw new NoSuchElementException
case Elt(e, _) => e
}
Of course this means each list has to create an extra object (End), and there are various other drawbacks, most of which can be gotten around to some extent. But for an exercise in avoiding null, you can ignore those complications.
i'm also scala learning my stack implementation ,it;s simple i used scala mutable array buffer
object Stack{
var stack=scala.collection.mutable.ArrayBuffer[Int]()
def isEmpty():Boolean={
if(stack.length==0) true
else false
}
def push(input:Int):Unit={
stack+=input
}
def size():Int={
stack.length
}
def pop():Int={
stack.remove(stack.length-1)
}
def peek()={
stack(stack.length-1)
}
}
This example is from one of the Scala books:
trait IO { self =>
def run: Unit
def ++(io: IO): IO = new IO {
def run = { self.run; io.run }
}
}
object IO {
def empty: IO = new IO { def run = () }
}
The explanation given in the book is as follows:
The self argument lets us refer to this object as self instead of this.
What would that statement mean?
self is just an alias to this in the object in which it is declared, and it can be any valid identifier (but not this, otherwise no alias is made). So self can be used to reference this from an outer object from within an inner object, where this would otherwise mean something different. Perhaps this example will clear things up:
trait Outer { self =>
val a = 1
def thisA = this.a // this refers to an instance of Outer
def selfA = self.a // self is just an alias for this (instance of Outer)
object Inner {
val a = 2
def thisA = this.a // this refers to an instance of Inner (this object)
def selfA = self.a // self is still an alias for this (instance of Outer)
}
}
object Outer extends Outer
Outer.a // 1
Outer.thisA // 1
Outer.selfA // 1
Outer.Inner.a // 2
Outer.Inner.thisA // 2
Outer.Inner.selfA // 1 *** From `Outer`