Assume I have a variable x which receives its value from the user at some point. Once that is done, I need to set up an Object which needs the value of x.
Naively, I'd like to write:
Object MyCoolObject(num:Double) {
//code
}
and then somewhere in the code:
val MCO = MyCoolObject(x)
But that's not possible in Scala. So how do I do it?
This is already discussed here: Pass Parameters to Scala Object
You can also use a case class:
case class MyCoolObject(num: Double)
{
//code
}
val n = 10 // external data
val myCoolObject = MyCoolObject(n)
Something like this:
class MyCoolObject(num:Double) {
}
object MyCoolObject{
def apply(x:Double) = new MyCoolObject(x)
}
val x : Double = 56.1
val MCO = MyCoolObject(x)
You can use this article i.e.
https://twitter.github.io/scala_school/basics2.html
Related
Primary goal
I want to use some static vals in a class so that I don't have to pass them as function parameters.
My approach
Since I want them to be static, I am declaring them in the companion object. But I cannot assign them values when I declare them, for some reasons. So I am following the below approach.
case class DemoParams(name: String)
class Demo {
def foo = {
println("Demo params name is: ", Demo.demoParams.name) // Works fine
anotherFoo(Demo.demoParams.name) // Throws NPE !
}
def anotherFoo(someName: String) = {
// some code
}
}
object Demo {
var demoParams: DemoParams = _ // Declare here
def apply() = new Demo()
def run = {
demoParams = DemoParams(name = "Salmon") // Define here
val demoObj = Demo()
demoObj.foo
}
def main() = {
run
}
}
Demo.main()
I am able to print Demo.demoParams but surprisingly, this throws a NullPointerException when I pass Demo.demoParams to another function, while running the Spark app on a cluster.
Questions
Firstly, is this the right way of declaring static values and defining them later? I would prefer to not use vars and use immutable vals. Is there a better alternative?
Second, could you think of any reason I would be getting a NPE while passing Demo.demoParams.name to another function?
Your code works fine and doesn't throw anything (after fixing a few compile errors).
But ... Don't do this, it's ... yucky :/
How about passing params to the class as ... well ... params instead?
case class Demo(params: DemoParams) {
def foo() = {
println("Demo params name is: " + params.name)
}
}
object Demo {
def run() = {
val demoParams = DemoParams(name = "Salmon")
val demoObj = Demo(demoParams)
demoObj.foo()
}
}
Not sure this is the best alternative, but consider using a trait, which still keeps you in the FP zone by avoiding the use of var:
case class DemoParams(name: String)
trait Demo {
val demoParams: DemoParams
}
Then just define it where you need it, and it's ready for use:
object MainApp extends App {
val demoObj = new Demo {
override val demoParams: DemoParams = DemoParams(name = "Salmon")
}
println("Demo params name is: ", demoObj.demoParams.name) // (Demo params name is: ,Salmon)
anotherFoo(demoObj.demoParams.name) // Salmon
def anotherFoo(name: String): Unit = println(name)
}
About the second question, without the actual code one can only guess (this sample code does not throw NPE). Probably somewhere you are using it without defining it previously, because var demoParams: DemoParams = _ just initializes demoParams to the default value of the reference type DemoParams, which is null in this case, and you get NPE when you try to access the name value of a null object. This is why using var is discouraged.
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
Is there a way to share a variable among all objects (instantiated from the same type)? Consider the following simple program. Two objects name and name2 have the same type A. Is there way to connect the properyList inside the two instantiation name and name2?
class A {
var properyList = List[String]()
def printProperties(): Unit = {
println(properyList)
}
}
object Experiment {
def main(args: Array[String]): Unit = {
val name = new A
val name2 = new A
name.properyList = List("a property")
name.printProperties()
name2.printProperties()
}
}
The output is
List(a property)
List()
Any way to change the class definition so that by just changing the .properyList in one of the objects, it is changed in all of the instatiations?
What you seem to be looking for is a class variable. Before I get into why you should avoid this, let me explain how you can do it:
You can attach propertyList to the companion object instead of the class:
object A {
var properyList = List[String]()
}
class A {
def printProperties(): Unit = {
println(A.properyList)
}
}
Now, to the why you shouldn't:
While scala let's you do pretty much anything that the JVM is capable of, its aims are to encourage a functional programming style, which generally eschews mutable state, especially shared, mutable state. I.e. the anti-pattern in A is not only that propertyList is a var, not a val but by sharing it via the companion object, you further allow anyone, from any thread to change the state of all instances at anytime.
The benefit of declaring your data as val is that you can safely pass it around, since you can be sure that nobody can change from under you at any time in the future.
You seem to be looking for something like java static fields.
In scala you usually achieve something like that by using a companion object:
object Main extends App {
class A {
import A._
def printProperties(): Unit = {
println(properyList)
}
}
object A {
private var properyList = List[String]()
def addProperty(prop: String): Unit = {
properyList ::= prop
}
}
val name = new A
val name2 = new A
A.addProperty("a property")
name.printProperties()
name2.printProperties()
}
If you want to have something similar to java's static fields you will have to use companion objects.
object Foo {
private var counter = 0
private def increment = {
counter += 1;
counter
}
}
class Foo {
val i = Foo.increment
println(i)
}
Code copied from:
"Static" field in Scala companion object
http://daily-scala.blogspot.com/2009/09/companion-object.html
Based on Arne Claassen's answer, but using private mutable collection with the companion object, which makes it visible only to the companion classes. Very simplistic example tried out in scala 2.11.7 console:
scala> :paste
// Entering paste mode (ctrl-D to finish)
object A {
private val mp = scala.collection.mutable.Map("a"->1)
}
class A {
def addToMap(key:String, value:Int) = { A.mp += (key -> value) }
def getValue(key:String) = A.mp.get(key)
}
// Exiting paste mode, now interpreting.
defined object A
defined class A
// create a class instance, verify it can access private map in object
scala> val a = new A
a: A = A#6fddee1d
scala> a.getValue("a")
res1: Option[Int] = Some(1)
// create another instance and use it to change the map
scala> val b = new A
b: A = A#5e36f335
scala> b.addToMap("b", 2)
res2: scala.collection.mutable.Map[String,Int] = Map(b -> 2, a -> 1)
// verify that we cannot access the map directly
scala> A.mp // this will fail
<console>:12: error: value mp is not a member of object A
A.mp
^
// verify that the previously created instance sees the updated map
scala> a.getValue("b")
res4: Option[Int] = Some(2)
I try to define a parametric type alias :
case class A
case class B
case class C
// We need an Int to load instances of A and B, and a String to load C
object Service {
def loadA(i: Int) : A = ???
def loadB(i: Int) : B = ???
def loadC(s: String) : C = ???
}
trait Location[T] { def get : T}
class IntLocation(val i: Int)
class StringLocation(val s: String)
trait EntityLocation[E] extends Location[_]
// Aim : make the loader typesafe
// Problem : I need something like that : type EntityLocation[Composite] = IntLocation
object Family {
trait EntityLoader[EntityT] extends (EntityLocation[EntityT] => EntityT)
val ALoader = new EntityLoader[A] {def load[A](l: EntityLocation[A]) = Service.loadA(l.get)
}
I am not sure what you are trying to achieve here. Could you please explain how you want to use these types in your code?
Assuming just want to use the types IdLocation and FileLocation in your code, maybe you want to try
trait Location[T] { def get : T }
type IdLocation = Location[Id]
type FileLocation = Location[java.io.File]
Seems rather convoluted, so I'm not sure I follow exactly what your purpose here is. You seem to go into many layers of factories that create factories, that call factory methods, etc.
Seems to me that at the end of the day you need you want to have a val ALoader value that you can use to get instances of A from Location[Int] objects, so I'll go with that assumption:
// Not sure what you want this one, but let's assume that you need a wrapper class per your example.
trait Location[P] { def get: P }
class IntLocation(val i: Int) extends Location[Int]
{
override def get: Int = i
}
// P for parameter, O for output class.
def loader[O, P](creator: P => O)(param: Location[P]) = { creator(param.get) }
object Service
{
// A function somewhere, capable of taking your parameter and creating something else (in your example, an Int to an 'A')
// here Int to String to make something concrete.
// This could be any function, anywhere
def loadA(someParam: Int) = someParam.toString
}
def main(args: Array[String])
{
val myStringLoader: Location[Int] => String = loader(Service.loadA)
// Alternatively, you could have written `val myStringLoader = loader(Service.loadA)(_)`. Either the type or the underscore are needed to tell the compiler that you expect a function, not a value.
// Some definition for you wrapper class
val location3 = new Location[Int]{
override def get: Int = 3
}
// ... or just a plain old instance of it.
val otherLocation = new IntLocation(5)
// This would 'load' the kind of thing you want using the method you specified.
val myString = myStringLoader(location3)
val myOtherString = myStringLoader(otherLocation)
// This prints "3 - 5"
print(myString + " - " + myOtherString)
}
This might seem like a long answer, but in truth the line def loader[O, P](creator: P => O)(param: Location[P]) = { creator(param.get) } is the one that does it all, the rest is to make it as similar to your sample as possible and to provide a working main you can use to start from.
Of course, this would be even simpler if you don't really need the Location wrapper for your integer.
I'm looking to create a way to dynamically call logic depending on template id within scala. So template id 1 calls logic a, template id 2 call logic b, etc. The logic will be diverse but will have the same inputs/outputs. Also the number of different template ids will get into the thousands and will not be known ahead of time, so a loose coupling feels the way to go.
I've started looking at reflection to do this using scala 2.11.1 and can statically use reflection when I know the logic to be used ahead of time but have not found the correct way to dynamically use reflection, so for example passing in template id 2 will call logic b.
Below is a cut down example showing how the static version works and the skeleton I have so far for the dynamic version.
package thePackage
import scala.reflect.runtime.{universe => ru}
trait theTrait { def theMethod(x: String): Unit }
// the different logic held in different objects
object object1 extends theTrait {
def theMethod(x: String) = { println("a " + x ) }
}
object object2 extends theTrait {
def theMethod(x: String) = { println("b " + x ) }
}
object object3 extends theTrait {
def theMethod(x: String) = { println("c " + x ) }
}
// run static/dynamic reflection methods
object ReflectionTest {
// "static" invocation calling object1.theMethod
def staticInvocation() = {
val m = ru.runtimeMirror(getClass.getClassLoader)
val im = m.reflect(thePackage.object1)
val method = ru.typeOf[thePackage.object1.type]
.decl(ru.TermName("theMethod")).asMethod
val methodRun = im.reflectMethod(method)
methodRun("test")
}
staticInvocation
// "dynamic" invocation using integer to call different methods
def dynamicInvocation( y: Integer) = {
val m = ru.runtimeMirror(getClass.getClassLoader)
val module = m.staticModule("thePackage.object" + y)
val im = m.reflectModule(module)
// stuck... static approach does not work here
}
dynamicInvocation(1)
dynamicInvocation(2)
dynamicInvocation(3)
}
What needs to be added/changed to the dynamicInvocation method to make this work, or should I be using a different approach?
You need to get an instance mirror for your module, on which you can reflect the method.
def dynamicInvocation( y: Integer) = {
val m = ru.runtimeMirror(getClass.getClassLoader)
val module = m.staticModule("thePackage.object" + y)
val im = m.reflectModule(module)
val method = im.symbol.info.decl(ru.TermName("theMethod")).asMethod
val objMirror = m.reflect(im.instance)
objMirror.reflectMethod(method)("test")
}
It seems that TermName method in above solution has been replaced by newTermName and also the info.decl seems to not work. Below line worked for me
val method = im.symbol.typeSignature.member(ru.newTermName("testMethod")).asMethod