coming from my other question is there a way to get by-name-parameters for constructors working? I need a way to provide a code-block which is executed on-demand/lazy/by-name inside an object and this code-block must be able to access the class-methods as if the code-block were part of the class.
Following Testcase fails:
package test
class ByNameCons(code: => Unit) {
def exec() = {
println("pre-code")
code
println("post-code")
}
def meth() = println("method")
def exec2(code2: => Unit) = {
println("pre-code")
code2
println("post-code")
}
}
object ByNameCons {
def main(args: Array[String]): Unit = {
val tst = new ByNameCons {
println("foo")
meth() // knows meth() as code is part of ByNameCons
}
tst.exec() // ByName fails (executed right as constructor)
println("--------")
tst.exec2 { // ByName works
println("foo")
//meth() // does not know meth() as code is NOT part of ByNameCons
}
}
}
Output:
foo
method
pre-code
post-code
--------
pre-code
foo
post-code
This is because when you're making an instance like this:
val tst = new ByNameCons {
...
}
.. you're actually creating an anonymous class, like in java.
The above code is the same as:
val tst = new ByNameCons() { ... }
.. while the correct syntax for passing by-name is:
val tst = new ByNameCons( { ... } )
You cant omit parentheses the same way for constructors as with functions.
val tst = new ByNameCons( {
println("foo")
} )
Thought it is probably just easier to do this:
object ByNameCons {
def apply(code: => Unit) = new ByNameCons(code)
}
val tst = ByNameCons { // no "new" here -- can't mix traits either
println("foo")
}
I dont know why, but it appears that using {} or () when creating the class changes the behavior. Using the following class,
class Coder(code: => Unit) {
def exec = {
println("before")
code
println("after")}
}
}
scala> val brackets = new Coder {println("testing")}
testing
brackets: Coder = $anon$1#1af7a03
scala> brackets exec
before
after
Now instead if defined another way,
scala> val parens = new Coder(println("testing"))
parens: Coder = Coder#dca3ed
scala> parens exec
before
testing
after
as desired. It seems as if in the first notation, the compiler is interpreting the brackets as a block to be evaluated to Unit, instead of an anonymous function which, when called, evaluates to Unit.
FWIW, using ({ ... }) also works fine.
Related
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
I am trying to write a function which can add a context to those functions given in parameters.
The idea is here
object example {
def withOne(f : => T) = {
val a = 1 //some context
f
}
def foo() = withOne {
println(a)
}
}
I think the context could be passed in implicit.
The idea is to not have the content of f constraint by the surrounding function f should be able to use the context or not.
For now the only way i seen to do that is like that
object example {
def withOne(f : => Int => T) = {
val a = 1 //some context
f(a)
}
def foo() = withOne { a =>
println(a)
}
}
But this forces to declare a 'a' witch is not obvious for others devs :x
I'm afraid you cannot work around this, since you cannot inject an implicit into a function.
There's a proposal to add this feature in the typelevel/scala fork, but it seems hard to achieve as of today.
My suggestion here is to use proper naming, so that you won't surprise your users. For instance if you provide a method like:
def withConnection[A](f: Connection => A): A = {
try {
val conn = ???
f(conn)
} finally {
conn.close()
}
}
it won't surprise me to do:
withConnection { implicit c =>
// db stuff
}
I am trying to parse commandline arguments and execute a function that takes the parameters upon successful extraction of the parameters. I have an object called CurrencyExchangeRunner where the main method is. I have envisioned the structure of the class as follows:
object CurrencyExtractionRunner {
def main(args:Array[String]){
parseArgs(args){
(currencyType,currencyTypeArgs) =>
CurrencyExchanger(curencyType,currencyTypeArgs){
(exchanger) => exchanger.startExchange
}
}
}
}
}
What I want to accomplish above is to parse the arguments using parseArgs(args), get the (currencyType,currencyTypeArgs) as parameters and pass those into the CurrencyExchanger factory object and then that would return the appropriate exchanger on which I will execute the startExchange method. This is what I have envisioned but I am a little clueless on how would I go about creating this flow. The first thing I tried was to create a trait that parses the command-line args as follows(I am using the jcommander library for the commandline parse):
object Args {
#Parameter(
names = Array("-h", "--help"), help = true)
var help = false
#Parameter(
names = Array("-c", "--currency-type"),
description = "Type of currency exchange that needs to be performed",
required = true)
var currencyType: String = null
#Parameter(
names = Array("-d", "--denominations"),
description = "Specific denominations to be used during the exchage")
var exchangeDenomination: String = null
#Parameter(
names = Array("-s", "--someotheroptionalarg"),
description = "Additional argument for a specific currency exchange")
var someOtherOptionalArg: String = null
}
trait ParseUtils {
//How do I do this, take the args and return a function.
def parseArgs(args: Array[String]){
val jCommander = new JCommander(Args, args.toArray: _*)
if (Args.help) {
jCommander.usage()
System.exit(0)
}
//What do I do now? How do I proceed with executing the function with
//the specific arguments?
//What do I need to do to wrap the commandline arguments so that it could
//be passed to the next function
}
}
I am pretty stuck here since I am not sure how would I make the code flexible enough to take the arbitrary sequence of commandline args and execute the next step which is the factory that returns that takes these arguments and returns the correct exchanger.
It will be great if someone could point me in the right direction.
I'm not sure why you'd use such unusual syntax to pass return values to the following methods.
I would go for a simpler solution that looks like
trait ParseUtils {
//Why would you return a function here?
//Is it a strict constraint you need to fulfill?
def parseArgs(args: Array[String]): (String, String) {
val jCommander = new JCommander(Args, args.toArray: _*)
if (Args.help) {
jCommander.usage()
System.exit(0)
}
//This is the return value of the method, a pair of parameters
(Args.currencyType, Args.exchangeDenomination)
//If you need to embed additional params, you should append them to existing one
// or you could create optional values from the Args members...
// e.g. (Args.currencyType, Args.exchangeDenomination, Option(Args.someOtherOptionalArg))
// with return type (String, String, Option[String])
}
}
object CurrencyExtractionRunner with ParseUtils {
def main(args:Array[String]){
val (currencyType,currencyTypeArgs) = parseArgs(args)
CurrencyExchanger(currencyType,currencyTypeArgs).startExchange
}
}
case class CurrencyExchanger(currencyType: String, currencyTypeArgs: String) {
def startExchange = //implementation details using the costructor arguments
}
Alternative solution
since I prefer parseArgs to be more "functional" I'd change it to
trait ParseUtils {
def parseArgs(args: Array[String]): Option[(String, String)] {
val jCommander = new JCommander(Args, args.toArray: _*)
if (Args.help) {
jCommander.usage()
None
} else
Some(Args.currencyType, Args.exchangeDenomination)
}
}
object CurrencyExtractionRunner with ParseUtils {
def main(args:Array[String]){
parseArgs(args).foreach {
case (currencyType,currencyTypeArgs) =>
CurrencyExchanger(currencyType,currencyTypeArgs).startExchange
}
}
}
case class CurrencyExchanger(currencyType: String, currencyTypeArgs: String) {
def startExchange = //implementation details using the costructor arguments
}
I am fairly new to Scala macros, and am trying to write a very basic DSL.
I have the following Scala class:
abstract class SpecialFunction {
def apply(): Unit
}
and the following Scala macro:
def mImpl(c: Context)(bodyBlock: c.Expr[Unit]): c.Expr[X] =
c.universe.reify {
new X(new SpecialFunction {
override def apply() {
bodyBlock.splice
}
})
}
def m(bodyBlock: Unit): X = macro mImpl
So far so good. For example, it allows me to write:
def example = m {
println("Hello, world")
}
And this compiles down to:
def example = new X(new SpecialFunction {
override def apply() {
println("Hello, world")
}
})
But this formulation does not allow me to have local variables in such an "m block". For example, I cannot write:
def example = m {
val x = 7
println(x.toString)
}
At compile time I get the error:
symbol value x does not exist in example
However, what I want to achieve is this:
def example = new X(new SpecialFunction {
override def apply() {
val x = 7
println(x.toString)
}
})
(I think I understand why this is the case: sub-expressions are evaluated before being passed to the macro, and thus the reference to x is invalid).
So my question is this: How can I get the above to work? (I just want the extra code defined in the macro to be "copy-pasted" around the code in the "m block", like in a C++ macro.)
Any help would be greatly appreciated :-)
Your output expression is keeping a reference to the old x symbol, but instead it should be a reference to a new one. Therefore, you can reset all the junk references in the tree you aim to reuse by applying resetLocalAttrs from Context.
How about...?
def mImpl(c: Context)(bodyBlock: c.Expr[Unit]): c.Expr[X] =
c.universe.reify {
new X(new SpecialFunction {
override def apply() {
c.Expr[Unit](c.resetLocalAttrs(bodyBlock.tree)).splice
}
})
}
In this question, you can read a great explanation by the Scala Macros guy.
As far as I understand it, Scala creates an anonymous class if I create a class using the new keyword and follow the class name with a constructor:
class MyClass {
def doStuff() {
// ...
}
}
val mc = new MyClass {
doStuff()
}
The nice thing being that all the code in the constructor is in the scope of the new object.
Is there a way I can reproduce this syntax where the class is created by a factory method rather than the new keyword? i.e. make the following code work:
val mf = new MyFactory
val mc = mf.MyClass {
doStuff()
}
I can't find a way to do it but Scala has so much to it that this might be pretty easy!
Using an import as suggested by #Ricky below I can get:
val mf = MyFactory;
val mc = mf.MyClass
{
import mc._
doStuff()
}
(Where the blank line before the block is needed) but that code block is not a constructor.
You can do this, but you still have to keep the new keyword, and create the nested class as a path-dependent type:
class Bippy(x: Int) {
class Bop {
def getIt = x
}
}
val bip = new Bippy(7)
val bop = new bip.Bop
bop.getIt // yields 7
val bop2 = new bip.Bop{ override def getIt = 42 }
bop2.getIt // yields 42
I don't think it's possible. However, a common pattern is to add a parameter to factory methods which takes a function modifying the created object:
trait MyClass {
var name = ""
def doStuff():Unit
}
class Foo extends MyClass {
def doStuff() { println("FOO: " + name) }
}
trait MyClassFactory {
def make: MyClass
def apply( body: MyClass => Unit ) = {
val mc = make
body(mc)
mc
}
}
object FooFactory extends MyClassFactory {
def make = new Foo
}
You can then create and modify instance with a syntax close to your example:
val foo = FooFactory { f=>
f.name = "Joe"
f.doStuff
}
It sounds like you're just looking to mix in a trait. Instead of calling myFactoryMethod(classOf[Foo]] which ideally would do (if Scala permitted it):
new T {
override def toString = "My implementation here."
}
you can instead write
trait MyImplementation {
override def toString = "My implementation here."
}
new Foo with MyImplementation
However, if you are just looking to get the members of the new object accessible without qualification, remember you can import from any stable identifier:
val foo = new Bar
import foo._
println(baz) //where baz is a member of foo.