I am fairly new to Scala macros, and am trying to write a very basic DSL.
I have the following Scala class:
abstract class SpecialFunction {
def apply(): Unit
}
and the following Scala macro:
def mImpl(c: Context)(bodyBlock: c.Expr[Unit]): c.Expr[X] =
c.universe.reify {
new X(new SpecialFunction {
override def apply() {
bodyBlock.splice
}
})
}
def m(bodyBlock: Unit): X = macro mImpl
So far so good. For example, it allows me to write:
def example = m {
println("Hello, world")
}
And this compiles down to:
def example = new X(new SpecialFunction {
override def apply() {
println("Hello, world")
}
})
But this formulation does not allow me to have local variables in such an "m block". For example, I cannot write:
def example = m {
val x = 7
println(x.toString)
}
At compile time I get the error:
symbol value x does not exist in example
However, what I want to achieve is this:
def example = new X(new SpecialFunction {
override def apply() {
val x = 7
println(x.toString)
}
})
(I think I understand why this is the case: sub-expressions are evaluated before being passed to the macro, and thus the reference to x is invalid).
So my question is this: How can I get the above to work? (I just want the extra code defined in the macro to be "copy-pasted" around the code in the "m block", like in a C++ macro.)
Any help would be greatly appreciated :-)
Your output expression is keeping a reference to the old x symbol, but instead it should be a reference to a new one. Therefore, you can reset all the junk references in the tree you aim to reuse by applying resetLocalAttrs from Context.
How about...?
def mImpl(c: Context)(bodyBlock: c.Expr[Unit]): c.Expr[X] =
c.universe.reify {
new X(new SpecialFunction {
override def apply() {
c.Expr[Unit](c.resetLocalAttrs(bodyBlock.tree)).splice
}
})
}
In this question, you can read a great explanation by the Scala Macros guy.
Related
In JS, I can access the function arguments through the arguments object.
I want to do the same thing somehow in ScalaJS, because I want to do some logging for functions and what parameters they got.
Is it possible?
Hmm -- interesting question. I honestly don't know if there is a way to access arguments per se. I would probably address the desire for logging the function arguments by using the sourcecode library, which is designed for that sort of thing, but I'll admit that I haven't tried that from Scala.js yet...
You cannot directly access the arguments object in Scala.js. However, you can export a method with varargs and it will work as expected in JavaScript:
object Logger {
#JSExportTopLevel("log")
def log(xs: js.Any*): Unit = {
xs.foreach(println)
}
}
This defines and exports log to the top level scope. In the JavaScript code, you can now call:
log(1, {}, {a: 1}, "foo");
As #justin-du-coeur suggested, you can use sourcecode for this. For example:
object Test extends js.JSApp {
def main(): Unit = {
a(1, "a")
b()
c("foo", "bar", "baz")
}
def trace()(implicit name: sourcecode.Name, args: sourcecode.Args): Unit = {
def makeArgList(as: Seq[sourcecode.Text[_]]): String =
as.map(a => f"${a.source} = ${a.value}").mkString("(", ", ", ")")
val argStr = args.value.map(makeArgList).mkString("")
println(f"${name.value}$argStr")
}
def a(arg1: Int, arg2: String): Unit = {
trace()
}
def b(): Unit = {
trace()
}
def c(x: String*): Unit = {
trace()
}
}
The output is as follows:
[info] Running Test
a(arg1 = 1, arg2 = a)
b()
c(x = WrappedArray(foo, bar, baz))
As you can see, trace can capture everything it needs from the context, so the result is even more boilerplate free than any JS solution I can think of.
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
Edit2
Okay, so maybe I should parse out two desires here.
I had in mind that when it came time to get to the setSendTimeout(0) part, I would be using something like implicitly[Socket].
new ZContext(1) {
createSocket(ZMQ.PUB).setSendTimeout(0).//RATS!
}
I also had in mind a more generic approach to it, that would be (in pseudo code terms):
This is how you wrap a reference of T at a point in time without copying it, so that moving forward, you can tease out state of the reference after potential state changes from the value of whatever expression used it.
If it could be thought of as a chain of map map map s from T to where ever it ended up, then it is easy to append / apply a value onto it - just map again...
This is the motivating example.
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) {
logger.info("Initializing ZMQ context.")
val context = new ZContext(1)
logger.info(s"Binding PUB socket to ${endpoint}")
val socket = {
val s = context.createSocket(ZMQ.PUB)
s.setSendTimeOut(0)
s.bind(endpoint)
s
}
Look at socket down there. For some reason that feels uglier than it needs to be to me, but is a consequence of the fact that setters don't return stuff like setSendTimeOut.
I would normally try to improve it as follows:
new ZContext(1) {
createSocket(ZMQ.PUB).setSendTimeout(0).//RATS!
}
Here a version of #Dima's answer. Again the setup:
trait Instance {
def createPort(): Port
}
trait Port {
def makeSpecial(): Unit
def bindTo(address: Any): Unit
}
trait Provider {
def getTheInstance(i: Int): Instance
}
Now the trick:
implicit class InstanceOps(i: Instance) {
def withCreatePort(fun: (Unit => Port) => Any): Port = {
val res = i.createPort()
fun(_ => res)
res
}
}
And if you add an implicit modifier to the argument of the function passed into withCreatePort, you "import" the implicit conversion:
trait ConnectTest extends Provider {
getTheInstance(2).withCreatePort { implicit p =>
().makeSpecial().bindTo("foo")
}
}
This is potentially more dangerous, because you have an implicit conversion from Unit to Port, although it is locally encapsulated. This is generic because Connect is generic.
This trick is perhaps too clever and difficult to understand by some outside standing person reading your code.
Yes, you can create two wrappers, one giving you withCreatePort, the other giving you variants of the port method that return this:
trait Instance {
def createPort(): Port
}
trait Port {
def makeSpecial(): Unit
def bindTo(address: Any): Unit
}
class PortOps(p: Port) {
def makeSpecial() : this.type = { p.makeSpecial() ; this }
def bindTo(address: Any): this.type = { p.bindTo(address); this }
}
implicit class InstanceOps(i: Instance) {
def withCreatePort[A](fun: PortOps => A): A = fun(new PortOps(i.createPort()))
}
Example:
trait Provider {
def getTheInstance(i: Int): Instance
}
trait Plain extends Provider {
val instance = getTheInstance(2)
val port = instance.createPort()
port.makeSpecial()
port.bindTo("foo")
}
trait Rich extends Provider {
getTheInstance(2).withCreatePort { p =>
p.makeSpecial().bindTo("foo")
}
}
The question is if the effort is worth it. You can also experiment with import:
trait Import extends Provider {
val instance = getTheInstance(2)
val port = instance.createPort()
locally {
import port._
makeSpecial(); bindTo("foo")
}
}
I am not sure where you are going with this Zipped thingy ... But what you described in the beginning of your question (assuming that port in the end of that snippet is a typo, and you really meant to return instance) can be done with something like this:
object Taps {
implicit class Tap[T](t: T) extends Anyval {
def tap(f: T => Unit) = { f(t); t }
}
}
Then you can write:
import Taps._
val instance = getTheInstance(2).tap {
_.createPort
.makeSpecial
.bindTo(...)
}
Is this what you are looking for?
In Scala, I'd like to have a subclass method conditionally elided out (and revert to the base class implementation) based on the command line priority. As an example, suppose I have the following code:
// File: mini.scala
import scala.annotation._, elidable._
trait FooBase {
def bar(msg: String) = println("FooBase: " + msg)
}
object Foo extends FooBase {
#elidable(INFO)
override def bar(msg: String) = println("Foo: " + msg)
}
object App {
def main(args: Array[String]) {
println("before")
Foo.bar("message")
println("after")
}
}
If I compile with:
scalac -Xelide-below MINIMUM mini.scala
And run it, I see (as expected):
before
Foo: message
after
Now if I compile with:
scalac -Xelide-below MAXIMUM mini.scala
Then, I hoped to see:
before
FooBase: message
after
But I actually see:
before
after
So instead of just the subclass method being elided out (as hoped) the base class seems to be gone too.
Any explanation appreciated.
Based on the response from #som-snytt, it seems that the best way to achieve the effect I'm after is something like:
trait DebugBase {
def on = false
}
object Debug extends DebugBase {
#elidable(INFO)
override def on = true
}
object Foo {
def bar(msg: String) = if (Debug.on) println("Foo (Debug): " + msg) else println("Foo: " + msg)
}
But then there is still a runtime check on the status of Debug.on instead of completely eliding out the debug version as you might with an #ifdef in C/C++.
That's a good one. I've looked at the mechanism once upon a time, but I wasn't sure about the use case.
The answer is commented:
/** Called if a tree's symbol is elidable. If it's a DefDef,
* replace only the body/rhs with 0/false/()/null; otherwise replace
* the whole tree with it.
*/
So in eliding Foo.bar, you don't remove the method definition. You merely turn it into def bar = () and any invocation into the unit value ().
You can elide a template method usefully:
scala> class X { def f() = { if (g() < 1) println("default") } ; def g() = 1 }
defined class X
scala> :se -Xelide-below 200
scala> class Y extends X { #elidable(100) override def g() = 2 }
defined class Y
scala> new Y().f()
default
scala> :se -Xelide-below 5
scala> class Y extends X { #elidable(100) override def g() = 2 }
defined class Y
scala> new Y().f()
coming from my other question is there a way to get by-name-parameters for constructors working? I need a way to provide a code-block which is executed on-demand/lazy/by-name inside an object and this code-block must be able to access the class-methods as if the code-block were part of the class.
Following Testcase fails:
package test
class ByNameCons(code: => Unit) {
def exec() = {
println("pre-code")
code
println("post-code")
}
def meth() = println("method")
def exec2(code2: => Unit) = {
println("pre-code")
code2
println("post-code")
}
}
object ByNameCons {
def main(args: Array[String]): Unit = {
val tst = new ByNameCons {
println("foo")
meth() // knows meth() as code is part of ByNameCons
}
tst.exec() // ByName fails (executed right as constructor)
println("--------")
tst.exec2 { // ByName works
println("foo")
//meth() // does not know meth() as code is NOT part of ByNameCons
}
}
}
Output:
foo
method
pre-code
post-code
--------
pre-code
foo
post-code
This is because when you're making an instance like this:
val tst = new ByNameCons {
...
}
.. you're actually creating an anonymous class, like in java.
The above code is the same as:
val tst = new ByNameCons() { ... }
.. while the correct syntax for passing by-name is:
val tst = new ByNameCons( { ... } )
You cant omit parentheses the same way for constructors as with functions.
val tst = new ByNameCons( {
println("foo")
} )
Thought it is probably just easier to do this:
object ByNameCons {
def apply(code: => Unit) = new ByNameCons(code)
}
val tst = ByNameCons { // no "new" here -- can't mix traits either
println("foo")
}
I dont know why, but it appears that using {} or () when creating the class changes the behavior. Using the following class,
class Coder(code: => Unit) {
def exec = {
println("before")
code
println("after")}
}
}
scala> val brackets = new Coder {println("testing")}
testing
brackets: Coder = $anon$1#1af7a03
scala> brackets exec
before
after
Now instead if defined another way,
scala> val parens = new Coder(println("testing"))
parens: Coder = Coder#dca3ed
scala> parens exec
before
testing
after
as desired. It seems as if in the first notation, the compiler is interpreting the brackets as a block to be evaluated to Unit, instead of an anonymous function which, when called, evaluates to Unit.
FWIW, using ({ ... }) also works fine.