To give you a minimal example:
object Main extends JSApp
{
val someThing: String = determineSomething("test")
def main(): Unit =
{
println(someThing)
}
}
Now, two possibilities here:
private def determineSomething(s: String): String = "succeeded"
If the project is executed like this, well, I get a log entry saying
succeeded
But when I use the more functional syntax:
private val determineSomething: (s: String) => "succeeded"
I get
TypeError: this.determineSomething$1 is null
I am curious as to the why this happens as in the (JVM) repl, both ways work perfectly fine.
I think what you want is something like this:
object Main extends JSApp {
private val determineSomething: String => String = (s: String) => "succeeded"
val someThing: String = determineSomething("test")
def main(): Unit = {
println(someThing)
}
}
The declaration of determineSomething needs to come before the declaration of something, otherwise the former will be uninitialized when the compiler attempts to initialize the latter.
Related
I'm creating a DSL where users have a given when then, where they pass a string with the action they want.
All those actions are controlled using some regular expressions.
Now using shapeless or macros I would like to control what users compones using the DSL, and in case the string does not fit the string as we expect make the code don't compile.
Si for instance
Given("a good action") -> fine
When("wrong string action") -> compilation error
Can anybody please point me to a good blog or documentation of how to achieve this?
Regards
After the solution of Gabriele now I´m facing a design issue. This is my vaidator class
object DSLValidator {
import scala.language.experimental.macros
def Given(message: String): Any = macro checkActionImpl
def When(message: String): Any = macro checkActionImpl
def Then(message: String): Any = macro checkActionImpl
def And(message: String): Any = macro checkActionImpl
def checkActionImpl(c: blackbox.Context)(message: c.Tree): c.Tree = {
import c.universe._
def isValidAction(s: String): Boolean = checkMessage(s)
message match {
case _tree#Literal(Constant(s: String)) if isValidAction(s) => _tree
case _ => c.abort(c.enclosingPosition, "Invalid action for DSL. Check the allowed actions in RegexActions")
}
}
val PARAMS = "(.*)=(.*)"
val PAYLOAD_VALUE_REGEX = s"^Payload $PARAMS".r
def checkMessage(action: String): Boolean = {
action match {
case PAYLOAD_VALUE_REGEX(c, c1) => true
case "Make a request to server" => true
case _ => false
}
}
}
And if I invoke the Given/When/Then from another class with wrong arguments the compiler complain as we expect.
#Test
def TestDSL(): Unit = {
DSLValidator.Given("Make a request to wrong")--> not compiling
DSLValidator.When("Payload code=works") --> compiling
println("End")
}
But now thinking the whole idea of was not only to check the values on compilation time but also execute the logic of the Given/When/then
And I realize than when I use macros in a function there´s nothing else that we can do so.
def Given(message: String): Any = {
macro checkActionImpl
//My bussiness logic
}
is not compiling, so my question is, where can I put now the business logic and where the invocation of the macro function to continue working.
Also using a static class as bridge does not work and wont compile since the string message does not fit.
object Macros {
def Given(message: String) = {
DSLValidator.Given(message)
//Logic here
println("Given")
}
}
Macros.Given("Make a request to wrong")
Any idea?
Regards
I think shapeless cannot help you in this case, so a macro would be the way to go.
Here's a skeleton for a macro you can start from:
import scala.language.experimental.macros
def Given(message: String): /* the return type of Given */ = macro givenImpl
def givenImpl(c: Context)(message: c.Tree): c.Tree = {
import c.universe._
def isValid(s: String): Boolean = ??? // your validation logic here
message match {
case case t # Literal(Constant(s: String)) if isValid(s) => t
case _ => c.abort(c.enclosingPosition, "Invalid message")
}
}
In order to fully understand this I can recommend you to read:
http://docs.scala-lang.org/overviews/macros/overview.html
https://blog.scalac.io/2016/02/25/def-hello-macro-world.html
I'm trying to come up with a CSV Parser that can be called like this:
parser parse "/path/to/csv/file" using parserConfiguration
Where the parser will be a class that contains the target case class into which the CSV file will be parsed into:
class CSVParser[A] {
def parse(path: String) = Source.fromFile(fromFilePath).getLines().mkString("\n")
def using(cfg: ParserConfig) = ??? How do I chain this optionally???
}
val parser = CSVParser[SomeCaseClass]
I managed to get up to the point where I can call:
parser parse "/the/path/to/the/csv/file/"
But I do not want to run the parse method yet as I want to apply the configuration using the using like DSL as mentioned above! So there are two rules here. If the caller does not supply a parserConfig, I should be able to run with the default, but if the user supplies a parserConfig, I want to apply the config and then run the parse method. I tried it with a combination of implicits, but could not get them to work properly!
Any suggestions?
EDIT: So the solution looks like this as per comments from "Cyrille Corpet":
class CSVReader[A] {
def parse(path: String) = ReaderWithFile[A](path)
case class ReaderWithFile[A](path: String) {
def using(cfg: CSVParserConfig): Seq[A] = {
val lines = Source.fromFile(path).getLines().mkString("\n")
println(lines)
println(cfg)
null
}
}
object ReaderWithFile {
implicit def parser2parsed[A](parser: ReaderWithFile[A]): Seq[A] = parser.using(defaultParserCfg)
}
}
object CSVReader extends App {
def parser[A] = new CSVReader[A]
val sss: Seq[A] = parser parse "/csv-parser/test.csv" // assign this to a val so that the implicit conversion gets applied!! Very important to note!
}
I guess I need to get the implicit in scope at the location where I call the parser parse, but at the same time I do not want to mess up the structure that I have above!
If you replace using with an operator with a higher precedence than parse you can get it to work without needing extra type annotations. Take for instance <<:
object parsedsl {
class ParserConfig
object ParserConfig {
val default = new ParserConfig
}
case class ParseUnit(path: String, config: ParserConfig)
object ParseUnit {
implicit def path2PU(path: String) = ParseUnit(path, ParserConfig.default)
}
implicit class ConfigSyntax(path: String) {
def <<(config: ParserConfig) = ParseUnit(path, config)
}
class CSVParser {
def parse(pu: ParseUnit) = "parsing"
}
}
import parsedsl._
val parser = new CSVParser
parser parse "path" << ParserConfig.default
parser parse "path"
Your parse method should just give a partial result, without doing anything at all. To deal with default implem, you can use implicit conversion to output type:
class CSVParser[A] {
def parse(path: String) = ParserWithFile[A](path)
}
case class ParserWithFile[A](path: String) {
def using(cfg: ParserConfig): A = ???
}
object ParserWithFile {
implicit def parser2parsed[A](parser: ParserWithFile[A]): A = parser.using(ParserConfig.default)
}
val parser = CSVParser[SomeCaseClass]
In JS, I can access the function arguments through the arguments object.
I want to do the same thing somehow in ScalaJS, because I want to do some logging for functions and what parameters they got.
Is it possible?
Hmm -- interesting question. I honestly don't know if there is a way to access arguments per se. I would probably address the desire for logging the function arguments by using the sourcecode library, which is designed for that sort of thing, but I'll admit that I haven't tried that from Scala.js yet...
You cannot directly access the arguments object in Scala.js. However, you can export a method with varargs and it will work as expected in JavaScript:
object Logger {
#JSExportTopLevel("log")
def log(xs: js.Any*): Unit = {
xs.foreach(println)
}
}
This defines and exports log to the top level scope. In the JavaScript code, you can now call:
log(1, {}, {a: 1}, "foo");
As #justin-du-coeur suggested, you can use sourcecode for this. For example:
object Test extends js.JSApp {
def main(): Unit = {
a(1, "a")
b()
c("foo", "bar", "baz")
}
def trace()(implicit name: sourcecode.Name, args: sourcecode.Args): Unit = {
def makeArgList(as: Seq[sourcecode.Text[_]]): String =
as.map(a => f"${a.source} = ${a.value}").mkString("(", ", ", ")")
val argStr = args.value.map(makeArgList).mkString("")
println(f"${name.value}$argStr")
}
def a(arg1: Int, arg2: String): Unit = {
trace()
}
def b(): Unit = {
trace()
}
def c(x: String*): Unit = {
trace()
}
}
The output is as follows:
[info] Running Test
a(arg1 = 1, arg2 = a)
b()
c(x = WrappedArray(foo, bar, baz))
As you can see, trace can capture everything it needs from the context, so the result is even more boilerplate free than any JS solution I can think of.
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
I'm new to the concept of using the Option type but I've tried to use it multiple places in this class to avoid these errors.
The following class is used to store data.
class InTags(val tag35: Option[String], val tag11: Option[String], val tag_109: Option[String], val tag_58: Option[String])
This following code takes a string and converts it into a Int -> String map by seperating on an equals sign.
val message= FIXMessage("8=FIX.4.29=25435=D49=REDACTED56=REDACTED115=REDACTED::::::::::CENTRAL34=296952=20151112-17:11:1111=Order7203109=CENTRAL1=TestAccount63=021=155=CSCO48=CSCO.O22=5207=OQ54=160=20151112-17:11:1338=5000040=244=2815=USD59=047=A13201=CSCO.O13202=510=127
")
val tag58 = message.fields(Some(58)).getOrElse("???")
val in_messages= new InTags(message.fields(Some(35)), message.fields(Some(11)), message.fields(Some(109)), Some(tag58))
println(in_messages.tag_109.getOrElse("???"))
where the FIXMessage object is defined as follows:
class FIXMessage (flds: Map[Option[Int], Option[String]]) {
val fields = flds
def this(fixString: String) = this(FIXMessage.parseFixString(Some(fixString)))
override def toString: String = {
fields.toString
}
}
object FIXMessage{
def apply(flds: Map[Option[Int], Option[String]]) = {
new FIXMessage(flds)
}
def apply(flds: String) = {
new FIXMessage(flds)
}
def parseFixString(fixString: Option[String]): Map[Option[Int], Option[String]] = {
val str = fixString.getOrElse("str=???")
val parts = str.split(1.toChar)
(for {
part <- parts
p = part.split('=')
} yield Some(p(0).toInt) -> Some(p(1))).toMap
}
}
The error I'm getting is ERROR key not found: Some(58) but doesnt the option class handle this? Which basically means that the string passed into the FIXMessage object doesnt contain a substring of the format 58=something(which is true) What is the best way to proceed?
You are using the apply method in Map, which returns the value or throw NoSuchElementException if key is not present.
Instead you could use getOrElse like
message.fields.getOrElse(Some(58), Some("str"))