Related
I want to evaluate a function passed as a variable string in scala (sorry but i'm new to scala )
def concate(a:String,b:String): String ={
a+" "+b
}
var func="concate" //i'll get this function name from config as string
I want to perform something like
eval(func("hello","world)) //Like in Python
so output will be like
hello world
Eventually I want to execute few in built functions on a string coming from my config and I don't want to hard code the function names in the code.
EDIT
To Be More clear with my exact usecase
I have a Config file which has multiple functions defined in it that are Spark inbuilt functions on Data frame
application.conf looks like
transformations = [
{
"table" : "users",
"function" : "from_unixtime",
"column" : "epoch"
},
{
"table" : "users",
"function" : "yearofweek",
"column" : "epoch"
}
]
Now functions yearofweek and from_unixtime are Spark inbuilt functions now I want to eval my Dataframe by the functions defined in config. #all the functions are applied to a column defined.
the Obvious way is to write an if else and do string comparison calling a particular inbuilt function but that is way to much..
i am looking for a better solution.
This is indeed possible in scala, as scala is JSR 223 compliant scripting language. Here is an example (running with scala 2.11.8). Note that you need to import your method because otherwise the interpreter will not find it:
package my.example
object EvalDemo {
// evalutates scala code and returns the result as T
def evalAs[T](code: String) = {
import scala.reflect.runtime.currentMirror
import scala.tools.reflect.ToolBox
val toolbox = currentMirror.mkToolBox()
import toolbox.{eval, parse}
eval(parse(code)).asInstanceOf[T]
}
def concate(a: String, b: String): String = a + " " + b
def main(args: Array[String]): Unit = {
var func = "concate" //i'll get this function name from config as string
val code =
s"""
|import my.example.EvalDemo._
|${func}("hello","world")
|""".stripMargin
val result: String = evalAs[String](code)
println(result) // "hello world"
}
}
Have Function to name mapping in the code
def foo(str: String) = str + ", foo"
def bar(str: String) = str + ", bar"
val fmap = Map("foo" -> foo _, "bar" -> bar _)
fmap("foo")("hello")
now based on the function name we get from the config, pass the name to the map and lookup the corresponding function and invoke the arguments on it.
Scala repl
scala> :paste
// Entering paste mode (ctrl-D to finish)
def foo(str: String) = str + ", foo"
def bar(str: String) = str + ", bar"
val fmap = Map("foo" -> foo _, "bar" -> bar _)
fmap("foo")("hello")
// Exiting paste mode, now interpreting.
foo: (str: String)String
bar: (str: String)String
fmap: scala.collection.immutable.Map[String,String => String] = Map(foo -> $$Lambda$1104/1335082762#778a1250, bar -> $$Lambda$1105/841090268#55acec99)
res0: String = hello, foo
Spark offers you a way to write your transformations or queries using SQL. So, you really don't have to worry about Scala functions, casting and evaluation in this case. You just have to parse your config to generate the SQL query.
Let's say you have registered a table users with Spark and want to do a select and transform based on provided config,
// your generated query will look like this,
val query = "SELECT from_unixtime(epoch) as time, weekofyear(epoch) FROM users"
val result = spark.sql(query)
So, all you need to do is - build that query from your config.
With a route like:
GET /users/:id controllers.Users.view(id)
In the controller, I can read a custom backTo query string param like this:
def view (id: String) = Action {r =>
val backTo = r.getQueryString("backTo") // read custom param
...
}
Assuming backTo is an optional query string parameter that I don't want to include in the routes definition (maybe all the actions can read it).
How do I construct a URL using reverse-routing that includes the backTo parameter?
I would expect something like:
routes.Users.view(id).withQueryString("backTo" -> Seq("previous"))
But that doesn't exist.
I ended up just pimping a withQueryString method onto Call that just parses the URL and rebuilds the URL with the extra parameters:
implicit class CallOps (c: Call) {
import org.jboss.netty.handler.codec.http.{QueryStringDecoder, QueryStringEncoder}
import scala.collection.JavaConverters._
def withQueryString(query: (String,Seq[String])*): Call = {
val decoded = new QueryStringDecoder(c.url)
val newUrl = new QueryStringEncoder(decoded.getPath)
val params = decoded.getParameters.asScala.mapValues(_.asScala.toSeq).toSeq
for {
(key, values) <- params ++ query
value <- values
} newUrl.addParam(key, value)
Call(c.method, newUrl.toString, c.fragment)
}
}
And now I can use it like this:
routes.Users.view(id).withQueryString("backTo" -> Seq("previous")).url
I wish I didn't have to re-parse the URL, but by the time I have a Call, the URL has already been constructed.
I think you would have to come down to String level as there is really no such method available (yet).
What the reverse router is giving you is actually a play.api.mvc.Call object. You can take a look at the source here:
https://github.com/playframework/playframework/blob/2.5.0/framework/src/play/src/main/scala/play/api/mvc/Http.scala#L359
You will see that you can get the absoluteURL() - which returns you a String - from here you would have to manipulate it by adding your query parameter:
val url = routes.Users.view(id).absoluteURL() + "?backTo=home"
What's wrong about using Parameters with default values (without resolving it manually by yourself)? sample:
def view(id: String, backTo: String, forwardTo: String, whatever: String) = Action {
Ok(
"ID: " + id
+ ", backTo: " + backTo
+ ", forwardTo: " + forwardTo
+ ", whatever: " + whatever
)
}
route:
GET /users/:id controllers.Users.view(id, backTo ?= null, forwardTo ?= null, whatever ?= null)
it allows you to make reverse routes with only ID as a required param:
// /users/john-doe
routes.Users.view("john-doe")
by ordered params:
// /users/john-doe?backTo=prev&forwardTo=next&whatever=bar
routes.Users.view("john-doe", "prev", "next", "bar")
or only with named optional params:
// /users/john-doe?whatever=baz
routes.Users.view("john-doe", whatever = "baz")
What's more important it's type-safe as it's absolutely under the Play's routing syntax, without any manual manipulation. Also you don't need to care if you should start your params with ? or & char: "?backTo=home" vs. "&backTo=home"...
I have code that deeply walks a case class' constructor fields, which of course may themselves be complex (list of things, maps, options, and other case classes). The code I found to extract field values at runtime works great on the highest-level fields but explodes when I try to access deeper fields. Example below.
I real life my application introspects the fields at each level, so I know that 'stuff' is another case class (I have the Symbol/Type), and I know Dos' field Symbols/Types. But this is obtained at runtime so I think it's blowing up because it doesn't know [T]/Manifest[T]. Is there a way to get this at runtime via reflection? How might my code change? The examples I found seemed to all require various things[T], which I wouldn't have for 'dos', right?
case class Uno( name:String, age:Int, pets:List[String], stuff:Dos )
case class Dos( foo:String )
object Boom extends App {
val ru = scala.reflect.runtime.universe
val m = ru.runtimeMirror(getClass.getClassLoader)
val u = Uno("Marcus",19,List("fish","bird"),Dos("wow"))
println("NAME: "+unpack(u,"name")) // Works
println("PETS: "+unpack(u,"pets")) // Works
// ----- Goes Boom -------
val dos = unpack(u,"stuff")
println("Other: "+unpack(dos,"foo")) // Boom!
// -----------------------
// Get object value for named parameter of target
def unpack[T]( target:T, name:String )(implicit man:Manifest[T]) : Any = {
val im = m.reflect(target)
val fieldX = ru.typeOf[T].declaration(ru.newTermName(name)).asTerm.accessed.asTerm
im.reflectField(fieldX).get
}
}
You're exactly right, the type of your dos is Any.
FieldMirror.symbol.typeSignature is what you'd get from typeOf[Dos].
So consider returning a pair (Any, Type) from unpack to have something to pass to unpack(target, type, name). Somewhat like:
case class Uno(name: String, age: Int, pets: List[String], stuff: Dos)
case class Dos(foo: String)
object Boom extends App {
import scala.reflect.runtime.universe._
import scala.reflect.runtime.{ currentMirror => cm }
import scala.reflect.ClassTag
val u = Uno("Marcus", 19, List("fish", "bird"), Dos("wow"))
println("NAME: " + unpack(u, "name")) // Works
println("PETS: " + unpack(u, "pets")) // Works
// ----- Goes Boom -------
val (dos, dosT) = unpack(u, "stuff")
println("Other: " + unpack(dos, dosT, "foo")) // Boom! ...or fizzle
// -----------------------
def unpack[T: TypeTag](target: T, name: String): (Any, Type) = unpack(target, typeOf[T], name)
// Get object value for named parameter of target
def unpack[T](target: T, t: Type, name: String): (Any, Type) = {
val im = cm.reflect(target)(ClassTag(target.getClass))
val fieldX = t.declaration(newTermName(name)).asTerm.accessed.asTerm
val fm = im.reflectField(fieldX)
(fm.get, fm.symbol.typeSignature)
}
}
Specs2's documentation and samples show some codes about the use of Given/Then/When style in an acceptance test.
Here one of them:
"A given-when-then example for the addition" ^
"Given the following number: ${1}" ^ number1 ^
"And a second number: ${2}" ^ number2 ^
"And a third number: ${3}" ^ number3
val number1: Given[Int] = (_:String).toInt
val number2: When[Int, (Int, Int)] = (n1: Int) => (s: String) => (n1, s.toInt)
val number3: When[Seq[Int], Seq[Int]] = (numbers: Seq[Int]) => (s: String) => numbers :+ s.toInt
However, it only deal with some primitive objects as Int here (normal since it's a sample).
But how to deal with complex objects? Using Datatables?
Ideal would be to define, within some file of "fixtures", datatables (or similar things) defining complex objects.
External datatable defining the User "Mik"
"name" | "job" | "e-mail"
"mik" | "engineer" | "mik#mik.com"
I would like to create a Given statement like this one:
"Given the following user: ${Mik}" ^ retrieve_user_from_database ^
val user: Given[User] = .....Some ideal codes here to map Mik's values to User model.
What is a good practice and what does Specs2 currently provide?
If you want to retrieve a user from an existing database, you can do the following:
"Given the following user: ${Mik}" ^ user ^
val user: Given[User] = (name: String) => database.getUserByName(name)
Now you might want to populate your database with users before doing that. One possibility is indeed to use Datatables:
def populateDatabase =
"name" || "job" | "e-mail" |
"mik" !! "engineer" ! "mik#mik.com" |
"eric" !! "engineer" ! "eric#eric.com" | { (name, job, email) =>
database.saveUser(User(name, job, email)) must not(throwAn[Exception])
}
and put this in a Step before your G/W/T group:
Step(populateDatabase) ^
"Given the following user: ${Mik}" ^ user ^
"This user should be an ${engineer} ^ jobOk ^
end
val user: Given[User] = (name: String) => database.getUserByName(name)
val jobOk: Then[User] = (user: User) => (job: String) => user.job must_== job
I want to take input from the user. Can you please tell me how to ask for user input as a string in Scala?
In Scala 2.11 use
scala.io.StdIn.readLine()
instead of the deprecated Console.readLine.
Here is a standard way to read Integer values
val a = scala.io.StdIn.readInt()
println("The value of a is " + a)
similarly
def readBoolean(): Boolean
Reads a Boolean value from an entire line from stdin.
def readByte(): Byte
Reads a Byte value from an entire line from stdin.
def readChar(): Char
Reads a Char value from an entire line from stdin.
def readDouble(): Double
Reads a Double value from an entire line from stdin.
def readFloat(): Float
Reads a Float value from an entire line from stdin.
def readInt(): Int
Reads an Int value from an entire line from stdin.
def readLine(text: String, args: Any*): String
Prints formatted text to stdout and reads a full line from stdin.
def readLine(): String
Reads a full line from stdin.
def readLong(): Long
Reads a Long value from an entire line from stdin.
def readShort(): Short
Reads a Short value from an entire line from stdin.
def readf(format: String): List[Any]
Reads in structured input from stdin as specified by the format specifier.
def readf1(format: String): Any
Reads in structured input from stdin as specified by the format specifier, returning
only the first value extracted, according to the format specification.
def readf2(format: String): (Any, Any)
Reads in structured input from stdin as specified by the format specifier, returning
only the first two values extracted, according to the format specification.
def readf3(format: String): (Any, Any, Any)
Reads in structured input from stdin as specified by the format specifier, returning
only the first three values extracted, according to the format specification.
Similarly if you want to read multiple user inputs from the same line ex: name, age, weight you can use the Scanner object
import java.util.Scanner
// simulated input
val input = "Joe 33 200.0"
val line = new Scanner(input)
val name = line.next
val age = line.nextInt
val weight = line.nextDouble
abridged from Scala Cookbook: Recipes for Object-Oriented and Functional Programming by Alvin Alexander
From the Scala maling list (formatting and links were updated):
Short answer:
readInt
Long answer:
If you want to read from the terminal, check out Console.scala.
You can use these functions like so:
Console.readInt
Also, for your convenience, Predef.scala
automatically defines some shortcuts to functions in Console. Since
stuff in Predef is always and everywhere imported automatically, you
can use them like so:
readInt
object InputTest extends App{
println("Type something : ")
val input = scala.io.StdIn.readLine()
println("Did you type this ? " + input)
}
This way you can ask input.
scala.io.StdIn.readLine()
You can take a user String input using readLine().
import scala.io.StdIn._
object q1 {
def main(args:Array[String]):Unit={
println("Enter your name : ")
val a = readLine()
println("My name is : "+a)
}
}
Or you can use the scanner class to take user input.
import java.util.Scanner;
object q1 {
def main(args:Array[String]):Unit={
val scanner = new Scanner(System.in)
println("Enter your name : ")
val a = scanner.nextLine()
println("My name is : "+a)
}
}
Simple Example for Reading Input from User
val scanner = new java.util.Scanner(System.in)
scala> println("What is your name") What is your name
scala> val name = scanner.nextLine()
name: String = VIRAJ
scala> println(s"My Name is $name")
My Name is VIRAJ
Also we can use Read Line
val name = readLine("What is your name ")
What is your name name: String = Viraj
In Scala 2:
import java.io._
object Test {
// Read user input, output
def main(args: Array[String]) {
// create a file writer
var writer = new PrintWriter(new File("output.txt"))
// read an int from standard input
print("Enter the number of lines to read in: ")
val x: Int = scala.io.StdIn.readLine.toInt
// read in x number of lines from standard input
var i=0
while (i < x) {
var str: String = scala.io.StdIn.readLine
writer.write(str + "\n")
i = i + 1
}
// close the writer
writer.close
}
}
This code gets input from user and outputs it:
[input] Enter the number of lines to read in: 2
one
two
[output] output.txt
one
two
Using a thread to poll the input-readLine:
// keystop1.sc
// In Scala- or SBT console/Quick-REPL: :load keystop1.sc
// As Script: scala -savecompiled keystop1.sc
#volatile var isRunning = true
#volatile var isPause = false
val tInput: Thread = new Thread {
override def run: Unit = {
var status = ""
while (isRunning) {
this.synchronized {
status = scala.io.StdIn.readLine()
status match {
case "s" => isRunning = false
case "p" => isPause = true
case "r" => isRunning = true;isPause = false
case _ => isRunning = false;isPause = false
}
println(s"New status is: $status")
}
}
}
}
tInput.start
var count = 0
var pauseCount = 0
while (isRunning && count < 10){
println(s"still running long lasting job! $count")
if (count % 3 == 0) println("(Please press [each + ENTER]: s to stop, p to pause, r to run again!)")
count += 1
Thread sleep(2000) // simulating heavy computation
while (isPause){
println(s"Taking a break ... $pauseCount")
Thread sleep(1000)
pauseCount += 1
if (pauseCount >= 10){
isPause = false
pauseCount = 0
println(s"Taking a break ... timeout occurred!")
}
}
}
isRunning = false
println(s"Computation stopped, please press Enter!")
tInput.join()
println(s"Ok, thank you, good bye!")
readLine() lets you prompt the user and read their input as a String
val name = readLine("What's your name? ")
please try
scala> readint
please try this method