I want to write a function to format Vaadin messages. These messages have the format
108|0ff1255e-e2be-4e7b-ac5c-1ff2709ce886[["0_11_12_13_login_username","v","v",["text",["s","Agent00232"]]]]
The first number is the length then there some kind of session id (later called vaadin_security_key) followed by the payload. (In this example I set the value "Agent00232" to the textfield with the connector id "0_11_12_13_login_username") I wrote a function like this:
def sendMessage(name: String, vaadinCommand: String) = {
def createMessage(session: Session) = {
val message = session("vaadin_security_key").as[String] + "\u001d" + vaadinCommand
val message2 = ELCompiler.compile(message)(classTag[String])(session).toString
val message3 = message2.substring(8, message2.length - 2)
val len = message3.length
val completeMessage = len.toString() + "|" + message3
completeMessage
}
exec(
ws(name)
.sendText(createMessage)
.check(wsAwait
.within(Vaadin.defaultTimeout)
.until(1)
.regex("""(.+)""")
.saveAs("lastResult")))
.exec { session =>
println(session("lastResult").as[String])
session
}
}
I'd like to use this method like this and use EL expressions in the string:
exec(Vaadin.sendMessage("setting Name", "[[\"0_11_12_13_login_username\",\"v\",\"v\",[\"text\",[\"s\",\"${username}\"]]]]"))
Therefore I have to evaluate the String before calculating the length. The method ELCompiler.compile()... returns a Expression[String]. The problem is that I need this string and concat it with the calculated length. When I do message2.toString it returns
Success(0ff1255e-e2be-4e7b-ac5c-1ff2709ce886[["0_11_12_13_login_username","v","v",["text",["s","Agent00232"]]]]) and therefor I have to use substring(8, message2.length - 2) to get the evaluated payload (with the security key) to calculate the length of it.
Is there a better (more elegant) way to extract the String out of the expression then the use of substring(...)?
Don't explicitly pass a classTag to ELCompiler.compile. If you really were to use this yourself, use ELCompiler.compile[String].
ELCompiler.compile returns an Expression[String], alias a function of Session to Validation[String], so if you apply it, you get a Validation[String] (because the evaluation could fail), hence toString prints the Success, not the value it contains. What you want is to map this Validation container.
You're compiling the expression over and over again, on each function execution. Move it outside the function.
Beware that session("lastResult").as[String] would crash if lastResult could not be saved. Use validate instead.
Use triple quotes so you don't have to escape the inner double quotes.
The first part of your code would then look like:
import io.gatling.core.session._
import io.gatling.core.session.el._
def sendMessage(name: String, vaadinCommand: String) = {
val message = ("${vaadin_security_key}\u001d" + vaadinCommand).el[String]
def createMessage = message.map { resolvedMessage =>
resolvedMessage.length + "|" + resolvedMessage
}
exec(
ws(name)
.sendText(createMessage)
.check(wsAwait
.within(Vaadin.defaultTimeout)
.until(1)
.regex("""(.+)""")
.saveAs("lastResult")))
.exec { session =>
println(session("lastResult").validate[String])
session
}
}
Then you'd call it with:
exec(Vaadin.sendMessage("setting Name", """[["0_11_12_13_login_username","v","v",["text",["s","${username}"]]]]"""))
Related
I'm iterating over the command line input arguments in my Scala program and want to set several variables in a loop. My problem is that I'm using var's.
I don't want to use big libraries for this task, I want to keep it simple. The arguments contain simple key value pairs (like "-v myVar"), that's why I try to use the sliding function. Probably not the best choice for setting val's.
object Main {
def main(args: Array[String]): Unit = {
// args: Array("-c", "myFile", "-j", "update", "-d", "debugPath")
var config: String = null
var jobType: String = null
var debug: String = null
args.sliding(2,2).toList.collect {
case Array("-c", argProperty: String) => config = argProperty
case Array("-j", argJobType: String) => jobType = argJobType
case Array("-d", argDebug: String) => debug = argDebug
}
println("Config: " + config)
println("Job type: " + jobType)
println("Debug: " + debug)
}
}
The code compiles and delivers the correct output, but the 3 var's are not so nice. However, I could not find a solution for declaring 3 val's in a loop which are used outside the loop.
This code turns your arg list in to a Map from argument name to value:
val argMap = args.grouped(2).collect{ case Array(k, v) => k -> v }.toMap
val config = argMap.getOrElse("-c", defaultConfig)
This avoids using null and allows you to easily tell whether a parameter was supplied or not.
I am trying to learn graphx on from this code click here in GitHub
On the spark-shell, when I try this:
def parseFlight(str: String): Flight = {
val line = str.split(",")
Flight(line(0), line(1), line(2), line(3), line(4).toInt, line(5).toLong, line(6), line(7).toLong, line(8), line(9).toDouble, line(10).toDouble, line(11).toDouble, line(12).toDouble, line(13).toDouble, line(14).toDouble, line(15).toDouble, line(16).toInt)
}
val textRDD = sc.textFile("/user/user01/data/rita2014jan.csv")
val flightsRDD = textRDD.map(parseFlight).cache()
val airports = flightsRDD.map(flight => (flight.org_id, flight.origin)).distinct
airports.take(1)
I get this exception which points at airports.take(1):
java.lang.NumberFormatException: empty String
Can anyone let me know if I'm missing something?
It most probably comes from a row within your input in which a field you're casting to Double is empty.
The error most likely comes from this function (applied at the begining of the spark pipeline):
def parseFlight(str: String): Flight = {
val line = str.split(",")
Flight(line(0), line(1), line(2), line(3), line(4).toInt, line(5).toLong, line(6), line(7).toLong, line(8), line(9).toDouble, line(10).toDouble, line(11).toDouble, line(12).toDouble, line(13).toDouble, line(14).toDouble, line(15).toDouble, line(16).toInt)
}
At some point a cast .toDouble is applied on "" (an empty String).
For instance you can reproduce the same error by doing this:
"aa,,,s".split(",")(2).toDouble
which produces:
java.lang.NumberFormatException: empty String
The error led you to think it comes from the line which contains airports.take(1) because it's the line which contains the first action of your pipeline (this is where the laziness of the RDD is lost - as opposed to transformations such as map).
With a route like:
GET /users/:id controllers.Users.view(id)
In the controller, I can read a custom backTo query string param like this:
def view (id: String) = Action {r =>
val backTo = r.getQueryString("backTo") // read custom param
...
}
Assuming backTo is an optional query string parameter that I don't want to include in the routes definition (maybe all the actions can read it).
How do I construct a URL using reverse-routing that includes the backTo parameter?
I would expect something like:
routes.Users.view(id).withQueryString("backTo" -> Seq("previous"))
But that doesn't exist.
I ended up just pimping a withQueryString method onto Call that just parses the URL and rebuilds the URL with the extra parameters:
implicit class CallOps (c: Call) {
import org.jboss.netty.handler.codec.http.{QueryStringDecoder, QueryStringEncoder}
import scala.collection.JavaConverters._
def withQueryString(query: (String,Seq[String])*): Call = {
val decoded = new QueryStringDecoder(c.url)
val newUrl = new QueryStringEncoder(decoded.getPath)
val params = decoded.getParameters.asScala.mapValues(_.asScala.toSeq).toSeq
for {
(key, values) <- params ++ query
value <- values
} newUrl.addParam(key, value)
Call(c.method, newUrl.toString, c.fragment)
}
}
And now I can use it like this:
routes.Users.view(id).withQueryString("backTo" -> Seq("previous")).url
I wish I didn't have to re-parse the URL, but by the time I have a Call, the URL has already been constructed.
I think you would have to come down to String level as there is really no such method available (yet).
What the reverse router is giving you is actually a play.api.mvc.Call object. You can take a look at the source here:
https://github.com/playframework/playframework/blob/2.5.0/framework/src/play/src/main/scala/play/api/mvc/Http.scala#L359
You will see that you can get the absoluteURL() - which returns you a String - from here you would have to manipulate it by adding your query parameter:
val url = routes.Users.view(id).absoluteURL() + "?backTo=home"
What's wrong about using Parameters with default values (without resolving it manually by yourself)? sample:
def view(id: String, backTo: String, forwardTo: String, whatever: String) = Action {
Ok(
"ID: " + id
+ ", backTo: " + backTo
+ ", forwardTo: " + forwardTo
+ ", whatever: " + whatever
)
}
route:
GET /users/:id controllers.Users.view(id, backTo ?= null, forwardTo ?= null, whatever ?= null)
it allows you to make reverse routes with only ID as a required param:
// /users/john-doe
routes.Users.view("john-doe")
by ordered params:
// /users/john-doe?backTo=prev&forwardTo=next&whatever=bar
routes.Users.view("john-doe", "prev", "next", "bar")
or only with named optional params:
// /users/john-doe?whatever=baz
routes.Users.view("john-doe", whatever = "baz")
What's more important it's type-safe as it's absolutely under the Play's routing syntax, without any manual manipulation. Also you don't need to care if you should start your params with ? or & char: "?backTo=home" vs. "&backTo=home"...
I have following code:
import java.io._
import com.twitter.chill.{Input, Output, ScalaKryoInstantiator}
import scala.reflect.ClassTag
object serializer {
val instantiator = new ScalaKryoInstantiator
instantiator.setRegistrationRequired(false)
val kryo = instantiator.newKryo()
def load[T](file:_=>_,name:String,cls:Class[T]):T = {
if (java.nio.file.Files.notExists(new File(name).toPath())) {
val temp = file
val baos = new FileOutputStream(name)
val output = new Output(baos, 4096)
kryo.writeObject(output, temp)
temp.asInstanceOf[T]
}
else {
println("loading from " + name)
val baos = new FileInputStream(name)
val input = new Input(baos)
kryo.readObject(input,cls)
}
}
}
I want to use it in this way:
val mylist = serializer.load((1 to 100000).toList,"allAdj.bin",classOf[List[Int]])
I don't want to run (1 to 100000).toList every time so I want to pass it to the serializer and then decide to compute it for the first time and serialize it for future or load it from file.
The problem is that the code block is running first in my code, how can I pass the code block without executing it?
P.S. Is there any scala tool that do the exact thing for me?
To have parameters not be evaluated before being passed, use pass-by-name, like this:
def method(param: =>ParamType)
Whatever you pass won't be evaluated at the time you pass, but will be evaluated each time you use param, which might not be what you want either. To have it be evaluated only the first time you use, do this:
def method(param: =>ParamType) = {
lazy val p: ParamType = param
Then use only p on the body. The first time p is used, param will be evaluated and the value will be stored. All other uses of p will use the stored value.
Note that this happens every time you invoke method. That is, if you call method twice, it won't use the "stored" value of p -- it will evaluate it again on first use. If you want to "pre-compute" something, then perhaps you'd be better off with a class instead?
My Query is, read input from a file and convert data lines of the file to List[Map[Int,String]] using scala. Here I give a dataset as the input. My code is,
def id3(attrs: Attributes,
examples: List[Example],
label: Symbol
) : Node = {
level = level+1
// if all the examples have the same label, return a new node with that label
if(examples.forall( x => x(label) == examples(0)(label))){
new Leaf(examples(0)(label))
} else {
for(a <- attrs.keySet-label){ //except label, take all attrs
("Information gain for %s is %f".format(a,
informationGain(a,attrs,examples,label)))
}
// find the best splitting attribute - this is an argmax on a function over the list
var bestAttr:Symbol = argmax(attrs.keySet-label, (x:Symbol) =>
informationGain(x,attrs,examples,label))
// now we produce a new branch, which splits on that node, and recurse down the nodes.
var branch = new Branch(bestAttr)
for(v <- attrs(bestAttr)){
val subset = examples.filter(x=> x(bestAttr)==v)
if(subset.size == 0){
// println(levstr+"Tiny subset!")
// zero subset, we replace with a leaf labelled with the most common label in
// the examples
val m = examples.map(_(label))
val mostCommonLabel = m.toSet.map((x:Symbol) => (x,m.count(_==x))).maxBy(_._2)._1
branch.add(v,new Leaf(mostCommonLabel))
}
else {
// println(levstr+"Branch on %s=%s!".format(bestAttr,v))
branch.add(v,id3(attrs,subset,label))
}
}
level = level-1
branch
}
}
}
object samplet {
def main(args: Array[String]){
var attrs: sample.Attributes = Map()
attrs += ('0 -> Set('abc,'nbv,'zxc))
attrs += ('1 -> Set('def,'ftr,'tyh))
attrs += ('2 -> Set('ghi,'azxc))
attrs += ('3 -> Set('jkl,'fds))
attrs += ('4 -> Set('mno,'nbh))
val examples: List[sample.Example] = List(
Map(
'0 -> 'abc,
'1 -> 'def,
'2 -> 'ghi,
'3 'jkl,
'4 -> 'mno
),
........................
)
// obviously we can't use the label as an attribute, that would be silly!
val label = 'play
println(sample.try(attrs,examples,label).getStr(0))
}
}
But How I change this code to - accepting input from a .csv file?
I suggest you use Java's io / nio standard library to read your CSV file. I think there is no relevant drawback in doing so.
But the first question we need to answer is where to read the file in the code? The parsed input seems to replace the value of examples. This fact also hints us what type the parsed CSV input must have, namely List[Map[Symbol, Symbol]]. So let us declare a new class
class InputFromCsvLoader(charset: Charset = Charset.defaultCharset()) {
def getInput(file: Path): List[Map[Symbol, Symbol]] = ???
}
Note that the Charset is only needed if we must distinguish between differently encoded CSV-files.
Okay, so how do we implement the method? It should do the following:
Create an appropriate input reader
Read all lines
Split each line at the comma-separator
Transform each substring into the symbol it represents
Build a map from from the list of symbols, using the attributes as key
Create and return the list of maps
Or expressed in code:
class InputFromCsvLoader(charset: Charset = Charset.defaultCharset()) {
val Attributes = List('outlook, 'temperature, 'humidity, 'wind, 'play)
val Separator = ","
/** Get the desired input from the CSV file. Does not perform any checks, i.e., there are no guarantees on what happens if the input is malformed. */
def getInput(file: Path): List[Map[Symbol, Symbol]] = {
val reader = Files.newBufferedReader(file, charset)
/* Read the whole file and discard the first line */
inputWithHeader(reader).tail
}
/** Reads all lines in the CSV file using [[java.io.BufferedReader]] There are many ways to do this and this is probably not the prettiest. */
private def inputWithHeader(reader: BufferedReader): List[Map[Symbol, Symbol]] = {
(JavaConversions.asScalaIterator(reader.lines().iterator()) foldLeft Nil.asInstanceOf[List[Map[Symbol, Symbol]]]){
(accumulator, nextLine) =>
parseLine(nextLine) :: accumulator
}.reverse
}
/** Parse an entry. Does not verify the input: If there are less attributes than columns or vice versa, zip creates a list of the size of the shorter list */
private def parseLine(line: String): Map[Symbol, Symbol] = (Attributes zip (line split Separator map parseSymbol)).toMap
/** Create a symbol from a String... we could also check whether the string represents a valid symbol */
private def parseSymbol(symbolAsString: String): Symbol = Symbol(symbolAsString)
}
Caveat: Expecting only valid input, we are certain that the individual symbol representations do not contain the comma-separation character. If this cannot be assumed, then the code as is would fail to split certain valid input strings.
To use this new code, we could change the main-method as follows:
def main(args: Array[String]){
val csvInputFile: Option[Path] = args.headOption map (p => Paths get p)
val examples = (csvInputFile map new InputFromCsvLoader().getInput).getOrElse(exampleInput)
// ... your code
Here, examples uses the value exampleInput, which is the current, hardcoded value of examples if no input argument is specified.
Important: In the code, all error handling has been omitted for convenience. In most cases, errors can occur when reading from files and user input must be considered invalid, so sadly, error handling at the boundaries of your program is usally not optional.
Side-notes:
Try not to use null in your code. Returning Option[T] is a better option than returning null, because it makes "nullness" explicit and provides static safety thanks to the type-system.
The return-keyword is not required in Scala, as the last value of a method is always returned. You can still use the keyword if you find the code more readable or if you want to break in the middle of your method (which is usually a bad idea).
Prefer val over var, because immutable values are much easier to understand than mutable values.
The code will fail with the provided CSV string, because it contains the symbols TRUE and FALSE which are not legal according to your programs logic (they should be true and false instead).
Add all information to your error-messages. Your error message only tells me what that a value for the attribute 'wind is bad, but it does not tell me what the actual value is.
Read a csv file ,
val datalines = Source.fromFile(filepath).getLines()
So this datalines contains all the lines from the csv file.
Next, convert each line into a Map[Int,String]
val datamap = datalines.map{ line =>
line.split(",").zipWithIndex.map{ case (word, idx) => idx -> word}.toMap
}
Here, we split each line with ",". Then construct a map with key as column number and value as each word after the split.
Next, If we want List[Map[Int,String]],
val datamap = datalines.map{ line =>
line.split(",").zipWithIndex.map{ case (word, idx) => idx -> word}.toMap
}.toList