I have two Options:
val name: Option[String] = ...
val shared: Option[Boolean] = ...
I would like to build an UPDATE query that SETs these fields if the above values are Some, but leaves them unchanged if they are None.
I have managed to achieve this like so, but I am not so keen on the check for the comma, which is not going to scale if I need to add extra columns. I'm also not so keen on having to use a var.
var query = Q.u + "UPDATE wishlist SET"
if(name.isDefined) query = query + " name = " +? name + (if (share.isDefined) ", " else "")
if(shared.isDefined) query = query + " shared = " +? share
If I wasn't concerned with SQL injection, I could do something along the lines of:
val fields = List(
name.map(n => s"name = '$n'"),
shared.map(e => s"shared = $e")
).flatten
s"UPDATE wishlist SET ${fields.mkString(", ")}"
Question: Is there a nicer way to do this with Slick's plain SQL/String interpolation or is my only option to move to lifted embedded?
It's not super elegant, but I think it's at least flexible in that it will expand to support many inputs w/o changing the underlying logic:
val name:Option[String] = Some("foo")
val share:Option[String] = Some("bar")
val sets = List(
name.map(n => (" name = ", n)),
share.map(s => (" shared = ", s))
).flatten
val query2 = sets.zipWithIndex.foldLeft(Q.u + "UPDATE wishlist SET"){
case (q,(p,0)) => q + p._1 +? p._2
case (q,(p,i)) => q + "," + p._1 +? p._2
}
I'm putting the field name and value pairs into a list as Options for Tuples and then flattening to remove the Nones. I'm then folding to produce the final statement, taking into account that any piece of the query past the first piece will need a comma in it.
Related
Wording of my question might be confusing so let me explain. Say I have an array of strings. They are ranked in order of the best case scenario match. So at index 0 we want this to always exist in the dataframe column, but if it doesn't then index 1 is the next best option. I have written this logic like this, but I don't feel like this is the most efficient way to have done this. Is there another way of doing it that is better?
The datasets are quite small, but I fear this can't really scale very well.
val df = spark.createDataFrame(data)
val nameArray = Array[String]("Name", "Name%", "%Name%", "Person Name", "Person Name%", "%Person Name%")
nameArray.foreach(x => {
val nameDf = df.where("text like '" + x + "'")
if(nameDf.count() > 0){
nameDf.show(1)
break()
}
})
If values are order according to preference from left (highest precedence) to right (lowest precedence) and lower precedence patterns already cover higher precedence ones (it doesn't look like it is the case in your example) you generate expression like this
import org.apache.spark.sql._
def matched(df: DataFrame, nameArray: Seq[String], c: String = "text") = {
val matchIdx = nameArray.zipWithIndex.foldRight(lit(-1)){
case ((s, i), acc) => when(col(c) like s, lit(i)).otherwise(acc)
}
df.select(max(matchIdx)).first match {
case Row(-1) => None // No pattern matches all records
case Row(i: Int) => Some(nameArray(i))
}
}
Usage examples:
matched(Seq("Some Name", "Name", "Name Surname").toDF("text"), Seq("Name", "Name%", "%Name%"))
// Option[String] = Some(%Name%)
There are two advantages of this method:
Only one action is required.
Pattern matching can be short circuited.
If pre-conditions are not satisfied you can
import org.apache.spark.sql.functions._
val unmatchedCount: Map[String, Long] = df.select(
nameArray.map(s => count(when(not($"text" like s), 1)).alias(s)): _*
).first.getValuesMap[Long](nameArray)
Unlike the first approach it will check all patterns, but it requires only one action.
I'm pretty new to scala (and programming in general), but have come up with what feels like a less than perfect solution to an issue I have - I was wondering if anyone has a more elegant/efficient one?
I have a (very large) set of strings, a small example of which is below for replication purposes:
val brands = Set("Brand one", "BRAND-two!!!", "brand_Three1, brandThree2", "brand04")
Now what I want to do is clean up this set so that I have a new clean set where:
any strings separated by commas are split into separate strings
leading spaces and non-alphanumeric (and _ -) characters are
removed
any string with a space is replaced by three version of
that string (one with no space, one with "-" instead of a space, and one and one with "_")
The code I have so far does this, but it does it in two steps, thus iterating over the list twice (which is inefficient):
val brands_clean = brands.flatMap(
_.toLowerCase.split(",").map(
_.trim.replaceAll("[^A-Za-z0-9\\-\\_\\s]+", "")
)
)
def spaceVariations(v: String) = if (v.contains(" ")) Set(v.replaceAll(" ", "-"), v.replaceAll(" ", "_"), v.replaceAll(" ", "")) else Set(v)
val brands_final = brands_clean.flatMap(spaceVariations(_))
I have tried incorporating the spaceVariations function directly into the main code by appending to the replaceAll a map or flatMap:
// using the function call
.flatMap(spaceVariations(_))
// or using a function directly within the code
.flatMap {v => if (v.contains(" ")) Set(v.replaceAll(" ", "-"), v.replaceAll(" ", "_"), v.replaceAll(" ", "")) else Set(v) }
but I get the following error:
error: type mismatch;
found : Array[Nothing]
required: scala.collection.GenTraversableOnce[?]
I'm not sure I understand why this doesn't work here, or if there is a better way to achieve what I am trying to achieve?
brands.flatMap(
_.toLowerCase.split(",").map(
_.trim.replaceAll("""[^\w-]""", "")
)
).flatMap(spaceVariations)
Works for me, not sure where (or why) you are getting the error (I cleaned up your regex a little bit to make it more concise, but that shouldn't matter).
Note, that this still traverses the set twice though. Sets are not lazy in scala, so, it will complete the first flatMap with an intermediate set first, and then start with the next one.
If you want to save a scan, you should starts with an Iterator rather than a set, because iterators are lazy, and will send each element through the whole chain before moving on to the next one:
brands
.iterator
.flatMap { _.toLowerCase.split(",") }
.map(_.trim)
.map { _.replaceAll("""[^\w-]""", "") }
.flatMap(spaceVariations)
.toSet
Based on the assumption that your Set will always look like this:
def spaceVariations(v: String) = if (v.contains(" ")) Set(v.replaceAll(" ", "-"), v.replaceAll(" ", "_"), v.replaceAll(" ", "")) else Set(v)
val brands = Set("Brand one", "BRAND-two!!!", "brand_Three1, brandThree2", "brand04")
brands.map( x => if (x.contains(",") ) x.split(",") else x ).flatMap {
case str: String => Array(str)
case a : Array[String] => a
}.map(_.trim.toLowerCase.replaceAll("[^A-Za-z0-9\\-\\_\\s]+", "")).map(spaceVariations(_))
Gives the output :
Set(Set(brand-one, brand_one, brandone), Set(brandthree2), Set(brand04), Set(brand-two), Set(brand_three1))
Is it possible to do anything after a pattern match in a foreach statement?
I want to do a post match step e.g. to set a variable. I also want to force a Unit return as my foreach is String => Unit, and by default Scala wants to return the last statement.
Here is some code:
Iteratee.foreach[String](_ match {
case "date" => out.push("Current date: " + new Date().toString + "<br/>")
case "since" => out.push("Last command executed: " + (ctm - last) + "ms before now<br/>")
case unknow => out.push("Command: " + unknown + " not recognized <br/>")
} // here I would like to set "last = ctm" (will be a Long)
)
UPDATED:
New code and context. Also new questions added :) They are embedded in the comments.
def socket = WebSocket.using[String] { request =>
// Comment from an answer bellow but what are the side effects?
// By convention, methods with side effects takes an empty argument list
def ctm(): Long = System.currentTimeMillis
var last: Long = ctm
// Command handlers
// Comment from an answer bellow but what are the side effects?
// By convention, methods with side effects takes an empty argument list
def date() = "Current date: " + new Date().toString + "<br/>"
def since(last: Long) = "Last command executed: " + (ctm - last) + "ms before now<br/>"
def unknown(cmd: String) = "Command: " + cmd + " not recognized <br/>"
val out = Enumerator.imperative[String] {}
// How to transform into the mapping strategy given in lpaul7's nice answer.
lazy val in = Iteratee.foreach[String](_ match {
case "date" => out.push(date)
case "since" => out.push(since(last))
case unknown => out.push(unknown)
} // Here I want to update the variable last to "last = ctm"
).mapDone { _ =>
println("Disconnected")
}
(in, out)
}
I don't know what your ctm is, but you could always do this:
val xs = List("date", "since", "other1", "other2")
xs.foreach { str =>
str match {
case "date" => println("Match Date")
case "since" => println("Match Since")
case unknow => println("Others")
}
println("Put your post step here")
}
Note you should use {} instead of () when you want use a block of code as the argument of foreach().
I will not answer your question, but I should note that reassigning variables in Scala is a bad practice. I suggest you to rewrite your code to avoid vars.
First, transform your strings to something else:
val strings = it map {
case "date" => "Current date: " + new Date().toString + "<br/>"
case "since" => "Last command executed: " + (ctm - last) + "ms before now<br/>"
case unknow => "Command: " + unknown + " not recognized <br/>"
}
Next, push it
strings map { out.push(_) }
It looks like your implementation of push has side effects. Bad for you, because such methods makes your program unpredictable. You can easily avoid side effects by making push return a tuple:
def push(s: String) = {
...
(ctm, last)
}
And using it like:
val (ctm, last) = out.push(str)
Update:
Of course side effects are needed to make programs useful. I only meant that methods depending on outer variables are less predictable than pure one, it is hard to reason about it. It is easier to test methods without side effects.
Yes, you should prefer vals over vars, it makes your program more "functional" and stateless. Stateless algorithms are thread safe and very predictable.
It seems like your program is stateful by nature. At least, try to stay as "functional" and stateless as you can :)
My suggested solution of your problem is:
// By convention, methods with side effects takes an empty argument list
def ctm(): Long = // Get current time
// Command handlers
def date() = "Current date: " + new Date().toString + "<br/>"
def since(last: Long) = "Last command executed: " + (ctm() - last) + "ms before now<br/>"
def unknown(cmd: String) = "Command: " + unknown + " not recognized <br/>"
// In your cmd processing loop
// First, map inputs to responses
val cmds = inps map {
case "date" => date()
case "since" => since(last)
case unk => unknown(unk)
}
// Then push responses and update state
cmds map { response =>
out.push(response)
// It is a good place to update your state
last = ctm()
}
It is hard to test this without context of your code, so you should fit it to your needs yourself. I hope I've answered your question.
Right now I have many params that can be none, and I want to assign the whole expressiona default value.
Right now I'm doing things like
var name: Option[String] = None
var surname: Option[String] = Some("Smith")
val fullName:String = {
name.map { name =>
surname.map { surname =>
surname + ", " + name
}.getOrElse("unknown")
}.getOrElse("unknown")
}
but it's a bit too verbose. I'd like to know what would be a more idiomatic and elegant way to handle it, ideally it would be something like (it's pseudo code, of course!):
val fullName = (name + ", " + surname).getOrElse("unknown")
or something similar...
(just avoiding the double .getOrElse would be great...)
How about this
scala> val fullName = (for(n <-name;s <-surname) yield n + s).getOrElse("unknown")
fullName: String = unknown
You might want to learn a bit about applicative functors, as the same pattern can be used in all sorts of ways. Using scalaz, there is the applicative builder:
(name |#| surname)(_ + _)
What is going on is this:
(M[A] |#| M[B])(fabc) ~> M[C] //fabc is a function f: (A, B) => C
That is, the function f is being lifted into the realm of applicative functor M. This is particularly useful because it works with Option, Validation, Promise, List, Stream and many more. That is, just as you might use the expression:
(name |#| surname)(_ + _)
where name and surname are both Option[String], you could switch them to be ValidationNEL[Exception, String] or Promise[String] and the code would still do the exact same thing (appropriate to the higher kind being used). This is very powerful.
I've found this way:
val fullname = (name, surname) match {
case (Some(x), Some(y)) => x + ", " + y
case _ => "unknonw"
}
but it's still a bit verbose
It seems no easy way to use "in" clause in anorm:
val ids = List("111", "222", "333")
val users = SQL("select * from users where id in ({ids})").on('ids-> ???).as(parser *)
How to replace the ??? part?
I tried:
on('ids -> ids)
on('ids -> ids.mkString("'","','","'"))
on('ids -> ids.mkString("','")
But none works.
I see in the discussion the exactly same problem: https://groups.google.com/d/topic/play-framework/qls6dhhdayc/discussion, the author has a complex solution:
val params = List(1, 2, 3)
val paramsList = for ( i <- 0 until params.size ) yield ("userId" + i)
// ---> results in List("userId0", "userId1", "userId2")
User.find("id in ({%s})"
// produces "id in ({userId0},{userId1},{userId2})"
.format(paramsList.mkString("},{"))
// produces Map("userId0" -> 1, "userId1" -> 2, ...)
.on(paramsList.zip(params))
.list()
This is too much complicated.
Is there any easier way? Or should play provide something to make it easier?
Anorm now supports such a case (and more) since 2.3: "Using multi-value parameter"
Back to initial example it gives:
val ids = Seq("111", "222", "333")
val users = SQL("select * from users where id in ({ids})").on('ids-> ids).as(parser *)
Nailed it! There haven't really been any more updates on this thread, but it seems to still be relevant. Because of that, and because there isn't an answer, I thought I'd throw mine in for consideration.
Anorm doesn't support 'IN' clauses. I doubt they ever will. There's nothing you can do to make them work, I even read a post where anorm specifically took out those clauses because they made Anorm feel 'like an ORM'.
It's fairly easy, however, to wrap the SqlQuery in a short class that supports the IN clause, and then convert that class into a SqlQuery when needed.
Instead of pasting the code in here, because it gets a little long, here is the link to my blog, where I've posted the code and how to use it.
In clause with Anorm
Basically, when you have the code from my blog, your statements look like this:
RichSQL(""" SELECT * FROM users WHERE id IN ({userIds}) """).onList("userIds" -> userIds).toSQL.as(userParser *)(connection)
Maybe it's too late but here is a tip for using custom string interpolation that also works for solve the problem of IN clause.
I have implemented a helper class to define a string interpolation. You can see it below, and you can simply copy and paste, but first let's see how you can use it.
Instead of write something
SQL("select * from car where brand = {brand} and color = {color} and year = {year} order by name").on("brand" -> brand, "color" -> color, "year" -> year).as(Car.simple *)
You can simply write:
SQL"select * from car where brand = $brand and color = $color and year = $year order by name".as(Car.simple *)
So using string interpolation it's more concise and easier to read.
And for the case of using the IN clause, you can write:
val carIds = List(1, 3, 5)
SQLin"select * from car where id in ($carIds)".as(Car.simple *)
Or for your example:
val ids = List("111", "222", "333")
val users = SQLin"select * from users where id in ($ids)".as(parser *)
For more information about string interpolation, check this link
The code for this implicit class is the following:
package utils
object AnormHelpers {
def wild (str: String) = "%" + str + "%"
implicit class AnormHelper (val sc: StringContext) extends AnyVal {
// SQL raw -> it simply create an anorm.Sql using string interpolation
def SQLr (args: Any*) = {
// Matches every argument to an arbitrary name -> ("p0", value0), ("p1", value1), ...
val params = args.zipWithIndex.map(p => ("p"+p._2, p._1))
// Regenerates the original query substituting each argument by its name with the brackets -> "select * from user where id = {p0}"
val query = (sc.parts zip params).map{ case (s, p) => s + "{"+p._1+"}" }.mkString("") + sc.parts.last
// Creates the anorm.Sql
anorm.SQL(query).on( params.map(p => (p._1, anorm.toParameterValue(p._2))) :_*)
}
// SQL -> similar to SQLr but trimming any string value
def SQL (args: Any*) = {
val params = args.zipWithIndex.map {
case (arg: String, index) => ("p"+index, arg.trim.replaceAll("\\s{2,}", " "))
case (arg, index) => ("p"+index, arg)
}
val query = (sc.parts zip params).map { case (s, p) => s + "{"+ p._1 + "}" }.mkString("") + sc.parts.last
anorm.SQL(query).on( params.map(p => (p._1, anorm.toParameterValue(p._2))) :_*)
}
// SQL in clause -> similar to SQL but expanding Seq[Any] values separated by commas
def SQLin (args: Any*) = {
// Matches every argument to an arbitrary name -> ("p0", value0), ("p1", value1), ...
val params = args.zipWithIndex.map {
case (arg: String, index) => ("p"+index, arg.trim.replaceAll("\\s{2,}", " "))
case (arg, index) => ("p"+index, arg)
}
// Expands the Seq[Any] values with their names -> ("p0", v0), ("p1_0", v1_item0), ("p1_1", v1_item1), ...
val onParams = params.flatMap {
case (name, values: Seq[Any]) => values.zipWithIndex.map(v => (name+"_"+v._2, anorm.toParameterValue(v._1)))
case (name, value) => List((name, anorm.toParameterValue(value)))
}
// Regenerates the original query substituting each argument by its name expanding Seq[Any] values separated by commas
val query = (sc.parts zip params).map {
case (s, (name, values: Seq[Any])) => s + values.indices.map(name+"_"+_).mkString("{", "},{", "}")
case (s, (name, value)) => s + "{"+name+"}"
}.mkString("") + sc.parts.last
// Creates the anorm.Sql
anorm.SQL(query).on(onParams:_*)
}
}
}
It's probably late, but I add this for others looking for the same.
You could use some built-in database features to overcome this. This is one of the advantages Anorm has over ORMs. For example, if you are using PostgreSQL you could pass your list as an array and unnest the array in your query:
I assume ids are integer.
val ids = List(1, 2, 3)
val idsPgArray = "{%s}".format(ids.mkString(",")) //Outputs {1, 2, 3}
val users = SQL(
"""select * from users where id in (select unnest({idsPgArray}::integer[]))"""
).on('ids-> ???).as(parser *)
Executed query will be
select * from users where id in (select unnest('{1, 2, 3}'::integer[]))
which is equal to
select * from users where id in (1, 2, 3)
I had the same problem recently. Unfortunately there doesn't seem to be a way without using string interpolation and thus vulnerable to SQL injection.
What I ended up doing was kinda sanitizing it by transforming it to a list of ints and back:
val input = "1,2,3,4,5"
// here there will be an exception if someone is trying to sql-inject you
val list = (_ids.split(",") map Integer.parseInt).toList
// re-create the "in" string
SQL("select * from foo where foo.id in (%s)" format list.mkString(","))
User.find("id in (%s)"
.format(params.map("'%s'".format(_)).mkString(",") )
.list()
val ids = List("111", "222", "333")
val users = SQL("select * from users
where id in
(" + ids.reduceLeft((acc, s) => acc + "," + s) + ")").as(parser *)