Im using Squeryl with the function inhibitWhen to ignore a Where clause when a scala value is Empty. I tried this code but Squeryl still checking for the where clause that i want to ignore!
(l.book_id in params.get("book_id").toList.flatMap(_.split(",")).map(_.toInt)).inhibitWhen(params.get("book_id").map(_.isEmpty) == Some(true))
So what i need to do is when book_id has some values (1,4,6) it use the in clause but when it's empty the in clause should be ignored !
Related
I'm trying to do a case on a DF I have but I'm getting an error. I want to implement this with built in spark functions - withcolumn, when, otherwise:
CASE WHEN vehicle="BMW"
AND MODEL IN ("2020","2019","2018","2017")
AND value> 100000 THEN 1
ELSE 0 END AS NEW_COLUMN
Currently I have this
DF.withColumn(NEW_COLUMN, when(col(vehicle) === "BMW"
and col(model) isin(listOfYears:_*)
and col(value) > 100000, 1).otherwise(0))
But I'm getting an error due to data type mismatch, (boolean and string)... I understand my condition returns booleans and strings, which is causing the error. What's the correct syntax for executing a case like that one? also, I was using && instead of and but the third && was giving me a "cannot resolve symbol &&"
Thanks for the help!
I think && is correct - with the built-in spark functions, all of the expressions are of type Column, checking the API it looks like && is correct and should work fine. Could it be as simple as an order-of-operations issue, where you need parentheses around each of the boolean conditions? The function / "operator" isin would have a lower precedence than &&, which might trip things up.
I am a bit confused about using $ to reference columns in DataFrame operators like select or filter.
The following statements work:
df.select("app", "renders").show
df.select($"app", $"renders").show
But, only the first statement in the following works:
df.filter("renders = 265").show // <-- this works
df.filter($"renders" = 265).show // <-- this does not work (!) Why?!
However, this again works:
df.filter($"renders" > 265).show
Basically, what is this $ in DataFrame's operators and when/how should I use it?
Implicits are a major feature of the Scala language that take a lot of different forms--like implicit classes as we will see shortly. They have different purposes, and they all come with varying levels of debate regarding how useful or dangerous they are. Ultimately though, implicits generally come down to simply having the compiler convert one class to another when you bring them into scope.
Why does this matter? Because in Spark there is an implicitclass called StringToColumn that endows a StringContext with additional functionality. As you can see, StringToColumn adds the $ method to the Scala class StringContext. This method produces a ColumnName, which extends Column.
The end result of all this is that the $ method allows you to treat the name of a column, represented as a String, as if it were the Column itself. Implicits, when used wisely, can produce convenient conversions like this to make development easier.
So let's use this to understand what you found:
df.select("app","renders").show -- succeeds because select takes multiple Strings
df.select($"app",$"renders").show -- succeeds because select takes multiple Columnss that result after the implicit conversions are applied
df.filter("renders = 265").show -- succeeds because Spark supports SQL-like filters
df.filter($"renders" = 265).show -- fails because $"renders" is of type Column after implicit conversion, and Columns use the custom === operator for equality (unlike the case in SQL).
df.filter($"renders" > 265).show -- succeeds because you're using a Column after implicit conversion and > is a function on Column.
$ is a way to convert a string to the column with that name.
Both options of select work originally because select can receive either a column or a string.
When you do the filter $"renders" = 265 is an attempt at assigning a number to the column. > on the other hand is a comparison method. You should be using === instead of =.
I want to insert the literal '${a}' into a table using anorm 2.5.2, which means I want to execute the bare SQL query
INSERT INTO `db`.`table` (`a`) VALUES ('${a}');
without using any anorm / string interpolation. When I try to do the following
SQL("INSERT INTO `db`.`table` (`a`) VALUES ('${a}');").execute()
I get an anorm.Sql$MissingParameter: Missing parameter value exception because it tries to use anorm interpolation on ${a} but no value a is available in the scope.
How to escape the anorm / string interpolations $... and ${...}?
Escape a dollar sign in string interpolation doesn't seem to work here.
You can make ${a} the value of a parameter, i.e.
SQL("""INSERT INTO db.table (a) VALUES ({x})""").on("x" -> s"$${a}")
(s"$${a}" is the way to write "${a}" without getting a warning about possible missing interpolators).
The same can be written equivalently as
val lit = s"$${a}"
SQL"""INSERT INTO db.table (a) VALUES ($lit)"""
The below will probably work, but I am not sure:
SQL"INSERT INTO db.table (a) VALUES ('$${a}')"
It may also be worth asking if it's intentional behavior or a bug: when talking about parametrized SQL queries, it doesn't make sense to have a parameter inside '.
Consider a repository/DAO method like this, which works great:
def countReports(customerId: Long, createdSince: ZonedDateTime) =
DB.withConnection {
implicit c =>
SQL"""SELECT COUNT(*)
FROM report
WHERE customer_id = $customerId
AND created >= $createdSince
""".as(scalar[Int].single)
}
But what if the method is defined with optional parameters:
def countReports(customerId: Option[Long], createdSince: Option[ZonedDateTime])
Point being, if either optional argument is present, use it in filtering the results (as shown above), and otherwise (in case it is None) simply leave out the corresponding WHERE condition.
What's the simplest way to write this method with optional WHERE conditions? As Anorm newbie I was struggling to find an example of this, but I suppose there must be some sensible way to do it (that is, without duplicating the SQL for each combination of present/missing arguments).
Note that the java.time.ZonedDateTime instance maps perfectly and automatically into Postgres timestamptz when used inside the Anorm SQL call. (Trying to extract the WHERE condition as a string, outside SQL, created with normal string interpolation did not work; toString produces a representation not understood by the database.)
Play 2.4.4
One approach is to set up filter clauses such as
val customerClause =
if (customerId.isEmpty) ""
else " and customer_id={customerId}"
then substitute these into you SQL:
SQL(s"""
select count(*)
from report
where true
$customerClause
$createdClause
""")
.on('customerId -> customerId,
'createdSince -> createdSince)
.as(scalar[Int].singleOpt).getOrElse(0)
Using {variable} as opposed to $variable is I think preferable as it reduces the risk of SQL injection attacks where someone potentially calls your method with a malicious string. Anorm doesn't mind if you have additional symbols that aren't referenced in the SQL (i.e. if a clause string is empty). Lastly, depending on the database(?), a count might return no rows, so I use singleOpt rather than single.
I'm curious as to what other answers you receive.
Edit: Anorm interpolation (i.e. SQL"...", an interpolation implementation beyond Scala's s"...", f"..." and raw"...") was introduced to allow the use $variable as equivalent to {variable} with .on. And from Play 2.4, Scala and Anorm interpolation can be mixed using $ for Anorm (SQL parameter/variable) and #$ for Scala (plain string). And indeed this works well, as long as the Scala interpolated string does not contains references to an SQL parameter. The only way, in 2.4.4, I could find to use a variable in an Scala interpolated string when using Anorm interpolation, was:
val limitClause = if (nameFilter="") "" else s"where name>'$nameFilter'"
SQL"select * from tab #$limitClause order by name"
But this is vulnerable to SQL injection (e.g. a string like it's will cause a runtime syntax exception). So, in the case of variables inside interpolated strings, it seems it is necessary to use the "traditional" .on approach with only Scala interpolation:
val limitClause = if (nameFilter="") "" else "where name>{nameFilter}"
SQL(s"select * from tab $limitClause order by name").on('limitClause -> limitClause)
Perhaps in the future Anorm interpolation could be extended to parse the interpolated string for variables?
Edit2: I'm finding there are some tables where the number of attributes that might or might not be included in the query changes from time to time. For these cases I'm defining a context class, e.g. CustomerContext. In this case class there are lazy vals for the different clauses that affect the sql. Callers of the sql method must supply a CustomerContext, and the sql will then have inclusions such as ${context.createdClause} and so on. This helps give a consistency, as I end up using the context in other places (such as total record count for paging, etc.).
Finally got this simpler approach posted by Joel Arnold to work in my example case, also with ZonedDateTime!
def countReports(customerId: Option[Long], createdSince: Option[ZonedDateTime]) =
DB.withConnection {
implicit c =>
SQL( """
SELECT count(*) FROM report
WHERE ({customerId} is null or customer_id = {customerId})
AND ({created}::timestamptz is null or created >= {created})
""")
.on('customerId -> customerId, 'created -> createdSince)
.as(scalar[Int].singleOpt).getOrElse(0)
}
The tricky part is having to use {created}::timestamptz in the null check. As Joel commented, this is needed to work around a PostgreSQL driver issue.
Apparently the cast is needed only for timestamp types, and the simpler way ({customerId} is null) works with everything else. Also, comment if you know whether other databases require something like this, or if this is a Postgres-only peculiarity.
(While wwkudu's approach also works fine, this definitely is cleaner, as you can see comparing them side to side in a full example.)
Is it possible to use a function import in a where clause in entity framework? I have tried the following, but I get a rather cryptic exception to which I cannot find any information about:
var q = MyContext.MyEntities.Where("MyContext.MyFunction(it.ID)")
(The function is set to return a Boolean)
System.Data.EntitySqlException: 'MyContext.MyFunction' cannot be resolved into a valid type constructor or function., near WHERE predicate, line 6, column 21..
Regards
Lee
The query you are trying to write is composing a call to a FunctionImport with a query over an EntitySet.
But because FunctionImports are wrappers around StoredProcedures, which are non-composable, this just won't work.
In order for something like this to work, theoretically the function would need to be a wrapper around something composable like a TVF (Table Value Function). But unfortunately TVFs aren't supported in the Entity Framework today.
Alex
I don't think you want to pass this expression as a string. You want a proper lambda expression like:
MyContext.MyEntities.Where( entity => MyContext.MyFunction(entity.ID ) );