I'm writing server that executes any select query on my db and returns json. I've done most of that task but I stuck with parsing nullable column to string.
val result = SQL("SELECT * FROM Table limit 5;")().map(_.asList.map({_.toString})).toList
val jsonResp = Json.toJson(result)
And that generates if the column could have null value string Some(123) instead of 123. I tried with match but I failed with compose that command. Maybe you had some similar problem and you know how to deal with that kind of response?
Edit:
I made some progress by adding pattern matching:
val result = SQL(query)()
.map(_.asList.map(
{
case Some(s) => s.toString
case None => ""
case v => v.toString
}
)).toList
but I'm not sure is it good way to solve that problem. Still waiting for ideas
Anorm is supporting nullable column as optional value.
There you return row as a list of raw value. It would be better to use the parser API to indicate how to extract properly values. E.g.
SQL("SELECT a, b, c ...").as(get[Option[String]]("a") ~ int("b") ~ str("c) map { case a ~ b ~ c => MyClass(a, b, c) }.*)
Returns SQL results as list of MyClass, with properties being in order Option[String], Int and String.
Anorm documentation has plenty of other examples.
It would probably be best to map optional entries in the database to optional fields in the json; most scala json libraries will render a field with Option type appropriately. It also seems odd that you would want to render an integer value as a string for json - json has a perfectly good numeric type for integers.
If you definitely want to convert an Option[Int] to a string in this fashion, the clearest way is probably o.map(_.toString).getOrElse(""); you could therefore write _.asList.map{_.map{_.toString}.getOrElse("")} rather than your pattern-match. I don't know the specific SQL library well enough to know whether the results are statically known to be of type Option or not; if the values are of type Any then you probably do need to match options and non-options the way you have.
Related
Here is a code that requires a change:
val activityDate = validation.select("activity_date").first.get(0).toString
When we run a job, 'activityDate' might return null as a result of query since there might not be any data in db. In this case we get NullPointerException. I need to update this code to avoid NPE.
I tried to do it in different ways but there is always smth missing. I should probably use Match Expression here but have face some errors while initializing it.
The usual way to model some kind of data that might or might not be there in Scala is the Option type. The Option type has two concrete implementations, Some for a value which is there and the None singleton to represent any absent value. Option conveniently has a constructor that wraps a nullable value and turns it into either a Some(value) for non-null values and None for nulls. You can use it as follows:
Option(validation.select("activity_date").first.get(0))
You can apply transformations to it using various combinators. If you want to transform the piece of data itself into something more meaningful for your application, map is usually a good call. The following applies the logic you had before:
val activityDate: Option[String] =
Option(validation.select("activity_date").first.get(0)).
map { activityDate => activityDate.toString }
Note that now activityDate is an Option itself, which means that you have to explicitly handle the case in which the data is not there. You can do so with a match on the concrete type of the option as follows:
activityDate match {
case Some(date) => // `date` is there for sure!
case None => // handle the `select` returned nothing
}
Altenrnatively if you want to apply a default value you can use the getOrElse method on the Option:
val activityDate: String =
Option(validation.select("activity_date").first.get(0)).
map { activityDate => activityDate.toString }.
getOrElse("No Data")
Another possibility to apply a default value on a None and a function to the value in a Some is using fold:
val activityDate: String =
Option(validation.select("activity_date").first.get(0)).
fold("No Data")(activityDate => activityDate.toString)
As a final note, you can shorten anonymous functions in these cases as follows:
val activityDate: String =
Option(validation.select("activity_date").first.get(0)).
fold("No Data")(_.toString)
Where _ is used to refer to the only parameter.
I am new to Scala and would like to learn the idiomatic way to solve common problems, as in pythonic for Python. My question regards reading JSON data with upickle, where the JSON value contains a string when present, and null when not present. I want to use a custom value to replace null. A simple example:
import upickle.default._
val jsonString = """[{"always": "foo", "sometimes": "bar"}, {"always": "baz", "sometimes": null}]"""
val jsonData = ujson.read(jsonString)
for (m <- jsonData.arr) {
println(m("always").str.length) // this will work
println(m("sometimes").str.length) // this will fail, Exception in thread "main" ujson.Value$InvalidData: Expected ujson.Str (data: null)
}
The issue is with the field "sometimes": when null, we cannot apply .str (or any other function mapping to a static type other than null). I am looking for something like m("sometimes").str("DEFAULT").length, where "DEFAULT" is the replacement for null.
Idea 1
Using pattern matching, the following works:
val sometimes = m("sometimes") match {
case s: ujson.Str => s.str
case _ => "DEFAULT"
}
println(sometimes.length)
Given Scala's concise syntax, this looks a bit complicated and will be repetitive when done for a number of values.
Idea 2
Answers to a related question mention creating a case class with default values. For my problem, the creation of a case class seems inflexible to me when different replacement values are needed depending depending on context.
Idea 3
Anwers to another question (not specific to upickle) discuss using Try().getOrElse(), i.e.:
import scala.util.Try
// ...
println(Try(m("sometimes").str).getOrElse("DEFAULT").length)
However, the discussion mentions that throwing an exception for a regular program path is expensive.
What are idiomatic, yet concise ways to solve this?
Idiomatic or scala way to do this by using scala's Option.
Fortunately, upickle Values offers them. Refer strOpt method in this source code.
Your problem in code is str methods in m("always").str and m("sometimes").str
With this code, you are prematurely assuming that all the values are strings. That's where the strOpt method comes. It either outputs a string if its value is a string or a None type if it not. And we can use getOrElse method coupled with it to decide what to throw if the value is None.
Following would be the optimum way to handle this.
val jsonString = """[{"always": "foo", "sometimes": "bar"}, {"always": "baz", "sometimes": null}]"""
for (m <- jsonData.arr) {
println(m("always").strOpt.getOrElse("").length)
println(m("sometimes").strOpt.getOrElse("").length)
}
Output:
3
3
3
0
Here if we get any value other than a string (null, float, int), the code will output it as an empty string. And its length will be calculated as 0.
Basically, this is similar to your "Idea1" approach but this is the scala way. Instead of "DEFAULT", I am throwing an empty string because you wouldn't want to have null values' length to be 7 (Length of string "DEFAULT").
val vJsonLoc = new HashMap[String, String]();
def getPrevJson(s:String) = vJsonLoc.get(s)
val previousFile = getPrevJson(s"/${site.toLowerCase}/$languagePath/$channel/v$v/$segment")
this returns
Some(/Users/abc/git/abc-c2c/)
on trying to append string previousFile + "/" + index + ".json"
the result is Some(/Users/abc/git/abc-c2c/)/0.json when the desired result is /Users/abc/git/abc-c2c/0.json
Guess this is some concept of Option that have not understood. New to scala.
As you pointed out, you're getting back an Option type, and not a direct reference to the String contained in your data structure. This is a very standard Scala practice, allowing you to better handle cases where an expected value might not be present in your data structure.
For example, in Java, this type of method typically returns the value if it exists and null if it doesn't. This means, however, subsequent code could be operating on the null value and thus you'd need further protection against exceptions.
In Scala, you're getting a reference to an object which may, or may not, have the value you expect. This is the Option type, and can be either Some (in which case the reference is accessible) or None (in which case you have several options for handling it).
Consider your code:
val vJsonLoc = new HashMap[String, String]();
def getPrevJson(s:String) = vJsonLoc.get(s)
val previousFile = getPrevJson(s"/${site.toLowerCase}/$languagePath/$channel/v$v/$segment")
If the HashMap returned String, your previousFile reference could point to either a null value or to a String value. You'd need to protect against a potential exception (regular practice in Java).
But in Scala, get is returning an Option type, which can be handled in a number of ways:
val previousFile = getPrevJson("your_string").getOrElse("")
//or
val previousFile = getPrevJson("your_string") match {
case Some(ref) => ref
case None => ""
}
The resulting reference previousFile will point to a String value: either the expected value ("get") or the empty string ("OrElse").
Scala Map on get returns Option. Use vJsonLoc(s) instead of vJsonLoc.get(s)
I have this weird requirement where data comes in as name ->value pair from a service and all the name-> value type is string only (which really they are not but that's how data is stored)
This is a simplified illustration.
case class EntityObject(type:String,value:String)
EntityObject("boolean","true")
now when getting that EntityObject if type is "boolean" then I have to make sure value is not anything else but boolean so first get type out and check value and cast value to that type. e.g in this case check value is boolean so have to cast string value to boolean to validate. If it was anything else besides boolean then it should fail.
e.g. if data came in as below, casting will fail and it should report back to the caller about this error.
EntityObject("boolean","1")
Due to this weird requirement it forces type conversion in validation code which doesn't look elegant and against type safe programming. Any elegant way to handle this in scala (may be in a more type safe manner)?
Here is where I'm going to channel an idea taken from a tweet by Miles Sabin in regards to hereogenous mappings (see this gist on github.) If you know the type of object mapping names a head of time you can use a nifty little trick which involves dependent types. Hold on, 'cause it's a wild ride:
trait AssocConv[K] { type V ; def convert: String => V }
def makeConv[V0](name: String, con: String => V0) = new AssocConv[name.type]{
V = V0
val convert = con
}
implicit val boolConv = makeConv("boolean", yourMappingFunc)
def convEntity(name: String, value: String)(implicit conv: AssocConv[name.type]): Try[conv.V] = Try{ conv.convert(value) }
I haven't tested this but it "should" work. I've also enclosed it in a Scala Try so that it catches exceptions thrown by your conversion function (in case you're doing things like _.toInt as the converter.)
You're really talking about conversion, not casting. Casting would be if the value really were an instance of Boolean at runtime, whereas what you have is a String representation of a Boolean.
If you're already working with a case class, I think a pattern matching expression would work pretty well here.
For example,
def convert(entity : EntityObject) : Any = entity match {
case EntityObject("boolean", "true") => true
case EntityObject("boolean", "false") => false
case EntityObject("string", s) => s
// TODO: add Regex-based matchers for numeric types
}
Anything that doesn't match one of the specified patterns would cause a MatchError, or you could put a catchall expression at the end to throw your own exception.
In this particular example, since the function returns Any, the calling coffee would need to do an actual type cast to get the specific type, but at least by that point all validation/conversion would have already been performed. Alternatively, you could just put the code that uses the values directly into the above function and avoid casting. I don't know what your specific needs are, so I can't offer anything more detailed.
I'm writing a data structure that converts the results of a database query. The raw structure is a java ResultSet and it would be converted to a map or class which permits accessing different fields on that data structure by either a named method call or passing a string into apply(). Clearly different values may have different types. In order to reduce burden on the clients of this data structure, my preference is that one not need to cast the values of the data structure but the value fetched still has the correct type.
For example, suppose I'm doing a query that fetches two column values, one an Int, the other a String. The result then names of the columns are "a" and "b" respectively. Some ideal syntax might be the following:
val javaResultSet = dbQuery("select a, b from table limit 1")
// with ResultSet, particular values can be accessed like this:
val a = javaResultSet.getInt("a")
val b = javaResultSet.getString("b")
// but this syntax is undesirable.
// since I want to convert this to a single data structure,
// the preferred syntax might look something like this:
val newStructure = toDataStructure[Int, String](javaResultSet)("a", "b")
// that is, I'm willing to state the types during the instantiation
// of such a data structure.
// then,
val a: Int = newStructure("a") // OR
val a: Int = newStructure.a
// in both cases, "val a" does not require asInstanceOf[Int].
I've been trying to determine what sort of data structure might allow this and I could not figure out a way around the casting.
The other requirement is obviously that I would like to define a single data structure used for all db queries. I realize I could easily define a case class or similar per call and that solves the typing issue, but such a solution does not scale well when many db queries are being written. I suspect some people are going to propose using some sort of ORM, but let us assume for my case that it is preferred to maintain the query in the form of a string.
Anyone have any suggestions? Thanks!
To do this without casting, one needs more information about the query and one needs that information at compiole time.
I suspect some people are going to propose using some sort of ORM, but let us assume for my case that it is preferred to maintain the query in the form of a string.
Your suspicion is right and you will not get around this. If current ORMs or DSLs like squeryl don't suit your fancy, you can create your own one. But I doubt you will be able to use query strings.
The basic problem is that you don't know how many columns there will be in any given query, and so you don't know how many type parameters the data structure should have and it's not possible to abstract over the number of type parameters.
There is however, a data structure that exists in different variants for different numbers of type parameters: the tuple. (E.g. Tuple2, Tuple3 etc.) You could define parameterized mapping functions for different numbers of parameters that returns tuples like this:
def toDataStructure2[T1, T2](rs: ResultSet)(c1: String, c2: String) =
(rs.getObject(c1).asInstanceOf[T1],
rs.getObject(c2).asInstanceOf[T2])
def toDataStructure3[T1, T2, T3](rs: ResultSet)(c1: String, c2: String, c3: String) =
(rs.getObject(c1).asInstanceOf[T1],
rs.getObject(c2).asInstanceOf[T2],
rs.getObject(c3).asInstanceOf[T3])
You would have to define these for as many columns you expect to have in your tables (max 22).
This depends of course on that using getObject and casting it to a given type is safe.
In your example you could use the resulting tuple as follows:
val (a, b) = toDataStructure2[Int, String](javaResultSet)("a", "b")
if you decide to go the route of heterogeneous collections, there are some very interesting posts on heterogeneous typed lists:
one for instance is
http://jnordenberg.blogspot.com/2008/08/hlist-in-scala.html
http://jnordenberg.blogspot.com/2008/09/hlist-in-scala-revisited-or-scala.html
with an implementation at
http://www.assembla.com/wiki/show/metascala
a second great series of posts starts with
http://apocalisp.wordpress.com/2010/07/06/type-level-programming-in-scala-part-6a-heterogeneous-list%C2%A0basics/
the series continues with parts "b,c,d" linked from part a
finally, there is a talk by Daniel Spiewak which touches on HOMaps
http://vimeo.com/13518456
so all this to say that perhaps you can build you solution from these ideas. sorry that i don't have a specific example, but i admit i haven't tried these out yet myself!
Joschua Bloch has introduced a heterogeneous collection, which can be written in Java. I once adopted it a little. It now works as a value register. It is basically a wrapper around two maps. Here is the code and this is how you can use it. But this is just FYI, since you are interested in a Scala solution.
In Scala I would start by playing with Tuples. Tuples are kinda heterogeneous collections. The results can be, but not have to be accessed through fields like _1, _2, _3 and so on. But you don't want that, you want names. This is how you can assign names to those:
scala> val tuple = (1, "word")
tuple: ([Int], [String]) = (1, word)
scala> val (a, b) = tuple
a: Int = 1
b: String = word
So as mentioned before I would try to build a ResultSetWrapper around tuples.
If you want "extract the column value by name" on a plain bean instance, you can probably:
use reflects and CASTs, which you(and me) don't like.
use a ResultSetToJavaBeanMapper provided by most ORM libraries, which is a little heavy and coupled.
write a scala compiler plugin, which is too complex to control.
so, I guess a lightweight ORM with following features may satisfy you:
support raw SQL
support a lightweight,declarative and adaptive ResultSetToJavaBeanMapper
nothing else.
I made an experimental project on that idea, but note it's still an ORM, and I just think it may be useful to you, or can bring you some hint.
Usage:
declare the model:
//declare DB schema
trait UserDef extends TableDef {
var name = property[String]("name", title = Some("姓名"))
var age1 = property[Int]("age", primary = true)
}
//declare model, and it mixes in properties as {var name = ""}
#BeanInfo class User extends Model with UserDef
//declare a object.
//it mixes in properties as {var name = Property[String]("name") }
//and, object User is a Mapper[User], thus, it can translate ResultSet to a User instance.
object `package`{
#BeanInfo implicit object User extends Table[User]("users") with UserDef
}
then call raw sql, the implicit Mapper[User] works for you:
val users = SQL("select name, age from users").all[User]
users.foreach{user => println(user.name)}
or even build a type safe query:
val users = User.q.where(User.age > 20).where(User.name like "%liu%").all[User]
for more, see unit test:
https://github.com/liusong1111/soupy-orm/blob/master/src/test/scala/mapper/SoupyMapperSpec.scala
project home:
https://github.com/liusong1111/soupy-orm
It uses "abstract Type" and "implicit" heavily to make the magic happen, and you can check source code of TableDef, Table, Model for detail.
Several million years ago I wrote an example showing how to use Scala's type system to push and pull values from a ResultSet. Check it out; it matches up with what you want to do fairly closely.
implicit val conn = connect("jdbc:h2:f2", "sa", "");
implicit val s: Statement = conn << setup;
val insertPerson = conn prepareStatement "insert into person(type, name) values(?, ?)";
for (val name <- names)
insertPerson<<rnd.nextInt(10)<<name<<!;
for (val person <- query("select * from person", rs => Person(rs,rs,rs)))
println(person.toXML);
for (val person <- "select * from person" <<! (rs => Person(rs,rs,rs)))
println(person.toXML);
Primitives types are used to guide the Scala compiler into selecting the right functions on the ResultSet.