I'm trying to write a quick data browser for a database using Squeryl but I have difficulty iterating over all the tables in a generic way. Based on the Squeryl SchoolDb example i tried the following:
def browseTable(name: String) = {
SchoolDb.tables.find(_.name == name) map { t=>
val fields = t.posoMetaData.fieldsMetaData
val rows = from (t) (s => select(s))
// Print the columns
println(fields.map(_.columnName).mkString("\t"))
rows map { row =>
println(fields.map(f => f.get(row)).mkstring("\t"))
}
}
The compiler is not very happy with this attempt (Missing type type for 'row') and I can sort-of understand its dilemma. Explicitly declaring the parametr as Any just changes the comilation error to "No implicit view available from Any => org.squeryl.dsl.ast.TypedExpressionNode[_]" on 'f.get(row)'
How Can I either fix this issue or change the models (maybe adding a trait of some sort) to enable generic access to all data in all tables?
The compiler complains because f.get method expects an AnyRef parameter. AFAIK, in Scala everything can be safely cast to AnyRef - the compiler will force necessary boxing if needed. So I think this should work: f.get(row.asInstanceOf[AnyRef])
EDIT: I just tested this and it works.
Related
I have the following protobuf setup:
import "google/protobuf/any.proto";
message EntityEnvelope {
string id = 1;
string entity_type = 2;
google.protobuf.Any entity = 3;
}
message EntityABC {
string name = 1;
}
message EntityXYZ {
string desc = 1;
}
where EntityEnvelope.entity can be any type packed as google.protobuf.Any. Each protobuf message is stored in the disk encoded as Base64.
When reading these messages, it works perfectly if I use the specific entity type at compile type when unpacking:
import scalapb.spark.Implicits._
spark.read.format("text")
.load("/tmp/entity").as[String]
.map { s => Base64.getDecoder.decode(s) }
.map { bytes => EntityEnvelope.parseFrom(bytes) }
.map { envelope => envelope.getEntity.unpack[EntityXYZ]}
.show()
But I want to use the same code to read any kind of entity at runtime, without having to specify its type. The "closer" I got was (but doesn't even compile):
val entityClass = Class.forName(qualifiedEntityClassNameFromRuntime)
spark.read.format("text")
.load("/tmp/entity").as[String]
.map { s => Base64.getDecoder.decode(s) }
.map { bytes => EntityEnvelope.parseFrom(bytes) }
.map { envelope => toJavaProto(envelope.getEntity).unpack(entityClass)} // ERROR: No implicits found for Encoder[Any]
.show()
Since Any contains the typeUrl, would be possible to find the correct descriptor and unpack it automatically at runtime, without having to specify the type at compile time?
To unpack Anys that you don't know their type at compile time, you will need to build a map from typeUrl to the companion object of all the types you might expect, and use that to call unpack. It can be done like this:
val typeUrlToCompanion = Map[String, scalapb.GeneratedMessageCompanion[_ <: scalapb.GeneratedMessage]](
"type.googleapis.com/myexample.Person2" -> Person2
// more types
)
Then unpack with
envelope.getEntity.unpack(typeUrlToCompanion(envelope.getEntity.typeUrl)
which would give you back a GeneratedMessage - the parent trait of all ScalaPB messages.
However, after you do that, you are going to hit another problem. Spark dataframes are structured, they have a schema with consists of named columns and types, like a table in a relational database. The schema needs to be known at runtime when the dataframe is constructed. However, you seem to want to create a dataframe where each row has a different type (whatever the Any happens to be), and the collection of types only becomes known when unpacking - so that's incompatible with the design of dataframes.
Depends on what you want to do, there are few options to consider:
Instead of using Any, use the oneof feature of protobufs, so all the possible types are known and a schema could be automatically created (and you won't have to deal with unpacking Anys)
Alternatively, partition the dataframe by typeUrl, so you end up with a different dataframes where in each one, all the items are of of the same type. Then unpack each one with its known type. Can be done generically with the typeUrlToCompanion approach above.
I'm using Slick 3.1.0 and Slick-pg 0.10.0. I have an enum as:
object UserProviders extends Enumeration {
type Provider = Value
val Google, Facebook = Value
}
Following the test case, it works fine with the column mapper simply adding the following implicit mapper into my customized driver.
implicit val userProviderMapper = createEnumJdbcType("UserProvider", UserProviders, quoteName = true)
However, when using plain SQL, I encountered the following compilation error:
could not find implicit value for parameter e: slick.jdbc.SetParameter[Option[models.UserProviders.Provider]]
I could not find any document about this. How can I write plain SQL with enum in slick? Thanks.
You need to have an implicit of type SetParameter[T] in scope which tells slick how to set parameters from some custom type T that it doesn't already know about. For example:
implicit val setInstant: SetParameter[Instant] = SetParameter { (instant, pp) =>
pp.setTimestamp(new Timestamp(instant.toEpochMilli))
}
The type of pp is PositionedParameters.
You might also come across the need to tell slick how to extract a query result into some custom type T that it doesn't already know about. For this, you need an implicit GetResult[T] in scope. For example:
implicit def getInstant(implicit get: GetResult[Long]): GetResult[Instant] =
get andThen (Instant.ofEpochMilli(_))
I am experiencing a strange behaviour with slick and I would like some help figuring out why it happens. The issue is that I have a query that reads as follows:
db.withSession { implicit session =>
tables.users.where(_.id === UserId(1)).firstOption
}
This does not compile producing an error as follows:
inferred type arguments [Boolean] do not conform to method where's type parameter bounds [T <: scala.slick.lifted.Column[_]]
But if I rewrite the code as:
db.withSession { implicit session =>
(for {
u <- tables.users if u.id === UserId(1)
} yield u).firstOption
}
It compiles and works fine.
The table is defined as follows:
class Users(tag: Tag) extends Table[User](tag, "users") {
def id = column[UserId]("id", O.PrimaryKey, O.AutoInc, O.NotNull)
}
And I have an implicit conversion to map the UserId type:
implicit lazy val userIdColumnType = MappedColumnType.base[UserId, Int](_.value, UserId(_))
It looks like a type inference problem, but I can't really understand why it should happen.
Anyone has any on why this should behave differently in the two scenario I reported?
EDIT: After some investigation I found that when using where the implicit conversion for the userIdColumnType has to be in scope, while with the for comprehension it is not needed. Is there a good explanation for this?
You are using === from ScalaTest. It returns a Boolean. Slick's === returns a Column[Boolean]. The methods filter and where prevent using Boolean (at least in the latest version of Slick), to protect you from accidentally using == or also from using ScalaTest's === in your case, which does a local comparison of the underlying values instead of an equality comparison in the database, which is what you actualy want. For comprehensions are desugared to withFilter and can sometimes generate a Boolean value, so unfortunately we cannot disallow Boolean for comprehensions.
To fix this you need to make sure, that Slick's === is picked in queries. Maybe you can affect this with the import order or scope. Or if you are unlucky you can't and they are incompatible.
I am not sure how the userIdColumnType interacts here at the moment.
I fixed this by importing my driver api and making sure it was in scope
i.e. as I'm using a postgresql driver,
import PostgresProfile.api._
I was tired of always doing something like the following in order to do database access using slick for each of my domain entities.
database withSession {
implicit session =>
val entities = TableQuery[EntityTable]
val id = //Some ID
val q = for {
e <- entities if e.id === id
} yield (e)
val entity = q.first
}
(Note: EntityTable was defined like described here)
So I decided that I want a generic database access object that handles this for me. The usage should look something like
[...]
val entityDAO = new GenericDAO[Entity, EntityTable, String]()
[...]
database withSession { implicit session =>
val id = // Some ID
val entity = entityDAO.get(id)
}
My try of the implementation of the GenericDAO looks like this
class GenericDAO[T, TB, PK](implicit session: Session) {
val entities = TableQuery[TB] // Line 1
def get(id: PK): T = {
val q = for {
e <- entities
} yield (e)
val res: T = q.first
res
}
}
But line 1 leaves me with a compiler error stating that something is wrong with the TB argument.
Multiple markers at this line
- type arguments [TB] conform to the bounds of none of the overloaded alternatives of value apply: [E <:
scala.slick.lifted.AbstractTable[]]=>
scala.slick.lifted.TableQuery[E,E#TableElementType] [E <:
scala.slick.lifted.AbstractTable[]](cons: scala.slick.lifted.Tag =>
E)scala.slick.lifted.TableQuery[E,E#TableElementType]
- wrong number of type parameters for overloaded method value apply with alternatives: [E <: scala.slick.lifted.AbstractTable[]]=>
scala.slick.lifted.TableQuery[E,E#TableElementType] [E <:
scala.slick.lifted.AbstractTable[]](cons: scala.slick.lifted.Tag =>
E)scala.slick.lifted.TableQuery[E,E#TableElementType]
Any suggesions on this issue? Or maybe I am wrong and it is supposed to be implemented in another way. I am open for any solution. Thanks!
First of all, you could write
val entities = TableQuery[EntityTable] // put in a central place for re-use
and then
database.withSession(
(for {
e <- entities if e.id === /*Some ID*/
} yield e).first()(_)
)
or this
database.withSession(entities.filter(_.id === /*Some ID*/).first()(_))
or this
val get = entities.findBy(_.id) // <- reuse this
database.withSession(get(/*Some ID*/).first()(_))
for brevity. This probably makes your whole DAO unnecessary (which is great :)!).
Regarding the error message you got. TableQuery[TB]is a macro, which is a short-hand for TableQuery(tag => new TB(tag)), TB must be a Table and support object creation. You cannot just use the TableQuery macro on an unconstrained type parameter your got from a DAO wrapper. You could constraint TB <: Table[_] but it still wouldn't support object creation, which you cannot constraint in Scala. You could only provide a factory to your DAO (a common pattern is to fetch one as an implicit argument), but that all does not make sense, when you can just write your TableQuery once and store it in a globally accessible place.
Update:
The shortcut works for all of these methods the same way. It's plain Scala. You just turn the method into a function and pass it to the higher-order function withSession, which requires a function from session to anything. Just be aware, that some Slick methods have an empty argument list, which requires ()(_) to turn them into a function and some only have the implicit argument list, which requires only (_). E.g. database.withSession(entities.filter(_.id === /*Some ID*/).delete(_)).
If you wonder about the _. Scala distinguishes methods from functions. def foo(a: A, b: B, ...): R is a method but can be turned into a function of type (A,B,C) => R using foo _. This conversion is called eta expansion and googling for it will turn up more info. Sometimes when a function expected, but you provide a method, the Scala compiler infers the _ and you don't have to write it explicitly. You can also provide some parameters and use _ in place of the parameters you don't want to apply yet. In that case you partially apply the method and get a function back. This is what we do here. We use _ in the place where the methods usually expect a session and get a function back that takes a session. When exactly you have to use _ or (_) or ()(_) has to do with the method signatures and the details interplay between implicit argument lists, nullary methods, methods with empty argument lists, which is general Scala knowledge worth researching at some point.
This was a huge help to me. A complete working generic dao https://gist.github.com/lshoo/9785645
I've been working with Scala Macros and have the following code in the macro:
val fieldMemberType = fieldMember.typeSignatureIn(objectType) match {
case NullaryMethodType(tpe) => tpe
case _ => doesntCompile(s"$propertyName isn't a field, it must be another thing")
}
reify{
new TypeBuilder() {
type fieldType = fieldMemberType.type
}
}
As you can see, I've managed to get a c.universe.Type fieldMemberType. This represents the type of certain field in the object. Once I get that, I want to create a new TypeBuilder object in the reify. TypeBuilder is an abstract class with an abstract parameter. This abstract parameter is fieldType. I want this fieldType to be the type that I've found before.
Running the code shown here returns me a fieldMemberType not found. Is there any way that I can get the fieldMemberType to work inside the reify clause?
The problem is that the code you pass to reify is essentially going to be placed verbatim at the point where the macro is being expanded, and fieldMemberType isn't going to mean anything there.
In some cases you can use splice to sneak an expression that you have at macro-expansion time into the code you're reifying. For example, if we were trying to create an instance of this trait:
trait Foo { def i: Int }
And had this variable at macro-expansion time:
val myInt = 10
We could write the following:
reify { new Foo { def i = c.literal(myInt).splice } }
That's not going to work here, which means you're going to have to forget about nice little reify and write out the AST by hand. You'll find this happens a lot, unfortunately. My standard approach is to start a new REPL and type something like this:
import scala.reflect.runtime.universe._
trait TypeBuilder { type fieldType }
showRaw(reify(new TypeBuilder { type fieldType = String }))
This will spit out several lines of AST, which you can then cut and paste into your macro definition as a starting point. Then you fiddle with it, replacing things like this:
Ident(TypeBuilder)
With this:
Ident(newTypeName("TypeBuilder"))
And FINAL with Flag.FINAL, and so on. I wish the toString methods for the AST types corresponded more exactly to the code it takes to build them, but you'll pretty quickly get a sense of what you need to change. You'll end up with something like this:
c.Expr(
Block(
ClassDef(
Modifiers(Flag.FINAL),
anon,
Nil,
Template(
Ident(newTypeName("TypeBuilder")) :: Nil,
emptyValDef,
List(
constructor(c),
TypeDef(
Modifiers(),
newTypeName("fieldType"),
Nil,
TypeTree(fieldMemberType)
)
)
)
),
Apply(Select(New(Ident(anon)), nme.CONSTRUCTOR), Nil)
)
)
Where anon is a type name you've created in advance for your anonymous class, and constructor is a convenience method I use to make this kind of thing a little less hideous (you can find its definition at the end of this complete working example).
Now if we wrap this expression up in something like this, we can write the following:
scala> TypeMemberExample.builderWithType[String]
res0: TypeBuilder{type fieldType = String} = $1$$1#fb3f1f3
So it works. We've taken a c.universe.Type (which I get here from the WeakTypeTag of the type parameter on builderWithType, but it will work in exactly the same way with any old Type) and used it to define the type member of our TypeBuilder trait.
There is a simpler approach than tree writing for your use case. Indeed I use it all the time to keep trees at bay, as it can be really difficult to program with trees. I prefer to compute types and use reify to generate the trees. This makes much more robust and "hygienic" macros and less compile time errors. IMO using trees must be a last resort, only for a few cases, such as tree transforms or generic programming for a family of types such as tuples.
The tip here is to define a function taking as type parameters, the types you want to use in the reify body, with a context bound on a WeakTypeTag. Then you call this function by passing explicitly the WeakTypeTags you can build from universe Types thanks to the context WeakTypeTag method.
So in your case, that would give the following.
val fieldMemberType: Type = fieldMember.typeSignatureIn(objectType) match {
case NullaryMethodType(tpe) => tpe
case _ => doesntCompile(s"$propertyName isn't a field, it must be another thing")
}
def genRes[T: WeakTypeTag] = reify{
new TypeBuilder() {
type fieldType = T
}
}
genRes(c.WeakTypeTag(fieldMemberType))