I am trying to pass generalized class as parameter . If I give Case class and values then it is working fine . But , would like to make it generic.
class DB[T] {
lazy val ctx = new OracleJdbcContext(SnakeCase, "ctx")
import ctx._
def insert(input: T) = {
val q = quote {
query[T].insert(lift(input))
}
ctx.run(q)
}
}
I am getting errors saying::
"Can't find an implicit SchemaMeta for type T
Can't find Encoder for type 'T'. Note that Encoders are invariant"
But, If I give actual class name then it is going well.
case class Location(street:String,pinCode:Int)
class DB {
lazy val ctx = new OracleJdbcContext(SnakeCase, "ctx")
import ctx._
val q = quote {
query[Location].insert(Location("2ndcross",500001))
}
ctx.run(q)
}
You need to have SchemaMeta[T] in scope to be able to execute queries using this type. Dummy solution would be to demand it as parameter constraint (and so implicit class parameter) like this
class DB[T: SchemaMeta]
but this wouldn't work, because it's ctx who provides those instances.
I believe that you will need to follow examples shown here: https://getquill.io/#contexts-dependent-contexts
But even then what you want may not be achievable.
Imporant thing to understand when working with quill is that almost everything there is based on macros and if you abstract things away then there is not enough information for these macros to act. So you either need to duplicate code requiring macros or wrap the code that is meant to be generic in your own macro.
Related
I feel like I should preface this with the fact that I'm building my projects with sbt.
My problem is that, if at compile time a method returns something of an unimported type, in the file where I call the method, as long as I use type inference, everything compiles. Once I try to assign the unimported type to the var/val which I created with the return value of my function, I get a compiler error.
Lets say I have two classes in two package. Class App in package main and class Imported in package libraries. Lets further more say that we have a class ImportedFactory in the package main and that this class has a method for creating objects of the type Imported.
This code compiles just fine:
class App() {
// method return object of type Imported
val imp = ImportedFactory.createImportedObject()
}
This doesn't:
class App() {
// method return object of type Imported
val imp : Imported = ImportedFactory.createImportedObject()
}
This yet again does:
import libraries.Imported
class App() {
// method return object of type Imported
val imp : Imported = ImportedFactory.createImportedObject()
}
This seems like rather strange behavior. Is this normal for languages with type inference at compile time and I've yet to notice it until now in go/C++ due to my ignorance ?
Does one of the two valid approaches (import&explicit type vs infered) have advantages/drawback over the other ? (expect for, of course, one being more explicit and verbose and the other one being shorter)
Is this black magic or does the Scala compiler accomplish these deductions in a rather straight forward way ?
The only thing importing does is making a not fully qualified name available in the current scope. You could just as well write this:
class App() {
val imp: libraries.Imported = ImportedFactory.createImportedObject()
}
The reason you import libraries.Imported is for making the shorter name Imported available for you to write. If you let the compiler infer the type, you don't mention the type in your code, so you don't have to import its shorter name.
And by the way: this has nothing to do with dynamic casting in C++. The only mechanism at work in your code is type inference.
note: You'll get better search results with the term type inference
With val imp = ImportedFactory.createImportedObject() you are letting the compiler figure out what type imp should be based on type inference. Whatever type createImportObject returns, that's what type imp is.
With val imp : Imported = ImportedFactory.createImportedObject() you are explicitly stating that imp is an Imported. But the compiler doesn't know what you mean by that unless you... import... it.
Both approaches have merit:
inferred types
Inferred types are great for when you're throwing together code where the type should be obvious:
val i = 1 // obviously `i` is an int
val j = i + 10 // obviously still an int
It's also great for local vars/vals where the type would be too much of a pain to write
val myFoo: FancyAbstractThing[TypeParam, AnotherTypeParam[OhNoMoreTypeParams]] = ...
// vs
val myFoo = FancyThingFactory.makeANewOne()
The downside is that if you have allowed a public def/val to have an inferred type, it can be more difficult to determine how to use that method. For this reason, omitting type annotations is typically only used for simple constants, and in local vals/vars that "client code" doesn't have to look at.
explicit types
When you do want to write library-ish code (i.e. public vals/defs), the convention is to explicitly-type them.
Probably the simplest reason for this is because this:
def myLibraryMethod = {
// super complicated implementation
}
is harder to understand than
def myLibraryMethod: String = {
// super complicated implementation
}
Another benefit to explicitly-typing your code is when you want to expose a less-specific type than what the value actually is:
val invalidNumbers: Set[Int] = TreeSet(4, 8, 15, 16, 23, 42)
In this example, you don't want client code to need to care that your invalidNumbers is actually a TreeSet. That's an implementation detail. In this case you're hiding some information that, while true, would be distracting.
I have a following question. Our project has a lot of code, that runs tests in Scala. And there is a lot of code, that fills the fields like this:
production.setProduct(new Product)
production.getProduct.setUuid("b1253a77-0585-291f-57a4-53319e897866")
production.setSubProduct(new SubProduct)
production.getSubProduct.setUuid("89a877fa-ddb3-3009-bb24-735ba9f7281c")
Eventually, I grew tired from this code, since all those fields are actually subclasses of the basic class that has the uuid field, so, after thinking a while, I wrote the auxiliary function like this:
def createUuid[T <: GenericEntity](uuid: String)(implicit m : Manifest[T]) : T = {
val constructor = m.runtimeClass.getConstructors()(0)
val instance = constructor.newInstance().asInstanceOf[T]
instance.setUuid(uuid)
instance
}
Now, my code got two times shorter, since now I can write something like this:
production.setProduct(createUuid[Product]("b1253a77-0585-291f-57a4-53319e897866"))
production.setSubProduct(createUuid[SubProduct]("89a877fa-ddb3-3009-bb24-735ba9f7281c"))
That's good, but I am wondering, if I could somehow implement the function createUuid so the last bit would like this:
// Is that really possible?
production.setProduct(createUuid("b1253a77-0585-291f-57a4-53319e897866"))
production.setSubProduct(createUuid("89a877fa-ddb3-3009-bb24-735ba9f7281c"))
Can scala compiler guess, that setProduct expects not just a generic entity, but actually something like Product (or it's subclass)? Or there is no way in Scala to implement this even shorter?
Scala compiler won't infer/propagate the type outside-in. You could however create implicit conversions like:
implicit def stringToSubProduct(uuid: String): SubProduct = {
val n = new SubProduct
n.setUuid(uuid)
n
}
and then just call
production.setSubProduct("89a877fa-ddb3-3009-bb24-735ba9f7281c")
and the compiler will automatically use the stringToSubProduct because it has applicable types on the input and output.
Update: To have the code better organized I suggest wrapping the implicit defs to a companion object, like:
case class EntityUUID(uuid: String) {
uuid.matches("[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}") // possible uuid format check
}
case object EntityUUID {
implicit def toProduct(e: EntityUUID): Product = {
val p = new Product
p.setUuid(e.uuid)
p
}
implicit def toSubProduct(e: EntityUUID): SubProduct = {
val p = new SubProduct
p.setUuid(e.uuid)
p
}
}
and then you'd do
production.setProduct(EntityUUID("b1253a77-0585-291f-57a4-53319e897866"))
so anyone reading this could have an intuition where to find the conversion implementation.
Regarding your comment about some generic approach (having 30 types), I won't say it's not possible, but I just do not see how to do it. The reflection you used bypasses the type system. If all the 30 cases are the same piece of code, maybe you should reconsider your object design. Now you can still implement the 30 implicit defs by calling some method that uses reflection similar what you have provided. But you will have the option to change it in the future on just this one (30) place(s).
I have a copy of Programming MapReduce with Scalding by Antonios Chalkiopoulos. In the book he discusses the External Operations design pattern for Scalding code. You can see an example on his website here. I have made a choice to use the Type Safe API. Naturally, this introduces new challenges but I prefer it over the Fields API which is what is heavily discussed in the book I have previously mentioned and the site.
I am wondering how people have implemented the external operations pattern with the Type Safe API. My initial implementation is as follows:
I create a class that extends com.twitter.scalding.Job which will
serve as my Scalding job class where I will 'manage arguments, define
taps, and use external operations to construct data processing
pipelines'.
I create an object where I define my functions to be used in the Type
Safe pipes. Because the Type Safe pipes take as arguments a function,
I can then just pass the functions in the object as arguments to the
pipes.
This creates code that looks like this:
class MyJob(args: Args) extends Job(args) {
import MyOperations._
val input_path = args(MyJob.inputArgPath)
val output_path = args(MyJob.outputArgPath)
val eventInput: TypedPipe[(LongWritable, Text)] = this.mode match {
case m: HadoopMode => TypedPipe.from(WritableSequenceFile[LongWritable, Text](input_path))
case _ => TypedPipe.from(WritableSequenceFile[LongWritable, Text](input_path))
}
val eventOutput: FixedPathSource with TypedSink[(LongWritable, Text)] with TypedSource[(LongWritable, Text)] = this.mode match {
case m: HadoopMode => WritableSequenceFile[LongWritable, Text](output_path)
case _ => TypedTsv[(LongWritable, Text)](output_path)
}
val validatedEvents: TypedPipe[(LongWritable, Either[Text, Event])] = eventInput.map(convertTextToEither).fork
validatedEvents.filter(isEvent).map(removeEitherWrapper).write(eventOutput)
}
object MyOperations {
def convertTextToEither(v: (LongWritable, Text)): (LongWritable, Either[Text, Event]) = {
...
}
def isEvent(v: (LongWritable, Either[Text, Event])): Boolean = {
...
}
def removeEitherWrapper(v: (LongWritable, Either[Text, Event])): (LongWritable, Text) = {
...
}
}
As you can see, the functions that are passed to the Scalding Type Safe operations are kept separate from the job itself. While this is not as 'clean' as the external operations pattern presented, this is a quick way to write this kind of code. Additionally, I can use JUnitRunner for doing job level integration tests and ScalaTest for function level unit tests.
The main point of this post though is to ask how people are doing this sort of thing? The documentation around the internet for Scalding Type Safe API is sparse. Are there more functional Scala friendly ways for doing this? Am I missing a key component here for the design pattern? I sort of feel nervous about this because with the Fields API you can write unit tests on pipes with ScaldingTest. As far as I know, you can't do that with TypedPipes. Please let me know if there is a generally agreed upon pattern for Scalding Type Safe API or how you create reusable, modular, and testable Type Safe API code. Thanks for the help!
Update 2 after Antonios' reply
Thank you for the reply. That was basically the answer I was looking for. I wanted to continue the conversation. The main issue I see in your answer as I commented was that this implementation expects a specific type implementation but what if the types change throughout your job? I have explored this code and it seems to work but it seems hacked on.
def self: TypedPipe[Any]
def testingPipe: TypedPipe[(LongWritable, Text)] = self.map(
(firstVar: Any) => {
val tester = firstVar.asInstanceOf[(LongWritable, Text)]
(tester._1, tester._2)
}
)
The upside to this is I declare one implementation of self but the downside is this ugly type casting. Additionally, I have not tested this out in depth with a more complex pipeline. So basically, what are your thoughts on how to handle types as they change with only one self implementation for cleanliness/brevity?
Scala extension methods are implemented using implicit classes.
You add to the compiler the capability of converting a TypedPipe into a (wrapper) class that contains your external operations:
import com.twitter.scalding.TypedPipe
import com.twitter.scalding._
import cascading.flow.FlowDef
class MyJob(args: Args) extends Job(args) {
implicit class MyOperationsWrapper(val self: TypedPipe[Double]) extends MyOperations with Serializable
val pipe = TypedPipe.from(TypedTsv[Double](args("input")))
val result = pipe
.operation1
.operation2(x => x*2)
.write(TypedTsv[Double](args("output")))
}
trait MyOperations {
def self: TypedPipe[Double]
def operation1(implicit fd: FlowDef): TypedPipe[Double] =
self.map { x =>
println(s"Input: $x")
x / 100
}
def operation2(datafn:Double => Double)(implicit fd: FlowDef): TypedPipe[Double] =
self.map { x=>
val result = datafn(x)
println(s"Result: $result")
result
}
}
import org.apache.hadoop.util.ToolRunner
import org.apache.hadoop.conf.Configuration
object MyRunner extends App {
ToolRunner.run(new Configuration(), new Tool, (classOf[MyJob].getName :: "--local" ::
"--input" :: "doubles.tsv" ::
"--output":: "result.tsv" :: args.toList).toArray)
}
Regarding how to manage types across the pipes, my recommendation would be to try to work out some basic types that make sense and use case classes. To use your example i would rename the method convertTextToEither into extractEvents :
case class LogInput(l : Long, text: Text)
case class Event(data: String)
def extractEvents( line : LogInput ): TypedPipe[Event] =
self.filter( isEvent(line) )
.map ( getEvent(line.text) )
Then you would have
LogInputOperations for LogInput types
EventOperations for Event types
I am not sure what is the problem you see with the snippet you showed, and why you think it is "less clean". It looks fine to me.
As for the unit testing jobs using typed API question, take a look at JobTest, it seems to be just what you are looking for.
I want to improve the following Cassandra related Scala code. I have two unrelated user defined types which are actually in Java source files (leaving out the details).
public class Blob { .. }
public class Meta { .. }
So here is how I use them currently from Scala:
private val blobMapper: Mapper[Blob] = mappingManager.mapper(classOf[Blob])
private val metaMapper: Mapper[Meta] = mappingManager.mapper(classOf[Meta])
def save(entity: Object) = {
entity match {
case blob: Blob => blobMapper.saveAsync(blob)
case meta: Meta => metaMapper.saveAsync(meta)
case _ => // exception
}
}
While this works, how can you avoid the following problems
repetition when adding new user defined type classes like Blob or Meta
pattern matching repetition when adding new methods like save
having Object as parameter type
You can definitely use Mapper as a typeclass, doing:
def save[A](entity: A)(implicit mapper: Mapper[A]) = mapper.saveAsync(entity)
Now you have a generic method able to perform a save operation on every type A for which a Mapper[A] is in scope.
Also, the mappingManager.mapper implementation could be probably improved to avoid classOf, but it's hard to tell from the question in the current state.
A few questions:
Is mappingManager.mapper(cls) expensive?
How much do you care about handling subclasses of Blob or Meta?
Can something like this work for you?
def save[T: Manifest](entity: T) = {
mappingManager.mapper(manifest[T].runtimeClass).saveAsync(entity)
}
If you do care about making sure that subclasses of Meta grab the proper mapper then you may find isAssignableFrom helpful in your .mapper (and store found sub-classes in a HashMap so you only have to look once).
EDIT: Then maybe you want something like this (ignoring threading concerns):
private[this] val mapperMap = mutable.HashMap[Class[_], Mapper[_]]()
def save[T: Manifest](entity: T) = {
val cls = manifest[T].runtimeClass
mapperMap.getOrElseUpdate(cls, mappingManager.mapper(cls))
.asInstanceOf[Mapper[T]]
.saveAsync(entity)
}
The following code fails for me:
object Message {
def parse[T](bsonDoc: BSONDocument): Try[T] = {
implicit val bsonHandler = Macros.handler[T]
bsonDoc.seeAsTry[T]
}
}
Message.parse[messages.ClientHello](data)
The error is:
No apply function found for T
implicit val bsonHandler = Macros.handler[T]
^
However, if I hardcode a type (one of my case classes), it's fine:
object Message {
def parse(bsonDoc: BSONDocument): Try[ClientHello] = {
implicit val bsonHandler = Macros.handler[ClientHello]
bsonDoc.seeAsTry[ClientHello]
}
}
Message.parse(data)
So I presume this is a problem using generics. Incidentally, I have to import messages.ClientHello. If I just use messages.ClientHello I get:
not found: value ClientHello
implicit val bsonHandler = Macros.handler[messages.ClientHello]
^
How can I achieve what I'm trying to do, which is to have a single method that will take a BSON document and return an instance of the appropriate case class?
1) Macro applications get expanded immediately when encountered (well, modulo some fine details of type inference that are irrelevant here). This means that when you write handler[T], handler will try to expand with T as a type parameter. This won't lead to anything good, hence the error. To make this work, you need to turn Message.parse into a macro itself.
2) This happens because ReactiveMongo macros are unhygienic. Specifically, https://github.com/ReactiveMongo/ReactiveMongo/blob/v0.10.0/macros/src/main/scala/macros.scala#L142 isn't going to work correctly in situations like yours, because it uses simple name of the class, not a fully qualified name. I think the best way to make the macro work correctly would be using Ident(companion), not Ident(companion.name) - that would ensure that this identifier binds to the companion, not to something in scope having the same name.