I'm trying to create a data structure that has a PriorityQueue in it. I've succeeded in making a non-generic version of it. I can tell it works because it solves the A.I. problem I have.
Here is a snippet of it:
class ProntoPriorityQueue { //TODO make generic
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
}
val hashSet = new HashSet[Node]
val priorityQueue = new PriorityQueue[Node]()
...
I'm trying to make it generic, but if I use this version it stops solving the problem:
class PQ[T <% Ordered[T]] {
//[T]()(implicit val ord: T => Ordered[T]) {
//[T]()(implicit val ord: Ordering[T] {
val hashSet = new HashSet[T]
val priorityQueue = new PriorityQueue[T]
...
I've also tried what's commented out instead of using [T <% Ordered[T]]
Here is the code that calls PQ:
//the following def is commented out while using ProntoPriorityQueue
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
} //I've also tried making this return an Ordering[Node]
val frontier = new PQ[Node] //new ProntoPriorityQueue
//have also tried (not together):
val frontier = new PQ[Node]()(orderedNode)
I've also tried moving the implicit def into the Node object (and importing it), but essentially the same problem.
What am I doing wrong in the generic version? Where should I put the implicit?
Solution
The problem was not with my implicit definition. The problem was the implicit ordering was being picked up by a Set that was automatically generating in a for(...) yield(...) statement. This caused a problem where the yielded set only contained one state.
What's wrong with simply defining an Ordering on your Node (Ordering[Node]) and using the already-generic Scala PriorityQueue?
As general rule, it's better to work with Ordering[T] than T <: Ordered[T] or T <% Ordered[T]. Conceptually, Ordered[T] is an intrinsic (inherited or implemented) property of the type itself. Notably, a type can have only one intrinsic ordering relationship defined this way. Ordering[T] is an external specification of the ordering relationship. There can any be any number of different Ordering[T].
Also, if you're not already aware, you should know that the difference between T <: U and T <% U is that while the former includes only nominal subtype relations (actual inheritance), the latter also includes the application of implicit conversions that yield a value conforming to the type bound.
So if you want to use Node <% Ordered[Node] and you don't have a compare method defined in the class, an implicit conversion will be applied every time a comparison needs to be made. Additionally, if your type has its own compare, the implicit conversion will never be applied and you'll be stuck with that "built-in" ordering.
Addendum
I'll give a few examples based on a class, call it CIString that simply encapsulates a String and implements ordering as case-invariant.
/* Here's how it would be with direct implementation of `Ordered` */
class CIString1(val s: String)
extends Ordered[CIString1]
{
private val lowerS = s.toLowerCase
def compare(other: CIString1) = lowerS.compareTo(other.lowerS)
}
/* An uninteresting, empty ordered set of CIString1
(fails without the `extends` clause) */
val os1 = TreeSet[CIString1]()
/* Here's how it would look with ordering external to `CIString2`
using an implicit conversion to `Ordered` */
class CIString2(val s: String) {
val lowerS = s.toLowerCase
}
class CIString2O(ciS: CIString2)
extends Ordered[CIString2]
{
def compare(other: CIString2) = ciS.lowerS.compareTo(other.lowerS)
}
implicit def cis2ciso(ciS: CIString2) = new CIString2O(ciS)
/* An uninteresting, empty ordered set of CIString2
(fails without the implicit conversion) */
val os2 = TreeSet[CIString2]()
/* Here's how it would look with ordering external to `CIString3`
using an `Ordering` */
class CIString3(val s: String) {
val lowerS = s.toLowerCase
}
/* The implicit object could be replaced by
a class and an implicit val of that class */
implicit
object CIString3Ordering
extends Ordering[CIString3]
{
def compare(a: CIString3, b: CIString3): Int = a.lowerS.compareTo(b.lowerS)
}
/* An uninteresting, empty ordered set of CIString3
(fails without the implicit object) */
val os3 = TreeSet[CIString3]()
Well, one possible problem is that your Ordered[Node] is not a Node:
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
}
I'd try with an Ordering[Node] instead, which you say you tried but there isn't much more information about. PQ would be declared as PQ[T : Ordering].
Related
I have an interesting Problem matching a Case Class in Scala....
I am using Akka and I have functionality that I will use in every Actor in my System, so created a Base Class for my Actor and I try to Match that Command there....
My Command looks like the following...
sealed trait ReportCommand extends ProcessCommand
final case class onReport(key: Key, replyTo: ActorRef[ResponseBase[State]]) extend ReportCommand
while I constructed Base Class so that it might be used from different Actors, onReport is delivered to Base Actor as generic parameter to be used in pattern match with a case class ...
abstract class BaseActor[E: ClassTag, R <: ReportBase[STATE], COMMAND](signal: TypeCase[R]) {
private val report = signal
def base[B <: E: ClassTag](cmd: E, state: STATE)(f: B => ReplyEffect[COMMAND, STATE]): ReplyEffect[COMMAND, STATE] =
cmd match {
case report(report) =>
Effect.reply(report.replytTo)(new ResponseBase[STATE]{
override def state: STATE = state
})
}
}
First if you think this construct will not work, it works, I have another Command (which I didn't place here) which does not have a generic parameter in the Command Class and above snippet is able to match that Snippet.
Now when I first try this code, Shapeless is complained about there is no mapping to ActorRef for Typeable of TypeCase, so after researching the internet I found I have to do the following....
implicit def mapActorRef[T: ClassTag]: Typeable[ActorRef[T]] =
new Typeable[ActorRef[T]] {
private val typT = Typeable[T]
override def cast(t: Any) : Option[ActorRef[T]] = {
if(t==null) None
else if(t.isInstanceOf[ActorRef[_]]) {
val o= t.asInstanceOf[ActorRef[_]]
for {
_ <- typT.cast(myClassOf)
} yield o.asInstanceOf[ActorRef[T]]
} else None
}
}
def myClassOf[T: ClassTag] = implicitly[ClassTag[T]].runtimeClass
implicit def responseBaseIsTypeable[S: Typeable] : Typeable[ResponseBase[S]] =
new Typeable[ResponseBase[S]] {
private val typS = Typeable[S]
override def cast(t: Any) : Option[ResponseState[S]] = {
if(t==null) None
else if(t.isIntanceOf[ResponseBase[_]]) {
val o = t.asInstanceOf[ResponseBase[_]]
for {
_ <- typS.cast(o.state)
} yield o.asInstanceOf[ResponseBase[S]]
} else None
}
}
Now after this changes I don't receive any Exceptions from Shapeless but case report(report) is not matching, I have no idea how we get a reasoning from Scala why it decide it does not match. During my debugging session I observed the following.
I am using the Akka's Ask Pattern to communicate with this actor...
val future : Future[BaseActor.ResponseBase[Actor.State]] = actorRef.ask[BaseActor.ResponseBase[Actor.State]](ref =>
Actor.onReport(key, ref)
)
now if I observe the cmd object that BaseActor receives, I see that 'ask' Pattern of the Akka change ActorRef in the onReport Command class to an ActorRefAdapter, of course ActorRefAdapter is a subclass of an ActorRef but I am not sure what I defined in the implicit for mapping ActorRef to TypeCase can deal with that stuff but I can't figure a way to change implicit to be aware of the Subtypes....
Unfortunately ActorRefAdapter is private to package package akka.actor.typed.internal.adapter so I can't define an extra mapping for ActorRefAdapter.
So can anybody see why Scala is not matching over my Shapeless <-> TypeCase configuration and give me some tips...
Thx for answers...
Your instance Typeable[ActorRef[T]] is incorrect.
Why did you decide to substitute a ClassTag in typT.cast(myClassOf)? This can't be meaningful.
I guess you used something like "No default Typeable for parametrized type" using Shapeless 2.1.0-RC2
If your gole is to make case report(replyTo) matching then you can define
implicit def mapActorRef[T: Typeable]: Typeable[ActorRef[T]] =
new Typeable[ActorRef[T]] {
private val typT = Typeable[T]
override def cast(t: Any): Option[ActorRef[T]] = {
if (t == null) None
else util.Try(t.asInstanceOf[ActorRef[T]]).toOption
}
override def describe: String = s"ActorRef[${typT.describe}]"
}
The problem is that this instance is also bad. Now case report(replyTo) is matching too much.
val actorTestKit = ActorTestKit()
val replyToRef = actorTestKit.spawn(ReplyToActor(), "replyTo")
import BaseActor._ // importing implicits
import shapeless.syntax.typeable._
val future: Future[BaseActor.ResponseBase[Actor.State]] = replyToRef.cast[ActorRef[Int]].get.ask[BaseActor.ResponseBase[Actor.State]](ref =>
1
)(5.seconds, system.scheduler)
Await.result(future, 10.seconds) // ClassCastException
A legal instance of the type class Typeable can be defined not for every type.
Providing instances for (concrete instantiations of) polymorphic types (where well defined) is pretty much the whole point of Typeable, both here and in Haskell.
The key phrase in the above is "where well defined". It's well defined in the case of non-empty container-like things. It's clearly not well defined for function values.
https://github.com/milessabin/shapeless/issues/69
ResponseBase is a non-empty container-like thing. But ActorRef is like a function T => Unit, so there shouldn't be a Typeable for it
trait ActorRef[-T] extends ... {
def tell(msg: T): Unit
...
}
You should reconsider your approach.
I am working in Spark 1.6.3. Here are two functions that do the same thing:
def modelFromBytesCV(modelArray: Array[Byte]): CountVectorizerModel = {
val tempPath: Path = KAZOO_TEMP_DIR.resolve(s"model_${System.currentTimeMillis()}")
Files.write(tempPath, modelArray)
CountVectorizerModel.read.load(tempPath.toString)
}
def modelFromBytesIDF(modelArray: Array[Byte]): IDFModel = {
val tempPath: Path = KAZOO_TEMP_DIR.resolve(s"model_${System.currentTimeMillis()}")
Files.write(tempPath, modelArray)
IDFModel.read.load(tempPath.toString)
}
I would like to make these functions generic. What I am hung up on is that the common trait between the CountVectorizerModel object and IDFModel is MLReadable[T] which itself must take as a type either CountVectorizerModel or IDFModel. This is sort of a recursive parent class loop that I can't figure out a solution to.
By comparison, the generic model writer is easy, because MLWritable is a common trait extended by all the models I am interested in:
def modelToBytes[M <: MLWritable](model: M): Array[Byte] = {
val tempPath: Path = KAZOO_TEMP_DIR.resolve(s"model_${System.currentTimeMillis()}")
model.write.overwrite().save(tempPath.toString)
Files.readAllBytes(tempPath)
}
How can I make a generic reader that will turn turn a spark-ml model into a byte array?
To make it work you'll need access to a specific MlReadable object.
import org.apache.spark.ml.util.MLReadable
def modelFromBytes[M](obj: MLReadable[M], modelArray: Array[Byte]): M = {
val tempPath: Path = ???
...
obj.read.load(tempPath.toString)
}
which could be later used as:
val bytes: Array[Byte] = ???
modelFromBytes(CountVectorizerModel, bytes)
Note that, despite the first appearance, there is nothing recursive here - MLReadable[M] refers to companion object, not class as such. So for example CountVectorizerModel object is MLReadable, while CountVectorizeModel class isn't.
Internally, Spark MLReader handles this in a different way - it creates an instance of the class using reflection, and then sets its Params. However this path won't be very useful for you here*.
If compatibility with the current API is required, you can try making readable object implicit:
def modelFromBytes[M](modelArray: Array[Byte])(implicit obj: MLReadable[M]): M = {
...
}
and then
implicit val readable: MLReadable[CountVectorizerModel] = CountVectorizerModel
modelFromBytes[CountVectorizerModel](bytes)
* Technically speaking it is possible to get companion object via reflection
def modelFromBytesCV[M <: MLWritable](
modelArray: Array[Byte])(implicit ct: ClassTag[M]): M = {
val tempPath: Path = ???
...
val cls = Class.forName(ct.runtimeClass.getName + "$");
cls.getField("MODULE$").get(cls).asInstanceOf[MLReadable[M]]
.read.load(tempPath.toString))
}
but I don't think that is a path worth exploring here. In particular we cannot really provide strict type bounds here - using MLWritable is a hack to limit human errors, but is rather useless for compiler.
I'm designing sort of Services and faced with a design issue. Here is what I currently have:
trait Service {
def isFailed(): Boolean
def start(): Unit
def stop(): Unit
}
And in order to group Services related to each other in a group (in order to restart/recover the group, not other services) I created the following package object:
package object app {
type FaultTolerantServiceGroup = Seq[Service]
object FaultTolerantServiceGroup{
def apply(svcs: Service*): FaultTolerantServiceGroup = Seq(svcs: _*)
}
class FaultTolerantServiceGroupOps(val F: FaultTolerantServiceGroup){
def hasFailed: Boolean = F.forall(_.failed())
}
trait FaultTolerantServiceGroupSyntax{
implicit def serviceGroup2Ops(F: FaultTolerantServiceGroup) = new FaultTolerantServiceGroupOps(F)
}
}
So I added the method hasFailed to FaultTolerantServiceGroup. But I'm not sure about this decision.
Would it be better to define a typeclass, say
trait Watchable[T]{
def hasFailed(t: T): Boolean
}
And implicitly provide an instance of Watchable[FaultTolerantServiceGroup]?
In my humble opinion implicit functions become much harder to read afterwards. Even when reading my old code it can sometimes be confusing when objects have methods that appear out of the blue.
I have yet to see an instance where implicits are easier to reason about than declarative functions:
val failedGroup : FaultTolerantServiceGroup => Boolean = _.forall(_.failed())
The resulting code doesn't seem any better, or worse, than implicits but at least it's obvious where functionality is coming from:
val group : FaultTolerantServiceGroup = ???
//no implicit
val failed = failedGroup(group)
//with implicits : how does a Seq have a hasFailed method?
val failed = group.hasFailed
Explicit functions also make Iterable functions easier to read:
val groups : Iterable[FaultTolerantServiceGroup] = ???
val failedGroups = groups filter failedGroup
I am tinkling with Scala and would like to produce some generic code. I would like to have two classes, one "outer" class and one "inner" class. The outer class should be generic and accept any kind of inner class which follow a few constraints. Here is the kind of architecture I would want to have, in uncompilable code. Outer is a generic type, and Inner is an example of type that could be used in Outer, among others.
class Outer[InType](val in: InType) {
def update: Outer[InType] = new Outer[InType](in.update)
def export: String = in.export
}
object Outer {
def init[InType]: Outer[InType] = new Outer[InType](InType.empty)
}
class Inner(val n: Int) {
def update: Inner = new Inner(n + 1)
def export: String = n.toString
}
object Inner {
def empty: Inner = new Inner(0)
}
object Main {
def main(args: Array[String]): Unit = {
val outerIn: Outer[Inner] = Outer.empty[Inner]
println(outerIn.update.export) // expected to print 1
}
}
The important point is that, whatever InType is, in.update must return an "updated" InType object. I would also like the companion methods to be callable, like InType.empty. This way both Outer[InType] and InType are immutable types, and methods defined in companion objects are callable.
The previous code does not compile, as it is written like a C++ generic type (my background). What is the simplest way to correct this code according to the constraints I mentionned ? Am I completely wrong and should I use another approach ?
One approach I could think of would require us to use F-Bounded Polymorphism along with Type Classes.
First, we'd create a trait which requires an update method to be available:
trait AbstractInner[T <: AbstractInner[T]] {
def update: T
def export: String
}
Create a concrete implementation for Inner:
class Inner(val n: Int) extends AbstractInner[Inner] {
def update: Inner = new Inner(n + 1)
def export: String = n.toString
}
Require that Outer only take input types that extend AbstractInner[InType]:
class Outer[InType <: AbstractInner[InType]](val in: InType) {
def update: Outer[InType] = new Outer[InType](in.update)
}
We got the types working for creating an updated version of in and we need somehow to create a new instance with empty. The Typeclass Pattern is classic for that. We create a trait which builds an Inner type:
trait InnerBuilder[T <: AbstractInner[T]] {
def empty: T
}
We require Outer.empty to only take types which extend AbstractInner[InType] and have an implicit InnerBuilder[InType] in scope:
object Outer {
def empty[InType <: AbstractInner[InType] : InnerBuilder] =
new Outer(implicitly[InnerBuilder[InType]].empty)
}
And provide a concrete implementation for Inner:
object AbstractInnerImplicits {
implicit def innerBuilder: InnerBuilder[Inner] = new InnerBuilder[Inner] {
override def empty = new Inner(0)
}
}
Invoking inside main:
object Experiment {
import AbstractInnerImplicits._
def main(args: Array[String]): Unit = {
val outerIn: Outer[Inner] = Outer.empty[Inner]
println(outerIn.update.in.export)
}
}
Yields:
1
And there we have it. I know this may be a little overwhelming to grasp at first. Feel free to ask more questions as you read this.
I can think of 2 ways of doing it without referring to black magic:
with trait:
trait Updatable[T] { self: T =>
def update: T
}
class Outer[InType <: Updatable[InType]](val in: InType) {
def update = new Outer[InType](in.update)
}
class Inner(val n: Int) extends Updatable[Inner] {
def update = new Inner(n + 1)
}
first we use trait, to tell type system that update method is available, then we put restrains on the type to make sure that Updatable is used correctly (self: T => will make sure it is used as T extends Updatable[T] - as F-bounded type), then we also make sure that InType will implement it (InType <: Updatable[InType]).
with type class:
trait Updatable[F] {
def update(value: F): F
}
class Outer[InType](val in: InType)(implicit updatable: Updatable[InType]) {
def update: Outer[InType] = new Outer[InType](updatable.update(in))
}
class Inner(val n: Int) {
def update: Inner = new Inner(n + 1)
}
implicit val updatableInner = new Updatable[Inner] {
def update(value: Inner): Inner = value.update
}
First we define type class, then we are implicitly requiring its implementation for our type, and finally we are providing and using it. Putting whole theoretical stuff aside, the practical difference is that this interface is that you are not forcing InType to extend some Updatable[InType], but instead require presence of some Updatable[InType] implementation to be available in your scope - so you can provide the functionality not by modifying InType, but by providing some additional class which would fulfill your constrains or InType.
As such type classes are much more extensible, you just need to provide implicit for each supported type.
Among other methods available to you are e.g. reflection (however that might kind of break type safety and your abilities to refactor).
I have the following where I set information and extractors for different schemes of data:
trait DataScheme {
type Type <: List[Any]
class ExtractorMethods(ticker: String, dataList: List[Type]) {
def getDatetime(datum: Type): Date = new Date(datum(columnIndex(Names.datetime)).toString)
def upperDatum(date: Date): Type = dataList.minBy(datum => getDatetime(datum) >= date)
def lowerDatum(date: Date): Type = dataList.maxBy(datum => getDatetime(datum) <= date)
}
}
trait IndexScheme extends DataScheme {
type Type = (Date, Double, Double, Double, Double, Long)
class ExtractorMethods(ticker: String, dataList: List[Type]) extends super.ExtractorMethods(ticker: String, dataList: List[Type]){
def testing12(int: Int):Int = 12
val test123 = 123
}
}
I want anything extending DataScheme to use its ExtractorMethods methods (e.g. lowerDatum) but also have its own methods (e.g. testing12).
There is a class definition for lists of data elements:
class Data[+T <: DataScheme](val ticker: String, val dataList: List[T#Type], val isSorted: Boolean)
(implicit m: Manifest[T], mm: Manifest[T#Type]) extends Symbols {
def this(ticker: String, dataList: List[T#Type])(implicit m: Manifest[T], mm: Manifest[T#Type]) = this(ticker, dataList, false)(m: Manifest[T], mm: Manifest[T#Type])
val dataScheme: T
val extractorMethods = new dataScheme.ExtractorMethods(ticker, dataList.asInstanceOf[List[dataScheme.Type]])
}
A Data class should make accessible the methods in ExtractorMethods of the scheme so they can be used in the main program through the instance of Data that has been defined. For example if sortedData is an instance of Data[IndexScheme], the following works:
val lowerDatum = sortedData.extractorMethods.lowerDatum(new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2010-03-31 00:00:00"))
but this does not:
val testing = sortedData.extractorMethods.testing12(123)
because 'testing 123 is not a member of sortedData.dataScheme.extractorMethods'. So my question is how can the subclasses of ExtractorMethods in the subtraits of DataScheme like IndexScheme be made accessible? How is it possible using Manifests and TypeTags? Thanks.
So you want the generic class Data[DataScheme] or Data[IndexScheme] to have access to the methods of whichever type Data has been parameterised with. You've tried to do this several different ways, from the evidence in your code.
To answer your last question - manifests can't help in this particular case and TypeTags are only part of the answer. If you really want to do this, you do it with mirrors.
However, you will have to make some changes to your code. Scala only has instance methods; there are no such things as static methods in Scala. This means that you can only use reflection to invoke a method on an instance of a class, trait or object. Your traits are abstract and can't be instantiated.
I can't really tell you how to clean up your code, because what you have pasted up here is a bit of a mess and is full of different things you have tried. What I can show you is how to do it with a simpler set of classes:
import scala.reflect.runtime.universe._
class t1 {
class Methods {
def a = "a"
def b = "b"
}
def methods = new Methods
}
class t2 extends t1 {
class Methods extends super.Methods {
def one = 1
def two = 2
}
override def methods = new Methods
}
class c[+T <: t1](implicit tag: TypeTag[T]) {
def generateT = {
val mirror = runtimeMirror(getClass.getClassLoader)
val cMirror = mirror.reflectClass(typeOf[T].typeSymbol.asClass)
cMirror.reflectConstructor(typeOf[T].declaration(nme.CONSTRUCTOR).asMethod)
}
val t = generateT().asInstanceOf[T]
}
val v1 = new c[t1]
val v2 = new c[t2]
If you run that, you'll find that v1.t.methods gives you a class with only methods a and b, but v2.t.methods gives a class with methods one and two as well.
This really is not how to do this - reaching for reflection for this kind of job shows a very broken model. But I guess that's your business.
I stick by what I said below, though. You should be using implicit conversions (and possibly implicit parameters) with companion objects. Use Scala's type system the way it's designed - you are fighting it all the way.
ORIGINAL ANSWER
Well, I'm going to start by saying that I would never do things the way you are doing this; it seems horribly over-complicated. But you can do what you want to do, roughly the way you are doing it, by
Using mixins
Moving the extractorMethods creation code into the traits.
Here's a greatly simplified example:
trait t1 {
class Methods {
def a = "a"
def b = "b"
}
def methods = new Methods
}
trait t2 extends t1 {
class Methods extends super.Methods {
def one = 1
def two = 2
}
override def methods = new Methods
}
class c1 extends t1
val v1 = new c1
// v1.methods.a will return "a", but v1.methods.one does not exist
class c2 extends c1 with t2
val v2 = new c2
// v2.methods.a returns "a" and v2.methods.one returns 1
I could replicate your modus operandi more closely by defining c1 like this:
class c1 extends t1 {
val myMethods = methods
}
in which case v1.myMethods would only have methods a and b but v2.myMethods would have a, b, one and two.
You should be able to see how you can adapt this to your own class and trait structure. I know my example doesn't have any of your complex type logic in it, but you know better than I what you are trying to achieve there. I'm just trying to demonstrate a simple mechanism.
But dude, way to make your life difficult...
EDIT
There are so many things I could say about what is wrong with your approach here, both on the small and large scale. I'm going to restrict myself to saying two things:
You can't do what you are trying to do in the Data class because it is abstract. You cannot force Scala to magically replace an uninitialised, abstract method of a non-specific type with the specific type, just by littering everything with Type annotations. You can only solve this with a concrete class which provides the specific type.
You should be doing this with implicit conversions. Implicits would help you do it the wrong way you seem fixated on, but would also help you do it the right way. Oh, and use a companion object, either for the implicits or to hold a factory (or bot).