Is there a way to convert a vector of type Any to Shapeless HList (productelement)
val frame = Vector(Vector(1,"a","b",false),Vector(2,"y","z",false),Vector(3,"p","q",true))
frame.map(_.hlisted) or frame.map(_.productElements)
I am attempting to convert to the following structure
List[Int :: String :: String :: Boolean :: HNil](1 :: a :: b :: false :: HNil, 2 :: y :: z :: false :: HNil, 3 :: p :: q :: true :: HNil)
Based on Shapless Migration guide, it's possible with Typed Tuples
https://github.com/milessabin/shapeless/wiki/Migration-guide:-shapeless-1.2.4-to-2.0.0#productelements-is-the-new-name-for-hlisted
import shapeless._
import syntax.std.product._ // New import
scala> (23, "foo", true).productElements // was '.hlisted'
res0: Int :: String :: HNil = 23 :: foo :: true :: HNil
Is this possible with untyped Vectors or perhaps a vector -> Typed Tuples -> HList ?
Thanks in advance
Yes, it's possible, but you have to specify the types, and since this is a cast that can fail at runtime, you'll get the results wrapped in Option:
import shapeless._, syntax.std.traversable._
val hlists = frame.map(_.toHList[Int :: String :: String :: Boolean :: HNil])
Now hlists has type Vector[Option[Int :: String :: String :: Boolean :: HNil]], and in this case specifically all of the conversions are successful, so they're all wrapped in Some.
Related
There is very few readable documentation about HLists, and the answers I can find on SO come from outer space for a humble Scala beginner.
I encountered HLists because Slick can auto-generate some to represent database rows. They are slick.collection.heterogeneous.HList (not shapeless').
Example:
type MyRow = HCons[Int,HCons[String,HCons[Option[String],HCons[Int,HCons[String,HCons[Int,HCons[Int,HCons[Option[Int],HCons[Option[Float],HCons[Option[Float],HCons[Option[String],HCons[Option[String],HCons[Boolean,HCons[Option[String],HCons[Option[String],HCons[Option[String],HCons[Option[String],HCons[Option[String],HCons[Option[Int],HCons[Option[Float],HCons[Option[Float],HCons[Option[Float],HCons[Option[String],HCons[Option[String],HNil]]]]]]]]]]]]]]]]]]]]]]]]
def MyRow(a, b, c, ...): MyRow = a :: b :: c :: ... :: HNil
Now given one of these rows, I'd need to read one element, typed if possible. I just can't do that. I tried
row(4) // error
row._4 // error
row.toList // elements are inferred as Any
row match { case a :: b :: c :: x :: rest => x } // "Pattern type is incompatible. Expected MyRow."
row match { case MyRow(_,_,_,_,_,x,...) => x } // is not a case class like other rows
row match { HCons[Int,HCons[String,HCons[Option[String],HCons[Int,HCons[String, x]]]]] => x.head } // error
row.tail.tail.tail.tail.head // well, is that really the way??
Could somebody please explain how I can extract a specific value from that dinosaur?
I'd expect your row(0) lookup to work based on the HList API doc for apply. Here's an example I tried with Slick 3.1.1:
scala> import slick.collection.heterogeneous._
import slick.collection.heterogeneous._
scala> import slick.collection.heterogeneous.syntax._
import slick.collection.heterogeneous.syntax._
scala> type MyRow = Int :: String :: HNil
defined type alias MyRow
scala> val row: MyRow = 1 :: "a" :: HNil
row: MyRow = 1 :: a :: HNil
scala> row(0) + 99
res1: Int = 100
scala> val a: String = row(1)
a: String = a
Just one thing... if it is not too important than just stick to HList as the type. Do not alias it to MyRow unless necessary.
So.. you had
val row = a :: b :: c :: ... :: HNil
How about this ?
val yourX = row match { case a :: b :: c :: x ::: rest => x }
notice that ::: instead of :: at the end.
Or... how about this,
val yourX = row.tail.tail.tail.head
// this may change a little if you had,
def MyRow(a, b, c, ...): MyRow = a :: b :: c :: ... :: HNil
val row = MyRow(a, b, c, ...)
val yourX = row.asInstanceOf[HList].tail.tail.tail.head
The following program is suppose to calculate an expression with the possibility of having two different data types , Float and RDD. I already created an RPN from the infix expression and now I am trying to perform calculations on them.
Note: I have also overloaded :+,-,/,* for doing calculations on RDD and float.
def calcRPN(s: String): RDD[(Int,Array[Float])] =
(s.split(' ').toList.foldLeft(Nil: List[Either[Float, RDD[(Int,Array[Float])]]) {foldingFunction}).head
def foldingFunction(list: List[Either[Float, RDD[(Int,Array[Float])]]], next: String): List[Either[Float,RDD[(Int,Array[Float])]]] = (list, next) match {
//apply * on inputs
case (Right(x) :: Right(y) :: ys, "*") =>{(sv.*(x,y)) :: ys} //both RDD sv is another class containing overloads
case (Left(x) :: Right(y) :: ys, "*") =>{sv.*(x,y) :: ys} //x being float
case (Right(x) :: Left(y) :: ys, "*") =>{sv.*(x,y) :: ys} //y being float}
case (Left(x) :: Left(y) :: ys, "*") => (x * y) :: ys //both float
//apply + on inputs
case (Right(x) :: Right(y) :: ys, "+") => {(sv.+(x,y)) :: ys} //both RDD
case (Left(x) :: Right(y) :: ys, "+") =>{(sv.+(x,y)):: ys} //x being float
case (Right(x) :: Left(y) :: ys, "+") =>{(sv.+(x,y)):: ys} //y being float
case (Left(x) :: Left(y) :: ys, "+") => (y + x) :: ys //both float
//apply - on the inputs
case (Right(x) :: Right(y) :: ys, "-") => {(sv.-(x,y)):: ys} //both RDD
case (Left(x) :: Right(y) :: ys, "-") =>{(sv.-(x,y)) :: ys} //x being float
case (Right(x) :: Left(y) :: ys, "-") =>{(sv.-(x,y)):: ys} //y being float
case (Left(x) :: Left(y) :: ys, "-") => (y - x) :: ys //both float
//apply / on the inputs
case (Right(x) :: Right(y) :: ys, "/") => {(sv./(x,y)) :: ys} //both RDD
case (Left(x) :: Right(y) :: ys, "/") =>{(sv./(x,y)) :: ys} //x being float
case (Right(x) :: Left(y) :: ys, "/") =>{(sv./(x,y)):: ys} //y being float
case (Left(x) :: Left(y) :: ys, "/") => {(y / x) :: ys} //both float
case (xs, numString) => numString.toInt :: xs //**
case (xs, pathxml) => sv.getArrayRDD() :: xs //***
}
I know this code is ugly sorry about that. I can make it shorter but right now I need to make it work then brush it up!
So in the ** part it is working for two numbers but I added *** to make it accept RDD as well. Don't know if it works for both Float and RDD! plus I have faced the following error because of using Either and apparently Left and Right are not helping me here!
[error] type mismatch;
[error] found : Either[Float,org.apache.spark.rdd.RDD[(Int, Array[Float])]]
[error] required: org.apache.spark.rdd.RDD[(Int, Array[Float])]
[error] (s.split(' ').toList.foldLeft(Nil: List[Either[Float, RDD[(Int,Array[Float])]]]) {foldingFunction}).head
[error] ^
I also tried Scalaz but it made it more complex.
Ok first things first, lets split up things for a better understanding:
val creepyListOfoperatorsAndStuff: List[String] = s.split(' ').toList
val eitherList: List[Either[Float, RDD[(Int,Array[Float])]]] =
creepyListOfoperatorsAndStuff.foldLeft(
List.empty[Either[Float, RDD[(Int,Array[Float])]]
) (foldingFunction)
val headEither:Either[Float, RDD[(Int,Array[Float])]] = eitherList.head
The head of that List is an Either. Thus neither a Float nor a RDD.
That means we have to decide whether it is a Float or a RDD[(Int,Array[Float])].
If you are REALLY sure head contains an RDD, you can just do:
headEither.right.get
A better way to do this might be to deal with both cases:
headEither.fold[RDD[(Int,Array[Float])]](
// function to convert a Left() result to the RDD you want
fa = someFloat => <code to get to the RDD you want>,
// this is a function to transform the Right() result to what is desired
// as RDD is what you want you can just return the input
fb = anRDD => anRDD
)
Now onwards to the cases ** and ***:
in
case (xs, numString) => numString.toInt :: xs //**
case (xs, pathxml) => sv.getArrayRDD() :: xs //***
The second case seems unreachable, because both cases match the same input. You would probably be better off using regex to match the strings you expect there. I am not exactly an expert on regular expression matching but something like the following might point in the right direction.
val Numeric = """(\d+)""".r
// don't forget to put the matched string into a group
val XmlPath = """<some regular expression that matches your expected xml path""".r
...
case (xs, NumericString(numString)) => numString.toInt :: xs //**
case (xs, XmlPath(pathxml)) => sv.getArrayRDD() :: xs //***
However, there are more essential problems in theses two cases:
case (xs, numString) => numString.toInt :: xs //**
xs would be a List[Either[Float, RDD[(Int,Array[Float])]].
Thus I have to wonder, does this compile?
numString.toInt :: xs
If so then numString.toInt is probably converted to Float and then to Left[Float]. But I'm just guessing.
case (xs, pathxml) => sv.getArrayRDD() :: xs //***
While I donn't see what sv might possibley be and where it comes form, it might be ok, with the regex matcher.
I would only be able to help with that with more information form you.
I am having a problem with 2.10.3 and code generated by Slick (Codegen). It seems very similar to
Scalac hanging on phase typer of RegexParser
Other files generated by Codegen work, but this one just hangs forever in "scalac: phase typer Foo.scala"
The only difference I can see is the number of columns in the table resulting in lots of vals and large cons'ed lists like this
def * = WordRootID :: WordID :: WordHeadID :: SynonymID :: PronunciationID :: Rank :: BNCFrequency :: CompassDifficulty :: DifficultyNormalized :: DifficultySourceCode :: COCARank :: PartOfSpeech :: AttributeNounProper :: AttributeNounGerund :: AttributePronounType :: AttributeVerbIrregular :: AttributeAdjectiveParticiple :: AttributeArticleType :: AttributeNumber :: AttributeLanguage :: AttributeIdiom :: AttributeMultiWord :: AttributeExcluded :: AttributePrivate :: en_DefinitionDeprecated :: en_QuizDefinitionDeprecated :: en_SourceDeprecated :: ja_WordDeprecated :: ja_SourceDeprecated :: ja_DifficultyDeprecated :: ko_WordDeprecated :: ko_SourceDeprecated :: zh_WordDeprecated :: zh_SourceDeprecated :: es_WordDeprecated :: es_SourceDeprecated :: pt_WordDeprecated :: pt_SourceDeprecated :: tr_WordDeprecated :: tr_SourceDeprecated :: vi_WordDeprecated :: vi_Source :: ReviewCode :: Active :: DateModified :: DateCreated :: en_QuizDefinitionDateModifiedDeprecated :: WordRootAssetID :: Locked :: AttributeStudiable :: WhiteListingStatus :: HNil
When I comment out most of fields and cut down the lists to something small like this the compiler finishes
def * = WordRootID :: WordID :: WordHeadID :: AttributeLanguage :: HNil
However, this is hardly an ideal way to code. Any help would be very much appreciated!
Thank you very much
Peter
Thank you Cvogt, upgrading to 2.11 did the trick. As you said, this is a problem with the 2.10.3 compiler and seems to be fixed now.
In shapeless I'm trying to write a function such that takes two HLists l1 and l2 of arbitrary length which exhibit the following properties:
Length of l1 and l2 are the same.
l2 contains the exact types of l1, wrapped in a constant outer type constructor.
So, if l1 was
1 :: 1.2 :: "hello" :: HNil`
l2 could be
Ordering[Int] :: Ordering[Double] :: Ordering[String] :: HNil
Using UnaryTCConstraint and LengthAux lets me constrain the lengths and require a static outer constructor for l2, however having them conform has become a problem.
Any ideas on how I could go about it?
Mapped provides precisely this constraint without the additional need for Length. From the documentation:
Type class witnessing that the result of wrapping each element of
HList L in type constructor F is Out.
Here's how it looks in 1.2.4:
import shapeless._
def foo[L1 <: HList, L2 <: HList](l1: L1, l2: L2)(implicit
ev: MappedAux[L1, Ordering, L2]
) = ()
val l1 = 1 :: 1.2 :: "hello" :: HNil
val l2 = Ordering[Int] :: Ordering[Double] :: Ordering[String] :: HNil
val l3 = Ordering[Int] :: Ordering[Double] :: Ordering[Char] :: HNil
And then:
scala> foo(l1, l2)
scala> foo(l1, l3)
<console>:17: error: could not find implicit value for parameter ev: ...
As expected. For 2.0 just add a shapeless.ops.hlist._ import and replace MappedAux with Mapped.Aux and you're ready to go.
The next code
def f(chars: List[Char]): List[List[Char]] = chars match {
case Nil => List(Nil)
case x :: xs => for {
v <- f(xs)
} yield List(x) :: v
}
gives the error message
- type mismatch; found : List[List[Any]] required: List[List[Char]]
Please help me understand why 'for' chooses the most general Any instead of Char here? What topic in language spec should I read? Thanks.
The result, you are yielding is a mix of List[List[List[Char]]] and List[List[Char]]. Scala upcasts that to List[List[Any]]. For your case either of the following will do the job:
scala> def f(chars: List[Char]): List[List[Char]] = chars match {
| case Nil => List(Nil)
| case x :: xs => for {
| v <- f(xs)
| } yield x :: v
| }
f: (chars: List[Char])List[List[Char]]
scala> def f(chars: List[Char]): List[List[Char]] = chars match {
| case Nil => List(Nil)
| case x :: xs => for {
| v <- f(xs)
| } yield List(x) ++ v
| }
f: (chars: List[Char])List[List[Char]]
The problem is List(x) -- it needs to be x.
First, v iterates over the results of f(xs), and f returns List[List[Char]]. That means the result will be List[X], where X is the type returned by yield.
The type of v is List[Char], since it is iterating over the contents of f(xs). So we have to figure out the type of List(x) :: v, which is prepending a List[Char] on a List[Char]. It is not concatenating them: it is adding a list to a list containing only characters. The resulting list will have both Char and List[Char] in it.
Since the only type that satisfy both is Any, then X will be Any and the result of the for-comprehension List[Any].