I am using avro4s
https://github.com/sksamuel/avro4s
I wrote this code
implicit val schema = AvroSchema[SalesRecord]
val output = AvroOutputStream[SalesRecord](new File(outputLocation))
output.write(salesList)
output.flush
output.close
But I get a compile time error
could not find implicit value for parameter builder: shapeless.Lazy[....]
Not enough arguments for method apply
There was a bug in 1.2.x with private vals in a case class which caused the error you've seen here. That's fixed in 1.3.0 and should solve your problem.
(If it's not private vals, you'd need to post up your SalesRecord object for us to take a look at and I'll update this answer with a solution).
Related
Does Scala have a way to get the contained class(es) of a collection? i.e. if I have:
val foo = Vector[Int]()
Is there a way to get back classOf[Int] from it?
(Just checking the first element doesn't work since it might be empty.)
You can use TypeTag:
import scala.reflect.runtime.universe._
def getType[F[_], A: TypeTag](as: F[A]) = typeOf[A]
val foo = Vector[Int]()
getType(foo)
Not from the collection itself, but if you get it a parameter from a method, you could add an implicit TypeTag to that method to obtain the type at runtime. E.g.
def mymethod[T](x: Vector[T])(implicit tag: TypeTag[T]) = ...
See https://docs.scala-lang.org/.../typetags-manifests.html for details.
Technically you can do it by using TypeTag or Typeable/TypeCase from Shapless library (see link). But I just want to note that all these tricks are really very advanced solutions when there is no any better way get the task done without digging inside type parameters.
All type parameters in scala and java are affected by type erasure on runtime, and if you сatch yourself thinking about extracting these information from the class it might be a good sign to redesign the solution that you are trying to implement.
I am trying to write some use cases for Apache Flink. One error I run into pretty often is
could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[SomeType]
My problem is that I cant really nail down when they happen and when they dont.
The most recent example of this would be the following
...
val largeJoinDataGen = new LargeJoinDataGen(dataSetSize, dataGen, hitRatio)
val see = StreamExecutionEnvironment.getExecutionEnvironment
val newStreamInput = see.addSource(largeJoinDataGen)
...
where LargeJoinDataGen extends GeneratorSource[(Int, String)] and GeneratorSource[T] extends SourceFunction[T], both defined in separate files.
When trying to build this I get
Error:(22, 39) could not find implicit value for evidence parameter of type org.apache.flink.api.common.typeinfo.TypeInformation[(Int, String)]
val newStreamInput = see.addSource(largeJoinDataGen)
1. Why is there an error in the given example?
2. What would be a general guideline when these errors happen and how to avoid them in the future?
P.S.: first scala project and first flink project so please be patient
You may make an import instead of implicits
import org.apache.flink.streaming.api.scala._
It will also help.
This mostly happens when you have user code, i.e. a source or a map function or something of that nature that has a generic parameter. In most cases you can fix that by adding something like
implicit val typeInfo = TypeInformation.of(classOf[(Int, String)])
If your code is inside another method that has a generic parameter you can also try adding a context bound to the generic parameter of the method, as in
def myMethod[T: TypeInformation](input: DataStream[Int]): DataStream[T] = ...
My problem is that I cant really nail down when they happen and when they dont.
They happen when an implicit parameter is required. If we look at the method definition we see:
def addSource[T: TypeInformation](function: SourceFunction[T]): DataStream[T]
But we don't see any implicit parameter defined, where is it?
When you see a polymorphic method where the type parameter is of the form
def foo[T : M](param: T)
Where T is the type parameter and M is a context bound. It means that the creator of the method is requesting an implicit parameter of type M[T]. It is equivalent to:
def foo[T](param: T)(implicit ev: M[T])
In the case of your method, it is actually expanded to:
def addSource[T](function: SourceFunction[T])(implicit evidence: TypeInformation[T]): DataStream[T]
This is why you see the compiler complaining, as it can't find the implicit parameter the method is requiring.
If we go to the Apache Flink Wiki, under Type Information we can see why this happens :
No Implicit Value for Evidence Parameter Error
In the case where TypeInformation could not be created, programs fail to compile with an error stating “could not find implicit value for evidence parameter of type TypeInformation”.
A frequent reason if that the code that generates the TypeInformation has not been imported. Make sure to import the entire flink.api.scala package.
import org.apache.flink.api.scala._
For generic methods, you'll need to require them to generate a TypeInformation at the call-site as well:
For generic methods, the data types of the function parameters and return type may not be the same for every call and are not known at the site where the method is defined. The code above will result in an error that not enough implicit evidence is available.
In such cases, the type information has to be generated at the invocation site and passed to the method. Scala offers implicit parameters for that.
Note that import org.apache.flink.streaming.api.scala._ may also be necessary.
For your types this means that if the invoking method is generic, it also needs to request the context bound for it's type parameter.
For example Scala versions (2.11, 2.12, etc.) are not binary compatible.
The following is a wrong configuration even if you use import org.apache.flink.api.scala._ :
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<scala.version>2.12.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
</properties>
Correct configuration in Maven:
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<scala.version>2.12.8</scala.version>
<scala.binary.version>2.12</scala.binary.version>
</properties>
I'm working on writing some unit tests for my Scala Spark application
In order to do so I need to create different dataframes in my tests. So I wrote a very short DFsBuilder code that basically allows me to add new rows and eventually create the DF. The code is:
class DFsBuilder[T](private val sqlContext: SQLContext, private val columnNames: Array[String]) {
var rows = new ListBuffer[T]()
def add(row: T): DFsBuilder[T] = {
rows += row
this
}
def build() : DataFrame = {
import sqlContext.implicits._
rows.toList.toDF(columnNames:_*) // UPDATE: added :_* because it was accidently removed in the original question
}
}
However the toDF method doesn't compile with a cannot resolve symbol toDF.
I wrote this builder code with generics since I need to create different kinds of DFs (different number of columns and different column types). The way I would like to use it is to define some certain case class in the unit test and use it for the builder
I know this issue somehow relates to the fact that I'm using generics (probably some kind of type erasure issue) but I can't quite put my finger on what the problem is exactly
And so my questions are:
Can anyone show me where the problem is? And also hopefully how to fix it
If this issue cannot be solved this way, could someone perhaps offer another elegant way to create dataframes? (I prefer not to pollute my unit tests with the creation code)
I obviously googled this issue first but only found examples where people forgot to import the sqlContext.implicits method or something about a case class out of scope which is probably not the same issue as I'm having
Thanks in advance
If you look at the signatures of toDF and of SQLImplicits.localSeqToDataFrameHolder (which is the implicit function used) you'll be able to detect two issues:
Type T must be a subclass of Product (the superclass of all case classes, tuples...), and you must provide an implicit TypeTag for it. To fix this - change the declaration of your class to:
class DFsBuilder[T <: Product : TypeTag](...) { ... }
The columnNames argument is not of type Array, it's a "repeated parameter" (like Java's "varargs", see section 4.6.2 here), so you have to convert the array into arguments:
rows.toList.toDF(columnNames: _*)
With these two changes, your code compiles (and works).
I have a table column errorFixed of type TableColumn[Error, Boolean] inside a TableView[Error]. My Error class has a val fixed: Boolean which I try to put into this table column.
I tried
errorFixed.cellValueFactory = features =>
ReadOnlyBooleanWrapper(features.value.fixed)
but it fails with
type mismatch;
found : scalafx.beans.property.ReadOnlyBooleanWrapper
required: scalafx.beans.value.ObservableValue[Boolean,Boolean]
which I really don't understand as ObservableValue[Boolean,Boolean] is a supertype of ReadOnlyBooleanWrapper according to the documentation.
If I cast it myself using .asInstanceOf[ObservableValue[Boolean, Boolean]] it seems to work. What is going on here?
Gist with stripped down project to reproduce
Short answer is: instead of
errorFixed.cellValueFactory = features =>
ReadOnlyBooleanWrapper(features.value.fixed)
you should use
errorFixed.cellValueFactory = features =>
ObjectProperty[Boolean](features.value.fixed)
or ReadOnlyObjectWrapper[Boolean].
A brief version of long answer: there are certain "frictions" between Scala and Java when working with primitive Java types, like boolean or int. This inconvenience shows up in property binding in ScalaFX. Not everything is inherited in an intuitive way. In this case
ReadOnlyBooleanWrapper
is a subclass of
ObservableValue[scala.Boolean, java.lang.Boolean]
but scala.Boolean is not a subclass of java.lang.Boolean which internally, in ScalaFX this leads to complications. Interesting thing is that the casting .asInstanceOf[ObservableValue[scala.Boolean, scala.Boolean]] works, though type parameters do not match at compile time.
Thanks for positing a full code example (gist) this really helps in clarifying the question.
I have a beginners Scala question. I have a class, Sample which extends the trait SampleAPI. Now I'm trying to build a sequence of Sample instances using seq. I will look something like this:
var samples: Seq[SampleAPI] = Seq()
for(...) {
samples :+= new Sample(...))
}
This gives me the following compiler error: "type mismatch; found : Seq[java.lang.Object] required: Seq[se.uu.medsci.queue.setup.SampleAPI]"
So I tried:
samples :+= (new Sample(sampleName, this, illuminaXMLReportReader)).asInstanceOf[SampleAPI]
Which instead throws a runtime exception, saying that Sample cannot be bast to SampleAPI. I guess that this comes down to a problem in my understanding of the use of traits in Scala. Any help in figuring this out would be much appreciated.
Is the compiler error coming up on this line?
samples :+= new Sample(...))
If so, I think the problem is that your Sample class doesn't actually extend SampleAPI.
What's happening has to do with the contravariant type parameter of the List type in Scala. If you start with a List[SampleAPI], then add a Sample to that list, it needs to find the least-upper-bound on the types included in the list to use as the new type parameter. If Sample is a SampleAPI, then the least-upper-bound is just SampleAPI and you get a List[SampleAPI] as the result of the :+= operation. However, if Sample is not a SampleAPI then the least-upper-bound on the two types is just Object, hence your compiler error saying that it was expecting a Seq[SampleAPI] but found a Seq[Object].