I am trying to get a dialogue to get numeric input using this code
val dialog: TextInputDialog = new TextInputDialog{
initOwner(Main.stage)
title = "Set Programme Counter"
headerText = "Numeric format not supported."
contentText = "New PC value:"
import java.text.DecimalFormat
val format = new DecimalFormat("#.0")
import java.text.ParsePosition
editor.setTextFormatter(new TextFormatter( c => {
def foo(c:TextFormatter.Change): TextFormatter.Change = {
if (c.getControlNewText.isEmpty) return c
val parsePosition = new ParsePosition(0)
val o = format.parse(c.getControlNewText, parsePosition)
if (o == null || parsePosition.getIndex < c.getControlNewText.length) null
else c
}
foo(c)
}))
}
but get the missing parameter type compilation error
[error] editor.setTextFormatter(new TextFormatter( c => {
[error] ^
No idea what the parameter type should be and despite much googling cant find any useful hints.
IntelliJ does not think there is anything wrong but sbt gives the error on compile.
The constructor being used is of a UnaryOperator type, so the TextFormatter.Change => TextFormatter.Change type should be compatible.
With the Scala 2.12 REPL, this will work (using the above constructor signature with UnaryOperator):
import javafx.scene.control.TextFormatter
val tf = new TextFormatter((c: TextFormatter.Change) => c)
The missing type error may occur if the compiler cannot infer the matching types to a constructor, which may happen if you have an incorrect import, or possibly even an older version of Scala which doesn't match with the version being used by your IDE.
Since you say that IntelliJ did not find a problem, it seems like it's more likely the latter. Please check that your project settings in your IDE match with scalaVersion in build.sbt.
You may also want to reduce the inference that must be done by the compiler, by explicitly defining some of the terms. For example you can try explicit definition of a UnaryOperator:
import java.util.function._
val uo: UnaryOperator[TextFormatter.Change] = UnaryOperator.identity[TextFormatter.Change]()
val tf = new TextFormatter(uo)
EDIT#2: This might be memory related. Logs are showing out-of-heap.
Yes, definitely memory related. Basically docker logs reports all the
spewage of out-of-heap from the java, but the jupyter web notebook does not pass that to the user. Instead the user gets kernel failures and occasional weird behavior like code not compiling correctly.
Spark 1.6, particularly docker run -d .... jupyter/all-spark-notebook
Would like to count accounts in a file of ~ 1 million transactions.
This is simple enough, it can be done without spark but I've hit an odd error trying with spark scala.
Input data is type RDD[etherTrans] where etherTrans is a custom type enclosing a single transaction: a timestamp, the from and to accounts, and the value transacted in ether.
class etherTrans(ts_in:Long, afrom_in:String, ato_in:String, ether_in: Float)
extends Serializable {
var ts: Long = ts_in
var afrom: String = afrom_in
var ato: String = ato_in
var ether: Float = ether_in
override def toString():String = ts.toString+","+afrom+","+ato+","+ether.toString
}
data:RDD[etherTrans] looks ok:
data.take(10).foreach(println)
etherTrans(1438918233,0xa1e4380a3b1f749673e270229993ee55f35663b4,0x5df9b87991262f6ba471f09758cde1c0fc1de734,3.1337E-14)
etherTrans(1438918613,0xbd08e0cddec097db7901ea819a3d1fd9de8951a2,0x5c12a8e43faf884521c2454f39560e6c265a68c8,19.9)
etherTrans(1438918630,0x63ac545c991243fa18aec41d4f6f598e555015dc,0xc93f2250589a6563f5359051c1ea25746549f0d8,599.9895)
etherTrans(1438918983,0x037dd056e7fdbd641db5b6bea2a8780a83fae180,0x7e7ec15a5944e978257ddae0008c2f2ece0a6090,100.0)
etherTrans(1438919175,0x3f2f381491797cc5c0d48296c14fd0cd00cdfa2d,0x4bd5f0ee173c81d42765154865ee69361b6ad189,803.9895)
etherTrans(1438919394,0xa1e4380a3b1f749673e270229993ee55f35663b4,0xc9d4035f4a9226d50f79b73aafb5d874a1b6537e,3.1337E-14)
etherTrans(1438919451,0xc8ebccc5f5689fa8659d83713341e5ad19349448,0xc8ebccc5f5689fa8659d83713341e5ad19349448,0.0)
etherTrans(1438919461,0xa1e4380a3b1f749673e270229993ee55f35663b4,0x5df9b87991262f6ba471f09758cde1c0fc1de734,3.1337E-14)
etherTrans(1438919491,0xf0cf0af5bd7d8a3a1cad12a30b097265d49f255d,0xb608771949021d2f2f1c9c5afb980ad8bcda3985,100.0)
etherTrans(1438919571,0x1c68a66138783a63c98cc675a9ec77af4598d35e,0xc8ebccc5f5689fa8659d83713341e5ad19349448,50.0)
This next function parses ok and is written this way because earlier attempts were complaining of type mismatch between Array[String] or List[String] and TraversableOnce[?]:
def arrow(e:etherTrans):TraversableOnce[String] = Array(e.afrom,e.ato)
But then using this function with flatMap to get an RDD[String] of all accounts fails.
val accts:RDD[String] = data.flatMap(arrow)
Name: Compile Error
Message: :38: error: type mismatch;
found : etherTrans(in class $iwC)(in class $iwC)(in class $iwC)(in class $iwC) => TraversableOnce[String]
required: etherTrans(in class $iwC)(in class $iwC)(in class $iwC)(in class $iwC) => TraversableOnce[String]
val accts:RDD[String] = data.flatMap(arrow)
^
StackTrace:
Make sure you scroll right to see it complain that TraversableOnce[String]
doesn't match TraversableOnce[String]
This must be a fairly common problem as a more blatant type mismatch comes up in Generate List of Pairs and while there isn't enough context, is suggested in I have a Scala List, how can I get a TraversableOnce?.
What's going on here?
EDIT: The issue reported above doesn't appear, and code works fine in older spark-shell, Spark 1.3.1 running standalone in a docker container. Errors are generated running in the spark 1.6 scala jupyter environment with the jupyter/all-spark-notebook docker container.
Also #zero323 says that this toy example:
val rdd = sc.parallelize(Seq((1L, "foo", "bar", 1))).map{ case (ts, fr, to, et) => new etherTrans(ts, fr, to, et)}
rdd.flatMap(arrow).collect
worked for him in the terminal spark-shell 1.6.0/spark 2.10.5 and also Scala 2.11.7 and Spark 1.5.2 work as well.
I think you should switch to use case classes, and it should work fine. Using "regular" classes, might case weird issues when serializing them, and it looks like all you need are value objects, so case classes look like a better fit for your use case.
An example:
case class EtherTrans(ts: Long, afrom: String, ato: String, ether: Float)
val source = sc.parallelize(Array(
(1L, "from1", "to1", 1.234F),
(2L, "from2", "to2", 3.456F)
))
val data = source.as[EtherTrans]
val data = source.map { l => EtherTrans(l._1, l._2, l._3, l._4) }
def arrow(e: EtherTrans) = Array(e.afrom, e.ato)
data.map(arrow).take(5)
/*
res3: Array[Array[String]] = Array(Array(from1, to1), Array(from2, to2))
*/
data.map(arrow).take(5)
// res3: Array[Array[String]] = Array(Array(from1, to1), Array(from2, to2))
If you need to, you can just create some method / object to generate your case classes.
If you don't really need the "toString" method for your logic, but just for "presentation", keep it out of the case class: you can always add it with a map operation before storing if or showing it.
Also, if you are in Spark 1.6.0 or higher, you could try using the DataSet API instead, that would look more or less like this:
val data = sqlContext.read.text("your_file").as[EtherTrans]
https://databricks.com/blog/2016/01/04/introducing-spark-datasets.html
Another question lead me to the need to create a sequence of Scala expressions. I seem to be unable to do that.
I have a SchemaRDD object z:
org.apache.spark.sql.SchemaRDD =
SchemaRDD[0] at RDD at SchemaRDD.scala:103
== Query Plan ==
== Physical Plan ==
ParquetTableScan [event_type#0,timestamp#1,id#2,domain#3,code#4], (ParquetRelation ...., Some(Configuration: core-default.xml, core-site.xml, yarn-default.xml, yarn-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml), org.apache.spark.sql.SQLContext#e7f91e, []), []
and I want to project it on two columns. select should be the answer:
z.select _
res19: Seq[org.apache.spark.sql.catalyst.expressions.Expression] => org.apache.spark.sql.SchemaRDD = <function1>
However, I seem to be unable to generate a Seq[Expression], e.g.:
z.select(Seq('event_type,'code))
<console>:21: error: type mismatch;
found : Seq[Symbol]
required: org.apache.spark.sql.catalyst.expressions.Expression
z.select(Seq('event_type,'code))
^
or:
z.select('event_type,'code)
<console>:21: error: type mismatch;
found : Symbol
required: org.apache.spark.sql.catalyst.expressions.Expression
z.select('event_type,'code)
^
I thought that a symbol was an expression...
so, how do I invoke select?
from a private e-mail:
It looks like you're missing this import:
import sqc._
where sqc is your SqlContext variable (so it's subtly different from import SqlContext._).
This loads a bunch of implicit functions which convert the symbols to an Expression.
If you haven't encountered implicit functions in scala yet it's basically a way to define a function for transparently converting one type to another. e.g. if I define a function like this which takes an instance of Bar and returns a Foo:
implicit def toFoo(bar : Bar) : Foo = new Foo(bar.toString)
Then, anywhere in the code where I have a function which expects a Foo I can pass it a Bar instead and the compiler will introduce a toFoo() call on it in the background (i.e. "implicitly").
It's one of the most quirky features of scala and very much a double-edged sword. It lets you write very powerful and concise DSLs but at the same time introduces a lot of magic that can make code impossible to follow :-/
I use Scala 2.10.3.
I have result.iterator() returning Iterator<java.util.Map<String, Object>>. (in Java so)
I want to convert it to the Scala equivalent.
I use import scala.collection.JavaConversions._ to try to do the trick.
However it seems that it is unable to take into account type parameters, in this case, it can convert java.util.Iterator to the Scala equivalent, but fails to convert java.util.Map in the Scala equivalent.
Indeed, a compiler error occurs at this line:
val results: Iterator[Map[String, AnyRef]] = result.iterator()
type mismatch;
found : java.util.Iterator[java.util.Map[String,Object]]
required: scala.collection.Iterator[scala.collection.immutable.Map[String,AnyRef]]
val results: Iterator[Map[String, AnyRef]] = result.iterator()
^
Is there a short way to do the trick?
You could explicitly specify what you want to convert using JavaConverters instead of JavaConversions like this:
import scala.collection.JavaConverters._
def javaIt: java.util.Iterator[java.util.Map[String, Object]] = ???
def scalaIt = javaIt.asScala map {_.asScala}
// Iterator[scala.collection.mutable.Map[String,Object]]
How can I convert a java 1.4 Collection to a Scala Seq?
I am trying to pass a java-collection to a scala method:
import scala.collection.JavaConversions._
// list is a 1.4 java.util.ArrayList
// repository.getDir is Java legacy code
val list = repository.getDir(...)
perform(list)
def perform(entries: List[SVNDirEntry]) = ...
I always receive this error:
type mismatch; found : java.util.Collection[?0] where type ?0 required: List
[SVNDirEntry]
So I guess I have to create the parameterized Sequence myself as Scala is only able to create an unparameterized Iterable?
First you have to make sure that list has the type java.util.List[SVNDirEntry]. To do this, use a cast:
list.asInstanceOf[java.util.List[SVNDirEntry]]
After that, the implicit conversion will be resolved for you if you import the JavaConversions object. An implicit conversion to a Scala sequence exists in the JavaConversions object. See the following example with a list of strings being passed to a method that expects a Scala sequence:
scala> val jvlist: java.util.List[_] = new java.util.ArrayList[String]
jvlist: java.util.List[_] = []
scala> jvlist.asInstanceOf[java.util.List[String]]
res0: java.util.List[String] = []
scala> import collection.JavaConversions._
import collection.JavaConversions._
scala> def perform(scalaseq: Seq[String]) = println("ok")
perform: (scalaseq: scala.collection.Seq[String])Unit
scala> perform(res0)
ok
These conversions do not copy the elements - they simply construct a wrapper around a Java collection. Both versions point to the same underlying data. Thus, there is no implicit conversion in JavaConversions to immutable Scala lists from mutable Java lists, because that would enable changing the contents of a Scala collection that is guaranteed to be immutable.
In short - prefer Seq[...] to List[...] when defining parameters for methods if you can live with a less specific interface (as in perform above). Or write your own function that does the conversion by copying the elements.
You have to cast the legacy collection down to the target type. Something along the lines of:
perform(list.asInstanceOf[List[SVNDirEntry]])