How does one invoke the following java method from scala?
public static Config parseMap(Map<String, ? extends Object> values,
String originDescription) {
I have attempted to invoke it as follows:
val SAMPLE_PROPS_MAP : Map[String,AnyRef] = Map("hiveSaveFreq" -> new java.lang.Long(10L), "aggSeconds" -> new java.lang.Long(3))
val props = ConfigFactory.parseMap(SAMPLE_PROPS_MAP,"props"))
However Intellij complains:
Cannot resolve symbol parseMap
To ensure this were not other issues (e.g. incorrect version of libraries):
val props = ConfigFactory.parseMap(null,"props"))
works fine. Also, the intellisense bubble does confirm the signature as being shown above (having Map).
However Intellij complains:
So then what is the Scala equivalent to the ? extends Object
The issue is not on the ? extends Object type boundary declaration but rather on the Map definition. The signature you have is:
public static Config parseMap(**java.util.Map**<String, ? extends Object> values, String originDescription)
and you are passing it a scala.collection.immutable.Map
You need to convert your scala map to a java map, either by creating a java map "by hand" or using the facilities in the scala API:
import scala.collection.JavaConverters._
val SamplePropsMap : Map[String,AnyRef] = Map("hiveSaveFreq" -> new java.lang.Long(10L), "aggSeconds" -> new java.lang.Long(3))
val props = ConfigFactory.parseMap(SamplePropsMap.asJava,"props"))
(note: The implicit methods in JavaConverters are the recommended way to deal with java-scala conversions, by using javaCollection.asScala and scalaCollection.asJava. Also note that in Scala, the convention for 'constants' is camel-case starting with a capital. e.g. CamelCase)
The Map needs to be converted to java format, as follows:
import collection.JavaConversions._
val props = ConfigFactory.parseMap(mapAsJavaMap(SAMPLE_PROPS_MAP))
Related
I would like to understand is there a way to write a method to existing class at runtime and to create a jar dynamically in scala.
So far i tried to create a class dynamically and able to run it thru reflection, however the class is dynamic class which isnt generated.
val mirror = runtimeMirror(getClass.getClassLoader)
val tb = ToolBox(mirror).mkToolBox()
val function = q"def function(x: Int): Int = x + 2"
val functionWrapper = "object FunctionWrapper { " + function + "}"
data.map(x => tb.eval(q"$functionSymbol.function($x)"))
i got this from other source, however the class is available only for this run and will not be generated.
i would like to add a function to the existing class at runtime and able to compile it and create a jar for it.
Kindly suggest me the way
Thanks in advance.
I guess the code snippet you provided should actually look like
import scala.reflect.runtime.universe._
import scala.tools.reflect.ToolBox
val mirror = runtimeMirror(getClass.getClassLoader)
val tb = ToolBox(mirror).mkToolBox()
val function: Tree = q"def function(x: Int): Int = x + 2"
val functionWrapper: Symbol = tb.define(q"object FunctionWrapper { $function }".asInstanceOf[ImplDef])
val data: List[Tree] = List(q"1", q"2")
data.map(x => tb.eval(q"$functionWrapper.function($x)")) // List(3, 4)
... however the class is dynamic class which isnt generated.
... however the class is available only for this run and will not be generated.
How did you check that the class is not generated? (Which class, FunctionWrapper?)
is there a way to write a method to existing class at runtime and to create a jar dynamically in scala.
i would like to add a function to the existing class at runtime and able to compile it and create a jar for it.
What is "existing class"? Do you have access to its sources? Then you can modify the sources, compile them etc.
Does the class exist as a .class file? You can modify its byte code with Byte-buddy, ASM, Javassist, cglib etc., instrument the byte code with aspects etc.
Is it dynamic class (like FunctionWrapper above)? How did you create it? (For FunctionWrapper you have access to its Symbol so you can use it in further sources.)
Is the class already loaded? Then you'll have to play with class loaders (unload, modify, load modified).
Can a Java class add a method to itself at runtime?
In Java, given an object, is it possible to override one of the methods?
Why I can do this in Java:
import javax.swing.GroupLayout.Group;
but if I do the same in Scala (by using Ammonite), I get this:
value Group is not a member of object javax.swing.GroupLayout possible
cause: maybe a semicolon is missing before `value Group'? import
javax.swing.GroupLayout.Group
Is it due to the fact that Group is a public class derived from a private class called Spring?.
I can import neither SequentialGroup nor ParallelGroup.
Is it a bug in Scala?
I'm using Java 11 and Scala 2.12.10.
Scala 2.13.1 also fails. :-(
I need the import, for defining a generic method that can have a Group parameter, that could be either a ParallelGroup or a SequentialGroup.
I'd like to generate a generic method that takes as a parameter a Group, that could be either a ParallelGroup or a SequientialGroup
That would be a type projection
def method(group: GroupLayout#Group) = ...
or if you also have the layout the group belongs to,
def method(layout: GroupLayout)(group: layout.Group) = ...
or
val layout: GroupLayout = ...
def method(group: layout.Group) = ...
The following code executes fine in a scala shell given snakeyaml version 1.17
import org.yaml.snakeyaml.Yaml
import org.yaml.snakeyaml.constructor.Constructor
import scala.collection.mutable.ListBuffer
import scala.beans.BeanProperty
class EmailAccount {
#scala.beans.BeanProperty var accountName: String = null
override def toString: String = {
return s"acct ($accountName)"
}
}
val text = """accountName: Ymail Account"""
val yaml = new Yaml(new Constructor(classOf[EmailAccount]))
val e = yaml.load(text).asInstanceOf[EmailAccount]
println(e)
However when running in spark (2.0.0 in this case) the resulting error is:
org.yaml.snakeyaml.constructor.ConstructorException: Can't construct a java object for tag:yaml.org,2002:EmailAccount; exception=java.lang.NoSuchMethodException: EmailAccount.<init>()
in 'string', line 1, column 1:
accountName: Ymail Account
^
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:350)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127)
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:450)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:369)
... 48 elided
Caused by: org.yaml.snakeyaml.error.YAMLException: java.lang.NoSuchMethodException: EmailAccount.<init>()
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.createEmptyJavaBean(Constructor.java:220)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:190)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:346)
... 53 more
Caused by: java.lang.NoSuchMethodException: EmailAccount.<init>()
at java.lang.Class.getConstructor0(Class.java:2810)
at java.lang.Class.getDeclaredConstructor(Class.java:2053)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.createEmptyJavaBean(Constructor.java:216)
... 55 more
I launched the scala shell with
scala -classpath "/home/placey/snakeyaml-1.17.jar"
I launched the spark shell with
/home/placey/Downloads/spark-2.0.0-bin-hadoop2.7/bin/spark-shell --master local --jars /home/placey/snakeyaml-1.17.jar
Solution
Create a self-contained application and run it using spark-submit instead of using spark-shell.
I've created a minimal project for you as a gist here. All you need to do is put both files (build.sbt and Main.scala) in some directory, then run:
sbt package
in order to create a JAR. The JAR will be in target/scala-2.11/sparksnakeyamltest_2.11-1.0.jar or a similar location. You can get SBT from here if you haven't used it yet. Finally, you can run the project:
/home/placey/Downloads/spark-2.0.0-bin-hadoop2.7/bin/spark-submit --class "Main" --master local --jars /home/placey/snakeyaml-1.17.jar target/scala-2.11/sparksnakeyamltest_2.11-1.0.jar
The output should be:
[many lines of Spark's log)]
acct (Ymail Account)
[more lines of Spark's log)]
Explanation
Spark's shell (REPL) transforms all classes you define in it by adding $iw parameter to your constructors. I've explained it here. SnakeYAML expects a zero-parameter constructor for JavaBean-like classes, but there isn't one, so it fails.
You can try this yourself:
scala> class Foo() {}
defined class Foo
scala> classOf[Foo].getConstructors()
res0: Array[java.lang.reflect.Constructor[_]] = Array(public Foo($iw))
scala> classOf[Foo].getConstructors()(0).getParameterCount
res1: Int = 1
As you can see, Spark transforms the constructor by adding a parameter of type $iw.
Alternative solutions
Define your own Constructor
If you really need to get it working in the shell, you could define your own class implementing org.yaml.snakeyaml.constructor.BaseConstructor and make sure that $iw gets passed to constructors, but this is a lot of work (I actually wrote my own Constructor in Scala for security reasons some time ago, so I have some experience with this).
You could also define a custom Constructor hard-coded to instantiate a specific class (EmailAccount in your case) similar to the DiceConstructor shown in SnakeYAML's documentation. This is much easier, but requires writing code for each class you want to support.
Example:
case class EmailAccount(accountName: String)
class EmailAccountConstructor extends org.yaml.snakeyaml.constructor.Constructor {
val emailAccountTag = new org.yaml.snakeyaml.nodes.Tag("!emailAccount")
this.rootTag = emailAccountTag
this.yamlConstructors.put(emailAccountTag, new ConstructEmailAccount)
private class ConstructEmailAccount extends org.yaml.snakeyaml.constructor.AbstractConstruct {
def construct(node: org.yaml.snakeyaml.nodes.Node): Object = {
// TODO: This is fine for quick prototyping in a REPL, but in a real
// application you should probably add type checks.
val mnode = node.asInstanceOf[org.yaml.snakeyaml.nodes.MappingNode]
val mapping = constructMapping(mnode)
val name = mapping.get("accountName").asInstanceOf[String]
new EmailAccount(name)
}
}
}
You can save this as a file and load it in the REPL using :load filename.scala.
Bonus advantage of this solution is that it can create immutable case class instances directly. Unfortunately Scala REPL seems to have issues with imports, so I've used fully qualified names.
Don't use JavaBeans
You can also just parse YAML documents as simple Java maps:
scala> val yaml2 = new Yaml()
yaml2: org.yaml.snakeyaml.Yaml = Yaml:1141996301
scala> val e2 = yaml2.load(text)
e2: Object = {accountName=Ymail Account}
scala> val map = e2.asInstanceOf[java.util.Map[String, Any]]
map: java.util.Map[String,Any] = {accountName=Ymail Account}
scala> map.get("accountName")
res4: Any = Ymail Account
This way SnakeYAML won't need to use reflection.
However, since you're using Scala, I recommend trying
MoultingYAML, which is a Scala wrapper for SnakeYAML. It parses YAML documents to simple Java types and then maps them to Scala types (even your own types like EmailAccount).
I am trying use a custom annotation in Scala. In this example, I create a string that I want to annotate with metadata (in this case, another string). Then, given an instance of the data, and I want to read the annotation.
scala> case class named(name: String) extends scala.annotation.StaticAnnotation
defined class named
scala> #named("Greeting") val v = "Hello"
v: String = Hello
scala> def valueToName(x: String): String = ???
valueToName: (x: String)String
scala> valueToName(v) // returns "Greeting"
Is this even possible?
With scala 2.11.6, this works to extract values of a annotation:
case class Named(name: String) extends scala.annotation.StaticAnnotation
val myAnnotatedClass: ClassSymbol = u.runtimeMirror(Thread.currentThread().getContextClassLoader).staticClass("MyAnnotatedClass")
val annotation: Option[Annotation] = myAnnotatedClass.annotations.find(_.tree.tpe =:= u.typeOf[Named])
val result = annotation.flatMap { a =>
a.tree.children.tail.collect({ case Literal(Constant(name: String)) => doSomething(name) }).headOption
}
There are different kinds of annotations in Scala:
Java Annotations that you can access using the Java Reflection API, annotations that are just in the source code, static annotations that are available to the type checker across different compilation units (so they should be somewhere in a class file but not where normal reflections go) and classfile annotations which are stored like java annotations, but cannot be read using the java reflection api.
I have described how to access static and classfile annotations here: What is the (current) state of scala reflection capabilities, especially wrt annotations, as of version 2.11?
If you just need a annotation containing a string using a Java annotation that is loaded by the JVM for you might be the simpler alternative.
In Java, while type arguments are erased in runtime, it is possible to find the actual type arguments passed to a superclass:
class Derived extends Base<String> {
// ...
}
ParameterizedType type = (ParameterizedType)Derived.class.getGenericSuperclass();
Type[] args = type.getActualTypeArguments(); // gives {String.class}
While I can use the same Java reflection to Scala class, It does not catch Scala's value types:
class Base[T]
class Derived extends Base[Int]
classOf[Derived]
.getGenericSuperclass
.asInstanceOf[ParameterizedType]
.getActualTypeArguments // gives {Object.class}, not {int.class}
Is it possible to determine the value type used when extending from a generic superclass? I am loading classes from a jar file so it'd be best to achieve this only using a java.lang.Class instance.
In Java reflection you won't be able to obtain Int and other AnyVal types because they are handled specially by the compiler and if they are used generically, they will be represented by Object. However, you can use Scala reflection, and it is wholly possible to go from Java reflection to Scala reflection. Here's how:
import scala.reflect.runtime.universe._
class Base[T]
class Derived extends Base[Int]
object Main extends App {
val rm = runtimeMirror(getClass.getClassLoader) // whatever class loader you're using
val derivedSym = rm.staticClass(classOf[Derived].getName)
val baseSym = rm.staticClass(classOf[Base[_]].getName)
val TypeRef(_, _, params) = derivedSym.typeSignature.baseType(baseSym)
println(s"$derivedSym extends $baseSym[${params.mkString(", ")}]")
}
Unfortunately, unless you know exactly what you are searching for, you will have hard time finding proper documentation. I have found the answer on scala-users mailing list. Scala reflection is still experimental and, AFAIK, it will probably be superseded by a better one in future Scala versions.