Getting started with Scala and hunting around for Orika modules but for Scala. It is possible to create custom property builders in Orika but for stuff like case cases with value fields I would need to (with Scala 2.10+) reflex fields (mirroring) for setting immutables. Assuming there is a native approach with a Scala module?
I know Orika has lot more features but, if you just want to create a copy of an instance to a new instance and change some attributes of the new instance, scala has inbuilt feature for it. Use case classes and you can use copy method on it to create new instances.
case class Bird(name: String, color: String)
scala> val chicken = Bird("twitty", "yellow")
chicken: Bird = Bird(twitty,yellow)
scala> val coq = chicken.copy(color = "red")
coq: Bird = Bird(twitty,red)
Related
I have always seen that, when we are using a map function, we can create a dataframe from rdd using case class like below:-
case class filematches(
row_num:Long,
matches:Long,
non_matches:Long,
non_match_column_desc:Array[String]
)
newrdd1.map(x=> filematches(x._1,x._2,x._3,x._4)).toDF()
This works great as we all know!!
I was wondering , why we specifically need case classes here?
We should be able to achieve same effect using normal classes with parameterized constructors (as they will be vals and not private):-
class filematches1(
val row_num:Long,
val matches:Long,
val non_matches:Long,
val non_match_column_desc:Array[String]
)
newrdd1.map(x=> new filematches1(x._1,x._2,x._3,x._4)).toDF
Here , I am using new keyword to instantiate the class.
Running above has given me the error:-
error: value toDF is not a member of org.apache.spark.rdd.RDD[filematches1]
I am sure I am missing some key concept on case classes vs regular classes here but not able to find it yet.
To resolve error of
value toDF is not a member of org.apache.spark.rdd.RDD[...]
You should move your case class definition out of function where you are using it. You can refer http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-Scala-Error-value-toDF-is-not-a-member-of-org-apache/td-p/29878 for mode detail.
On your Other query - case classes are syntactic sugar and they provide following additional things
Case classes are different from general classes. They are specially used when creating immutable objects.
They have default apply function which is used as constructor to create object. (so Lesser code)
All the variables in case class are by default val type. Hence immutable. which is a good thing in spark world as all red are immutable
example for case class is
case class Book( name : string)
val book1 = Book("test")
you cannot change value of book1.name as it is immutable. and you do not need to say new Book() to create object here.
The class variables are public by default. so you don't need setter and getters.
Moreover while comparing two objects of case classes, their structure is compared instead of references.
Edit : Spark Uses Following class to Infer Schema
Code Link :
https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala
If you check. in schemaFor function (Line 719 to 791). It converts Scala types to catalyst types. I this the case to handle non case classes for schema inference is not added yet. so the every time you try to use non case class with infer schema. It goes to other option and hence gives error of Schema for type $other is not supported.
Hope this helps
I'm using the following code to instantiate a scala object. This works, but there seems to be one problem: the println is printed out twice, each time with another hashcode.
import scala.reflect.runtime.universe._
import scala.reflect.runtime.{universe => ru}
object Test2 { println("init"+hashCode())}
val mirror = ru.runtimeMirror(getClass.getClassLoader)
val m = ru.typeOf[Test2.type].members.filter(_.isConstructor).head.asMethod
val m2 = mirror.reflectClass(typeOf[Test2.type].typeSymbol.asClass)
val cm = m2.reflectConstructor(m)
val e = cm.apply()
Results in:
init472467991
init2051378291
e: Any = Test2$#7a458c73
the hashCode of e is equal to the latter one (2051378291). I'm wondering why this is because as far as I know there should be only one?
EDIT: using scala version 2.12.4
JVM has no singletons*
You're invoking a private constructor of a class. Scala reflection allows it. And when you invoke a constructor, you get a new instance back.
It's actually pretty hard to make a singleton in plain Java because there are ways to construct an instance except using new Something. For instance, de-serialization might not call any constructors besides one of Object. And there's sun.misc.Unsafe#allocateInstance that can conjure new instances of any class sans java.lang.Class without calling any constructor code.
Scala object does some job behind the hood to ensure you don't accidentally create a second instance during any normal usage (e.g. it hides the constructor and handles de-serialization), but it cannot protect you from deliberately creating one. When you start using reflection, you do exactly that.
Even Java enums can be instantiated at runtime using reflection. You cannot call Class#newInstance on an enum directly (the implementation forbids it), but knowing a bit of internal details can get you there**
import java.nio.file.StandardOpenOption // first std enum I could remember for a quick dirty sample
val ctor = classOf[StandardOpenOption].getDeclaredConstructors.head
val aac = ctor.getClass.getDeclaredMethod("acquireConstructorAccessor")
aac.setAccessible(true) // unlimited power!
val ctorAccess = aac.invoke(ctor)
val newInstanceCall = ctorAccess.getClass.getDeclaredMethod("newInstance", classOf[Array[AnyRef]])
newInstanceCall.setAccessible(true)
// note that it does not throw ClassCastException, so it's a fine instance
val uhOh = newInstanceCall.invoke(ctorAccess, Array("UhOh", 42)).asInstanceOf[StandardOpenOption]
assert(uhOh.name == "UhOh")
assert(uhOh.ordinal == 42)
(interactive version # Scastie)
To get the "default" instance, you can access a public static field named MODULE$ using reflection. You can also run whole scala compiler at runtime
It's likely to be the best bet for you to not rely on reflection in whatever you're trying to achieve.
BTW, it is possible to get ScalaReflectionException trying to run your code in IntelliJ worksheet or Scastie in worksheet mode, because these things wrap your code in another object with main method
* Only tested on few versions of HotSpot JVM
** Please don't do this in any serious code! I only use this to prove a point. This is also pretty useless because it does not change values or valueOf. And yes, I only checked it on HotSpot that comes with JDK8.
Scala case classes can have 22+ properties these days, but AFAIA compiler does not compile apply/unapply methods then.
Is there a way to generate apply/unapply by means of a plugin at compile time or at least generate methods using IDE etc?
Note
please don't start asking - why do you need this? It is for mapping existing JSON schema from a mongoDB using Reactive Mongo
please don't advise to group properties into smaller case classes and etc. Schema was created by someone else & already exists on production.
Thank you for your answers in advance.
Yes, Scala supports >22 fields from version 2.11. However, there are certain limitations - the case class will no more have unapply or unapplyseq and tupled(you'll no longer convert case class to tuple) functions because scala still don't support tuple with more that 22 values.
val tup = (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22) //will compile
val tup = (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23) //will fail
Because of this, case class is much more like regular class class and many other libraries will be unable to fully utilize this case class. Such as, json serializer libraries.
I have faced with this issue when I tried to use macros read/write to serialize case class to json and viceversa in a playframework project it won't compile because case class no longer contain unapply() method. The one work around for this is to provide custom implicit read/writes for the case class instead of using macros.
case class Person(name: String, age: Int, lovesChocolate: Boolean)
implicit val personReads = Json.reads[Person] //this wond work, need to write custom read as below.
implicit val personReads = (
(__ \ 'name).read[String] and
(__ \ 'age).read[Int] and
(__ \ 'lovesChocolate).read[Boolean]
)(Person)
please don't start asking - why do you need this? It is for mapping
existing JSON schema from a mongoDB using Reactive Mongo
I'm assuming your is the same situation, you are using reactivemongo macros for json to/from case class serialization.
implicit val personReader: BSONDocumentReader[Person] = Macros.reader[Person]
implicit val personWriter: BSONDocumentWriter[Person] = Macros.writer[Person]
//or Handler
Macros.handler[Person]
Therefore, I would suggest you to use custom BSON reader and writer for the case class as documented here.
I have tens of JSON fragments to parse, and for each one I need to get an instance of the right parser. My idea was to create a config file where to write the name of the class to instantiate for each parser (a kind of map url -> parser) . Getting back to your solution, I cannot call the method I implemented in each parser if I have a pointer to Any. I suppose this is a very common problem with a well-set solution, but I have no idea what the best practices could be.
I really have no experience with Java, Reflection, Class Loading and all that stuff. So,
can anyone write for me the body of the method below? I need to get an instance of a class passed as String (no arguments needed for the constructor, at least so far...)
def createInstance(clazzName: String) = {
// get the Class for the given clazzName, e.g. "net.my.BeautifulClazz"
// instantiate an object and return it
}
Thanks, as usual...
There is a very simple answer:
scala> def createInstance(clazzName: String) = Class.forName(clazzName).newInstance
createInstance: (clazzName: String)Any
scala> createInstance("java.lang.String")
res0: Any = ""
If it works for you, everything is fine. If it don't, we have to look into your class loader. This is usually the point when things will get dirty.
Depending in what you want to do, look into:
The cake pattern, if you want to combine your classes during compile time
OSGi when you want to build a plugin infrastructure (look here for a very simple example)
Google guice, if you really need dependency injection (e.g. when mixing Scala and Java code) and the cake pattern does not work for you
I'm making a strategic game and I try to apply what I learned, try to use immutable data. In my game I have Units, these units can have different special function. By exemple some plane can hide themselves. What I search is a way to be able to do some sort of
abstract class Units {
val life:Int
}
trait Hidable { self: Units =>
val hided:Boolean
def hide:Units with Hidable= ....
}
without having to copy paste:
def hide = copy(hided=true)
on every case class that mixin Hidable.
A common way to update an immutable data structure -- is using lenses.
There's a compiler plugin to generate lenses for your code, though it is not very production-ready. It is also available for an old scalaz only.
Here's a related question.