Scala: Can I reproduce anonymous class creation with a factory method? - scala

As far as I understand it, Scala creates an anonymous class if I create a class using the new keyword and follow the class name with a constructor:
class MyClass {
def doStuff() {
// ...
}
}
val mc = new MyClass {
doStuff()
}
The nice thing being that all the code in the constructor is in the scope of the new object.
Is there a way I can reproduce this syntax where the class is created by a factory method rather than the new keyword? i.e. make the following code work:
val mf = new MyFactory
val mc = mf.MyClass {
doStuff()
}
I can't find a way to do it but Scala has so much to it that this might be pretty easy!
Using an import as suggested by #Ricky below I can get:
val mf = MyFactory;
val mc = mf.MyClass
{
import mc._
doStuff()
}
(Where the blank line before the block is needed) but that code block is not a constructor.

You can do this, but you still have to keep the new keyword, and create the nested class as a path-dependent type:
class Bippy(x: Int) {
class Bop {
def getIt = x
}
}
val bip = new Bippy(7)
val bop = new bip.Bop
bop.getIt // yields 7
val bop2 = new bip.Bop{ override def getIt = 42 }
bop2.getIt // yields 42

I don't think it's possible. However, a common pattern is to add a parameter to factory methods which takes a function modifying the created object:
trait MyClass {
var name = ""
def doStuff():Unit
}
class Foo extends MyClass {
def doStuff() { println("FOO: " + name) }
}
trait MyClassFactory {
def make: MyClass
def apply( body: MyClass => Unit ) = {
val mc = make
body(mc)
mc
}
}
object FooFactory extends MyClassFactory {
def make = new Foo
}
You can then create and modify instance with a syntax close to your example:
val foo = FooFactory { f=>
f.name = "Joe"
f.doStuff
}

It sounds like you're just looking to mix in a trait. Instead of calling myFactoryMethod(classOf[Foo]] which ideally would do (if Scala permitted it):
new T {
override def toString = "My implementation here."
}
you can instead write
trait MyImplementation {
override def toString = "My implementation here."
}
new Foo with MyImplementation
However, if you are just looking to get the members of the new object accessible without qualification, remember you can import from any stable identifier:
val foo = new Bar
import foo._
println(baz) //where baz is a member of foo.

Related

Is there any way to rewrite the below code using Scala value class or other concept?

I need to write two functions to get the output format and the output index for file conversion. As part of this, I wrote a TransformSettings class for these methods and set the default value. And in the transformer class, I created a new object of TransformSettings class to get the default values for each job run. Also, I have another class called ParquetTransformer that extends Transformer where I want to change these default values. So I implemented like below.
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer{
def getTransformSettings: TransformSettings = {
new TransformSettings
}
def posttransform(table: AWSGlueDDL.Table):Dateframe ={
val indexAccess = getTransformSettings.getOuputIndex(table: AWSGlueDDL.Table)
........
}
}
class ParquetTransformer extends Transformer{
override def getTransformSettings: TransformSettings = {
val transformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
}
}
Is there a way to avoid creating a brand new object of TransformSettings in Transfomer class every time this is called?
Also is there a way to rewrite the code using Scala value class?
As #Dima proposed in the comments try to make TransformSettings a field / constructor parameter (a val) in the class Transformer and instantiate them outside
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer(val transformSettings: TransformSettings) {
def posttransform(table: AWSGlueDDL.Table): DataFrame ={
val indexAccess = transformSettings.getOuputIndex(table: AWSGlueDDL.Table)
???
}
}
val parquetTransformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
class ParquetTransformer extends Transformer(parquetTransformSettings)
You don't seem to need value classes (... extends AnyVal) now. They are more about unboxing, not about life-cycle management. TransformSettings and Transformer can't be value classes because they are not final (you're extending them in class ParquetTransformer extends Transformer... and new TransformSettings { ... }). By the way, value classes have many limatations
https://failex.blogspot.com/2017/04/the-high-cost-of-anyval-subclasses.html
https://github.com/scala/bug/issues/12271
Besides value classes, there are scala-newtype library in Scala 2 and opaque types in Scala 3.

The mock of my class isn't getting called

I have a HelperMethod class.
class HelperMethods {
def getUniqueID(): UUID = {
UUID.randomUUID()
}
def bucketIDFromEmail(email:String): Int = {
val bucketID= email(0).toInt
println("returning id "+bucketID+" for name "+email)
bucketID
}
}
And an object which has an instance of HelperMethods
package object utilities{
private val helper = new HelperMethods()
def getUniqueID(): UUID = helper.getUniqueID()
def bucketIDFromEmail(email:String): Int = helper.bucketIDFromEmail(email)
}
I wrote a spec to test that my mock works correctly.
class UserControllerUnitSpec extends PlaySpec {
val mockHelperMethods = mock(classOf[HelperMethods])
when(mockHelperMethods.getUniqueID()).thenReturn(UUID.fromString("87ea52b7-0a70-438f-81ff-b69ab9e57210"))
when(mockHelperMethods.bucketIDFromEmail(ArgumentMatchers.any[String])).thenReturn(1)
"mocking helper class " should {
"work" in {
val bucketId = utilities.bucketIDFromEmail("t#t.com")
println("user keys are " + userKeys)
val id: UUID = utilities.getUniqueID()
println("got id " + userKeys)
bucketId mustBe 1
id mustBe UUID.fromString("87ea52b7-0a70-438f-81ff-b69ab9e57210")
}
}
}
the test fails with reason 116 was not equal to 1. This corresponds to line
bucketId mustBe 1 in the spec. I can see the print returning id 116 for name t#t.com. I shouldn't see it as I am trying to mock this class. I suspect that it could be because the utilities object is getting created before the statement val mockHelperMethods = mock(classOf[HelperMethods]) in the spec.
Question 2- Is there a way to mock HelperMethods and make utilities use the mocked class?
You have mocked HelperMethods but not utilities.
Question 2- Is there a way to mock HelperMethods and make utilities use the mocked class?
It is not possible to mock an object.
If you want, you have to extract the behavior in a trait.
Here is a solution that would work:
package utils
// move the behavior to a trait:
trait UtitilitiesTrait {
private[utils] def helper = new HelperMethods()
def getUniqueID(): UUID = helper.getUniqueID()
def bucketIDFromEmail(email: String): Int = helper.bucketIDFromEmail(email)
}
// provide an object for real use
object Utilities extends UtitilitiesTrait
// override helper for test mock
object TestUtilities extends UtitilitiesTrait {
private[utils] override def helper = mock(classOf[HelperMethods])
}
And here is your test:
class UserControllerUnitSpec extends PlaySpec {
val mockHelperMethods = mock(classOf[HelperMethods])
object TestUtilities extends UtitilitiesTrait {
private[utils] override def helper = mockHelperMethods
}
when(mockHelperMethods.getUniqueID()).thenReturn(UUID.fromString("87ea52b7-0a70-438f-81ff-b69ab9e57210"))
when(mockHelperMethods.bucketIDFromEmail(ArgumentMatchers.any[String])).thenReturn(1)
"mocking helper class " should {
"work" in {
val bucketId = TestUtilities.bucketIDFromEmail("t#t.com")
println("user keys are " + userKeys)
val id: UUID = TestUtilities.getUniqueID()
println("got id " + userKeys)
bucketId mustBe 1
id mustBe UUID.fromString("87ea52b7-0a70-438f-81ff-b69ab9e57210")
}
}
}
The typical pattern that enables mocking objects used internally is to inject them, or at least provide a way to inject an alternate.
Since Utilities is an object, you can't inject using a constructor. You could still introduce a setter method.
If you'd like to discourage use of the setter for anything other than unit tests, make it package-private, and you might also prefix the name with "qa":
private[utils] def qaSetHelperMethods(qaHelper: HelperMethods): Unit

Access Spark broadcast variable in different classes

I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}

Class A cannot be cast to Class A after dynamic loading

Let's say I have:
object GLOBAL_OBJECT{
var str = ""
}
class A(_str: String){
GLOBAL_OBJECT.str = _str
}
and I would like to create 2 copies of GLOBAL_OBJECT (for tests), so I am using different classloader to create obj2:
val obj1 = new A("1")
val class_loader = new CustomClassLoader()
val clazz = class_loader.loadClass("my.packagename.A")
val obj2 = clazz.getDeclaredConstructor(classOf[String]).newInstance("2")
println("obj1.getSecret() == " + obj1.getSecret()) // Expected: 1
println("obj2.getSecret() == " + obj2.asInstanceOf[A].getSecret()) // Expected: 2
which results following error:
my.packagename.A cannot be cast to my.packagename.A.
IntelliJ Idea seems to do it correctly, I can run obj2.asInstanceOf[A].getSecret() in "expression" window during debug process without errors.
PS. I have seen similar questions, but I could not find any not regarding loading class from .jarfile.
You're not going to be able to get around Java's class casting, which requires strict typing, within the same ClassLoader. Same with traits/interfaces.
However, Scala comes to the rescue with structural typing (a.k.a. Duck Typing, as in "it quacks like a duck.") Instead of casting it to type A, cast it such that it has the method you want.
Here's an example of a function which uses structural typing:
def printSecret(name : String, secretive : { def getSecret : String } ) {
println(name+".getSecret = "+secretive.getSecret)
}
And here's sample usage:
printSecret("obj1", obj1) // Expected: 1
printSecret("obj2", obj2.asInstanceOf[ {def getSecret : String} ]) // Expected: 2
You could, of course, just call
println("secret: "+ obj2.asInstanceOf[ {def getSecret : String} ].getSecret
Here's full sample code that I wrote and tested.
Main code:
object TestBootstrap {
def createClassLoader() = new URLClassLoader(Array(new URL("file:///tmp/theTestCode.jar")))
}
trait TestRunner {
def runTest()
}
object RunTest extends App {
val testRunner = TestBootstrap.createClassLoader()
.loadClass("my.sample.TestCodeNotInMainClassLoader")
.newInstance()
.asInstanceOf[TestRunner]
testRunner.runTest()
}
In the separate JAR file:
object GLOBAL_OBJECT {
var str = ""
}
class A(_str: String) {
println("A classloader: "+getClass.getClassLoader)
println("GLOBAL classloader: "+GLOBAL_OBJECT.getClass.getClassLoader)
GLOBAL_OBJECT.str = _str
def getSecret : String = GLOBAL_OBJECT.str
}
class TestCodeNotInMainClassLoader extends TestRunner {
def runTest() {
println("Classloader for runTest: " + this.getClass.getClassLoader)
val obj1 = new A("1")
val classLoader1 = TestBootstrap.createClassLoader()
val clazz = classLoader1.loadClass("com.vocalabs.A")
val obj2 = clazz.getDeclaredConstructor(classOf[String]).newInstance("2")
def printSecret(name : String, secretive : { def getSecret : String } ) {
println(name+".getSecret = "+secretive.getSecret)
}
printSecret("obj1", obj1) // Expected: 1
printSecret("obj2", obj2.asInstanceOf[ {def getSecret : String} ]) // Expected: 2
}
}
Structural typing can be used for more than one method, the methods are separated with semicolons. So essentially you create an interface for A with all the methods you intend to test. For example:
type UnderTest = { def getSecret : String ; def myOtherMethod() : Unit }
One workaround to actually run some method from dynamically delivered object instead of casting it is to use reflection in order to extract particular method, from new class and then invoke it on our new object instance:
val m2: Method = obj2.getClass.getMethod("getSecret")
m2.invoke(obj2)
The class file that contains obj2.asInstanceOf[A].getSecret() should be reloaded by CustomClassLoader, too.
And you must not use any class that references to A unless you reload the class by the same class loader that reloads A.

Scala passing a reference to this class

I have a class, with multiple methods and members. When I create an instance of this class, I create an instance of another class within the first class. Some of the methods in this second class require to know which instance of the first class is running. Currently, I am trying to pass "this" into the argument that accepts type firstClass. What am I doing wrong? Again, I simply want the second class instance knowing what first class instance it belongs to so that it can call public methods and members from it.
EDIT: Code example:
def main(args:Array[String]) : Unit = {
val objectOne = new classOne
}
class classOne {
val mutableBuffer = mutable.Buffer[String]
val objectTwo = new classTwo
objectTwo.doThis(this)
}
class classTwo {
def doThis (exA:classOne) = {
exA.mutableBuffer += "Adding text to a Buffer in object One"
}
}
Self-typing is often the cleanest solution here
class Bippy {
outer =>
...
class Bop {
def demoMethod() = println(outer)
}
...
}
UPDATE
The example code changes everything, this clearly isn't about inner classes. I believe your problem is in this line:
val mutableBuffer = mutable.Buffer[String]
It isn't doing what you think it's doing, mutableBuffer is now pointing to the mutable.Buffer singleton, it isn't actually an instance of a Buffer
Instead, try one of these two:
val mutableBuffer = mutable.Buffer[String]()
//or
val mutableBuffer = mutable.Buffer.empty[String]
You should also stick to the convention of starting class/singleton/type names with an uppercase letter, turning your example code into:
import collection.mutable.Buffer
def main(args:Array[String]) : Unit = {
val one = new ClassOne()
}
class ClassOne {
val mutableBuffer = Buffer.empty[String]
val two = new ClassTwo()
two.doThis(this)
}
class ClassTwo {
def doThis(one: ClassOne) = {
one.mutableBuffer += "Adding text to a Buffer in object One"
}
}
I had to make some superficial changes to your example code in order to make it run:
import scala.collection.mutable
class classOne {
val mutableBuffer : mutable.Buffer[String] = new mutable.ArrayBuffer[String]
val objectTwo = new classTwo
objectTwo.doThis(this)
}
class classTwo {
def doThis (exA : classOne) = {
exA.mutableBuffer += "Adding text to a Buffer in object One"
}
}
val objectOne = new classOne
println(objectOne.mutableBuffer(0))
But it works as expected. The classTwo object is able to modify the classOne object. Do you need something beyond this functionality?