Factory method example in spark-shell giving error - scala

I am unable to run below code in spark-shell repl. This is a reproduction from Scala cookbook example provided by Alvin Alexander
I get the error:
<console>:22: error: reference to Animal is ambiguous;
it is imported twice in the same scope by
import $VAL11.Animal
and import INSTANCE.Animal
val test = Animal("dog")
This works fine with Scala repl. Can you please tell me how I can make this example work on spark-shell as well?
Thanks a lot for your help!
trait Animal {
def speak
}
object Animal {
private class Dog extends Animal {
override def speak = {
println("woof")
}
}
private class Cat extends Animal {
override def speak { println("meow") }
}
def apply(s: String):Animal = {
if (s == "dog") return new Dog
else return new Cat
}
}
# repl
Animal("dog")
# compiling
object test {
def main(args: Array[String]){
Animal(args(0)).speak
}
}

As som-snytt pointed out, this is most likely a version of an existing scala bug.
Animal("dog") // show
results in something like this:
import $line15.$read.INSTANCE.$iw.$iw.Animal;
val $line15$read = $line15.$read.INSTANCE;
import $line15$read.$iw.$iw.Animal;
Note that Animal is imported twice. To fix this you can wrap your code in an object:
object test{
trait Animal {
def speak
}
object Animal {
private class Dog extends Animal {
override def speak = {
println("woof")
}
}
private class Cat extends Animal {
override def speak { println("meow") }
}
def apply(s: String):Animal = {
if (s == "dog") return new Dog
else return new Cat
}
}
}
Now, when you call test.Animal("dog") // show you get this instead:
val $line15$read = $line15.$read.INSTANCE;
import $line15$read.$iw.$iw.$iw.$iw.$iw.$iw.$iw.$iw.test;
Note that you get only one, more concise import.
As an interesting side note, if you run your first code and then the second, then test.Animal("dog") results in this:
import $line15.$read.INSTANCE.$iw.$iw.Animal;
val $line16$read = $line16.$read.INSTANCE;
import $line16$read.$iw.$iw.$iw.$iw.$iw.$iw.$iw.$iw.test;
Where the Animal import remains.

Related

Is there any way to rewrite the below code using Scala value class or other concept?

I need to write two functions to get the output format and the output index for file conversion. As part of this, I wrote a TransformSettings class for these methods and set the default value. And in the transformer class, I created a new object of TransformSettings class to get the default values for each job run. Also, I have another class called ParquetTransformer that extends Transformer where I want to change these default values. So I implemented like below.
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer{
def getTransformSettings: TransformSettings = {
new TransformSettings
}
def posttransform(table: AWSGlueDDL.Table):Dateframe ={
val indexAccess = getTransformSettings.getOuputIndex(table: AWSGlueDDL.Table)
........
}
}
class ParquetTransformer extends Transformer{
override def getTransformSettings: TransformSettings = {
val transformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
}
}
Is there a way to avoid creating a brand new object of TransformSettings in Transfomer class every time this is called?
Also is there a way to rewrite the code using Scala value class?
As #Dima proposed in the comments try to make TransformSettings a field / constructor parameter (a val) in the class Transformer and instantiate them outside
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer(val transformSettings: TransformSettings) {
def posttransform(table: AWSGlueDDL.Table): DataFrame ={
val indexAccess = transformSettings.getOuputIndex(table: AWSGlueDDL.Table)
???
}
}
val parquetTransformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
class ParquetTransformer extends Transformer(parquetTransformSettings)
You don't seem to need value classes (... extends AnyVal) now. They are more about unboxing, not about life-cycle management. TransformSettings and Transformer can't be value classes because they are not final (you're extending them in class ParquetTransformer extends Transformer... and new TransformSettings { ... }). By the way, value classes have many limatations
https://failex.blogspot.com/2017/04/the-high-cost-of-anyval-subclasses.html
https://github.com/scala/bug/issues/12271
Besides value classes, there are scala-newtype library in Scala 2 and opaque types in Scala 3.

Scala Testing with Mock objects within objects

While Testing Called1.writeData and mocking using the traits, control is still going to the original implementation. Need help from experts.
object Caller1() {
def method1(param1:String, param2:Long) {
val param3=param1 //Some transformation
val param4=param2
Called1.writeData(param3, param4)
}
}
object Called1 extends Called1Trait {
val producer = Called2.createProducer
def writeData {
val data = createData(param1)
Called2.MethodA(...)
}
def createData(param1) {
// Some code
}
}
object Called2 extends Called2Trait {
def createProducer {
// Some code
}
def methodA {
// Some code
}
def methodB {
// methodA is used here
}
}
I have used mocking of the traits and using when and then to bypass the execution. Yet it is going to the methods. I have used any().
val CalledObject1Mock = mock[CalledObject1Trait]
val CalledObject2Mock = mock[CalledObject2Trait]
when(CalledObject1Mock.createProducer).thenReturn(ProducerMock)
when(CalledObject1Mock.writeData(any(),any())).thenReturn(true)
when(CalledObject1Mock.createData(any())).thenReturn(MockedData)
when(CalledObject2Mock.methodA(any(), any(), any())).thenReturn(true)
when(CalledObject2Mock.methodB(any(), any(), any())).thenReturn(true)
Called1.writeData(testParam1, testParam2) // this is failing
We have used scala Mockito. So do not intend to use any other framework.
import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.BeforeAndAfterAll
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatestplus.mockito.MockitoSugar
import org.scalatest.matchers.should._
trait BaseTestSuite extends AnyFunSuite with BeforeAndAfterAll with Matchers with MockitoSugar {
The current design does not allow to test it by Mock.
You mocked the called objects but don't use them. The original Called1 does not know about mocked objects.
You should not mock a class under the testing.
To fix it I would suggest redesigning your classes this way:
class Caller1(called1: Called1Trait) {
def method1(param1:String, param2:Long) {
val param3=param1 //Some transformation
val param4=param2
called1.writeData(param3, param4)
}
}
class Called1(called2: Called2Trait) extends Called1Trait {
val producer = called2.createProducer
def writeData {
val data = createData(param1)
called2.methodA(...)
}
def createData(param1) {
// Some code
}
}
class Called2 extends Called2Trait {
def createProducer {
// Some code
}
def methodA {
// Some code
}
def methodB {
// methodA is used here
}
}
So, assume you want to test the Called1. The test will look like this:
val called2Mock = mock[Called2Trait]
when(called2Mock.methodA(any(), any(), any())).thenReturn(true)
when(called2Mock.methodB(any(), any(), any())).thenReturn(true)
val called1 = new Called1(called2Mock)
called1.writeData(testParam1, testParam2)
Now the called1 knows that called2 is mocked object and will call the mocked methodA().

Specs2 with Scaldi - wrong implicit injector being invoked

I'm trying to run a test with scaldi and specs2. In the test I need to override a StringManipulator function that uses an injected ProxyManipulator. The ProxyManipulator takes a string and returns its upper case in a Future. The replacement manipulator in the test returns a Future("Test Message").
Here is the StringManipulator class where the injection occurs:
class StringManipulator {
def manip (str : String) (implicit inj: Injector) : String = {
val prox = inject[ProxyManipulator]
Await.result(prox.manipulate(str), 1 second)
}
}
I'm using a package.object that contains the implicit injector:
import modules.MyModule
package object controllers {
implicit val appModule = new MyModule
}
And here is the specs2 test with the new binding:
#RunWith(classOf[JUnitRunner])
class StringManipScaldiSpec extends Specification {
class TestModule extends Module {
bind [ProxyManipulator] to new ProxyManipulator {
override def manipulate(name: String) = Future("Test Message")
}
}
"Application" should {
"do something" in {
val myTestModule = new TestModule
val str = "my string"
val stringMan = new StringManipulator() //(myTestModule)
stringMan.manip(str)(myTestModule) === "Test Message"
}
}
}
The problem is that when the test runs the class StringManipulator is still using the original Proxy Manipulator instead of the one passed in the TestModule. Any ideas?

Scala Implicit Type Conversion of Classes with Type Parameters

I'm trying to add functionality to the scala.collection.Iterable trait, more specifically, a printerate function that iterates through the elements and prints them out (to the console if there are no parameters, otherwise to the outputstream param). I'm using a predefined extension method that I created for object, printSelf(). However, this is causing an compiler error, 'Value printSelf is not a member of type parameter Object.' I'd also like to have this an a separate file so that it's easy for me to use between several projects and applications.
Here's my current code for my conversion file:
import java.io.OutputStream
import scala.collection.Iterable
package conversion{
class Convert {
implicit def object2SuperObject(o:Object) = new ConvertObject(o)
implicit def iterable2SuperIterable[Object](i:Iterable[Object]) = new ConvertIterable[Object](i)
}
class ConvertObject(o:Object){
def printSelf(){
println(o.toString())
}
def printSelf(os:OutputStream){
os.write(o.toString().getBytes())
}
}
class ConvertIterable[Object](i:Iterable[Object]){
def printerate(){
i.foreach {x => x.printSelf() }
}
def printerate(os:OutputStream){
i.foreach { x => x.printSelf(os) }
}
}
}
I'm also getting a similar error in the code that's trying to test this out, 'value printerate is not a member of scala.collection.immutable.Range':
import conversion.Convert
package test {
object program extends App {
new testObj(10) test
}
class testObj(i: Integer) {
def test(){
val range = 0.until(i)
0.until(i).printerate()
}
}
}
What's wrong with the way that I'm approaching this type conversion?
Several things in fact:
Convert should be an object, not a class.
You use Object instead of Any
You use Object as a generic type identifier, instead of the much less confusing T.
You do not import the implicit definitions (it's not enough to import the object itself).
This should work:
package conversion {
object Convert {
implicit def object2SuperObject(o: Any) = new ConvertObject(o)
implicit def iterable2SuperIterable[T](i:Iterable[T]) = new ConvertIterable[T](i)
}
class ConvertObject(o: Any){
def printSelf(){
println(o.toString())
}
def printSelf(os:OutputStream){
os.write(o.toString().getBytes())
}
}
class ConvertIterable[T](i:Iterable[T]){
import Convert.object2SuperObject
def printerate(){
i.foreach {x => x.printSelf() }
}
def printerate(os:OutputStream){
i.foreach { x => x.printSelf(os) }
}
}
}
import conversion.Convert._
Second file:
package test {
object program extends App {
new testObj(10) test
}
class testObj(i: Integer) {
def test(){
val range = 0.until(i)
0.until(i).printerate()
}
}
}

Scala: Can I reproduce anonymous class creation with a factory method?

As far as I understand it, Scala creates an anonymous class if I create a class using the new keyword and follow the class name with a constructor:
class MyClass {
def doStuff() {
// ...
}
}
val mc = new MyClass {
doStuff()
}
The nice thing being that all the code in the constructor is in the scope of the new object.
Is there a way I can reproduce this syntax where the class is created by a factory method rather than the new keyword? i.e. make the following code work:
val mf = new MyFactory
val mc = mf.MyClass {
doStuff()
}
I can't find a way to do it but Scala has so much to it that this might be pretty easy!
Using an import as suggested by #Ricky below I can get:
val mf = MyFactory;
val mc = mf.MyClass
{
import mc._
doStuff()
}
(Where the blank line before the block is needed) but that code block is not a constructor.
You can do this, but you still have to keep the new keyword, and create the nested class as a path-dependent type:
class Bippy(x: Int) {
class Bop {
def getIt = x
}
}
val bip = new Bippy(7)
val bop = new bip.Bop
bop.getIt // yields 7
val bop2 = new bip.Bop{ override def getIt = 42 }
bop2.getIt // yields 42
I don't think it's possible. However, a common pattern is to add a parameter to factory methods which takes a function modifying the created object:
trait MyClass {
var name = ""
def doStuff():Unit
}
class Foo extends MyClass {
def doStuff() { println("FOO: " + name) }
}
trait MyClassFactory {
def make: MyClass
def apply( body: MyClass => Unit ) = {
val mc = make
body(mc)
mc
}
}
object FooFactory extends MyClassFactory {
def make = new Foo
}
You can then create and modify instance with a syntax close to your example:
val foo = FooFactory { f=>
f.name = "Joe"
f.doStuff
}
It sounds like you're just looking to mix in a trait. Instead of calling myFactoryMethod(classOf[Foo]] which ideally would do (if Scala permitted it):
new T {
override def toString = "My implementation here."
}
you can instead write
trait MyImplementation {
override def toString = "My implementation here."
}
new Foo with MyImplementation
However, if you are just looking to get the members of the new object accessible without qualification, remember you can import from any stable identifier:
val foo = new Bar
import foo._
println(baz) //where baz is a member of foo.