Mocking scala object called by under another object - scala

I am trying to write unit test for a function under object1.
object Object1 {
def main(sysArgs: Array[String]): Unit = {
val inputDF: DataFrame = UtilObject.getInput()
}
}
object UtilObject {
def getInput(){
...
}
}
To write Unit test, I am using MockitoSugar.
"object1Main" should "should make correct calls" in {
val inputArgs = Array("abc")
val util = mock[UtilObject.type]
when(util.getInput().thenReturn(inputData))
Object1.main(inputArgs)
}
While running the test, it doesn't consider the util mock and just execute the getInput() function.
I think I am missing some sort of injection here. Any ideas?
Thanks in advance!

Mocking Scala objects should be impossible conceptually speaking. An object in Scala is a pure singleton. That means there can only be one member of that type at any time.
mockito-scala can mock Scala objects via reflection. I'll use a result type of String, instead of a DataFrame, but the idea is the same:
object UtilObject {
def getInput(): String = {
// ...
"done"
}
}
object Object1 {
def main(sysArgs: Array[String]): String = {
val inputDF: String = UtilObject.getInput()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val inputArgs = Array("abc")
withObjectMocked[UtilObject.type] {
UtilObject.getInput() returns "mocked!"
Object1.main(inputArgs) shouldBe "mocked!"
}
Object1.main(inputArgs) shouldBe "done"
}
}
This mocks the singleton's method only inside the block of withObjectMocked.
Usually such powerful techniques often tend to be overused or misused, so I don't generally recommend them, unless the design cannot be refactored.
Luckily, yours can: the easiest way is to use Dependency Injection with a class or a function. For DI with a class you require to convert the object being mocked into a class:
class UtilObject {
def getInput(): String = {
// ...
"done"
}
}
object Object1 {
def main(sysArgs: Array[String], ut: UtilObject): String = {
val inputDF: String = ut.getInput()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val inputArgs = Array("abc")
val util = mock[UtilObject]
when(util.getInput()).thenReturn("mocked!")
Object1.main(inputArgs, util) shouldBe "mocked!"
}
}
For DI with a function you need to lift the method you want to mock into a function:
object UtilObject {
def getInput(): String = {
// ...
"done"
}
}
object Object1 {
def main(sysArgs: Array[String], f: () => String = UtilObject.getInput): String = {
val inputDF: String = f()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val inputArgs = Array("abc")
val f = mock[() => String]
when(f()).thenReturn("mocked!")
Object1.main(inputArgs, f) shouldBe "mocked!"
}
}
Since the function takes no arguments, you can convert it into a by-name parameter. I'll leave that to you.
Lastly, another way is to create a trait with the method you want to mock and extend that with the object. But now Object1 requires being a class and have a reference to the object being mocked:
object UtilObject extends Utils {
def getInput(): String = {
// ...
"done"
}
}
trait Utils {
def getInput(): String
}
class Object1 {
val uo: Utils = UtilObject
def main(sysArgs: Array[String]): String = {
val inputDF: String = uo.getInput()
inputDF
}
}
// in test file:
"object1Main" should {
"should make correct calls" in {
val classUnderTest = new Object1 {
override val uo = mock[Utils]
}
val inputArgs = Array("abc")
when(classUnderTest.uo.getInput()).thenReturn("mocked!")
classUnderTest.main(inputArgs) shouldBe "mocked!"
}
}
As you can see, there are a few ways to go. Neither is inherently wrong. It mostly depends on your requirements(e.g. you can't afford adding a dependency just for one UT), needs(e.g. does the object I am testing really needs to be an object or can I make it a class?), guidelines (e.g. your team decided to avoid using powerful testing frameworks that rely on reflection and use DI as much as possible instead) and personal preferences.

Related

Is there any way to rewrite the below code using Scala value class or other concept?

I need to write two functions to get the output format and the output index for file conversion. As part of this, I wrote a TransformSettings class for these methods and set the default value. And in the transformer class, I created a new object of TransformSettings class to get the default values for each job run. Also, I have another class called ParquetTransformer that extends Transformer where I want to change these default values. So I implemented like below.
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer{
def getTransformSettings: TransformSettings = {
new TransformSettings
}
def posttransform(table: AWSGlueDDL.Table):Dateframe ={
val indexAccess = getTransformSettings.getOuputIndex(table: AWSGlueDDL.Table)
........
}
}
class ParquetTransformer extends Transformer{
override def getTransformSettings: TransformSettings = {
val transformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
}
}
Is there a way to avoid creating a brand new object of TransformSettings in Transfomer class every time this is called?
Also is there a way to rewrite the code using Scala value class?
As #Dima proposed in the comments try to make TransformSettings a field / constructor parameter (a val) in the class Transformer and instantiate them outside
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer(val transformSettings: TransformSettings) {
def posttransform(table: AWSGlueDDL.Table): DataFrame ={
val indexAccess = transformSettings.getOuputIndex(table: AWSGlueDDL.Table)
???
}
}
val parquetTransformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
class ParquetTransformer extends Transformer(parquetTransformSettings)
You don't seem to need value classes (... extends AnyVal) now. They are more about unboxing, not about life-cycle management. TransformSettings and Transformer can't be value classes because they are not final (you're extending them in class ParquetTransformer extends Transformer... and new TransformSettings { ... }). By the way, value classes have many limatations
https://failex.blogspot.com/2017/04/the-high-cost-of-anyval-subclasses.html
https://github.com/scala/bug/issues/12271
Besides value classes, there are scala-newtype library in Scala 2 and opaque types in Scala 3.

Access Spark broadcast variable in different classes

I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}

How to initialize fixture once for all tests in scalatest?

According to documentation, fixtures can be passed to tests like this:
def withFixture(test: OneArgTest) = {
val f = veryLengthyInitialization(test.configMap)
withFixture(test.toNoArgTest(f))
}
describe("Some test") {
it("should succeed") { f =>
f.doSomething()
// ...
}
// many other tests
// ...
}
Problem is that the initialization is run for each test, and it takes a very long time, so I would like it to run just once. Documentation suggests an alternative:
override def beforeAll(configMap: ConfigMap) {
val fOnce = veryLengthyInitialization(configMap)
// but how do I pass f to the tests?
// ugly workaround:
f = fOnce
}
var f: InitType = null
describe("Some test") {
it("should succeed") { // hack: no parameter, use var instead
f.doSomething()
// ...
}
// many other tests
// ...
}
It works, but my understanding is that null should be avoided, and it's a bit silly to rely on mutation for something as basic as this. What is the correct way to write such tests?
I don't think that there is a solution that is good in all aspects for this issue. One thing I do sometime to avoid the var is structuring the test as follows:
class SomeTest extends Spec {
def initFixtures(): SomeFixtureType = ???
lazy val fixtures = initFixtures()
describe("Some test") {
it("should succeed") in {
fixtures.doSomething()
// ...
}
}
}
In this way I don't need to use the beforeAndAfterAll thing, but you can if you want to guarantee that the fixtures are initialised before the test as follows:
override def beforeAll() {
super.beforeAll()
fixtures
}
It can also be handy when dealing with asynchronous initialisation since you could have something like:
class SomeTest extends Spec {
def initFixtures(): Future[SomeFixtureType] = ???
lazy val fixtures = initFixtures()
describe("Some test") {
it("should succeed") in {
whenReady(fixtures.flatMap {
f.doSomething()
// ...
}) { res => doSomeChecks(res) }
}
}
}
Having both initFixtures and fixtures is most useful in this case as you can have some tests re-initialise the system and act in isolation if they need to do effectful stuffs (e.g. use a database).
I'm not sure if the below is something better:
trait DoSomething {
var f:InitType = _
def doSomething = f.doSomething
}
class MyTestSpec extends DoSomething {
override def beforeAll(configMap: ConfigMap) {
val fOnce = veryLengthyInitialization(configMap)
f = fOnce
}
it("should succeed") {
doSomething()
}
}
For any tests you need a f variable to do something, you could extend the trait and initialize it in your beforeAll methods, in addition, you could even override the doSomething in your test.

Do I have to avoid using objects if I want to unit-test them?

Say, I'm using some json libraries which provides an object Json:
object Json {
def unapply[T](jsonStr:String)(implicit converter:Converter[T]) { ... }
}
I can use it in my scala code like this:
class MyLoginController {
def login(request:Request) = {
val loginInfo = Json.unapply[LoginInfo](request.body)
// check(loginInfo.email)
// check(loginInfo.password)
}
}
It's works perfectly, but soon I found my self can't test it easily with mock. I can't find a way to mock the Json.unapply.
So I have to change my code to provide a trait and use dependency injection:
trait JsonParsable[T] {
def parse(jsonStr:String)(implicit converter:Converter[T])
}
object JsonParser extends JsonParsable[T] {
def parse(jsonStr:String)(implicit converter:Converter[T]) = Json.unapply(jsonStr)
}
class MyLoginController(jsonParser:JsonParsable[T]) {
def login(request:Request) = {
val loginInfo = jsonParser.parse[LoginInfo](request.body)
// check(loginInfo.email)
// check(loginInfo.password)
}
}
And when I write unit test, I will mock a JsonParsable for the MyLoginController:
val fakeParser = mock[JsonParsable]
// ...
val controller = new MyLoginController(fakeParser)
// ...
My question is, do I have to do this to avoid using objects, just in order to make it testable?
I found previous code is simple and easy, but the later one is more complex :(

Scala: Can I reproduce anonymous class creation with a factory method?

As far as I understand it, Scala creates an anonymous class if I create a class using the new keyword and follow the class name with a constructor:
class MyClass {
def doStuff() {
// ...
}
}
val mc = new MyClass {
doStuff()
}
The nice thing being that all the code in the constructor is in the scope of the new object.
Is there a way I can reproduce this syntax where the class is created by a factory method rather than the new keyword? i.e. make the following code work:
val mf = new MyFactory
val mc = mf.MyClass {
doStuff()
}
I can't find a way to do it but Scala has so much to it that this might be pretty easy!
Using an import as suggested by #Ricky below I can get:
val mf = MyFactory;
val mc = mf.MyClass
{
import mc._
doStuff()
}
(Where the blank line before the block is needed) but that code block is not a constructor.
You can do this, but you still have to keep the new keyword, and create the nested class as a path-dependent type:
class Bippy(x: Int) {
class Bop {
def getIt = x
}
}
val bip = new Bippy(7)
val bop = new bip.Bop
bop.getIt // yields 7
val bop2 = new bip.Bop{ override def getIt = 42 }
bop2.getIt // yields 42
I don't think it's possible. However, a common pattern is to add a parameter to factory methods which takes a function modifying the created object:
trait MyClass {
var name = ""
def doStuff():Unit
}
class Foo extends MyClass {
def doStuff() { println("FOO: " + name) }
}
trait MyClassFactory {
def make: MyClass
def apply( body: MyClass => Unit ) = {
val mc = make
body(mc)
mc
}
}
object FooFactory extends MyClassFactory {
def make = new Foo
}
You can then create and modify instance with a syntax close to your example:
val foo = FooFactory { f=>
f.name = "Joe"
f.doStuff
}
It sounds like you're just looking to mix in a trait. Instead of calling myFactoryMethod(classOf[Foo]] which ideally would do (if Scala permitted it):
new T {
override def toString = "My implementation here."
}
you can instead write
trait MyImplementation {
override def toString = "My implementation here."
}
new Foo with MyImplementation
However, if you are just looking to get the members of the new object accessible without qualification, remember you can import from any stable identifier:
val foo = new Bar
import foo._
println(baz) //where baz is a member of foo.