Scala: mock Hadoop File System - scala

I need to mock (with scalamock 4.4.0/scalatest 3.2.9) the behaviour of Hadoop file system. I found out, that I can't mock it directly, so I'm trying to get around this problem.
On the first, try I created wrapper (in the scope of main logic):
class HadoopFsWrapper(path: Path)(implicit fs: org.apache.hadoop.fs.FileSystem) {
def listStatus(path) = fs.listStatus(path)
}
class HadoopService(path: Path)(implicit val fs: HadoopFsWrapper){
def someLogic() = fs.listStatus(path)
}
I declare this class both in main and test code with defining of implicit HadoopFsWrapper.
implicit val fs = mock[HadoopFsWrapper]
val hs = HadoopService(path)
// other test logic
And this done well.
But I don't think, its a good idea to modify main code for test purpose.
Other way I considered is to create wrapper only in test scope and apply some kind of "decorator" to deceive HadoopService, which expects FileSystem to be defined in main scope:
// only in test scope
class HadoopFsWrapper(path: Path) extends org.apache.hadoop.fs.FileSystem {
def listStatus(path) = listStatus(path)
// dummies for implemented FileSystem-methods
}
object HadoopFsWrapper {
def apply(): org.apache.hadoop.fs.FileSystem = new HadoopFsWrapper
}
// main logic
class HadoopService(path: Path)(implicit val fs: org.apache.hadoop.fs.FileSystem){
def someLogic() = fs.listStatus(path)
}
But when I'm trying to mock HadoopFsWrapper it's defined as null.

Related

How To Create Temporary Directory in Scala Unit Tests

In scala how can a unit test create a temporary directory to use as part of the testing?
I am trying to unit test a class which depends on a directory
class UsesDirectory(directory : java.io.File) {
...
}
I'm looking for something of the form:
class UsesDirectorySpec extends FlatSpec {
val tempDir = ??? //missing piece
val usesDirectory = UsesDirectory(tempDir)
"UsesDirectory" should {
...
}
}
Also, any comments/suggestions on appropriately cleaning up the resource after the unit testing is completed would be helpful.
Thank you in advance for your consideration and response.
Krzysztof's answer provides a good strategy for avoiding the need for temp directories in your tests altogether.
However if you do need UsesDirectory to work with real files, you can do something like the following to create a temporary directory:
import java.nio.file.Files
val tempDir = Files.createTempDirectory("some-prefix").toFile
Regarding cleanup, you could use the JVM shutdown hook mechanism to delete your temp files.
(java.io.File does provide deleteOnExit() method but it doesn't work on non-empty directories)
You could implement a custom shutdown hook using sys.addShutdownHook {}, and use Files.walk or Files.walkTree to delete the contents of your temp directory.
Also you may want to take a look at the better-files library, which provides a less verbose scala API for common files operations including File.newTemporaryDirectory() and file.walk()
File in Java is very cumbersome to test. There is no simple way to create some kind of virtual filesystem abstraction, which can be used for tests.
A cool way around it is to create some kind of wrapper, that can be used for stubbing and mocking.
For example:
trait FileOps { //trait which works as proxy for file
def getName(): String
def exists(): Boolean
}
object FileOps {
class FileOpsImpl(file: File) extends FileOps {
override def getName(): String = file.getName //delegate all methods you need
override def exists(): Boolean = file.exists()
}
implicit class FromFile(file: File) { //implicit method to convert File to FileOps
def toFileOps: FileOpsImpl = new FileOpsImpl(file)
}
}
Then you'd have to use it instead of File in your class:
class UsesDirectory(directory : FileOps) {
...
}
//maybe you can even create implicit conversion, but it's better to do it explicitly
val directory = new UserDirectory(file.toFileOps)
And what is benefit of that?
In your tests you can provide custom implementation of FileOps:
class UsesDirectorySpec extends FlatSpec {
val dummyFileOps = new FileOps {
override def getName(): String = "mock"
override def exists(): Boolean = true
}
//OR
val mockFileOps = mock[FileOps] //you can mock it easily since it's only trait
val usesDirectory = UsesDirectory(dummyFileOps)
"UsesDirectory" should {
...
}
}
If you use this or a similar approach, you don't even need to touch filesystem in your unit test.

Scala - Injecting Generic type using guice when dependency class is also using same generic type

I want to inject dependency with Generic type using Guice. Find below example in scala which replicate the issue.
ProductModel.scala
trait BaseProduct
case class Product() extends BaseProduct
CartService.scala
class CartService[A <: BaseProduct] #Inject()(productService : ProductService[A]) {
def getCartItems = productService.getProduct
}
ProductService.scala
class ProductService[A]{
def getProduct = println("ProductService")
}
Main.scala
object Main extends App {
val injector = Guice.createInjector(new ShoppingModule)
val cartService = injector.getInstance(classOf[CartService[Product]])
cartService.getCartItems
}
class ShoppingModule extends AbstractModule with ScalaModule {
override def configure(): Unit = {
bind[BaseProduct].to(scalaguice.typeLiteral[Product])
}
}
while running this Main.scala app getting below error.
service.ProductService<A> cannot be used as a key; It is not fully specified.
I have tried binding using codingwell library. But it doesn't help to identify ProductService Type.
When you create instance of cartService at that time use typeLiteral to create instance like
val cartService = injector.getInstance(Key.get(scalaguice.typeLiteral[CartService[Product]])
if you create instance like above you don't need to create module.
Create injector using default Module (i.e. useful if you have any other bindings in default Module.scala at application level)
val appBuilder = new GuiceApplicationBuilder()
val injector = Guice.createInjector(appBuilder.applicationModule())
and if you don't have any module you can skip passing module as argument and create injector without passing any module as well like
val injector = Guice.createInjector()

Override modifier is not allowed in Main however it is allowed when I extend App trait in scala

I am trying to create a Launcher program for my application and was going through the App trait and main method to decide which one to use. The only difference that I found between both is:
(1) Threaded code that references the object will block until static initialization is complete. However, because the entire execution of an object extending Application takes place during static initialization, concurrent code will always deadlock if it must synchronize with the enclosing object.
In my case I have a Launcher which basically initialize a Kafka loader object which keeps on running kafka poll. Below is my Launcher trait:
trait Launcher extends LazyLogging {
val config: Config
val actorSystem: ActorSystem
val sourceMessagesType = config.getString("app.source.dataType")
val targetTopic = config.getString("app.kafka.targetTopic")
val targetTopicPartitions =config.getInt("app.kafka.targetTopicPartitions")
val routingManager = HashedPartitionRoutingManager(targetTopic, targetTopicPartitions)
logger.info(s"Initializing MorpheusStreamDataLoader for $sourceMessagesType type")
sourceMessagesType.toLowerCase match {
case "json" => JSonDataLoader(config, routingManager)
case "avro" => AvroDataLoader(config, routingManager)
case _ => throw new IllegalArgumentException
s"Messages of type ${sourceMessagesType.toLowerCase} are not supported.\n"
}
}
Now to launch my application I was trying to find which is best to use, App or main method. However main method implementation doesn't work at all:
object ClientLauncher extends Launcher {
def main(args: Array[String]): Unit = {
override val config=ConfigFactory.load(args(0))
override val actorSystem: ActorSystem=ActorSystem("ClientActorSystem")
}
}
When I do this I get error on override modifier that override modifier is not allowed here. However if I use App trait it doesn't gives me any compile time error.
object ClientLauncher extends App with Launcher {
override val config=ConfigFactory.load(args(0))
override val actorSystem: ActorSystem=ActorSystem("ClientActorSystem")
}
The information that I got after reading couple of posts about App trait and main was that there is not difference other than the delayed initialization that happens when we use App trait. Why override doesn't works for main method and works for App? and what is the best way for me to launch my application?
You need to move them outside of the method so that they are object fields instead of local variables.
object ClientLauncher extends Launcher {
override val config=ConfigFactory.load()
override val actorSystem: ActorSystem=ActorSystem("ClientActorSystem")
def main(args: Array[String]): Unit = {
/*code*/
}
}
You won't be able to access command-line arguments or anything else local to main this way though.
If you need access to that and don't want to extend App, an alternative is to use a separate class to extend launcher.
class ClientLauncher(configFile: String) extends Launcher {
override val config=ConfigFactory.load(configFile)
override val actorSystem: ActorSystem=ActorSystem("ClientActorSystem")
}
object Main {
def main(args: Array[String]): Unit = {
new ClientLauncher(args(0))
}
}
Or to pass in those arguments as system properties instead of command line arguments.
object ClientLauncher extends Launcher {
override val config=ConfigFactory.load(sys.props("configFile"))
override val actorSystem: ActorSystem=ActorSystem("ClientActorSystem")
def main(args: Array[String]): Unit = {
/*code*/
}
}
and pass in the system property when you run your code
java -DconfigFile="config.conf" -jar myJar.jar

How to call an overriden method using reflection?

Consider we have the following:
class Base { def name = "Base" }
class Successor extends Base {
override def name = "Successor"
}
I have tried to do the following (took from How to call a superclass method using Java reflection):
import java.lang.invoke.{MethodHandles, MethodHandle, MethodType}
object TestApp {
def main(args: Array[String]) {
val a = new Successor;
val h1 = MethodHandles.lookup().findSpecial(classOf[Base],
"name",
MethodType.methodType(classOf[String]),
classOf[Successor]);
println(h1.invoke(a));
}
}
but I get a runtime exception:
java.lang.IllegalAccessException: no private access for invokespecial: class Successor, from TestApp$
I was told that it is possible that Java reflection may not work correctly for Scala. Is it true? Or I simply do something wrong?
Actually, you can NOT even do it in Java. Note, in the answer of "How to call a superclass method using Java reflection", it works because the Test extends the Base: public class Test extends Base {...}.
It looks like it is possible and Java reflection works for Scala as well, I just didn't read all answers for How to call a superclass method using Java reflection.
The following code works:
object TestApp {
def main(args: Array[String]) {
val a = new Successor;
val impl_lookup = classOf[MethodHandles.Lookup].getDeclaredField("IMPL_LOOKUP")
impl_lookup.setAccessible(true)
val lkp = impl_lookup.get(null).asInstanceOf[MethodHandles.Lookup];
val h1 = lkp.findSpecial(classOf[Base],
"name",
MethodType.methodType(classOf[String]),
classOf[Successor])
println(h1.invoke(a)) // prints "Base"
println(a.name) // prints "Successor"
}
}
Thanks to Jesse Glick for this solution.

Scala's delayedInit and Scallop

SLS 5.1 says "Delayed Initializaton. The initialization code of an object or class (but not a trait) that follows the superclass constructor invocation and the mixin-evaluation of the template’s base classes is passed to a special hook, which is inaccessible from user code. Normally, that hook simply executes the code that is passed to it. But templates inheriting the scala.DelayedInit trait can override the hook by re-implementing the delayedInit method, which is defined as follows:"
def delayedInit(body: => Unit)
The ScallopConf command-line parser extends DelayedInit and using it according to the docs generates the warning Selecting value apples from class Conf, which extends scala.DelayedInit, is likely to yield an uninitialized value.
How should the following simple example be rewritten so the warning is not generated?
import org.rogach.scallop._
class Conf(arguments: Seq[String]) extends ScallopConf(arguments) {
val help = opt[Boolean](name = "help", short = 'h', descr = "Help me please.")
}
object Gen {
def main(args: Array[String]) {
val conf: Conf = new Conf(args)
if (conf.help()) {
println(s"""Usage: Gen [--help]""")
sys.exit(-1)
}
println("Do some work here")
}
}
Making help a lazy val and calling conf.afterInit() before accessing conf.help should clear the compiler warnings.