I have the following test case:
test("check foo") {
val conf = new SparkConf()
val sc = new SparkContext(conf)
val sqlc = new SQLContext(sc)
val res = foo("A", "B")
assert(true)
}
Which checks the following method:
def foo(arg1: String, arg2: String) (implicit sqlContext: SQLContext) : Seq[String] = {
//some other code
}
When running the tests I get the following issue:
Error:(65, 42) could not find implicit value for parameter sqlContext: org.apache.spark.sql.SQLContext
val res = foo("A", "B")
How can I share the SqlContext instance I create in the test method with foo?
Put implicit in front of val sqlc:
implicit val sqlc = new SQLContext(sc)
Related
Let's say I have a Scala object:
object SomeObject {
private def someMethod(msg: String): Unit = println(msg)
}
I can invoke someMethod with the following code:
import scala.reflect.runtime.{universe => ru}
import scala.reflect.ClassTag
def invokeObjectPrivateMethod[R](methodName: String, args: AnyRef*): R = {
val rm = ru.runtimeMirror(getClass.getClassLoader)
val instanceMirror = rm.reflect(SomeObject)
val methodSymbol = ru.typeOf[SomeObject.type].decl(ru.TermName(methodName)).asMethod
val method = instanceMirror.reflectMethod(methodSymbol)
method(args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod("someMethod", "it works")
But above I've hardcoded SomeObject into the function. What I'd really like is to pass an arbitrary object name/classtag/whatever so I can invoke a private function generically in ANY object.
These attempts have NOT worked:
// With explicit ClassTag parameter
def invokeObjectPrivateMethod2[R](classTag: ClassTag[_], methodName: String, args: AnyRef*): R = {
val rm = ru.runtimeMirror(getClass.getClassLoader)
val instanceMirror = rm.reflect(classTag)
val methodSymbol = ru.typeOf[classTag.type].decl(ru.TermName(methodName)).asMethod
val method = instanceMirror.reflectMethod(methodSymbol)
method(args: _*).asInstanceOf[R]
}
// This fails at runtime with: `scala.ScalaReflectionException: <none> is not a method`
invokeObjectPrivateMethod2[Unit](ClassTag(SomeObject.getClass), "someMethod", "it doesn't work")
// With implicit ClassTag/TypeTag
def invokeObjectPrivateMethod3[T: ClassTag, S: ru.TypeTag, R](methodName: String, args: AnyRef*)(implicit ct: ClassTag[T]): R = {
val rm = ru.runtimeMirror(getClass.getClassLoader)
val instanceMirror = rm.reflect(ct)
val methodSymbol = ru.typeOf[S].decl(ru.TermName(methodName)).asMethod
val method = instanceMirror.reflectMethod(methodSymbol)
method(args: _*).asInstanceOf[R]
}
// This fails at compile time: `not found: type SomeObject`
invokeObjectPrivateMethod3[SomeObject, SomeObject, Unit]("someMethod", "it also doesn't work")
I'm somewhat out of my depth, so what I'm trying with the 3rd option might not even make sense.
Thanks!
Try the following if you have String of object name
def invokeObjectPrivateMethod[R](objectName: String, methodName: String, args: AnyRef*): R = {
val rm = scala.reflect.runtime.currentMirror
val moduleSymbol = rm.staticModule(objectName)
val classSymbol = moduleSymbol.moduleClass.asClass
val moduleMirror = rm.reflectModule(moduleSymbol)
val objectInstance = moduleMirror.instance
val objectType = classSymbol.toType
val methodSymbol = objectType.decl(ru.TermName(methodName)).asMethod
val instanceMirror = rm.reflect(objectInstance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
methodMirror(args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod("com.example.App.SomeObject", "someMethod", "it works") //it works
or if you have object ClassTag
def invokeObjectPrivateMethod[R](classTag: ClassTag[_], methodName: String, args: AnyRef*): R = {
val rm = scala.reflect.runtime.currentMirror
val clazz = classTag.runtimeClass
val classSymbol = rm.classSymbol(clazz)
val moduleSymbol = classSymbol.owner.info.decl(classSymbol.name.toTermName).asModule //see (*)
val moduleMirror = rm.reflectModule(moduleSymbol)
val objectInstance = moduleMirror.instance
val objectType = classSymbol.toType
val methodSymbol = objectType.decl(ru.TermName(methodName)).asMethod
val instanceMirror = rm.reflect(objectInstance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
methodMirror(args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod(classTag[SomeObject.type], "someMethod", "it works") //it works
or if you have object TypeTag
def invokeObjectPrivateMethod[R](typeTag: TypeTag[_], methodName: String, args: AnyRef*): R = {
val rm = scala.reflect.runtime.currentMirror
val objectType = typeTag.tpe
val clazz = rm.runtimeClass(objectType) // see (**)
val classSymbol = rm.classSymbol(clazz)
val moduleSymbol = classSymbol.owner.info.decl(classSymbol.name.toTermName).asModule // see (*)
val moduleMirror = rm.reflectModule(moduleSymbol)
val objectInstance = moduleMirror.instance
val methodSymbol = objectType.decl(ru.TermName(methodName)).asMethod
val instanceMirror = rm.reflect(objectInstance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
methodMirror(args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod(typeTag[SomeObject.type], "someMethod", "it works") //it works
or if you have object Class
def invokeObjectPrivateMethod[R](clazz: Class[_], methodName: String, args: AnyRef*): R = {
val rm = scala.reflect.runtime.currentMirror
val classSymbol = rm.classSymbol(clazz)
val moduleSymbol = classSymbol.owner.info.decl(classSymbol.name.toTermName).asModule //see (*)
val moduleMirror = rm.reflectModule(moduleSymbol)
val objectInstance = moduleMirror.instance
val objectType = classSymbol.toType
val methodSymbol = objectType.decl(ru.TermName(methodName)).asMethod
val instanceMirror = rm.reflect(objectInstance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
methodMirror(args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod(SomeObject.getClass, "someMethod", "it works") //it works
or if you have object Type
def invokeObjectPrivateMethod[R](typ: Type, methodName: String, args: AnyRef*): R = {
val rm = scala.reflect.runtime.currentMirror
val classSymbol = typ.typeSymbol.asClass
val moduleSymbol = classSymbol.owner.info.decl(classSymbol.name.toTermName).asModule //see (*)
val moduleMirror = rm.reflectModule(moduleSymbol)
val objectInstance = moduleMirror.instance
val methodSymbol = typ.decl(ru.TermName(methodName)).asMethod
val instanceMirror = rm.reflect(objectInstance)
val methodMirror = instanceMirror.reflectMethod(methodSymbol)
methodMirror(args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod(typeOf[SomeObject.type], "someMethod", "it works") //it works
(*) Get the module symbol, given I have the module class, scala macro
(**) How to get ClassTag form TypeTag, or both at same time?
I assume that the method didn't have overloaded versions, otherwise you should work with typ.decl(...).alternatives.find(...).get.asMethod or typ.decl(...).alternatives.head.asMethod.
Using Java reflection
def invokeObjectPrivateMethod[R](packageName: String, objectName: String, methodName: String, args: AnyRef*): R = {
val javaClassName = s"$packageName.${objectName.replace('.', '$')}$$"
val clazz = Class.forName(javaClassName)
val method = clazz.getDeclaredMethods.find(_.getName == methodName).get
method.setAccessible(true)
val field = clazz.getField("MODULE$")
val instance = field.get(null) // null because field is static
method.invoke(instance, args: _*).asInstanceOf[R]
}
invokeObjectPrivateMethod("com.example", "App.SomeObject", "someMethod", "it works") //it works
Using Java method handles
def invokeObjectPrivateMethod[R](packageName: String, objectName: String, methodName: String, args: AnyRef*): R = {
val javaClassName = s"$packageName.${objectName.replace('.', '$')}$$"
val clazz = Class.forName(javaClassName)
val lookup = MethodHandles.lookup
val field = clazz.getField("MODULE$")
val fieldMethodHandle = lookup.unreflectGetter(field)
val instance = fieldMethodHandle.invokeWithArguments()
val method = clazz.getDeclaredMethods.find(_.getName == methodName).get
method.setAccessible(true)
val methodHandle = lookup.unreflect(method)
methodHandle.invokeWithArguments(instance +: args : _*).asInstanceOf[R]
}
invokeObjectPrivateMethod("com.example", "App.SomeObject", "someMethod", "it works") //it works
I'm creating shapeless HMap using the code like below from scala-exercises.
import shapeless.HMap
class BiMapIS[K, V]
implicit val intToString = new BiMapIS[Int, String]
implicit val stringToInt = new BiMapIS[String, Int]
val hm = HMap[BiMapIS](23 -> "foo", "bar" -> 13)
I would like to create HMap from variable arguments as below (I'm having long list of arguments so just checking whether I can simplify the code littlebit) -
import shapeless.{HMap, HNil}
import java.util.{List => JList}
val entities: JList[(_, _)] = ???
class BiMapIS[K, V]
implicit val intToString = new BiMapIS[Int, String]
implicit val stringToInt = new BiMapIS[String, Int]
import collection.JavaConverters._
val entitiesSeq = entities.asScala.toList
val hm = HMap[BiMapIS](entitiesSeq:_*)
Is there any way I can create HMap from variable args?
I'm using shapless 2.33 with scala 2.12 https://mvnrepository.com/artifact/com.chuusai/shapeless_2.12/2.3.3
Try
val entitiesSeq = entities.asScala.toMap[Any, Any]
val hm = new HMap[BiMapIS](entitiesSeq)
I am new to scala and just understanding how can I transform via Map call. The foo function is not getting called. What is that I am missing ?
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object Expt {
def main(args: Array[String]): Unit = {
var conf = new SparkConf().setAppName("Test").setMaster("local[*]")
val sc = new SparkContext(conf)
val a1 = sc.parallelize(List((1,"one"),(2,"two"),(3,"three"))
val a3 = a1.map(x => Expt.foo(x))
}
def foo(x: (Int,String)) : (Int,String) = {
println(x)
x
}
}
You don't execute any action so map is never evaluated. Also println in map usually won't have visible effect.
trigger computation using collect or foreach (do some action)
scala> val a1 = sc.parallelize(List((1,"one"),(2,"two"),(3,"three")))
a1: org.apache.spark.rdd.RDD[(Int, String)] = ParallelCollectionRDD[6] at parallelize at <console>:24
scala> val a3 = a1.map(x => foo(x))
a3: org.apache.spark.rdd.RDD[(Int, String)] = MapPartitionsRDD[7] at map at <console>:28
scala> a3.collect
(1,one)
(2,two)
(3,three)
res3: Array[(Int, String)] = Array((1,one), (2,two), (3,three))
scala> a3.foreach(_ => ())
(1,one)
(3,three)
(2,two)
Is it possible to 'invert' a Scala Future?
Sometimes the result of a Future being a Success means an error. In that case it would be nice to flip a Future, i.e. call a function that returns a Future, which succeeds with a specified value if the original Future fails and fails with a specified Error in case the original Future succeeds.
def flip[T](original: Future[T])(value: => T)(error: Throwable): Future[T] = ???
def craziness[A](future: Future[A])(default : => A)(error: Throwable)(implicit ec: ExecutionContext): Future[A] = {
val p = Promise[A]()
import scala.util.{Success, Failure, Try}
future.onComplete {
case _: Success[_] => p.failure(error)
case _: Failure[_] => p.complete(Try(default))
}
p.future
}
Here's a repl session showing it work:
scala> val f1 = craziness[String](Future("hello!"))("my crazy default")(new Throwable("boom"))
f1: scala.concurrent.Future[String] = scala.concurrent.impl.Promise$DefaultPromise#4d154ccd
scala> f1 onComplete { println }
Failure(java.lang.Throwable: boom)
scala> val f2 = craziness[String](Future(throw new Exception("boom!")))("my crazy default")(new Throwable("boom"))
f2: scala.concurrent.Future[String] = scala.concurrent.impl.Promise$DefaultPromise#1890516e
scala> f2 onComplete { println }
Success(my crazy default)
EDIT:
for completeness, def craziness[A](future: Future[A]) should probably be def craziness[A](future: => Future[A])
I think you are after recover and recoverWith constructs. Here's a quick REPL session to show its usage.
$ scala
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_45).
Type in expressions to have them evaluated.
Type :help for more information.
scala> import scala.concurrent.Future
import scala.concurrent.Future
scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global
scala> val failFuture = Future(sys.error("BOOM"))
failFuture: scala.concurrent.Future[Nothing] = scala.concurrent.impl.Promise$DefaultPromise#6e06451e
scala> val defaultValue = 100
defaultValue: Int = 100
scala> val futureRecoveredWithDefaultFuture = failFuture.recoverWith { case e: RuntimeException => Future.successful(defaultValue) }
futureRecoveredWithDefaultFuture: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise#130161f7
scala> val futureRecoveredWithDefaultValue = failFuture.recover { case e: RuntimeException => defaultValue }
futureRecoveredWithDefaultValue: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise#3b69e7d1
Checking if this really works:
scala> import scala.concurrent.duration._
import scala.concurrent.duration._
scala> import scala.concurrent.Await
import scala.concurrent.Await
scala> val res1 = Await.result(futureRecoveredWithDefaultFuture, 1.second)
res1: Int = 100
scala> val res2 = Await.result(futureRecoveredWithDefaultValue, 1.second)
res2: Int = 100
In Scala 2.12 you'll be able to use transform and transformWith to make this trivial.
But until then this should get you there:
implicit class InvertFuture[T](val fut: Future[T]) extends AnyVal {
def flip(recover: Throwable => T)(fail: T => Throwable)(implicit ec: ExecutionContext): Future[T] =
fut.recover({ case t => recover(t) }).map(t => throw fail(t))
}
// And usage:
scala> Future(1).flip(_ => 2)(_ => throw new IllegalStateException("ohnoes!")) onComplete println
Failure(java.lang.IllegalStateException: ohnoes!)
If I have a Shapeless HMap[MappingA] (with implicits properly defined for the type MappingA[K, V]), can I type-safely transform/map it to a HMap[MappingB].
class MappingA[K, V]
implicit val intToString = new MappingA[Int, String]
implicit val stringToInt = new MappingA[String, Int]
class MappingB[K, V]
implicit val longToString = new MappingA[Long, String]
implicit val stringToLong = new MappingA[String, Long]
val hm1 = HMap[MappingA](1 -> "one", "two" -> 2)
// How to...
val hm2: HMap[MappingB] = ??? // transform/map hm1
// expected for hm2 in this basic example
// HMap[MappingB](1L -> "one", "two" -> 2L)