This question seems pretty obvious I know, but i've tried everything that is written on the documentation and I cant't mock a single method on any classes.
For this test, I am using scalaMock 3 for Scala 2.10 and ScalaTest 2
DateServiceTest.scala
#RunWith(classOf[JUnitRunner])
class DateServiceTest extends FunSuite with MockFactory{
val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
val sc = new SparkContext(conf)
implicit val sqlc = new SQLContext(sc)
val m = mock[DateService]
(m.getDate _).expects().returning(Some(new DateTime)) // Error here
}
DateService.scala
class DateService extends Serializable {
def getDate(implicit sqlc: SQLContext): Option[DateTime] = {
Some(new DateTime(loadDateFromDatabase(sqlc)))
}
}
This seems pretty simple for me, but the expectation is Throwing me this error
type mismatch; found : Option[org.joda.time.DateTime] required: ? ⇒ ?
Am I doing something wrong here? Is there another way to set the expectations of a method ?
Your getDate method expects a SQLContext - try adding a wildcard (*) or pass the particular sqlc:
(m.getDate _).expects(*).returning(Some(new DateTime))
// Or, alternatively
(m.getDate _).expects(sqlc).returning(Some(new DateTime))
Related
I am trying to use scala TypeClass on Spark Types, here is a small code snippet I wrote.
trait ThresholdMethods[A] {
def applyThreshold(): Boolean
}
object ThresholdMethodsInstances {
def thresholdMatcher[A](v:A)(implicit threshold: ThresholdMethods[A]):Boolean =
threshold.applyThreshold()
implicit val exactMatchThresholdStringType = new ThresholdMethods[StringType] {
def applyThreshold(): Boolean = {
print("string")
true
}
}
implicit val exactMatchThresholdNumericType = new ThresholdMethods[IntegerType] {
def applyThreshold(): Boolean = {
print("numeric")
true
}
}
}
object Main{
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().appName("ZenDataValidationLib").config("spark.some.config.option", "some-value")
.master("local").getOrCreate()
import spark.sqlContext.implicits._
val df1 = Seq(
("2016-04-02", "14", "NULL", 9874, 880, "xyz"), ("2016-04-30", "14", "FR", 9875, 13,"xyz"), ("2017-06-10", "15", "PQR", 9867, 57721,"xyz")
).toDF("WEEK", "DIM1", "DIM2","T1","T2","T3")
import ThresholdMethodsInstances._
println(df1.schema("T1").dataType)
ThresholdMethodsInstances.thresholdMatcher(df1.schema("T1").dataType)
}
}
When I run this on my local intellij, following error is thrown
Error:(46, 51) could not find implicit value for parameter threshold: com.amazon.zen.datavalidation.activity.com.amazon.zen.datavalidation.activity.ThresholdMethods[org.apache.spark.sql.types.DataType]
ThresholdMethodsInstances.thresholdMatcher(df1.schema("T1").dataType)
Error:(46, 51) not enough arguments for method thresholdMatcher: (implicit threshold: com.amazon.zen.datavalidation.activity.com.amazon.zen.datavalidation.activity.ThresholdMethods[org.apache.spark.sql.types.DataType])Boolean.
Unspecified value parameter threshold.
ThresholdMethodsInstances.thresholdMatcher(df1.schema("T1").dataType)
I also tried the same things using String and Int and it worked perfectly fine. Can someone help me in doing this on SparkTypes ?
Please note that StructField.dataType returns DataType, not a specific subclass and your code defines implicit ThresholdMethods only for IntegerType and StringType.
Because implicit resolution happens at compilation time there is not enough information for compile to determine the right instance, which would make the types match.
This would work if you'd (obviously you don't want that):
ThresholdMethodsInstances.thresholdMatcher(
df1.schema("T1").dataType.asInstanceOf[IntegerType]
)
but in practice, you'll need something more explicit like pattern matching, not implicit argument.
You can also consider switching to strongly typed Dataset, which be a better choice here.
I don't understand a behavior of spark.
I create an udf which returns an Integer like below
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}
object Show {
def main(args: Array[String]): Unit = {
val (sc,sqlContext) = iniSparkConf("test")
val testInt_udf = sqlContext.udf.register("testInt_udf", testInt _)
}
def iniSparkConf(appName: String): (SparkContext, SQLContext) = {
val conf = new SparkConf().setAppName(appName)//.setExecutorEnv("spark.ui.port", "4046")
val sc = new SparkContext(conf)
sc.setLogLevel("WARN")
val sqlContext = new SQLContext(sc)
(sc, sqlContext)
}
def testInt() : Int= {
return 2
}
}
I work perfectly but if I change the return type of method test from Int to String
val testString_udf = sqlContext.udf.register("testString_udf", testString _)
def testString() : String = {
return "myString"
}
I get the following error
Error:(34, 43) No TypeTag available for String
val testString_udf = sqlContext.udf.register("testString_udf", testString _)
Error:(34, 43) not enough arguments for method register: (implicit evidence$1: reflect.runtime.universe.TypeTag[String])org.apache.spark.sql.UserDefinedFunction.
Unspecified value parameter evidence$1.
val testString_udf = sqlContext.udf.register("testString_udf", testString _)
here are my embedded jars:
datanucleus-api-jdo-3.2.6
datanucleus-core-3.2.10
datanucleus-rdbms-3.2.9
spark-1.6.1-yarn-shuffle
spark-assembly-1.6.1-hadoop2.6.0
spark-examples-1.6.1-hadoop2.6.0
I am a little bit lost... Do you have any idea?
Since I can't reproduce the issue copy-pasting just your example code into a new file, I bet that in your real code String is actually shadowed by something else. To verify this theory you can try to change you signature to
def testString() : scala.Predef.String = {
return "myString"
}
or
def testString() : java.lang.String = {
return "myString"
}
If this one compiles, search for "String" to see how you shadowed the standard type. If you use IntelliJ Idea, you can try to use "Ctrl+B" (GoTo) to find it out. The most obvious candidate is that you used String as a name of generic type parameter but there might be some other choices.
I'm trying to pass a bunch of classes to the SparkConf.registerKryoClasses method, which has the following signature:
registerKryoClasses: Array[Class[_]]) => SparkConf
Since I may or may not have classes that need to be registered, I'm wrapping it in an Option and tried this (in a simplified version):
class SomeClass(val app: String, val classes: Option[Array[Class[_]]]) {
val conf = classes match {
case Some(cs) ⇒ new SparkConf()
.setAppName(app)
.registerKryoClasses(cs)
case None ⇒ new SparkConf()
.setAppName(app)
}
// more stuff
}
IntelliJ gives me that there is a type mismatch on cs and then lists the expected and actual types. They are the same.
What am I missing here?
The following works fine for me,
def someClass(app: String, classes: Option[Array[Class[Any]]]): SparkConf = {
val conf = classes match {
case Some(cs) ⇒ new SparkConf()
.setAppName(app)
.registerKryoClasses(cs)
case None ⇒ new SparkConf()
.setAppName(app)
// more stuff
}
conf
}
val conf = someClass("lol", Some(Array(classOf[String], classOf[Int])))
//org.apache.spark.SparkConf = org.apache.spark.SparkConf#685d92cf
I am new to Slick and having a really hard time getting mapping of java.sql.date/time/timestamp mapped into jodatime.
trait ColumnTypeMappings {
val profile: JdbcProfile
import profile.api._
val localTimeFormatter = DateTimeFormat.forPattern("HH:mm:ss")
val javaTimeFormatter = new SimpleDateFormat("HH:mm:ss")
implicit val myDateColumnType = MappedColumnType.base[LocalDate, Date](
ld => new java.sql.Date(ld.toDateTimeAtStartOfDay(DateTimeZone.UTC).getMillis),
d => new LocalDateTime(d.getTime).toLocalDate
)
implicit val myTimeColumnType = MappedColumnType.base[LocalTime, Time](
lt => new java.sql.Time(javaTimeFormatter.parse(lt.toString(localTimeFormatter)).getTime),
t => new LocalTime(t.getTime)
)
implicit val myTimestampColumnType = MappedColumnType.base[DateTime, Timestamp](
dt => new java.sql.Timestamp(dt.getMillis),
ts => new DateTime(ts.getTime, DateTimeZone.UTC)
)
}
In the auto generated Tables.scala I include the mapping like this:
trait Tables extends ColumnTypeMappings {
val profile: slick.driver.JdbcDriver
import profile.api._
import scala.language.implicitConversions
// + rest of the auto generated code by slick codegen
}
And to wrap it all up I use this like this:
object TestTables extends Tables {
val profile = slick.driver.MySQLDriver
}
import Tables._
import profile.api._
val db = Database.forURL("url", "user", "password", driver = "com.mysql.jdbc.Driver")
val q = Company.filter(_.companyid === 1).map(._name)
val action = q.result
val future = db.run(action)
val result = Await.result(future, Duration.Inf)
I get an NullPointerException on: implicit val myDateColumnType.... when running this. I've verified that this last block of code works if I remove the mapping.
Try changing implicit val to implicit def in your definitions of the MappedColumnTypes. The reason why is related to the answer given by Maksym Chernenko to this question. Generally, the JdbcProfile driver (that defines api.MappedColumnType) has not been injected yet, and:
that causes NPE. You can either make your "mapper" val lazy, or change it
from val to def (as shown below)
implicit def myDateColumnType = MappedColumnType.base[LocalDate, Date](
ld => new java.sql.Date(ld.toDateTimeAtStartOfDay(DateTimeZone.UTC).getMillis),
d => new LocalDateTime(d.getTime).toLocalDate
)
implicit def myTimeColumnType = MappedColumnType.base[LocalTime, Time](
lt => new java.sql.Time(javaTimeFormatter.parse(lt.toString(localTimeFormatter)).getTime),
t => new LocalTime(t.getTime)
)
implicit def myTimestampColumnType = MappedColumnType.base[DateTime, Timestamp](
dt => new java.sql.Timestamp(dt.getMillis),
ts => new DateTime(ts.getTime, DateTimeZone.UTC)
)
So i think the issue may be that you are extending ColumnTypeMappings in your Tables.scala. The documentation doesn't make it clear but I think the auto generated code relating to the database should not be touched, as this is used by slick to map the rows in the DB, and then extend TestTables by ColumnTypeMappings to do the implicit conversion when you get the result back from the database.
I haven't particularly delved into slick 3.x yet so I may be wrong, but I think that makes sense.
Edit: No, i was wrong :(. Apologies
I have an conf/application.conf setting like
mongodb.replicaSetSeeds = ["bobk-mbp.local:27017","bobk-mbp.local:27018"]
I'm pulling it out in my code like (the actual extraction is a little different, but this is the gist of it)
val replicaSetSeeds = Play.current.configuration.getStringList("mongodb.replicaSetSeeds")
val listOfString: List[String] = replicaSetSeeds.getOrElse(List("localhost"))
but the compiler hates me
type mismatch; found : Object required: List[String]
The signature of getStringList is
def getStringList(path: String): Option[java.util.List[String]]
How do I handle the None case here or is my problem List[String] is not the same as List[java.util.String]?
Give this a shot:
import collection.JavaConversions._
val optList:Option[List[String]] = Play.current.configuration.getStringList("mongodb.replicaSetSeeds").map(_.toList)
val list = optList.getOrElse(List("localhost"))
There are multiple things going on here. First, you need to import the JavaConversions implicits because what's being returned is an Option[java.util.List[String]] and we want that to be a scala List instead. By doing the map(_.toList), I'm forcing the implicit conversion to kick in and get me an Option[List[String]] and from there things are pretty straight forward.
In play 2.5, you need to use dependency injection, the following works well for me:
1) in your class inject Configuration
class Application #Inject()(
configuration: play.api.Configuration
) ...
2) in your method
import scala.collection.JavaConversions._
val optlist = configuration.getStringList("mongodb.replicaSetSeeds").map{_.toList}
val list = optList.getOrElse(List("localhost"))