I am using Scala with Play framework to create a webapplication. I have a class that is connecting to a Cassandra DB. I am using the constructor to connect to the database, but it doesn't work, in fact, I can't call any function function from the constructor. I'm new to Scala but from what I read on the tutorials on Scala it should work. Here is the code:
class Database
{
var cluster = Cluster.builder().addContactPoint(Play.application.configuration.getString("cassandra.node")).build()
var session = cluster.connect("acm")
}
I removed the rest of the class body for clarity.
These functions don't get called when I make an instance of the class and the variables will be left unnassigned when using them in another function. they work fine from a regular function. I also tested it with the logger, but nothing is written. So what is going on here?
scala> class A {
var x = 1
println(s"x = $x")
}
val a = new A
and I got the expected result
scala> x = 1
from what given in the context, I think it should work. If it doesn't, it should resides on some place else.
Related
I'm having some issues when trying to execute a class function inside a "dataframe.foreach" function. My custom class is persisting the data into a DynamoDB table.
What happens is that if I have the following code, it won't work and will raise a "Null Pointer Exception" that points to the line of code where the "writer.writeRow(r)" is executed:
object writeToDynamoDB extends App {
val df: DataFrame = ...
val writer: DynamoDBWriter = new DDBWriter(...)
df
.foreach(
r => writer.writeRow(r)
)
}
If I use the same code, but having the code inside a code block or an if clause, it will work:
object writeToDynamoDB extends App {
val df: DataFrame = ...
if(true) {
val writer: DynamoDBWriter = new DDBWriter(...)
df
.foreach(
r => writer.writeRow(r)
)
}
}
I guess it has something to do with the variable scope. Even in IntelliJ the color of the variable is purple + Italic in the first case and "regular" grey in the second case. I read about it, and we have the method, field and local scope in Scala, but I'm can't relate that with what I'm trying to do.
Some questions after this introduction:
Can anyone explain why does Scala and/or Spark have this behaviour?
The solution here is to put some code inside a function, code block
or a "fake" if clause as far as I know. Is there any possible issue regarding Spark properties (shuffles, etc)?
Is there any other way to do this type of operations?
Hope I was clear.
Thanks in advance.
Regards
As said above, your issue is caused by delayed initialization when using the App trait. Spark docs strongly discourage that:
Note that applications should define a main() method instead of extending scala.App. Subclasses of scala.App may not work correctly.
The reason can be found in the Javadocs of the App trait itself:
It should be noted that this trait is implemented using the DelayedInit functionality, which means that fields of the object will not have been initialized before the main method has been executed.
This basically means that writer is still uninitialized (so null) by the time the closure passed to foreach is created.
If you put respective code into a block, writer becomes a local variable and is initialized at the time when the block is evaluated. That way your closure will contain the correct value of writer. In this case it doesn't matter anymore when the code is evaluated, because everything get's evaluated together.
The correct and recommended solution is to use a standard main method for your Spark applications:
object writeToDynamoDB {
def main(args: Array[String]): Unit = {
val df: DataFrame = ...
val writer: DynamoDBWriter = new DDBWriter(...)
df.foreach(r => writer.writeRow(r))
}
}
I try to get names of all trait a class extends using getInterfaces which returns an array of trait's names. When I manually access each member of the array, the method getName returns simple names like this
trait A
trait B
class C() extends A, B
val c = C()
val arr = c.getClass.getInterfaces
arr(0).getName // : String = A
arr(1).getName // : String = B
However, when I use map function on arr. The resulting array contains a cryptic version of trait's names
arr.map(t => t.getName) // : Array[String] = Array(repl$.rs$line$1$A, repl$.rs$line$2$B)
The goal of this question is not about how to get the resulting array that contains simple names (for that purpose, I can just use arr.map(t => t.getSimpleName).) What I'm curious about is that why accessing array manually and using a map do not yield a compatible result. Am I wrong to think that both ways are equivalent?
I believe you run things in Scala REPL or Ammonite.
When you define:
trait A
trait B
class C() extends A, B
classes A, B and C aren't defined in top level of root package. REPL creates some isolated environment, compiles the code and loads the results into some inner "anonymous" namespace.
Except this is not true. Where this bytecode was created is reflected in class name. So apparently there was something similar (not necessarily identical) to
// repl$ suggest object
object repl {
// .rs sound like nested object(?)
object rs {
// $line sounds like nested class
class line { /* ... */ }
// $line$1 sounds like the first anonymous instance of line
new line { trait A }
// import from `above
// $line$2 sounds like the second anonymous instance of line
new line { trait B }
// import from above
//...
}
}
which was made because of how scoping works in REPL: new line creates a new scope with previous definitions seen and new added (possibly overshadowing some old definition). This could be achieved by creating a new piece of code as code of new anonymous class, compiling it, reading into classpath, instantiating and importing its content. Byt putting each new line into separate class REPL is able to compile and run things in steps, without waiting for you to tell it that the script is completed and closed.
When you are accessing class names with runtime reflection you are seeing the artifacts of how things are being evaluated. One path might go trough REPLs prettifiers which hide such things, while the other bypass them so you see the raw value as JVM sees it.
The problem is not with map rather with Array, especially its toString method (which is one among the many reasons for not using Array).
Actually, in this case it is even worse since the REPL does some weird things to try to pretty-print Arrays which in this case didn't work well (and, IMHO, just add to the confusion)
You can fix this problem calling mkString directly like:
val arr = c.getClass.getInterfaces
val result = arr.map(t => t.getName)
val text = result.mkString("[", ", ", "]")
println(text)
However, I would rather suggest just not using Array at all, instead convert it to a proper collection (e.g. List) as soon as possible like:
val interfaces = c.getClass.getInterfaces.toList
interfaces .map(t => t.getName)
Note: About the other reasons for not using Arrays
They are mutable.
Thet are invariant.
They are not part of the collections hierarchy thus you can't use them on generic methods (well, you actually can but that requires more tricks).
Their equals is by reference instead of by value.
Hello I would like to create a generic wrapper in scala in order to track the changes of the value of any type. I don't know/haven't found any other ways so far and I was thinking of creating a class and I've been trying to use the Dynamic but it has some limitations.
case class Wrapper[T](value: T) extends Dynamic {
private val valueClass = value.getClass
def applyDynamic(id: String)(parameters: Any*) = {
val objectParameters = parameters map (x => x.asInstanceOf[Object])
val parameterClasses = objectParameters map (_.getClass)
val method = valueClass.getMethod(id, parameterClasses:_*)
val res = method.invoke(value, objectParameters:_*)
// TODO: Logic that will eventually create some kind of event about the method invoked.
new Wrapper(res)
}
}
With this code I have trouble when invoking the plus("+") method on two Integers and I don't understand why. Isn't there a "+" method in the Int class? The error I am getting when I try addition with both a type of Wrapper/Int is:
var wrapped1 = Wrapper(1)
wrapped1 = wrapped1 + Wrapper[2] // or just 2
type mismatch;
found : Wrapper[Int]/Int
required: String
Why is it expecting a string?
If possible it would also be nice to be able to work with both the Wrapper[T] and the T methods seamlessly, e.g.
val a = Wrapper[Int](1)
val b = Wrapper[Int](2)
val c = 3
a + b // Wrapper[Int].+(Wrapper[Int])
a + c // Wrapper[Int].+(Int)
c + a // Int.+(Wrapper[Int])
Well, if youre trying to make a proxy which will get any changes of desired values you'l probably fail without agents(https://dzone.com/articles/java-agent-1) because it will force you make code modifications for bytecode that accepts final classes and primitives to accept your proxy instead of that and it would require more than intercepting changes of "just class" but also all classes of members and produce origin-of-value analysis and so on. It's by no way trivial problem.
Another approach is to produce diffs of case classes by comparing classes in certain points of execution and there's generic implementation like that, it uses derivation for computing diffs: https://github.com/ivan71kmayshan27/ShapelesDerivationExample I believe you can came with easier solution with magnolia. Actualy this one is unable to work for just classes unless you write your own macro and have some problems regarding ordered and unordered collections.
Is there any benefit to using partially applied functions vs injecting dependencies into a class? Both approaches as I understand them shown here:
class DB(conn: String) {
def get(sql: String): List[Any] = _
}
object DB {
def get(conn: String) (sql: String): List[Any] = _
}
object MyApp {
val conn = "jdbc:..."
val sql = "select * from employees"
val db1 = new DB(conn)
db1.get(sql)
val db2 = DB.get(conn) _
db2(sql)
}
Using partially-applied functions is somewhat simpler, but the conn is passed to the function each time, and could have a different conn each time it is called. The advantage of using a class is that it can perform one-off operations when it is created, such as validation or caching, and retain the results in the class for re-use.
For example the conn in this code is a String but this is presumably used to connect to a database of some sort. With the partially-applied function it must make this connection each time. With a class the connection can be made when the class is created and just re-used for each query. The class version can also prevent the class being created unless the conn is valid.
The class is usually used when the dependency is longer-lived or used by multiple functions. Partial application is more common when the dependency is shorter-lived, like during a single loop or callback. For example:
list.map(f(context))
def f(context: Context)(element: Int): Result = ???
It wouldn't really make sense to create a class just to hold f. On the other hand, if you have 5 functions that all take context, you should probably just put those into a class. In your example, get is unlikely to be the only thing that requires the conn, so a class makes more sense.
I am trying to pass generalized class as parameter . If I give Case class and values then it is working fine . But , would like to make it generic.
class DB[T] {
lazy val ctx = new OracleJdbcContext(SnakeCase, "ctx")
import ctx._
def insert(input: T) = {
val q = quote {
query[T].insert(lift(input))
}
ctx.run(q)
}
}
I am getting errors saying::
"Can't find an implicit SchemaMeta for type T
Can't find Encoder for type 'T'. Note that Encoders are invariant"
But, If I give actual class name then it is going well.
case class Location(street:String,pinCode:Int)
class DB {
lazy val ctx = new OracleJdbcContext(SnakeCase, "ctx")
import ctx._
val q = quote {
query[Location].insert(Location("2ndcross",500001))
}
ctx.run(q)
}
You need to have SchemaMeta[T] in scope to be able to execute queries using this type. Dummy solution would be to demand it as parameter constraint (and so implicit class parameter) like this
class DB[T: SchemaMeta]
but this wouldn't work, because it's ctx who provides those instances.
I believe that you will need to follow examples shown here: https://getquill.io/#contexts-dependent-contexts
But even then what you want may not be achievable.
Imporant thing to understand when working with quill is that almost everything there is based on macros and if you abstract things away then there is not enough information for these macros to act. So you either need to duplicate code requiring macros or wrap the code that is meant to be generic in your own macro.