I'm curious why passing "context" dependencies is so often done implicitly in Scala. I'm looking at the fs2-kafka library, and as usual, deserializers are passed implicitly. I'm having a hard time seeing the advantages of this, it seems like it just obfuscates the code by hiding dependencies. Does anyone know what the upsides are of passing parameters in this way?
Edit: To be clear, I'm not asking whether this is a good practice; that's subjective. I'm wondering what the reasoning is. We wouldn't pass most dependencies implicitly just to avoid the inconvenience of having to pass them in explicitly, so why is it so often done with context dependencies like deserializers?
Deserializers are usually typeclasses, and the way that Scala implements typeclasses is via implict arguments. Doing this manually is going to be a lot more code and require the programmer to do type matching that the compiler can do itself with a typeclass.
A better example of a simple dependency would be ExecutionContext in scala.util.Future methods, and in this case it simplifies the calling code and makes it easier to provide a different context within a given scope.
Related
I have read various old StackOverflow discussions on this general topic but there is still one part of the puzzle which appears, to me at least, to be missing.
It is simply this: what is the actual mechanism by which the anonymous function is serialized? And, where could we find its source code?
Or is it all just magic?
Other relevant SO articles (the third of these itself points to some useful articles outside StockOverflow):
Serialization of Scala Functions
Why Scala can serialize...
How to serialize functions in Scala
I'm going to answer my own question with what, I believe is the correct answer. The reason I'm doing it this way is that it seems to me that this aspect of serialization is never explained and it does appear to work just by magic. I essentially confirmed (to my satisfaction) the answer as part of the research I was doing to ensure that my question above was indeed appropriate.
But the main reason I'm offering my own answer is that I invite knowledgeable users either to agree with it, to correct it, to expand upon it, or to destroy it. Here goes...
It's all magic. No, I'm just kidding. But essentially the mechanism, once Scala has taken the step of representing the anonymous function as a Class, is entirely provided for by Java. In addition, we, the programmer, need to ensure that an anonymous function is as much pure code as possible: no references to any objects that might not be serializable. The secret sauce is to be found in the Java class: ObjectStreamClass. Which, in turn, is invoked by the Java serialization classes: ObjectInputStream and ObjectOutputStream.
Essentially the serialized bytes contain the full pathname of the class, its serialVersionUID, and whatever other relevant information is necessary. When deserializing, the system will simply look up the class in the appropriate classpath and return a reference to it. This obviously assumes that the deserializing system has the class in its classpath. The mechanism for that is a little beyond the scope of my research but it's clear that in a system like Spark, it should be easy to arrange.
No (additional) compilation/decompilation of byte code is necessary as the classLoader has everything necessary. I'm slightly surprised to find the ObjectStreamClass in java.io rather than in the reflection package, but I suppose there's an argument for it being there, given the tight coupling with ObjectInputStream and ObjectOutputStream.
One thing to keep in mind is that while we think in terms of serializing/deserializing objects, rather than classes, what we are dealing with here is an object of type Class.
One more thing to note is that in Scala 2.12, anonymous functions are now implemented differently: as Java8 lambdas. This has broken the mechanism described above in a rather serious way. So serious, that Spark is currently having trouble supporting Scala 2.12. The holdup appears to be this issue: SPARK-14540.
I have a very simple question. This is not only true with spray-json but I have read similar claims with argonaut and circe. So please enlighten me.
In spray-json, I have come across the statement saying There is no reflection involved. I understand for type class based approach, if the user provides JsonFormat then all is well. But is this claim also true when it comes to using DefaultJsonProtocol?
Because when we you look at this, you can see the usage of clazz.getMethods, clazz.getDeclaredFields, etc. Isn't this the usage of reflection? Though of course thanks to object#apply that we do not need to worry about setting unlike in Java world using reflection. But at least for reading the field names, I do not understand on how reflection can be overlooked.
I'm not very familiar with spray-json, so I won't defend its claims about reflection, which definitely seem to be at odds with the parts of ProductFormats you point to.
I do know more about circe and Argonaut and argonaut-shapeless and Play JSON, all of which do use a kind of reflection to derive codecs for case classes and other user-defined types. The important point is that these libraries don't use runtime reflection—they determine the field names and other information they need at compile time through Scala's macro system.
Generally when people talk about "reflection" in the context of Java or Scala, they mean runtime reflection, but macros also support a kind of reflection, so when I personally talk about how derivation works in these libraries, I try to be careful to specify that there's no runtime reflection involved.
You can argue that compile-time reflection (or metaprogramming, or whatever you want to call it) is much less bad than runtime reflection. It may make your code more complex, and it's very easy to abuse, but it doesn't introduce the same kinds of fragility as runtime reflection, and it doesn't undermine your ability to reason about your code in the same ways that runtime reflection does. If you understand what the macro does (which is a big if), you'll never be surprised at runtime.
Types are fundamentally about rejecting bad potential programs before you run them, and introspection on types at runtime muddles this all up (as Erik Osheim says, "If you meet a Type in the Runtime, kill it"). On the other hand, introspection on types at compile-time is exactly what compilers do, and macros just give you as the programmer a clean way of getting involved in that process (or at least relatively clean, compared to writing compiler plugins, etc.).
There may also be performance benefits to avoiding runtime reflection, but for me personally that's generally a secondary concern—I hate runtime reflection because I've wasted too much of my life debugging horrible Java code that uses horrible Java libraries that depend heavily on runtime reflection—not because runtime reflection might make my programs marginally slower.
That's all a very long-winded way to say that you should read "there is no reflection involved" in this context as "there is no runtime reflection involved" (and even then you shouldn't take the author at their word, I guess, given all that getMethods stuff in spray-json).
I am learning Scala right in these days. I have a slight familiarity with Haskell, although I cannot claim to know it well.
Parenthetical remark for those who are not familiar with Haskell
One trait that I like in Haskell is that not only functions are first-class citizens, but side effects (let me call them actions) are. An action that, when executed, will endow you with a value of type a, belongs to a specific type IO a. You can pass these actions around pretty much like any other value, and combine them in interesting ways.
In fact, combining the side effects is the only way in Haskell to do something with them, as you cannot execute them. Rather, the program that will be executed, is the combined action which is returned by your main function. This is a neat trick that allows functions to be pure, while letting your program actually do something other than consuming power.
The main advantage of this approach is that the compiler is aware of the parts of the code where you perform side effects, so it can help you catch errors with them.
Actual question
Is there some way in Scala to have the compiler type check side effects for you, so that - for instance - you are guaranteed not to execute side effects inside a certain function?
No, this is not possible in principle in Scala, as the language does not enforce referential transparency -- the language semantics are oblivious to side effects. Your compiler will not track and enforce freedom from side effects for you.
You will be able to use the type system to tag some actions as being of IO type however, and with programmer discipline, get some of the compiler support, but without the compiler proof.
The ability to enforce referential transparency this is pretty much incompatible with Scala's goal of having a class/object system that is interoperable with Java.
Java code can be impure in arbitrary ways (and may not be available for analysis when the Scala compiler runs) so the Scala compiler would have to assume all foreign code is impure (assigning them an IO type). To implement pure Scala code with calls to Java, you would have to wrap the calls in something equivalent to unsafePerformIO. This adds boilerplate and makes the interoperability much less pleasant, but it gets worse.
Having to assume that all Java code is in IO unless the programmer promises otherwise would pretty much kill inheriting from Java classes. All the inherited methods would have to be assumed to be in the IO type; this would even be true of interfaces, since the Scala compiler would have to assume the existence of an impure implementation somewhere out there in Java-land. So you could never derive a Scala class with any non-IO methods from a Java class or interface.
Even worse, even for classes defined in Scala, there could theoretically be an untracked subclass defined in Java with impure methods, whose instances might be passed back in to Scala as instances of the parent class. So unless the Scala compiler can prove that a given object could not possibly be an instance of a class defined by Java code, it must assume that any method call on that object might call code that was compiled by the Java compiler without respecting the laws of what functions returning results not in IO can do. This would force almost everything to be in IO. But putting everything in IO is exactly equivalent to putting nothing in IO and just not tracking side effects!
So ultimately, Scala encourages you to write pure code, but it makes no attempt to enforce that you do so. As far as the compiler is concerned, any call to anything can have side effects.
In limited sense it is very easy to write out and ref classes on your own, but my question is not how to do it -- but are there some features (or classes) ready to use?
The closest thing I found is Reference trait (but it is a trait).
I need those, not tuple, not Option, and not Either as pure result, because only ref/out makes chaining ifs elegant.
No, Scala supports parameter passing by value or by name. Parameter passing by reference is actually quite difficult to accomplish correctly in the JVM, which is probably one reason why none of the popular JVM languages have it. Additionally, out and ref parameters encourage programming via side-effect, something the at design of Scala attempts to avoid wherever possible.
As for chaining of if's, there are a variety of ways to achieve some effects like that in Scala. "match" expressions are the most obvious, and you might also look into monadic compositions using Option or Either.
similar to the one in Ruby
Yes, as of Scala 2.9 with the -Xexperimental option, one can use the Dynamic trait
(scaladoc). Classes that extend Dynamic get the magical method applyDynamic(methodName, args) which behaves like Ruby's method_missing.
Among other things, the Dynamic trait can be useful for interfacing with dynamic languages on the JVM.
The following is no longer strictly true with the Dynamic trait found in [experimental] Scala 2.9. See the answer from Kipton Barros, for example.
However, Dynamic is still not quite like method_missing, but rather employs compiler magic to effectively rewrite method calls to "missing" methods, as determined statically, to a proxy (applyDynamic). It is the approach of statically-determining the "missing" methods that differentiates it from method_missing from a polymorphism viewpoint: one would need to try and dynamically forward (e.g. with reflection) methods to get true method_missing behavior. (Of course this can be avoided by avoiding sub-types :-)
No. Such a concept does not exist in Java or Scala.
Like Java, all the methods in Scala are 'bound' at compile time (this also determines what method is used for overloading, etc). If a program does compile, said method exists (or did according to the compiler), otherwise it does not. This is why you can get the NoSuchMethodError if you change a class definition without rebuilding all affected classes.
If you are just worried about trying to call a method on an object which conforms to a signature ("typed duck typing"), then perhaps you may be able to get away with structural typing. Structural typing in Scala is magical access over reflection -- thus it defers the 'binding' until runtime and a runtime error may be generated. Unlike method_missing this does not allow the target to handle the error, but it does allow the caller to (and the caller could theoretically call a defined methodMissing method on the target... but this is probably not the best way to approach Scala. Scala is not Ruby :-)
Not really. It doesn't make sense. Scala is a statically-typed language in which methods are bound at compile time; Ruby is a dynamically-typed language in which messages are passed to objects, and these messages are evaluated at runtime, which allows Ruby to handle messages that it doesn't directly respond to, à la method_missing.
You can mimic method_missing in a few ways in Scala, notably by using the Actors library, but it's not quite the same (or nearly as easy) as Ruby's method_missing.
No, this is not possible in Scala 2.8 and earlier.