Is there anyway we can read scala doc comments using reflection. My requirement is to read the #group tag value and use it for counting how many functions are there for each group
No, you can't use Scala reflection to access documentation comments. The reason is simple: comments are, almost by definition, not part of the program. Therefore, it is logically impossible for them to be available via reflection.
In Python, for example, documentation is available from the running program (in fact, even without using reflection), because the documentation is not hidden away in comments, but rather simply assigned to a field of the object that is being documented. Many Lisps (e.g. Clojure), and also Ioke and Seph work that way, too.
In Newspeak, what they call "comments" is available using reflection, but that's because what they call "comments" are not really comments, it is more like arbitrary metadata that can be attached to objects. It is in fact more similar to an annotation in Scala than a comment.
In Scala, documentation is written in comments, and comments are not part of the program (they are literally equivalent to whitespace in the Scala Language Specification), and therefore, cannot possibly be part of the program and thus cannot possibly be accessed via reflection.
Related
I have read various old StackOverflow discussions on this general topic but there is still one part of the puzzle which appears, to me at least, to be missing.
It is simply this: what is the actual mechanism by which the anonymous function is serialized? And, where could we find its source code?
Or is it all just magic?
Other relevant SO articles (the third of these itself points to some useful articles outside StockOverflow):
Serialization of Scala Functions
Why Scala can serialize...
How to serialize functions in Scala
I'm going to answer my own question with what, I believe is the correct answer. The reason I'm doing it this way is that it seems to me that this aspect of serialization is never explained and it does appear to work just by magic. I essentially confirmed (to my satisfaction) the answer as part of the research I was doing to ensure that my question above was indeed appropriate.
But the main reason I'm offering my own answer is that I invite knowledgeable users either to agree with it, to correct it, to expand upon it, or to destroy it. Here goes...
It's all magic. No, I'm just kidding. But essentially the mechanism, once Scala has taken the step of representing the anonymous function as a Class, is entirely provided for by Java. In addition, we, the programmer, need to ensure that an anonymous function is as much pure code as possible: no references to any objects that might not be serializable. The secret sauce is to be found in the Java class: ObjectStreamClass. Which, in turn, is invoked by the Java serialization classes: ObjectInputStream and ObjectOutputStream.
Essentially the serialized bytes contain the full pathname of the class, its serialVersionUID, and whatever other relevant information is necessary. When deserializing, the system will simply look up the class in the appropriate classpath and return a reference to it. This obviously assumes that the deserializing system has the class in its classpath. The mechanism for that is a little beyond the scope of my research but it's clear that in a system like Spark, it should be easy to arrange.
No (additional) compilation/decompilation of byte code is necessary as the classLoader has everything necessary. I'm slightly surprised to find the ObjectStreamClass in java.io rather than in the reflection package, but I suppose there's an argument for it being there, given the tight coupling with ObjectInputStream and ObjectOutputStream.
One thing to keep in mind is that while we think in terms of serializing/deserializing objects, rather than classes, what we are dealing with here is an object of type Class.
One more thing to note is that in Scala 2.12, anonymous functions are now implemented differently: as Java8 lambdas. This has broken the mechanism described above in a rather serious way. So serious, that Spark is currently having trouble supporting Scala 2.12. The holdup appears to be this issue: SPARK-14540.
Just wondering whether there is any support for comments considered as documentation for enums and classes members (methods and fields / properties) in PowerShell?
If you're referring to the equivalent of Comment-Based Help, I've seen no indication of any. However, if you're writing classes or enums, you are also presumably writing scripts or script cmdlets/advanced functions that rely on them; it wouldn't be unreasonable to document the classes or enums in the scripts/etc. that use them.
Trevor Sullivan provides a link below to building your own about_Topic help data files; this represents another possibility, and one that may well be more appropriate, depending on how you're using your classes/enums.
I'm new to scala and struggling with the documentation a little bit. I was looking at a piece of code in the spark codebase (cosine similarity for RowMatrix) and saw that they use Iterator.tabulate. Not knowing what that function does I looked in the scala API docs, only to find out the function does not exist. Except that it does exist, because I can use it in the repl (hmm, maybe I'm looking at the wrong API docs version ... no, this this is the current version).
After a bit of searching I find out that tabulate is defined (at least) in scala.collection.generic.SeqFactory and scala.collection.generic.TraversableFactory. These two however appear not the be connected in the dependency graph. I can't find any path between the two, and hence no way of actually knowing - from looking at the API docs - that .tabulate even exists.
So the question is: how do you find .tabulate and it's documentation from looking at the API docs for the class (say Iterator or Seq). Do I just have to google my way around it, or is there some magic button in the scala docs that will make the thing appear?
This doesn't seem to be limited to just .tabulate but a more common issue (at least for me), looking at library code functions seem to exist that are never mentioned in the API. Another example is
org.apache.spark.mllib.linalg.distributed.RowMatrix.toBreeze
I still don't know if that function exists, some code seems to use it, but I can't find any documentation about it.
In Scala source code all logic of Iterator defined in one file Iterator.scala. Function tabulate that you're looking for is defined in object Iterator in Scala API you make search by trait Iterator so this is why you can't find it.
In right corner of doc you can switch to object iterator and here you will find Iterator$#tabulate util function.
I'm having a hard time deciphering Scala API documentation.
For example, I've defined a timestamp for use in a database.
def postedDate = column[Timestamp]("posted_date", O NotNull, O Default new Timestamp(Calendar.getInstance.getTimeInMillis), O DBType("timestamp"))
If I hadn't read several examples, of which none were in the API doc, how could I construct this statement? From the Column documentation how could I know the parameters?
I guessed it had something to do with TimestampTypeMapperDelegate but it is still not crystal clear how to use it.
The first thing to note from the scaladoc for Column is that it is abstract, so you probably want to deal directly with one if its subclasses. For example, NamedColumn.
Other things to note are that it has a type parameter and the constructor takes an implicit argument of a TypeMapper of the same parameter type. The docs for TypeMapper provide an example of how to create a custom one, but if you look at the subclasses, there are plenty of provided ones (such as timestamp). The fact that the argument is declared as implicit suggests that there could be one in scope, and if so, it will automatically be used as the parameter without explicitly stating that. If there isn't an implicit in scope that satisfies the requirement, you'll have to provide it.
The next think to note is that a TypeMapper is a trait that extends a function with an argument of a BasicProfile and a TypeMapperDelegate result. Basically what's going on here is the definition of a type mapper is separated from the implementation. This is done to support multiple flavors of database. If look at the subclasses of BasicProfile, it will become apparent that ScalaQuery supports quite a few, and as we know, their implementations are sometimes quite different.
If you chase the docs for a while, you end up at the BasicTypeMapperDelegates trait that has a bunch of vals in it with delegates for each of the basic types (including timestamps).
BasicTable defines a method called column (which you've found), and the intent of the column method is to shield you from having to know anything about TypeMappers and Delegates as long as you are using standard types.
So, I guess to answer your question about whether there is enough information in the API docs, I'd personally say yes, but the docs could be enhanced with better descriptions of classes, objects, traits and methods.
All that said, I've always found that leveraging examples, API docs, and even the source code of the project provides a robust way of getting up to speed on most open source projects. To be quite blunt, many of these projects (including ScalaQuery) have saved me countless hours of work, but probably cost the author(s) countless hours of personal time to create and make available. These are not necessarily commercial products, and we as consumers shouldn't hold them to the same standards that we hold for-fee products. If you find docs inadequate, contribute!
I was a little surprised when I started using Lift how heavily it uses reflection (or appears to), it was a little unexpected in a statically-typed functional language. My experience with JSP was similar.
I'm pretty new to web development, so I don't really know how these tools work, but I'm wondering,
What aspects of web development encourage using reflection?
Are there any tools (in statically typed languages) that handle (1) referring to code from a template page (2) object-relational mapping, in a way that does not use reflection?
Please see lift source. It doesn't use reflection for most of the code that I have studied. Almost everything is statically typed. If you are referring to lift views they are processed as Xml nodes, that too is not reflection.
Specifically referring to the <lift:Foo.bar/> issue:
When <lift:Foo.bar/> is encountered in the code, Lift makes a few guesses, how the original name should have been (different naming conventions) and then calls java.lang.Class.forName to get the class. (Relevant code in LiftSession.scala and ClassHelpers.scala.) It will only find classes registered with addToPackages during boot.
Note that it is also possible (and common) to register classes and methods manually. Convention is still that all transformations must be of the form NodeSeq => NodeSeq because that is the only thing which makes sense for an untyped HTML/XHTML output.
So, what you have is Lift‘s internal registry of node transformations on one side, and on the other side the implicit registry of the module. Both types use a simple string lookup to execute a method. I guess it is arguable if one is more reflection based than the other.