Is there any way to test a scala compiler plugin? - scala

I am trying to setup tests for a scala compiler-plugin I am developping.
According to this similar question, it is possible to invoke the plugin programmatically. However, when I tried to do this the program could not find my plugin.
Error:(34, 25) not found: type GetFileFromAnnotation
for (phase <- new GetFileFromAnnotation(this).components)
Both files are in the same package.
Here is the code which is trying to test the plugin (pulled from the question above), the commented out code is an alternative to the code above it and throws the same error:
package compilerPlugin
import scala.reflect.internal.util.BatchSourceFile
import scala.tools.nsc.io.VirtualDirectory
import scala.tools.nsc.reporters.ConsoleReporter
import scala.tools.nsc.{Global, Settings}
object AnnotationFinderTest extends App {
// prepare the code you want to compile
val code =
"""
|class Typestate(filename:String) extends scala.annotation.StaticAnnotation
|
|#Typestate(filename = "MyProtocol.txt")
|class Cat{
| def comeAlive(): Unit = println("The cat is alive")
|}
|
|object Main extends App {
| val cat = new Cat()
| cat.comeAlive()
|}""".stripMargin
val sources = List(new BatchSourceFile("<test>", code))
println("sources "+sources)
val settings = new Settings
settings.usejavacp.value = true
settings.outputDirs.setSingleOutput(new VirtualDirectory("(memory)", None))
val compiler = new Global(settings, new ConsoleReporter(settings)) {
override protected def computeInternalPhases () {
super.computeInternalPhases
for (phase <- new GetFileFromAnnotation(this).components)
phasesSet += phase
}
}
new compiler.Run() compileSources (sources)
}
The code for the GetFileFromAnnotation plugin is:
package compilerPlugin
import java.io.{FileNotFoundException, IOException}
import scala.tools.nsc
import nsc.Global
import nsc.Phase
import nsc.plugins.Plugin
import nsc.plugins.PluginComponent
import scala.io.BufferedSource
import scala.io.Source._
class GetFileFromAnnotation(val global: Global) extends Plugin {
import global._
val name = "GetFileFromAnnotation"
val description = "gets file from typestate annotation"
val components: List[PluginComponent] = List[PluginComponent](Component)
private object Component extends PluginComponent {
val global: GetFileFromAnnotation.this.global.type = GetFileFromAnnotation.this.global
val runsAfter: List[String] = List[String]("parser")
val phaseName: String = GetFileFromAnnotation.this.name
def newPhase(_prev: Phase) = new GetFileFromAnnotationPhase(_prev)
class GetFileFromAnnotationPhase(prev: Phase) extends StdPhase(prev) {
override def name: String = GetFileFromAnnotation.this.name
def printFile(filename: String): Unit ={
val source = fromFile(filename)
try {
val it = source.getLines()
while (it.hasNext)
println(it.next())
}
catch{
case e: IOException => println(s"Had an IOException trying to use file $filename")
} finally {
source.close
}
}
def getFilenameFromAnnotation(annotation: Apply): Option[String] ={
annotation match{
case Apply(Select(New(Ident(TypeName("Typestate"))), con),List(NamedArg(Ident(TermName("filename")), Literal(Constant(filename))))) => Some(filename.toString)
case Apply(Select(New(Ident(TypeName("Typestate"))), con),List(Literal(Constant(filename)))) => Some(filename.toString)
case _ => None
}
}
def apply(unit: CompilationUnit): Unit = {
for (tree#q"$mods class $tpname[..$tparams] $ctorMods(...$paramss) extends { ..$earlydefns } with ..$parents { $self => ..$stats }" <- unit.body) {
val annotations = mods.annotations
for(annotation#Apply(arg1,arg2) <- annotations){
getFilenameFromAnnotation(annotation) match{
case Some(filename) => printFile(filename)
case None => println("Not a Typestate annotation")
}
}
}
}
}
}
}
Is it even feasible to use this method to test a plugin or is the only way to do it to use mocking? Is there another way of doing this entirely?
UPDATE:
Thanks for the comment and the github example.
It ended up working fine with the edited test code above and prints out the protocol file as expected. I am not sure why it wasn't working before.

Related

Run Object notebook in Databricks

I am trying to execute this code on databricks in scala. Everything is in an object, then I have a case class and def main and other def functions.
Trying to work with "package cells" but I got Warning: classes defined within packages cannot be redefined without a cluster restart.
Compilation successful.
removing the object didn't work either
package x.y.z
import java.util.Date
import java.io.File
import java.io.PrintWriter
import org.apache.hadoop.fs.{FileSystem, Path}
object Meter {
val dateFormat = new SimpleDateFormat("yyyyMMdd")
case class Forc (cust: String, Num: String, date: String, results: Double)
def main(args: Array[String]): Unit = {
val inputFile = "sv" //
val outputFile = "ssv" //
val fileSystem = getFileSystem(inputFile)
val inputData = readLines(fileSystem, inputFile, skipHeader = true).toSeq
val filtinp = inputData.filter(x => x.nonEmpty)
.map(x => Results(x(6), x(5), x(0), x(8).toDouble))
def getTimestamp(date: String): Long = dateFormat.parse(date).getTime
def getDate(timeStampInMills: Long): String = {
val time = new Date(timeStampInMills)
dateFormat.format(time)
}
def getFileSystem(path: String): FileSystem = {
val hconf = new Configuration()
new Path(path).getFileSystem(hconf)
}
override def next(): String = {
val result = line
line = inputData.readLine()
if (line == null) {
inputData.close()
}
result
}
}
}
}

Facing issue while using reflection class concept in scala

I have one main class like this:
class Test {
def exe(first:String, second:String, task:String):String = {
task match {
case "A" => {
val obj = new A(first)
obj.defineSecond(second)
}
case "B" => {
val obj = new B(first)
obj.defineSecond(second)
}
case "C" => {
val obj = new C(first)
obj.defineSecond(second)
}
....so many cases
}
}
}
Instead of writing case in my Test class everytime a new class is added, I tried using the concept of reflection in scala.
Below is what I trying:
val m = ru.runtimeMirror(getClass.getClassLoader)
val classTest = ru.typeOf[Test].typeSymbol.asClass
val cm = m.reflectClass(classTest)
But getting error as "class Test is inner class, use reflectClass on an InstaneMirror to obtain its classMirror".
Can anyone knows how can I can avoid adding cases to my main class everytime a new class is created, instead I can write my main class in a way it will work for every case.
I guess you haven't provided all necessary information in your question. It's written that "class Test is inner class" in your error message but Test is not inner in your code snippet. If you want your runtime-reflection code to be fixed please provide code snippet that reflects actual use case.
Meanwhile you can try a macro (working at compile time)
import scala.language.experimental.macros
import scala.reflect.macros.blackbox
class Test {
def exe(first: String, second: String, task: String): String = macro Test.exeImpl
}
object Test {
def exeImpl(c: blackbox.Context)(first: c.Tree, second: c.Tree, task: c.Tree): c.Tree = {
import c.universe._
val cases = Seq("A", "B", "C").map(name =>
cq"""${Literal(Constant(name))} => {
val obj = new ${TypeName(name)}($first)
obj.defineSecond($second)
}"""
)
q"$task match { case ..$cases }"
}
}
Usage:
class A(s: String) {
def defineSecond(s1: String): String = ""
}
class B(s: String) {
def defineSecond(s1: String): String = ""
}
class C(s: String) {
def defineSecond(s1: String): String = ""
}
new Test().exe("first", "second", "task")
//scalac: "task" match {
// case "A" => {
// val obj = new A("first");
// obj.defineSecond("second")
// }
// case "B" => {
// val obj = new B("first");
// obj.defineSecond("second")
// }
// case "C" => {
// val obj = new C("first");
// obj.defineSecond("second")
// }
//}

How to modify Scala actor to return List[String]?

Following the Akka tutorials https://doc.akka.io/docs/akka/current/typed/guide/tutorial_1.html I have modified sample code to send a message to an akka actor every 3 seconds:
scheduler.scheduleAtFixedRate(
initialDelay = Duration(0, TimeUnit.SECONDS),
interval = Duration(3, TimeUnit.SECONDS))(
runnable = task)
}
I'm unable to safely compiling changing the message return in Main from String to List[String]. So instead of firstRef ! "printit" , change to firstRef ! List("printit") but this causes compiler error:
To fix the compiler error I make changes to:
class Main(context: ActorContext[String]) extends AbstractBehavior[List[String]](context) {
override def onMessage(msg: String): Behavior[List[String]] =
def apply(): Behavior[List[String]] =
where previously this contained:
class Main(context: ActorContext[String]) extends AbstractBehavior[String](context) {
override def onMessage(msg: String): Behavior[String] =
def apply(): Behavior[String] =
What needs to change in order to return List[String] instead of String in the Main actor ?
complete code (without changes):
import java.util.concurrent.TimeUnit
import akka.actor.typed.ActorSystem
import akka.actor.typed.Behavior
import akka.actor.typed.scaladsl.AbstractBehavior
import akka.actor.typed.scaladsl.ActorContext
import akka.actor.typed.scaladsl.Behaviors
import map.QTableRow
import scala.concurrent.ExecutionContext
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
import scala.collection.mutable.ListBuffer
object PrintMyActorRefActor {
def apply(): Behavior[String] =
Behaviors.setup(context => new PrintMyActorRefActor(context))
}
class PrintMyActorRefActor(context: ActorContext[String]) extends AbstractBehavior[String](context) {
override def onMessage(msg: String): Behavior[String] =
msg match {
case "printit" =>
val secondRef = context.spawn(Behaviors.empty[String], "second-actor")
println(s"Second: $secondRef")
this
}
}
object Main {
def apply(): Behavior[String] =
Behaviors.setup(context => new Main(context))
}
class Main(context: ActorContext[String]) extends AbstractBehavior[String](context) {
override def onMessage(msg: String): Behavior[String] =
msg match {
case "getdata" =>
val firstRef = context.spawn(PrintMyActorRefActor(), "first-actor"+String.valueOf(System.currentTimeMillis()))
println(s"First: $firstRef")
firstRef ! "printit"
this
}
}
object ActorHierarchyExperiments extends App {
val testSystem = ActorSystem(Main(), "testSystem")
val scheduler = testSystem.scheduler
val task = new Runnable { def run() {
testSystem ! "getdata" }
}
scheduler.scheduleAtFixedRate(
initialDelay = Duration(0, TimeUnit.SECONDS),
interval = Duration(3, TimeUnit.SECONDS))(
runnable = task)
}
The firstRef actor reference references an instance of PrintMyActorRefActor, which is a Behavior[ String ], so to send any List[ String ]to the ref, the PrintMyActorRefActor needs to be able to recieve it. Changing this, including the signature of its recieve message, should make the snippet compile. Now, of course, you'll need to change the behaviour of the PrintMyActorRefActor.onMessage( msg: List[ String ] ): Behavior[ List[ String ] ] to deal with the list instead of the single string...
Complete transformed code snippet:
import java.util.concurrent.TimeUnit
import akka.actor.typed.scaladsl.{AbstractBehavior, ActorContext, Behaviors}
import akka.actor.typed.{ActorSystem, Behavior}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
object PrintMyActorRefActor {
def apply(): Behavior[List[String]] =
Behaviors.setup(context => new PrintMyActorRefActor(context))
}
class PrintMyActorRefActor(context: ActorContext[List[String]]) extends AbstractBehavior[List[String]](context) {
override def onMessage(msg: List[String]): Behavior[List[String]] =
msg match {
case "printit" :: xs => // ignores all but the head element
val secondRef = context.spawn(Behaviors.empty[String], "second-actor")
println(s"Second: $secondRef")
this
}
}
object Main {
def apply(): Behavior[String] =
Behaviors.setup(context => new Main(context))
}
class Main(context: ActorContext[String]) extends AbstractBehavior[String](context) {
override def onMessage(msg: String): Behavior[String] =
msg match {
case "getdata" =>
val firstRef = context.spawn(PrintMyActorRefActor(), "first-actor" + String.valueOf(System.currentTimeMillis()))
println(s"First: $firstRef")
firstRef ! List("printit")
this
}
}
object ActorHierarchyExperiments extends App {
val testSystem = ActorSystem(Main(), "testSystem")
val scheduler = testSystem.scheduler
val task = new Runnable {
def run() {
testSystem ! "getdata"
}
}
scheduler.scheduleAtFixedRate(
initialDelay = Duration(0, TimeUnit.SECONDS),
interval = Duration(3, TimeUnit.SECONDS))(
runnable = task)
}

Scala: expecting Try[Stream[(String)]] but is giving Try[String]

I am trying to read a text file to compare 2 files. I have written the code to read the first file and I am expecting readFileStream function to give me Collections of String but I am getting only String.
Could you see where I have made wrong?
import java.io.{BufferedReader, FileInputStream, InputStreamReader}
import java.net.URI
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FSDataInputStream, FileSystem, Path}
import scala.util.{Failure, Success, Try}
object TestCompareHDFSFiles {
def main(args: Array[String]): Unit = {
val hdfs = FileSystem.get(new Configuration())
val path1 = new Path(args(0))
val path2 = new Path(args(1))
readHDFSFile(hdfs, path1, path2)
}
// Accept a parameter which implements a close method
def using[A <: { def close(): Unit }, B](resource: A)(f: A => B): B =
try {
f(resource)
} finally {
resource.close()
}
def readHDFSFile(hdfs: FileSystem, path1: Path, path2: Path): Option[Stream[(String,String)]] = {
Try(using(new BufferedReader(new InputStreamReader(hdfs.open(path1))))(readFileStream))
} match {
case Success(result) => {
}
case Failure(ex) => {
println(s"Could not read file $path1, detail ${ex.getClass.getName}:${ex.getMessage}")
None
}
}
def readFileStream(br: BufferedReader)= {
for {
line <- Try(br.readLine())
if (line != null )
} yield line
}
}
I am struck here. Any help please.
Thanks,

File I/O with Free Monads

I have a CSV file that I need to parse and do some action on every record. How do I use Free Monads with it? Currently, I'm loading the entire file into memory and would like to know if there is any better solution. Below is my program:
for {
reader <- F.getReader("my_file.csv")
csvRecords <- C.readCSV(reader)
_ <- I.processCSV(csvRecords)
_ <- F.close(reader)
} yield()
This code works for smaller files, but if I have very large files (over 1 GB), this wouldn't work very well. I'm using Commons CSV for reading the CSVRecords.
Looking into the code at your gist I think that the line with the comment is exactly the line you don't want at all:
object CSVIOInterpreter extends (CSVIO ~> Future) {
import scala.collection.JavaConverters._
override def apply[A](fa: CSVIO[A]): Future[A] = fa match {
case ReadCSV(reader) => Future.fromTry(Try {
CSVFormat.RFC4180
.withFirstRecordAsHeader()
.parse(reader)
.getRecords // Loads the complete file
.iterator().asScala.toStream
})
}
}
Just remove the whole getRecords line. CSVFormat.parse returns an instance of CSVParser which already implements Iterable<CSVRecord>. And the getRecords call is the only thing that force it to read the whole file.
Actually you can see CSVParser.getRecords implementation and it is
public List<CSVRecord> getRecords() throws IOException {
CSVRecord rec;
final List<CSVRecord> records = new ArrayList<>();
while ((rec = this.nextRecord()) != null) {
records.add(rec);
}
return records;
}
So it just materializes the whole file using this.nextRecord call which is obviously a more "core" part of the API.
So when I do a simplified version of your code without the getRecords call:
import cats._
import cats.free.Free
import java.io._
import org.apache.commons.csv._
import scala.collection.JavaConverters._
trait Action[A] {
def run(): A
}
object F {
import Free.liftF
case class GetReader(fileName: String) extends Action[Reader] {
override def run(): Reader = new FileReader(fileName)
}
case class CloseReader(reader: Reader) extends Action[Unit] {
override def run(): Unit = reader.close()
}
def getReader(fileName: String): Free[Action, Reader] = liftF(GetReader(fileName))
def close(reader: Reader): Free[Action, Unit] = liftF(CloseReader(reader))
}
object C {
import Free.liftF
case class ReadCSV(reader: Reader) extends Action[CSVParser] {
override def run(): CSVParser = CSVFormat.DEFAULT.parse(reader)
}
def readCSV(reader: Reader): Free[Action, CSVParser] = liftF(ReadCSV(reader))
}
object I {
import Free.liftF
case class ProcessCSV(parser: CSVParser) extends Action[Unit] {
override def run(): Unit = {
for (r <- parser.asScala)
println(r)
}
}
def processCSV(parser: CSVParser): Free[Action, Unit] = liftF(ProcessCSV(parser))
}
object Runner {
import cats.arrow.FunctionK
import cats.{Id, ~>}
val runner = new (Action ~> Id) {
def apply[A](fa: Action[A]): Id[A] = fa.run()
}
def run[A](free: Free[Action, A]): A = {
free.foldMap(runner)
}
}
def test() = {
val free = for {
// reader <- F.getReader("my_file.csv")
reader <- F.getReader("AssetsImportCompleteSample.csv")
csvRecords <- C.readCSV(reader)
_ <- I.processCSV(csvRecords)
_ <- F.close(reader)
} yield ()
Runner.run(free)
}
it seems to work OK in line-by-line mode.
Here how I use the CSV file to read and do some operation on that -
I use scala.io.Source.fromFile()
I create one case class of the type of header of CSV file to make the data more accessible and operational.
PS: I don't have knowledge of monads, as well as I am in beginner in Scala. I posted this as it may be helpful.
case class AirportData(id:Int, ident:String, name:String, typeAirport:String, latitude_deg:Double,
longitude_deg:Double, elevation_ft:Double, continent:String, iso_country:String, iso_region:String,
municipality:String)
object AirportData extends App {
def toDoubleOrNeg(s: String): Double = {
try {
s.toDouble
} catch {
case _: NumberFormatException => -1
}
}
val source = scala.io.Source.fromFile("resources/airportData/airports.csv")
val lines = source.getLines().drop(1)
val data = lines.flatMap { line =>
val p = line.split(",")
Seq(AirportData(p(0).toInt, p(1).toString, p(2).toString, p(3).toString, toDoubleOrNeg(p(4)), toDoubleOrNeg(p(5)),
toDoubleOrNeg(p(6)), p(7).toString, p(8).toString, p(9).toString, p(10).toString))
}.toArray
source.close()
println(data.length)
data.take(10) foreach println
}