Node based Iterable implementation in Scala - scala

Continuing my travel through exercises from Sedgwick and Wayne's Algorithms I came across one in which I have to implement a RandomBag. Originally RandomBag is supposed to implement Iterable (in java) and its Iterator had to serve items in random order.
This is the companion object of my ImmutableRandomBag:
object ImmutableRandomBag{
case class Node[Item](item: Item, next: Option[Node[Item]])
def apply[Item](maybeNode: Option[Node[Item]], size: Int): ImmutableRandomBag[Item] = new ImmutableRandomBag(maybeNode, size)
}
And this is beginning of the class itself:
class ImmutableRandomBag[Item](maybeNode: Option[Node[Item]], size: Int) extends Iterable[Item]{
override def isEmpty: Boolean = size == 0
def add(item: Item) = {
ImmutableRandomBag(Some(Node(item, maybeNode)), size +1)
}
...
}
My understanding was that the val size should have overridden the def size from Iterable trait. When testing the add method I am getting the IndexOutOfBounException:
class RandomBagSpec extends BaseSpec {
trait RandomBag{
val begin = new ImmutableRandomBag[Connection](None, 0)
}
...
"Adding an item to empty RandomBag" should "return another bag with size 1" in new RandomBag {
val bag = begin.add(Connection(0,1))
bag.size should equal(1)
}
}
While debugging size is correctly evaluated in the constructor parameter, so I am not sure where the IndexOutOfBoundException comes from, but I get it whenever I call the add method. Maybe the problem sprouts from the following. In ImmutableRandomBag there's also Iterator implementation:
...
override def iterator: Iterator[Item] = new RandomIterator[Item](maybeNode)
private class RandomIterator[Item](first: Option[Node[Item]]) extends Iterator[Item]{
first match {
case Some(node) => random(node)
case None =>
}
var current: Int = 0
var container: Vector[Item] = Vector()
override def hasNext: Boolean = current < ImmutableRandomBag.this.size
override def next(): Item = {
val item = container(current)
current += 1
item
}
def random(first: Node[Item]) = {
#tailrec
def randomHelper(next: Option[Node[Item]], acc: List[Item]):List[Item]= next match {
case None => acc
case Some(node) => randomHelper(node.next, node.item::acc)
}
val items = randomHelper(Some(first), List[Item]())
container = Random.shuffle(items).toVector
}
}
}
And I have a different test in the same spec for it:
...
"Random Bag's iterator" should "contain all items passed to parent iterable" in new RandomBag{
val connections = List(Connection(0,1), Connection(1,0), Connection(1,1))
var localRB = begin
for(c <- connections) localRB = localRB.add(c)
assert(localRB.iterator.forall(conn=> connections.contains(conn)) == true)
}
...
I also get an IndexOutOfBoundException with the following stack:
[info] RandomBagSpec:
[info] Random Bag's iterator
[info] - should contain all items passed to parent iterable *** FAILED ***
[info] java.lang.IndexOutOfBoundsException: 0
[info] at scala.collection.immutable.Vector.checkRangeConvert(Vector.scala:123)
[info] at scala.collection.immutable.Vector.apply(Vector.scala:114)
[info] at ca.vgorcinschi.algorithms1_3_34.ImmutableRandomBag$RandomIterator.next(ImmutableRandomBag.scala:31)
[info] at scala.collection.Iterator.forall(Iterator.scala:956)
[info] at scala.collection.Iterator.forall$(Iterator.scala:954)
[info] at ca.vgorcinschi.algorithms1_3_34.ImmutableRandomBag$RandomIterator.forall(ImmutableRandomBag.scala:18)
[info] at ca.vgorcinschi.algorithms1_5_19.RandomBagSpec$$anon$1.<init>(RandomBagSpec.scala:16)
[info] at ca.vgorcinschi.algorithms1_5_19.RandomBagSpec.$anonfun$new$1(RandomBagSpec.scala:12)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
The issue seems to come from calling Iterator's next method and indeed the container Vector doesn't contain any elements:
but why is next being called before random?

val size should have overridden the def size from Iterable trait
A val would have, but you don't have one; you just have a constructor parameter in a non-case class. Effectively it's a private val and can't override anything.
but why is next being called before random?
It isn't; in RandomIterator's constructor, random is called (as part of first match ...) before the initializer container = Vector(). next is called only after constructor.

Related

How can I override a class' method in a Scala 3 compiler plugin?

I want to auto-generate an overriden method in a compiler plugin (Scala 3) for a class like:
trait SpecialSerialize {
def toJson(sb: StringBuilder, c:SJConfig): Unit = {println("wrong")}
}
case class Person(name:String, age:Int) extends SpecialSerialize
The plugin would generate:
case class Person(name:String, age:Int) extends SpecialSerialize {
override def toJson(sb: StringBuilder, c:SJConfig): Unit = ... // code here
}
I have a phase:
class ReflectionWorkerPhase extends PluginPhase {
import tpd._
val phaseName = "reflectionWorker"
override val runsAfter = Set(Pickler.name)
override def transformTypeDef(tree: TypeDef)(implicit ctx: Context): Tree =
if tree.isClassDef && !tree.rhs.symbol.isStatic then // only look at classes
// 0. Get a FreshContext so we can set the tree to this tree. (for '{} later)
implicit val fresh = ctx.fresh
fresh.setTree(tree)
QuotesCache.init(fresh)
implicit val quotes:Quotes = QuotesImpl.apply() // picks up fresh
import quotes.reflect.*
// 1. Set up method symbol, define parameters and return type
val toJsonSymbol = Symbol.newMethod(
Symbol.spliceOwner,
"toJson",
MethodType(
List("sb","config"))( // parameter list
_ => List( // types of the parameters
TypeRepr.of[StringBuilder],
TypeRepr.of[SJConfig],
),
_ => TypeRepr.typeConstructorOf(classOf[Unit]) // return type
),
Flags.Override, // Note override here
Symbol.noSymbol
)
// 2. Get our class' Symbol for ownership reassignment
val classDef = tree.asInstanceOf[ClassDef]
val classSymbol = classDef.symbol
// 3. Define our method definition (DefDef) using our method symbol defined above
val toJsonMethodDef = DefDef(
toJsonSymbol,
{
case List(List(sb: Term, config: Term)) =>
given Quotes = toJsonSymbol.asQuotes
Some({
// Multiple quotes here intentional...
// Real code will generate a list of quoted statements
quoted.Expr.ofList(List(
'{ println("Hello") },
'{ println("World") }
))
}.asTerm.changeOwner(toJsonSymbol))
}
).changeOwner(classSymbol)
// 4. Add toJsonMethodDef to tree and return
val cd = ClassDef.copy(classDef)(
name = classDef.name,
constr = classDef.constructor,
parents = classDef.parents,
selfOpt = classDef.self,
body = toJsonMethodDef +: classDef.body
)
cd.asInstanceOf[dotty.tools.dotc.ast.tpd.Tree]
else
tree
}
When I use the plugin on a sample Person class, I get this error on compile:
Exception in thread "sbt-bg-threads-1" java.lang.ClassFormatError: Duplicate method name "toJson" with signature "(Lscala.collection.mutable.StringBuilder;Lco.blocke.scala_reflection.SJConfig;)V" in class file com/foo/Person
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1013)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:524)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:427)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
...
So something in the other compile phases failed to recognize my generated method as being an override and tried to copy in the "master" method from the trait, and of course the JVM lost its mind at runtime, finding 2 copies of toJson with the same signature.
How can I fix this so my generated method is recognized as a valid override such that a 2nd toJson method won't be generated in a later phase?

Scala not recognizing type after pimping JSoup

I'm trying to do some web scraping in Scala, and is currently using JSoup. Now I found that the iterator is not working in Scala, so I did some pimpin' and wrote an iterator myself. It looks like this:
object Pimp {
implicit class PimpElements(es: Elements) extends Iterable[Element] {
def iterator = new Iterator[Element] {
var currentElem = 0
def hasNext = currentElem < size
def next(): Element = {
currentElem += 1
es.get(currentElem - 1)
}
}
}
}
Now, the code that does not work, because intelliJ or Scala does not recognize my variable cider to be of type Element I guess:
for (cider <- ciders; if cider.getElementsByClass("info").text() != "") {
ciderArray += Drink(DrinkType.CIDER, cider)
}
But why not? My next() method returns es.get(i) which supposedly should be an Element and works in the code below:
for (i <- 0 to ciders.size() - 1; if ciders.get(i).getElementsByClass("info").text() != "") {
ciderArray += Drink(DrinkType.CIDER, ciders.get(i))
}
Isn't this code basically doing the same as the iterator, but gets recognized for some reason? The type of cider is, according to intelliJ, Any and not Element.
The for comprehension is translated to c.withFilter(p).foreach(f).
Possibly you expected it to call iterator.
This question is interesting because these encodings can result in more inferred type parameters or other effects.
I see Elements is an ArrayList.
TraversableLike.withFilter does turn out to be different from Iterator.withFilter.
Your example works, after fixing the call to size (which stackoverflows). It also works with Java types for Elements and Element.
object Test extends App {
case class Element(value: String)
type Elements = java.util.ArrayList[Element]
implicit class PimpElements(es: Elements) extends Iterable[Element] {
def iterator = new Iterator[Element] {
var currentElem = 0
def hasNext = currentElem < es.size
def next(): Element = {
currentElem += 1
es.get(currentElem - 1)
}
}
}
val vs = new java.util.ArrayList[Element]
vs.add(new Element("hi"))
vs.add(new Element("bye"))
for (v <- vs if v.value.startsWith("h")) println(v)
}
But it will also work this way:
object Test extends App {
implicit class PimpElements(es: Elements) extends Iterator[Element] {
var currentElem = 0
def hasNext = currentElem < es.size
def next(): Element = {
currentElem += 1
es.get(currentElem - 1)
}
}
val vs = new Elements
vs.add(new Element("hi"))
vs.add(new Element("bye"))
for (v <- vs if v.value.startsWith("h")) println(v)
}
The Traversable tracks its representation as a type parameter, which might make for type inference issues. Both classes incur a wrapper for filtering. But the Iterator doesn't override foreach when filtering, so it saves the last unfiltered element on hasNext for the call to next. Possibly, the Traversable.foreach is more efficient.

how to ignore test utility methods when scalatest detects failures?

I have this convenience method in my tests:
def assertFormat[T: SexpFormat](start: T, expect: Sexp): Unit = {
val sexp = start.toSexp
assert(sexp === expect, s"${sexp.compactPrint} was not ${expect.compactPrint}")
expect.convertTo[T] should be(start)
}
which is basically a convenience for running an assertion pattern that I do a lot.
It's not possible to rewrite this as a Matcher because of the implicit requirement on SexpFormat[T] (although I'd be interested in hearing of ways to do this that don't require me to write the type MyFormat in foo should roundTrip[MyFormat](...))
If any tests fail inside this utility method, scalatest will flag the internals of assertFormat as being the cause of the test failure. But I really want scalatest to detect the caller of this method to be the cause of the test. How can I do that?
i.e. current output is
[info] - should support custom missing value rules *** FAILED ***
[info] SexpNil did not equal SexpCons(SexpSymbol(:duck),SexpCons(SexpNil,SexpNil)) nil was not (:duck nil) (FormatSpec.scala:11)
[info] org.scalatest.exceptions.TestFailedException:
[info] at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:529)
[info] at org.scalatest.FlatSpec.newAssertionFailedException(FlatSpec.scala:1691)
[info] at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:502)
[info] at org.ensime.sexp.formats.FormatSpec$class.assertFormat(FormatSpec.scala:11)
[info] at org.ensime.sexp.formats.test.FamilyFormatsSpec.assertFormat(FamilyFormatsSpec.scala:151)
[info] at org.ensime.sexp.formats.test.FamilyFormatsSpec.roundtrip(FamilyFormatsSpec.scala:156)
[info] at org.ensime.sexp.formats.test.FamilyFormatsSpec$$anonfun$12.apply(FamilyFormatsSpec.scala:222)
[info] at org.ensime.sexp.formats.test.FamilyFormatsSpec$$anonfun$12.apply(FamilyFormatsSpec.scala:221)
FormatSpec.scala:11 is where my assertFormat is defined. The real failure is in FamilyFormatsSpec.scala:222 (which is calling another convenience method FamilyFormatsSpec.scala:156)
This is possible in ScalaTest 3.0 by taking an implicit org.scalactic.source.Position in your custom assertion. The position will then be computed (via a macro) whenever your assertFormat method is called, and that position will be picked up by the assert and matcher expression inside assertFormat. Here is how it would look:
import org.scalactic.source
def assertFormat[T: SexpFormat](start: T, expect: Sexp)(implicit pos: source.Position): Unit = {
val sexp = start.toSexp
assert(sexp === expect, s"${sexp.compactPrint} was not ${expect.compactPrint}")
expect.convertTo[T] should be(start)
}
The following example illstrates it. If you have ScalaTest 3.0 on the class path, just :load the following file into the Scala REPL:
:paste
import org.scalatest._
import org.scalactic._
import Matchers._
case class Sexp(o: Any) {
def compactPrint: String = o.toString
def convertTo[T: SexpFormat]: Sexp = implicitly[SexpFormat[T]].convertIt(o)
override def toString = "I'm too sexp for my shirt."
}
trait SexpFormat[T] {
def convertIt(o: Any): Sexp = new Sexp(o)
}
implicit class Sexpify(o: Any) {
def toSexp: Sexp = new Sexp(o)
}
implicit def universalSexpFormat[T]: SexpFormat[T] = new SexpFormat[T] {}
def assertFormat[T: SexpFormat](start: T, expect: Sexp): Unit = {
val sexp = start.toSexp
assert(sexp === expect, s"${sexp.compactPrint} was not ${expect.compactPrint}")
expect.convertTo[T] should be(start)
}
import org.scalatest.exceptions.TestFailedException
val before = intercept[TestFailedException] { assertFormat(1, new Sexp) }
println(s"${before.failedCodeStackDepth} - This stack depth points to the assert call inside assertFormat")
import org.scalactic.source
def betterAssertFormat[T: SexpFormat](start: T, expect: Sexp)(implicit pos: source.Position): Unit = {
val sexp = start.toSexp
assert(sexp === expect, s"${sexp.compactPrint} was not ${expect.compactPrint}")
expect.convertTo[T] should be(start)
}
val after = intercept[TestFailedException] { betterAssertFormat(1, new Sexp) }
println(s"${after.failedCodeStackDepth} - This stack depth is the betterAssertFormat call itself in your test code")
It will print:
3 - This stack depth points to the assert call inside assertFormat
4 - This stack depth is the betterAssertFormat call itself in your test code

Assertion for overloaded equality operator fails for triple-equals but passes for double-equals

I overloaded a case class's equality operator:
case class Polygon(points: Seq[(Double, Double)]) extends Shape {
def ==(that: Polygon): Boolean = { ... }
}
My unit tests pass when using double-equals in assertions:
import org.scalatest.FunSpec
class ShapeSpec extends FunSpec {
describe("Polygon") {
it("is equal to all its rotations") {
val p = Polygon(Seq( ... ))
for { i ← 0 until p.points.size } {
val q = Polygon(p.points.drop(i) ++ p.points.take(i))
assert(p == q)
}
}
}
}
But when using === instead of ==, the same tests fail:
[info] Polygon
[info] - is equal to all its rotations *** FAILED ***
[info] Polygon(List((-1.0,-1.0), (4.0,-1.0), (4.0,2.0), (1.0,2.0),
(1.0,1.0), (-1.0,1.0))) did not equal PolygonM(List((4.0,-1.0),
(4.0,2.0), (1.0,2.0), (1.0,1.0), (-1.0,1.0), (-1.0,-1.0)))
(ShapeSpec.scala:28)
Why does this happen?
You spelled it wrong. ;)
It should be:
override def equals(x: Any): Boolean

Inferring result type in continuations

Is it possible to remove some types from the following code:
import util.continuations._
object TrackingTest extends App {
implicit def trackable(x: Int) = new {
def tracked[R] = shift { cf: (Int => (R, Set[Int])) =>
cf(x) match {
case (r, ints) => (r, ints + x)
}
}
}
def track[R](body: => R #cpsParam[(R, Set[Int]), (R, Set[Int])]) = reset {
(body, Set[Int]())
}
val result = track(7.tracked[Int] + 35.tracked[Int])
assert(result == (42, Set(7, 35)))
val differentTypes = track(9.tracked[String].toString)
assert(differentTypes == ("9", Set(9)))
}
track function tracks calls of tracked on Int instances (e.g. 7.tracked).
Is it possible to infer type parameter on tracked implicit, so the following would compile:
track(7.tracked + 35.tracked)
Your question made me think of how continuations can track state. So I adapted that to your case and came up with this:
import util.continuations._
object TrackingTest extends App {
type State = Set[Int]
type ST = State => State
implicit class Tracked(val i: Int) extends AnyVal {
def tracked = shift{ (k: Int=>ST) => (state:State) => k(i)(state + i) }
}
def track[A](thunk: => A#cps[ST]): (A, State) = {
var result: A = null.asInstanceOf[A]
val finalSate = (reset {
result = thunk
(state:State) => state
}).apply(Set[Int]())
(result, finalSate)
}
val result = track(7.tracked + 35.tracked)
assert(result == (42, Set(7, 35)))
val differentTypes = track(9.tracked.toString)
assert(differentTypes == ("9", Set(9)))
}
This is using 2.10.1 but it works fine with 2.9.1 as well provided you replace the 2.10.x implicit value class with:
implicit def tracked(i: Int) = new {
def tracked = shift{ (k: Int=>ST) => (state:State) => k(i)(state + i) }
}
The key change I made is to have tracked not use any type inference, fixing to Int#cps[ST]. The CPS plugin then maps the computation to the right type (like String#cps[ST]) as appropriate. The state is threaded by the continuation returning a State=>State function that takes the current state (the set of ints) and returns the next state. The return type of reset is a function from state to state (of type ST) that will take the initial state and will return the final state.
The final trick is to use a var to capture the result while still keeping the expected type for reset.
While the exact answer to this question can be given only by the authors of the compiler, we can guess it is not possible by giving a look to the continuation plugin source code.
If you look to the source of the continuations you can see this:
val anfPhase = new SelectiveANFTransform() {
val global = SelectiveCPSPlugin.this.global
val runsAfter = List("pickler")
}
val cpsPhase = new SelectiveCPSTransform() {
val global = SelectiveCPSPlugin.this.global
val runsAfter = List("selectiveanf")
}
The anfPhase phase is executed after the pickler phase, and the cpsPhase after selectiveAnf. If you look to SelectiveANFTransform.scala
abstract class SelectiveANFTransform extends PluginComponent with Transform with
TypingTransformers with CPSUtils {
// inherits abstract value `global' and class `Phase' from Transform
import global._ // the global environment
import definitions._ // standard classes and methods
import typer.atOwner // methods to type trees
/** the following two members override abstract members in Transform */
val phaseName: String = "selectiveanf"
If we use scalac -Xshow-phases, we can see the phases during the compilation process:
parser
namer
packageobjects
typer
superaccessors
pickler
refchecks
selectiveanf
liftcode
selectivecps
uncurry
......
As you can see the typer phase is applied before the selectiveAnf and selectiveCps phases. It should be confirmed that type inference occurs in the typer phase, but if this is really the case and it would make sense, it should be now clear why you can't omit the Int type on 7.tracked and 35.tracked.
Now if you are not satisfied yet, you should know that the compiler works by performing a set of transformations on "trees", which you might look, using the following options:
-Xprint: shows your scala code after a certain phase have been executed
-Xprint: -Yshow-trees shows your scala code and the trees after the phase have been executed
-YBrowse: opens a GUI to surf both.