How to write a cache loader for Caffeine LoadingCache in "Scala" for "refreshAfterWrite" to work - scala

Scala application use case:
We have a Scala based that module reads the data from global cache (Redis) and save the same into local cache(Caffeine LoadingCache). As we want this data to be refreshed asynchronously, we are using LoadingCache with refreshAfterWrite duration set to refresh window of 2.second.
Question:
Not question but need help with the following code that is giving warning and also compile time errors
Warning: For build method, it gives warning as Implements member load in CacheLoader (com.github.benmanes.caffeine.cache)
Compile time error 1: type arguments [Int,redisToCaffeine.DataObject] conform to the bounds of none of the overloaded alternatives of value build: [K1 <: Object, V1 <: Object](x$1: com.github.benmanes.caffeine.cache.CacheLoader[_ >: K1, V1])com.github.benmanes.caffeine.cache.LoadingCache[K1,V1] <and> [K1 <: Object, V1 <: Object]()com.github.benmanes.caffeine.cache.Cache[K1,V1] .build[Int, DataObject](key => loader(key))
Compile time error 2: wrong number of type parameters for overloaded method value build with alternatives: [K1 <: Object, V1 <: Object](x$1: com.github.benmanes.caffeine.cache.CacheLoader[_ >: K1, V1])com.github.benmanes.caffeine.cache.LoadingCache[K1,V1] <and> [K1 <: Object, V1 <: Object]()com.github.benmanes.caffeine.cache.Cache[K1,V1] .build[Int, DataObject](key => loader(key))
Code:
package redisToCaffeine
import scala.concurrent.duration._
import com.github.benmanes.caffeine.cache.{ CacheLoader, Caffeine, LoadingCache }
import com.twitter.finagle.stats.InMemoryStatsReceiver
import javax.annotation.Nullable
import redisToCaffeine.CacheImplicits.StatsReportingCaffeineCache
class LocalDealService {
class DataObject(data: String) {
override def toString: String = {
"[ 'data': '" + this.data + "' ]"
}
}
val defaultCacheExpireDuration: FiniteDuration = 2.second
val stats: InMemoryStatsReceiver = new InMemoryStatsReceiver
// loader helper
#Nullable
#throws[Exception]
protected def loader(key: Int): DataObject = { // this will replace to read the data from Redis Cache
new DataObject(s"LOADER_HELPER_$key")
}
def initCache(maximumSize: Int = 5): LoadingCache[Int, DataObject] = {
Caffeine
.newBuilder()
.maximumSize(maximumSize)
.refreshAfterWrite(defaultCacheExpireDuration.length, defaultCacheExpireDuration.unit)
.recordStats()
.build[Int, DataObject](key => loader(key))
.enableCacheStatsReporting("deal-service", stats)
}
}
I'm new to Scala and Caffeine both so not sure what I'm be doing wrong; I tried different ways mentioned here and here to write loader but nothing worked (mainly they are in Java). Little research around Scala bounds doesn't helped here any way. Kindly help.

Depends on which Scala version is being used here.
Although Scala (2.12 and later) Functions support conversions to Java SAM, these are done only when explicitly required. So if you are using Scala 2.12 or later, you can explicitly ask the compiler to convert the Scala function to SAM,
Also, don't use Int as key for the cache. Although it will work because of implicit conversions to Integer, that is not a good practice.
def initCache(maximumSize: Int = 5): LoadingCache[Integer, DataObject] = {
Caffeine
.newBuilder()
.maximumSize(maximumSize)
.refreshAfterWrite(defaultCacheExpireDuration.length, defaultCacheExpireDuration.unit)
.recordStats()
.build[Integer, DataObject]((key => loader(key)): CacheLoader[Integer, DataObject])
.enableCacheStatsReporting("deal-service", stats)
}
And if you are dealing with older Scala versions, then just forget that SAM exists and do it old style.
def initCache(maximumSize: Int = 5): LoadingCache[Integer, DataObject] = {
Caffeine
.newBuilder()
.maximumSize(maximumSize)
.refreshAfterWrite(defaultCacheExpireDuration.length, defaultCacheExpireDuration.unit)
.recordStats()
.build[Int, DataObject](new CacheLoader[Integer, DataObject] {
override def load(key: Integer): DataObject = loader(key)
})
.enableCacheStatsReporting("deal-service", stats)
}

Related

In Scala 3, is it possible to use declared type of an object in runtime?

In Scala 2, most generic type information of an object is erased at runtime. At this moment, all 3 binary execution environments (JVM, javascript, and LLVM) abide this behaviour, they only differs in minor details in metadata formats.
In a rare case if it incurs critical data loss, or if it triggers a rare binary error. A mechanism should be used to preserve declared type information in an adjoint data structure. The following code gave a short example of such data structure in Scala 2:
import scala.reflect.runtime.universe
import scala.collection.concurrent.TrieMap
import scala.language.implicitConversions
case class Unerase[T](self: T)(
implicit
ev: universe.TypeTag[T]
) {
import Unerase._
cache += {
val inMemoryId = System.identityHashCode(this)
inMemoryId -> ev
}
}
object Unerase {
lazy val cache = TrieMap.empty[Int, universe.TypeTag[_]]
def get[T](v: T): Option[universe.TypeTag[T]] = {
val inMemoryId = System.identityHashCode(v)
cache.get(inMemoryId).map { tt =>
tt.asInstanceOf[universe.TypeTag[T]]
}
}
implicit def unbox[T](v: Unerase[T]): T = v.self
implicit def box[T](v: T)(
implicit
ev: universe.TypeTag[T]
): Unerase[T] = Unerase(v)
}
Any variable declared as type Unerase[T] instead of T will be guaranteed to have its full declared type visible at runtime. Unfortunately, this example no longer works in Scala 3:
implicitly[TypeTag[Int]] // works in Scala 2
summon[Type[Int]] // doesn't work in Scala 3: No given instance of type quoted.Quotes was found for parameter x$1 ...
Is there a mechanism that I can use to implement the same mechanism to fully mitigate type erasure?

Is there anyway, in Scala, to get the Singleton type of something from the more general type?

I have a situation where I'm trying to use implicit resolution on a singleton type. This works perfectly fine if I know that singleton type at compile time:
object Main {
type SS = String with Singleton
trait Entry[S <: SS] {
type out
val value: out
}
implicit val e1 = new Entry["S"] {
type out = Int
val value = 3
}
implicit val e2 = new Entry["T"] {
type out = String
val value = "ABC"
}
def resolve[X <: SS](s: X)(implicit x: Entry[X]): x.value.type = {
x.value
}
def main(args: Array[String]): Unit = {
resolve("S") //implicit found! No problem
}
}
However, if I don't know this type at compile time, then I run into issues.
def main(args: Array[String]): Unit = {
val string = StdIn.readLine()
resolve(string) //Can't find implicit because it doesn't know the singleton type at runtime.
}
Is there anyway I can get around this? Maybe some method that takes a String and returns the singleton type of that string?
def getSingletonType[T <: SS](string: String): T = ???
Then maybe I could do
def main(args: Array[String]): Unit = {
val string = StdIn.readLine()
resolve(getSingletonType(string))
}
Or is this just not possible? Maybe you can only do this sort of thing if you know all of the information at compile-time?
If you knew about all possible implementations of Entry in compile time - which would be possible only if it was sealed - then you could use a macro to create a map/partial function String -> Entry[_].
Since this is open to extending, I'm afraid at best some runtime reflection would have to scan the whole classpath to find all possible implementations.
But even then you would have to embed this String literal somehow into each implementations because JVM bytecode knows nothing about mappings between singleton types and implementations - only Scala compiler does. And then use that to find if among all implementations there is one (and exactly one) that matches your value - in case of implicits if there are two of them at once in the same scope compilation would fail, but you can have more than one implementation as long as the don't appear together in the same scope. Runtime reflection would be global so it wouldn't be able to avoid conflicts.
So no, no good solution for making this compile-time dispatch dynamic. You could create such dispatch yourself by e.g. writing a Map[String, Entry[_]] yourself and using get function to handle missing pices.
Normally implicits are resolved at compile time. But val string = StdIn.readLine() becomes known at runtime only. Principally, you can postpone implicit resolution till runtime but you'll be able to apply the results of such resolution at runtime only, not at compile time (static types etc.)
object Entry {
implicit val e1 = ...
implicit val e2 = ...
}
import scala.reflect.runtime.universe._
import scala.reflect.runtime
import scala.tools.reflect.ToolBox
val toolbox = ToolBox(runtime.currentMirror).mkToolBox()
def resolve(s: String): Any = {
val typ = appliedType(
typeOf[Entry[_]].typeConstructor,
internal.constantType(Constant(s))
)
val instanceTree = toolbox.inferImplicitValue(typ, silent = false)
val instance = toolbox.eval(toolbox.untypecheck(instanceTree)).asInstanceOf[Entry[_]]
instance.value
}
resolve("S") // 3
val string = StdIn.readLine()
resolve(string)
// 3 if you enter S
// ABC if you enter T
// "scala.tools.reflect.ToolBoxError: implicit search has failed" otherwise
Please notice that I put implicits into the companion object of type class in order to make them available in the implicit scope and therefore in the toolbox scope. Otherwise the code should be modified slightly:
object EntryImplicits {
implicit val e1 = ...
implicit val e2 = ...
}
// val instanceTree = toolbox.inferImplicitValue(typ, silent = false)
// should be replaced with
val instanceTree =
q"""
import path.to.EntryImplicits._
implicitly[$typ]
"""
In your code import path.to.EntryImplicits._ is import Main._.
Load Dataset from Dynamically generated Case Class

sbt could not find implicit value for parameter valName

I'm using sbt to build some of the riscv boom from the source code, but the sbt complains that it "could not find implicit value for parameter valName: : freechips.rocketchip.diplomacy.ValName". The detailed error message are as below:
[error] F:\hiMCU\my_proj\src\main\scala\freechips\rocketchip\tile\BaseTile.scala:170:42: could not find implicit value for parameter valName: freechips.rocketchip.diplomacy.ValName
[error] Error occurred in an application involving default arguments.
[error] protected val tlMasterXbar = LazyModule(new TLXbar)
The code where sbt complains is as below:
abstract class BaseTile private (val crossing: ClockCrossingType, q: Parameters)
extends LazyModule()(q)
with CrossesToOnlyOneClockDomain
with HasNonDiplomaticTileParameters
{
// Public constructor alters Parameters to supply some legacy compatibility keys
def this(tileParams: TileParams, crossing: ClockCrossingType, lookup: LookupByHartIdImpl, p: Parameters) = {
this(crossing, p.alterMap(Map(
TileKey -> tileParams,
TileVisibilityNodeKey -> TLEphemeralNode()(ValName("tile_master")),
LookupByHartId -> lookup
)))
}
def module: BaseTileModuleImp[BaseTile]
def masterNode: TLOutwardNode
def slaveNode: TLInwardNode
def intInwardNode: IntInwardNode // Interrupts to the core from external devices
def intOutwardNode: IntOutwardNode // Interrupts from tile-internal devices (e.g. BEU)
def haltNode: IntOutwardNode // Unrecoverable error has occurred; suggest reset
def ceaseNode: IntOutwardNode // Tile has ceased to retire instructions
def wfiNode: IntOutwardNode // Tile is waiting for an interrupt
protected val tlOtherMastersNode = TLIdentityNode()
protected val tlSlaveXbar = LazyModule(new TLXbar)
protected val tlMasterXbar = LazyModule(new TLXbar)
protected val intXbar = LazyModule(new IntXbar)
....
}
The LazyModule object code is as below:
object LazyModule
{
protected[diplomacy] var scope: Option[LazyModule] = None
private var index = 0
def apply[T <: LazyModule](bc: T)(implicit valName: ValName, sourceInfo: SourceInfo): T = {
// Make sure the user put LazyModule around modules in the correct order
// If this require fails, probably some grandchild was missing a LazyModule
// ... or you applied LazyModule twice
require (scope.isDefined, s"LazyModule() applied to ${bc.name} twice ${sourceLine(sourceInfo)}")
require (scope.get eq bc, s"LazyModule() applied to ${bc.name} before ${scope.get.name} ${sourceLine(sourceInfo)}")
scope = bc.parent
bc.info = sourceInfo
if (!bc.suggestedNameVar.isDefined) bc.suggestName(valName.name)
bc
}
}
I think the sbt should find some val of type freechips.rocketchip.diplomacy.ValName, but it didn't find such kind of val.
You need to have an object of type ValName in the scope where your LazyModules are instantiated:
implicit val valName = ValName("MyXbars")
For more details on Scala's implicit please see https://docs.scala-lang.org/tutorials/tour/implicit-parameters.html.html
You generally shouldn't need to manually create a ValName, the Scala compiler can materialize them automatically based on the name of the val you're assigning the LazyModule to. You didn't include your imports in your example, but can you try importing ValName?
import freechips.rocketchip.diplomacy.ValName
In most of rocket-chip code, this is imported via wildcard importing everything in the diplomacy package
import freechips.rocketchip.diplomacy._

Recursively wrapping method invocations with compiler plugins/macros

OUTLINE
I have an API that looks something like this:
package com.example
object ExternalApi {
def create[T <: SpecialElement](elem: T): TypeConstructor[T] =
TypeConstructor(elem)
def create1[T <: SpecialElement](elem: T): TypeConstructor[T] =
TypeConstructor(elem)
def create2[T <: SpecialElement](elem: T): TypeConstructor[T] =
TypeConstructor(elem)
//...
}
object MyApi {
def process[T <: TypeConstructor[_ <: SpecialElement]](
l: T,
metadata: List[String]): T = {
println("I've been called!")
//do some interesting stuff with the List's type parameter here
l
}
}
case class TypeConstructor[E](elem: E)
trait SpecialElement
The ExternalApi (which is actually external to my lib, so no modifying that) has a series of calls that I'd like to automatically wrap with MyApi.process calls, with the metadata argument derived from the final type of T.
To illustrate, the calls to be wrapped could have any form, including nested calls, and calls within other AST subtree types (such as Blocks), e.g. :
package com.example.test
import com.example.{ExternalApi, SpecialElement}
object ApiPluginTest extends App {
//one possible form
val targetList = ExternalApi.create(Blah("name"))
//and another
ExternalApi.create2(ExternalApi.create1(Blah("sth")).elem)
//and yet another
val t = {
val sub1 = ExternalApi.create(Blah("anything"))
val sub2 = ExternalApi.create1(sub1.elem)
sub2
}
}
case class Blah(name: String) extends SpecialElement
Since compiler plugins handle matching structures within ASTs recursively "for free", I've decided to go with them.
However, due to the fact that I need to match a specific type signature, the plugin follows the typer phase.
Here's the code of the PluginComponent:
package com.example.plugin
import com.example.{SpecialElement, TypeConstructor}
import scala.tools.nsc.Global
import scala.tools.nsc.plugins.PluginComponent
import scala.tools.nsc.transform.Transform
class WrapInApiCallComponent(val global: Global)
extends PluginComponent
with Transform {
protected def newTransformer(unit: global.CompilationUnit) =
WrapInApiCallTransformer
val runsAfter: List[String] = List("typer") //since we need the type
val phaseName: String = WrapInApiCallComponent.Name
import global._
object WrapInApiCallTransformer extends Transformer {
override def transform(tree: global.Tree) = {
val transformed = super.transform(tree)
transformed match {
case call # Apply(_, _) =>
if (call.tpe != null && call.tpe.finalResultType <:< typeOf[
TypeConstructor[_ <: SpecialElement]]) {
println(s"Found relevant call $call")
val typeArguments = call.tpe.typeArgs.map(_.toString).toList
val listSymbOf = symbolOf[List.type]
val wrappedFuncSecondArgument =
q"$listSymbOf.apply(..$typeArguments)"
val apiObjSymbol = symbolOf[com.example.MyApi.type]
val wrappedCall =
q"$apiObjSymbol.process[${call.tpe.finalResultType}]($call, $wrappedFuncSecondArgument)"
//explicit typing, otherwise later phases throw NPEs etc.
val ret = typer.typed(wrappedCall)
println(showRaw(ret))
println("----")
ret
} else {
call
}
case _ => transformed
}
}
}
}
object WrapInApiCallComponent {
val Name = "api_embed_component"
}
This seems to resolve identifiers, as well as types, correctly, outputting e.g.:
Apply(TypeApply(Select(TypeTree().setOriginal(Ident(com.example.MyApi)), TermName("process")), List(TypeTree())), List(Apply(TypeApply(Select(Select(Select(Ident(com), com.example), com.example.MyApi), TermName("create")), List(TypeTree())), List(Apply(Select(Ident(com.example.test.Blah), TermName("apply")), List(Literal(Constant("name")))))), Apply(TypeApply(Select(TypeTree().setOriginal(Ident(scala.collection.immutable.List)), TermName("apply")), List(TypeTree())), List(Literal(Constant("com.example.test.Blah"))))))
Unfortunately, I get an error during compilation starting with:
scala.reflect.internal.FatalError:
[error]
[error] Unexpected tree in genLoad: com.example.MyApi.type/class scala.reflect.internal.Trees$TypeTree at: RangePosition([projectpath]/testPluginAutoWrap/compiler_plugin_test/src/main/scala/com/example/test/ApiPluginTest.scala, 108, 112, 112)
[error] while compiling: [projectpath]/testPluginAutoWrap/compiler_plugin_test/src/main/scala/com/example/test/ApiPluginTest.scala
[error] during phase: jvm
[error] library version: version 2.12.4
[error] compiler version: version 2.12.4
[error] reconstructed args: -Xlog-implicits -classpath [classpath here]
[error]
[error] last tree to typer: TypeTree(class String)
[error] tree position: line 23 of [projectpath]/testPluginAutoWrap/compiler_plugin_test/src/main/scala/com/example/test/ApiPluginTest.scala
QUESTION
It looks I'm screwing something up with type definitions, but what is it?
Specifically:
How do I correctly wrap every ExternalApi.createX call with an MyApi.process call, constrained by the requirements provided above?
NOTES
Given the amount of boilerplate required, I've set up a complete example project. It's available here.
The answer does not have to define a compiler plugin. If you're able to cover all the relevant calls with a macro, that is fine as well.
Originally the wrapped call was to something like: def process[T <: TypeConstructor[_ <: SpecialElement] : TypeTag](l: T): T, the setup here is actually a workaround. So if you are able to generate a wrapped call of this type, i.e. one that includes a runtime TypeTag[T], that's fine as well.

Trouble using Implicit Ordered with PriorityQueue (Scala)

I'm trying to create a data structure that has a PriorityQueue in it. I've succeeded in making a non-generic version of it. I can tell it works because it solves the A.I. problem I have.
Here is a snippet of it:
class ProntoPriorityQueue { //TODO make generic
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
}
val hashSet = new HashSet[Node]
val priorityQueue = new PriorityQueue[Node]()
...
I'm trying to make it generic, but if I use this version it stops solving the problem:
class PQ[T <% Ordered[T]] {
//[T]()(implicit val ord: T => Ordered[T]) {
//[T]()(implicit val ord: Ordering[T] {
val hashSet = new HashSet[T]
val priorityQueue = new PriorityQueue[T]
...
I've also tried what's commented out instead of using [T <% Ordered[T]]
Here is the code that calls PQ:
//the following def is commented out while using ProntoPriorityQueue
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
} //I've also tried making this return an Ordering[Node]
val frontier = new PQ[Node] //new ProntoPriorityQueue
//have also tried (not together):
val frontier = new PQ[Node]()(orderedNode)
I've also tried moving the implicit def into the Node object (and importing it), but essentially the same problem.
What am I doing wrong in the generic version? Where should I put the implicit?
Solution
The problem was not with my implicit definition. The problem was the implicit ordering was being picked up by a Set that was automatically generating in a for(...) yield(...) statement. This caused a problem where the yielded set only contained one state.
What's wrong with simply defining an Ordering on your Node (Ordering[Node]) and using the already-generic Scala PriorityQueue?
As general rule, it's better to work with Ordering[T] than T <: Ordered[T] or T <% Ordered[T]. Conceptually, Ordered[T] is an intrinsic (inherited or implemented) property of the type itself. Notably, a type can have only one intrinsic ordering relationship defined this way. Ordering[T] is an external specification of the ordering relationship. There can any be any number of different Ordering[T].
Also, if you're not already aware, you should know that the difference between T <: U and T <% U is that while the former includes only nominal subtype relations (actual inheritance), the latter also includes the application of implicit conversions that yield a value conforming to the type bound.
So if you want to use Node <% Ordered[Node] and you don't have a compare method defined in the class, an implicit conversion will be applied every time a comparison needs to be made. Additionally, if your type has its own compare, the implicit conversion will never be applied and you'll be stuck with that "built-in" ordering.
Addendum
I'll give a few examples based on a class, call it CIString that simply encapsulates a String and implements ordering as case-invariant.
/* Here's how it would be with direct implementation of `Ordered` */
class CIString1(val s: String)
extends Ordered[CIString1]
{
private val lowerS = s.toLowerCase
def compare(other: CIString1) = lowerS.compareTo(other.lowerS)
}
/* An uninteresting, empty ordered set of CIString1
(fails without the `extends` clause) */
val os1 = TreeSet[CIString1]()
/* Here's how it would look with ordering external to `CIString2`
using an implicit conversion to `Ordered` */
class CIString2(val s: String) {
val lowerS = s.toLowerCase
}
class CIString2O(ciS: CIString2)
extends Ordered[CIString2]
{
def compare(other: CIString2) = ciS.lowerS.compareTo(other.lowerS)
}
implicit def cis2ciso(ciS: CIString2) = new CIString2O(ciS)
/* An uninteresting, empty ordered set of CIString2
(fails without the implicit conversion) */
val os2 = TreeSet[CIString2]()
/* Here's how it would look with ordering external to `CIString3`
using an `Ordering` */
class CIString3(val s: String) {
val lowerS = s.toLowerCase
}
/* The implicit object could be replaced by
a class and an implicit val of that class */
implicit
object CIString3Ordering
extends Ordering[CIString3]
{
def compare(a: CIString3, b: CIString3): Int = a.lowerS.compareTo(b.lowerS)
}
/* An uninteresting, empty ordered set of CIString3
(fails without the implicit object) */
val os3 = TreeSet[CIString3]()
Well, one possible problem is that your Ordered[Node] is not a Node:
implicit def orderedNode(node: Node): Ordered[Node] = new Ordered[Node] {
def compare(other: Node) = node.compare(other)
}
I'd try with an Ordering[Node] instead, which you say you tried but there isn't much more information about. PQ would be declared as PQ[T : Ordering].