I am trying to determine a good way to close opened resource. I am using Micrometer and some resources (all coming from micrometer) that I have added in the binderList extends AutoCloseable and overrides close method but few of them do not.
https://mvnrepository.com/artifact/io.micrometer
class MetricConfiguration extends AutoCloseable{
var binderList : List[MeterBinder]
def initializePrometheusMetricsConfiguration(): Unit = {
binderList = List (
new JvmGcMetrics(),
new JvmHeapPressureMetrics(),
new JvmMemoryMetrics()
)
init(binderList)
}
private def init(binders: List[MeterBinder]): Unit = {
Metrics.globalRegistry.config().
meterFilter(new MeterFilter {
override def map(id: Meter.Id): Meter.Id = {
id.withName("hello" + id.getName)
}
})
binders.foreach(b => b.bindTo(Metrics.globalRegistry))
}
}
override def close(): Unit = ???
I am new to scala, I know I would have done in Java by extending Closeable and implementing the method
#Override
public void close() {
this.binderList.stream()
.filter(b -> b instanceof AutoCloseable)
.map(b -> (AutoCloseable) b)
.forEach(Closeables::closeQuietly);
}
Your Java close method translates nicely to Scala:
binderList
.collect { case b: AutoCloseable => b }
.foreach(Closeables.closeQuietly)
Related
I need to write two functions to get the output format and the output index for file conversion. As part of this, I wrote a TransformSettings class for these methods and set the default value. And in the transformer class, I created a new object of TransformSettings class to get the default values for each job run. Also, I have another class called ParquetTransformer that extends Transformer where I want to change these default values. So I implemented like below.
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer{
def getTransformSettings: TransformSettings = {
new TransformSettings
}
def posttransform(table: AWSGlueDDL.Table):Dateframe ={
val indexAccess = getTransformSettings.getOuputIndex(table: AWSGlueDDL.Table)
........
}
}
class ParquetTransformer extends Transformer{
override def getTransformSettings: TransformSettings = {
val transformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
}
}
Is there a way to avoid creating a brand new object of TransformSettings in Transfomer class every time this is called?
Also is there a way to rewrite the code using Scala value class?
As #Dima proposed in the comments try to make TransformSettings a field / constructor parameter (a val) in the class Transformer and instantiate them outside
class TransformSettings{
def getOuputFormat: String = {
"orc"
}
def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("orc.column.index.access")
}
}
class Transformer(val transformSettings: TransformSettings) {
def posttransform(table: AWSGlueDDL.Table): DataFrame ={
val indexAccess = transformSettings.getOuputIndex(table: AWSGlueDDL.Table)
???
}
}
val parquetTransformSettings = new TransformSettings {
override def getOuputFormat: String = {
"parquet"
}
override def getOuputIndex(table: AWSGlueDDL.Table): Option[String] = {
table.StorageDescriptor.SerdeInfo.Parameters.get("parquet.column.index.access")
}
}
class ParquetTransformer extends Transformer(parquetTransformSettings)
You don't seem to need value classes (... extends AnyVal) now. They are more about unboxing, not about life-cycle management. TransformSettings and Transformer can't be value classes because they are not final (you're extending them in class ParquetTransformer extends Transformer... and new TransformSettings { ... }). By the way, value classes have many limatations
https://failex.blogspot.com/2017/04/the-high-cost-of-anyval-subclasses.html
https://github.com/scala/bug/issues/12271
Besides value classes, there are scala-newtype library in Scala 2 and opaque types in Scala 3.
trait paymentTasks{
def calculatePayment()
def getData()
}
class paymentcalculator{
override def calculatePayment() ={
//Implementation }
}
override defGetData() ={
}
}
How do I call the method to keep the code less dependent and don't want to use new keyword
class MyApp{
val payment = new paymentcalculator
//I don't want to instantiate like this wants this to be decoupled
}
Please suggest how do I implement this ,also new Changes in future can be done without changing existing code.Also,suggest if doing this functionally would be more effective
If you want compile-time selection then just put the code into an object
trait PaymentTasks {
def calculatePayment()
def getData()
}
object PaymentCalculator extends PaymentTasks {
def calculatePayment() = ???
def getData() = ???
}
class MyApp {
val payment = PaymentCalculator.calculatePayment()
}
If you want run-time selection than the best way is to use dependency injection and create the payment object at the top level and pass it down to the code that requires it.
object SimplePaymentCalculator extends PaymentTasks
object ComplexPaymentCalculator extends PaymentTasks
class MyApp {
def myImplementation(tasks: PaymentTasks) = {
val payment = tasks.calculatePayment()
}
val payment =
if (???) {
SimplePaymentCalculator
} else {
ComplexPaymentCalculator
}
myImplementation(payment)
}
I have a graph that reads from sqs, writes to another system and then deletes from sqs. In order to delete from sqs i need a receipt handle on the SqsMessage object
In the case of Http flows the signature of the flow allows me to say which type gets emitted downstream from the flow,
Flow[(HttpRequest, T), (Try[HttpResponse], T), HostConnectionPool]
In this case i can set T to SqsMessage and i still have all the data i need.
However some connectors e.g google cloud pub sub emits a completely useless (to me) pub sub id.
Downstream of the pub sub flow I need to be able to access the sqs message id which i had prior to the pub sub flow.
What is the best way to work around this without rewriting the pub sub connector
I conceptually want something a bit like this:
Flow[SqsMessage] //i have my data at this point
within(
.map(toPubSubMessage)
.via(pubSub))
... from here i have the same type i had before within however it still behaves like a linear graph with back pressure etc
You can use PassThrough integration pattern.
As example of usage look on akka-streams-kafka -> Class akka.kafka.scaladsl.Producer -> Mehtod def flow[K, V, PassThrough]
So just implement your own Stage with PassThrough element, example inakka.kafka.internal.ProducerStage[K, V, PassThrough]
package my.package
import java.util.concurrent.atomic.AtomicInteger
import scala.concurrent.Future
import scala.util.{Failure, Success, Try}
import akka.stream._
import akka.stream.ActorAttributes.SupervisionStrategy
import akka.stream.stage._
final case class Message[V, PassThrough](record: V, passThrough: PassThrough)
final case class Result[R, PassThrough](result: R, message: PassThrough)
class PathThroughStage[R, V, PassThrough]
extends GraphStage[FlowShape[Message[V, PassThrough], Future[Result[R, PassThrough]]]] {
private val in = Inlet[Message[V, PassThrough]]("messages")
private val out = Outlet[Result[R, PassThrough]]("result")
override val shape = FlowShape(in, out)
override protected def createLogic(inheritedAttributes: Attributes) = {
val logic = new GraphStageLogic(shape) with StageLogging {
lazy val decider = inheritedAttributes.get[SupervisionStrategy]
.map(_.decider)
.getOrElse(Supervision.stoppingDecider)
val awaitingConfirmation = new AtomicInteger(0)
#volatile var inIsClosed = false
var completionState: Option[Try[Unit]] = None
override protected def logSource: Class[_] = classOf[PathThroughStage[R, V, PassThrough]]
def checkForCompletion() = {
if (isClosed(in) && awaitingConfirmation.get == 0) {
completionState match {
case Some(Success(_)) => completeStage()
case Some(Failure(ex)) => failStage(ex)
case None => failStage(new IllegalStateException("Stage completed, but there is no info about status"))
}
}
}
val checkForCompletionCB = getAsyncCallback[Unit] { _ =>
checkForCompletion()
}
val failStageCb = getAsyncCallback[Throwable] { ex =>
failStage(ex)
}
setHandler(out, new OutHandler {
override def onPull() = {
tryPull(in)
}
})
setHandler(in, new InHandler {
override def onPush() = {
val msg = grab(in)
val f = Future[Result[R, PassThrough]] {
try {
Result(// TODO YOUR logic
msg.record,
msg.passThrough)
} catch {
case exception: Exception =>
decider(exception) match {
case Supervision.Stop =>
failStageCb.invoke(exception)
case _ =>
Result(exception, msg.passThrough)
}
}
if (awaitingConfirmation.decrementAndGet() == 0 && inIsClosed) checkForCompletionCB.invoke(())
}
awaitingConfirmation.incrementAndGet()
push(out, f)
}
override def onUpstreamFinish() = {
inIsClosed = true
completionState = Some(Success(()))
checkForCompletion()
}
override def onUpstreamFailure(ex: Throwable) = {
inIsClosed = true
completionState = Some(Failure(ex))
checkForCompletion()
}
})
override def postStop() = {
log.debug("Stage completed")
super.postStop()
}
}
logic
}
}
I have a CategoryRepository class which implements several methods such that saves a category to database.
I also have an object Product that contains a list of categories.
What I want to do is to trigger an event that the Product object will listen to, and will update the product itself with the new data of its category.
In C# I know I can use delegates but I don't know if I can do something like this in Scala.
I don't want the CategoryRepository class to know the class Product so I won't call some method in Product that will update it through CategoryRepository.
My CategoryRepository class:
trait CategoryRepositoryComponentImpl extends CategoryRepositoryComponent {
val categoryRepository = new categoryRepositoryImpl
class CategoryRepositoryImpl extends CategoryRepository {
val dbRepository = CategoryDbRepository
def updateAttribute(id:String, request:UpdateCategoryItem): String = {
val cat = dbRepository.get(id)
cat.update(request)
dbRepository.save(cat)
}
}
}
The product repository looks the same as this category's repository.
Now I want to add a line after dbRepository.save(cat) that will trigger an event that will call the updateProduct() function within ProductRepository.
Please give an implementation example.
Thanks.
a (not so) basic implementation for an update channel based on events.
I took care to only generalize to the bare bone, as to give a hint to future evolution and code reuse.
Base infrastructure
We introduce an updates channel
//Listens for updates to A's and notifies interested listeners
class UpdateChannel[A] {
//simplified register
var listenMap: Map[A, Listening[A]] = Map()
//update call
def apply(a: A): Unit = listenMap.get(a).foreach(_.event(a))
//update call
def registerFor(a: value, listen: Listening[A]) = listenMap += (a, listen)
}
and a generic listener interested in corrisponding updates
//Listens to changes for type A
trait Listening[A] {
def event(upd: A): Unit
}
Application
Now we adapt the Repo Component to inject the channel
trait CategoryRepositoryComponentImpl extends CategoryRepositoryComponent {
val categoryRepository = new categoryRepositoryImpl
/************** NEW CODE HERE **************
* define a channel to send category updates
*******************************************/
def updateChannel: UpdateChannel[Category]
class CategoryRepositoryImpl extends CategoryRepository {
val dbRepository = CategoryDbRepository
def updateAttribute(id:String, request:UpdateCategoryItem): String = {
val cat = dbRepository.get(id)
cat.update(request)
dbRepository.save(cat)
//send the update to the channel
updateChannel(cat) //***************** << AND HERE
}
}
}
We also need to enable the product to event listening
//product must be listening to category updates
class Product(val category: Category) extends Listening[Category] {
def event(cat: Category) = ??? //strut your stuff here
...business stuff here too
}
Finally, here we put the ingredients together
//put the pieces together
def wireup() = {
//the channel
val catChan: UpdateChannel[Category] = new UpdateChannel[Category]
//the Repository component wired to the channel
val catRep = new CategoryRepositoryComponentImpl {
val updateChannel = catChan
}
//a nice cat
val myCat: Category = ???
//a nice prod with her nice cat
val p: Product = new Product(myCat)
//prod wants to know what happens to her cat
catChan.registerFor(myCat, p)
}
Remarks
we can make the Product independent of the whole framework by using refinement types
val product = new Product(myCat) with Listening[Category] {
def event(cat: Category) = ??? //strut your stuff here
}
a different solution would be to avoiding all the wirings and simply register a list of update closures in the RepositoryComponent
trait CategoryRepositoryComponentImpl extends CategoryRepositoryComponent {
val categoryRepository = new categoryRepositoryImpl
//public listeners, they should be encapsulated
var categoryUpdates: Seq[Category => Unit]
[...]
def updateAttribute(id:String, request:UpdateCategoryItem): String = {
val cat = dbRepository.get(id)
cat.update(request)
dbRepository.save(cat)
//send the update to the channel
categoryUpdates.foreach(_.apply(cat))
}
}
}
and the product needs only to add his own update function
catRep.categoryUpdates +:= (cat) => p.event(cat)
I think this would be more or less how it's done in C# with events and delegates.
When you define Event like this you'd have to use tuples (Event[(Any,EventArgs)] for instance) if you want to pass more than one parameter to the event listeners.
class Event[Arg] {
type L = Arg => Unit
private val listeners = scala.collection.mutable.ListBuffer.empty[L]
def +=(listener: L) {
listeners.append(listener)
}
def apply(arg: Arg) {
listeners.foreach(_(arg))
}
}
class CategoryRepository {
val event = new Event[String]
def fireEvent(data: String) {
event(data)
}
}
object Product {
def update(data: String) {
println(s"updating: $data")
}
}
object Main extends App {
val repo = new CategoryRepository()
repo.event += Product.update
repo.fireEvent("new data")
}
I'm trying to find a cleaner alternative (that is idiomatic to Scala) to the kind of thing you see with data-binding in WPF/silverlight data-binding - that is, implementing INotifyPropertyChanged. First, some background:
In .Net WPF or silverlight applications, you have the concept of two-way data-binding (that is, binding the value of some element of the UI to a .net property of the DataContext in such a way that changes to the UI element affect the property, and vise versa. One way to enable this is to implement the INotifyPropertyChanged interface in your DataContext. Unfortunately, this introduces a lot of boilerplate code for any property you add to the "ModelView" type. Here is how it might look in Scala:
trait IDrawable extends INotifyPropertyChanged
{
protected var drawOrder : Int = 0
def DrawOrder : Int = drawOrder
def DrawOrder_=(value : Int) {
if(drawOrder != value) {
drawOrder = value
OnPropertyChanged("DrawOrder")
}
}
protected var visible : Boolean = true
def Visible : Boolean = visible
def Visible_=(value: Boolean) = {
if(visible != value) {
visible = value
OnPropertyChanged("Visible")
}
}
def Mutate() : Unit = {
if(Visible) {
DrawOrder += 1 // Should trigger the PropertyChanged "Event" of INotifyPropertyChanged trait
}
}
}
For the sake of space, let's assume the INotifyPropertyChanged type is a trait that manages a list of callbacks of type (AnyRef, String) => Unit, and that OnPropertyChanged is a method that invokes all those callbacks, passing "this" as the AnyRef, and the passed-in String). This would just be an event in C#.
You can immediately see the problem: that's a ton of boilerplate code for just two properties. I've always wanted to write something like this instead:
trait IDrawable
{
val Visible = new ObservableProperty[Boolean]('Visible, true)
val DrawOrder = new ObservableProperty[Int]('DrawOrder, 0)
def Mutate() : Unit = {
if(Visible) {
DrawOrder += 1 // Should trigger the PropertyChanged "Event" of ObservableProperty class
}
}
}
I know that I can easily write it like this, if ObservableProperty[T] has Value/Value_= methods (this is the method I'm using now):
trait IDrawable {
// on a side note, is there some way to get a Symbol representing the Visible field
// on the following line, instead of hard-coding it in the ObservableProperty
// constructor?
val Visible = new ObservableProperty[Boolean]('Visible, true)
val DrawOrder = new ObservableProperty[Int]('DrawOrder, 0)
def Mutate() : Unit = {
if(Visible.Value) {
DrawOrder.Value += 1
}
}
}
// given this implementation of ObservableProperty[T] in my library
// note: IEvent, Event, and EventArgs are classes in my library for
// handling lists of callbacks - they work similarly to events in C#
class PropertyChangedEventArgs(val PropertyName: Symbol) extends EventArgs("")
class ObservableProperty[T](val PropertyName: Symbol, private var value: T) {
protected val propertyChanged = new Event[PropertyChangedEventArgs]
def PropertyChanged: IEvent[PropertyChangedEventArgs] = propertyChanged
def Value = value;
def Value_=(value: T) {
if(this.value != value) {
this.value = value
propertyChanged(this, new PropertyChangedEventArgs(PropertyName))
}
}
}
But is there any way to implement the first version using implicits or some other feature/idiom of Scala to make ObservableProperty instances function as if they were regular "properties" in scala, without needing to call the Value methods? The only other thing I can think of is something like this, which is more verbose than either of the above two versions, but is still less verbose than the original:
trait IDrawable {
private val visible = new ObservableProperty[Boolean]('Visible, false)
def Visible = visible.Value
def Visible_=(value: Boolean): Unit = { visible.Value = value }
private val drawOrder = new ObservableProperty[Int]('DrawOrder, 0)
def DrawOrder = drawOrder.Value
def DrawOrder_=(value: Int): Unit = { drawOrder.Value = value }
def Mutate() : Unit = {
if(Visible) {
DrawOrder += 1
}
}
}
I couldn't claim that this is a canonical property change framework in Scala, but I've used a class like this before:
abstract class Notifier[T,U](t0: T) {
import java.util.concurrent.atomic.AtomicReference
import scala.actors.OutputChannel
type OCUT = OutputChannel[(U,AtomicReference[T])]
val data = new AtomicReference[T](t0)
def id: U
protected var callbacks = Nil:List[T => Unit]
protected var listeners = Nil:List[OCUT]
def apply() = data.get
def update(t: T) {
val told = data.getAndSet(t)
if (t != told) {
callbacks.foreach(_(t))
listeners.foreach(_ ! (id,data))
}
}
def attend(f: T=>Unit) { callbacks ::= f }
def attend(oc: OCUT) { listeners ::= oc }
def ignore(f: T=>Unit) { callbacks = callbacks.filter(_ != f) }
def ignore(oc: OCUT) { listeners = listeners.filter(_ != oc) }
}
The motivation for creating this class was that I wanted a flexible thread-safe way to react to changes, which this provides (as it delivers both callbacks and can push messages to actors).
It seems to me--unless I'm misunderstanding exactly what you want because I haven't had occasion to learn the WPF/Silverlight stuff--that this can implement everything you want and more.
For example,
class IDrawable extends SomethingWithOnPropertyChanged {
val drawOrder = new Notifier[Int,Symbol](0) { def id = 'DrawOrder }
val visible = new Notifier[Boolean,Symbol](false) { def id = 'Visible }
drawOrder.attend((i:Int) => OnPropertyChanged(drawOrder.id))
def mutate {
if (visible()) drawOrder() += 1
}
}
should be roughly equivalent to what you want. (Again, I'm not sure how flexible you want this to be; you could create a set of symbol -> notifier mappings that you would look up with an apply method so the target would have an easier time of doing something when it gets the DrawOrder symbol.)
The only significant difference from your usage is that the Notifier uses its apply/update methods to save boilerplate; you don't have to write def x and def x_= methods every time, but you do have to use () for access.