I am trying to implement object factory design pattern in scala. However I am not able to undertand how to return the object based on condition.
I tried to return Option however it's not working as expected.
import java.io.File
import java.util.Properties
import scala.io.Source
abstract class FileSystem {
def moveFile(propFileURI: String): Unit
}
object FileSystem {
private class HDFSystem extends FileSystem{
override def moveFile(propFileURI: String): Unit = {
println(" HDFS move file")
}
}
private class S3System extends FileSystem {
override def moveFile(propFileURI: String): Unit = {
println("S3 Move File ")
}
}
// How to handle this??
def apply(propFileURI: String): Option[FileSystem] = {
val properties: Properties = new Properties()
val source = Source.fromFile( System.getProperty("user.dir")+"\\src\\main\\resources\\"+propFileURI).reader
properties.load(source)
val srcPath = properties.getProperty("srcPath")
val destPath = properties.getProperty("destPath")
var Obj = None:Option[FileSystem]
if (destPath.contains("hdfs")){
Obj = Option(new HDFSystem())
}
if (srcPath.contains("s3") && destPath.contains("s3")){
Obj = Option(new S3System())
}
Obj
}
def main(args: Array[String]): Unit = {
val obj = FileSystem("test.properties")
obj match {
case test: FileSystem => test.moveFile("test.properties")
case None => None
}
}
}
How to handle the Apply method based on condition I have mentioned? Do I really need to return option ?
There are ways to clean up your implementation, for example, usage of var could be avoided
if (destPath.contains("hdfs"))
Some(new HDFSystem())
else if (srcPath.contains("s3") && destPath.contains("s3"))
Some(new S3System())
else
None
however as far as I understand the main point of the question is
Do I really need to return Option?
This is a design decision with tradeoffs and no clear answer. Ask yourself how should the system react if .properties file is missing the appropriate configuration key:
Could we construct a meaningful default FileSystem object? In this case there is no need for Option, just return the default and continue processing.
If we cannot construct a meaningful default, is there any point in continuing? Should the system crash? In which case we might throw.
If the system can continue operating despite not being able to construct FileSystem, then we could model this information as Option or Either etc., perhaps log the event, and continue processing.
These are just some considerations to take into account. Personally, I have found that a misconfigured system is hard to recover from.
Options aren't Java's #nullables. You pattern match on Some if value is present:
val obj = FileSystem("test.properties") // Option[FileSystem]
obj match {
case Some(test) => test.moveFile("test.properties")
case None =>
}
Also in Scala almost everything is an expression so if-else, blocks, loops and functions returns their last value as the value of all. Also we should take IO errors into consideration, so Try could be better than Option:
def apply(propFileURI: String): Try[FileSystem] =
Try {
// reading properties could fail
val p = new Properties()
val source = Source.fromFile( System.getProperty("user.dir")+"\\src\\main\\resources\\"+propFileURI).reader
p.load(source)
p
}.flatMap { properties =>
// reading from properties could fail
val srcPath = properties.getProperty("srcPath")
val destPath = properties.getProperty("destPath")
if (destPath.contains("hdfs")) Success(new HDFSystem())
else if (srcPath.contains("s3") && destPath.contains("s3")) Success(new S3System())
else Failure(new Exception("Unable to recognize filesystem"))
}
def main(args: Array[String]): Unit =
FileSystem("test.properties") match {
case Success(fileSystem) => fileSystem.moveFile("test.properties")
case Failure(error) => error.printStackTrace()
}
We could convert Try at any moment to Option with .toOption and pattern-match on it. But this way we have no information on error. We could also create a FileSystemError type for storing error information and return Either[FileSystemError, FileSystem] instead of Try. Lastly, we could just throw Exception, but this way we return to Java-like practices that doesn't tell us that error can happen, and it surprises us at runtime.
What I would surely do is to rename apply - we usually expect object's apply to always return success, so if it isn't possible and we use some sort of smart constructor we should give it some other name. E.g. here we could name it
def resolveFor(propFileURI: String): Either[FileSystemError, FileSystem] = ...
so that everyone using it would know what behavior expect just from the signature.
Related
Currently my error handling is not working as I want to, this is what I am trying to do:
UserApi.insert fails, return its error and don't continue
WorkApi.insert fails, return its error after calling UserApi.delete
WorkApi.assign fails, return its error after calling WorkApi.delete and UserApi.delete
So in summary, UserApi.insert is called, if it is successfull, continue to #2. If WorkApi.insert is successfull, continue. And so on, if the current step fails, you have to reverse the previous one.
Also it is important to return the most relevant error for the Api call that failed.
If all calls were successful, I want to return the first calls value which is a User.
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.{Success, Failure}
val u1 = User("u1")
val w1 = Work("w1")
val resp = for {
insertResp <- UserApi.insert(u1)
workInsertResp <- WorkApi.insert(w1)
workAssignResp <- WorkApi.assign(w1)
} yield insertResp
println("ending...")
resp onComplete {
case Success(r) => println(r)
case Failure(t) => println(t)
}
case class User(name: String)
case class Work(name: String)
case class MyError(name: String)
object UserApi {
def insert(user: User): Future[Either[MyError, User]] =
if (user.name == "u1") Future(Right(user))
else Future(Left(MyError("UserApi.insert")))
def delete(user: User): Future[Either[MyError, String]] =
Future(Right("UserApi.delete"))
}
object WorkApi {
def insert(work: Work): Future[Either[MyError, Work]] =
if (work.name == "w1") Future(Right(work))
else Future(Left(MyError("WorkApi.insert")))
def delete(work: Work): Future[Either[MyError, Work]] = Future(Right(work))
def assign(work: Work): Future[Either[MyError, Work]] =
if (work.name == "w1") Future(Right(work))
else Future(Left(MyError("WorkApi.assign")))
}
Currently I am not sure how to bubble the correct error up.
Note: I am using scala 2.13.x, and I am not using other frameworks just plain Scala.
https://scastie.scala-lang.org/OV4Ax58qQ1S3R3fFUikSbw
I believe this does what you've described.
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
val resp: Future[Either[MyError,User]] =
UserApi.insert(u1).flatMap{_.fold(
{err => Future.successful(Left(err))}
,usr => WorkApi.insert(w1).flatMap{_.fold(
{err => UserApi.delete(u1); Future.successful(Left(err))}
, _ => WorkApi.assign(w1).map{_.fold(
{err => WorkApi.delete(w1); UserApi.delete(u1); Left(err)}
, _ => Right(usr)
)}
)}
)}
. . . //and the rest of your code
testing:
import scala.concurrent.duration.DurationInt
concurrent.Await.result(resp, 9999.millis)
//res0: Either[MyError,User] = Right(User(u1))
As you can see, your current code design is not well suited for the task you've laid out.
First of all, I would recommend against mixing Future and Either this way. Future has its own way of representing failures, and wrapping an Either in a Future means you will need to handle both the failed Future case and the Left case of Either, which can lead to some confusing code.
In the code provided in the question, there's no asynchronous execution, so using Future is redundant, and you could use Either types directly. However, I assume you want to replace these methods with ones that make actual (asynchronous) API calls, in which case you'll want to use Future without Either. Future requires that failure values extend Throwable, so this would require a change to MyError:
case class MyError(name: String) extends Exception(name)
Second, it's not a good practice to use Future.apply for non-blocking construction as in Future(Right(user)) or Future(Left(MyError("UserApi.insert"))). It isn't obvious, but this actually causes Right(user) to be scheduled as a task on the implicit execution context, rather than being computed synchronously on the current thread. It's better to use Future.successful or Future.failed to create a completed Future when the result is trivial.
With these changes, the new method implementations are:
object UserApi {
def insert(user: User): Future[User] =
if (user.name == "u1") Future.successful(user)
else Future.failed(MyError("UserApi.insert"))
def delete(user: User): Future[String] =
Future.successful("UserApi.delete")
}
object WorkApi {
def insert(work: Work): Future[Work] =
if (work.name == "w1") Future.successful(work)
else Future.failed(MyError("WorkApi.insert"))
def delete(work: Work): Future[Work] =
Future.successful(work)
def assign(work: Work): Future[Work] =
if (work.name == "w1") Future.successful(work)
else Future.failed(MyError("WorkApi.assign"))
}
I am working on a todo list app using
scalajs,
cats (free monads), and
scalajs-react
When I am using a simple model like the code below, everything works like expected.
class TodoModel() {
private object State {
var todos = Seq.empty[Todo]
def mod(f: Seq[Todo] => Seq[Todo]): Callback = {
val newTodos = f(todos)
Callback(todos = newTodos)
}
}
def add(t: Todo): Callback = State.mod(_ :+ t)
def todos: Seq[Todo] = State.todos
}
Once I use the free monads from cats, I have an odd behaviour. The first click always inserts two todo entries. Every click afterwards works like expected. See the pictures below.
What is wrong here?
import cats.free.Free
import cats.free.Free.liftF
import japgolly.scalajs.react._
import japgolly.scalajs.react.vdom.html_<^._
import org.scalajs.dom
case class Todo(text: String)
sealed trait TodoModelOp[A]
case class Add(todo: Todo) extends TodoModelOp[Unit]
case class Todos() extends TodoModelOp[Seq[Todo]]
object FreeTodoModelOps {
// type alias for lifted TodoModelOp
type TodoModelOpF[A] = Free[TodoModelOp, A]
def add(Todo: Todo): TodoModelOpF[Unit] = liftF[TodoModelOp, Unit](Add(Todo))
def todos: TodoModelOpF[Seq[Todo]] = liftF[TodoModelOp, Seq[Todo]](Todos())
}
object StateInterpreter {
import cats.arrow.FunctionK
import cats.{ Id, ~> }
val interpet: TodoModelOp ~> Id = new (TodoModelOp ~> Id) {
val todos = scala.collection.mutable.ArrayBuffer.empty[Todo]
def apply[A](fa: TodoModelOp[A]): Id[A] = fa match {
case Add(todo) => todos += todo; ()
case Todos() => todos.toSeq
}
}
}
class TodoModel() {
import cats.instances.list._
import cats.syntax.traverse._
import FreeTodoModelOps._
def add(t: Todo): Callback = {
def program: TodoModelOpF[Unit] = for {
_ <- FreeTodoModelOps.add(t)
} yield ()
Callback(program.foldMap(StateInterpreter.interpet))
}
def todos: Seq[Todo] = {
def program: TodoModelOpF[Seq[Todo]] = for {
n <- FreeTodoModelOps.todos
} yield n
program.foldMap(StateInterpreter.interpet)
}
}
object TodoPage {
case class Props(model: TodoModel)
case class State(todos: Seq[Todo])
class Backend($: BackendScope[Props, State]) {
val t = Todo("a new todo")
def onSubmit(e: ReactEventFromInput) =
e.preventDefaultCB >>
$.modState(s => State(s.todos :+ t)) >>
$.props.flatMap(P => P.model.add(t))
def render(S: State) =
<.div(
<.form(
^.onSubmit ==> onSubmit,
<.button("Add #", S.todos.length + 1)),
<.ul(S.todos.map(t => <.li(t.text)): _*))
}
val component = ScalaComponent.builder[Props]("Todo")
.initialStateFromProps(p => State(p.model.todos))
.renderBackend[Backend]
.build
def apply(model: TodoModel) = component(Props(model))
}
object Test {
val model = new TodoModel()
def main(args: Array[String]): Unit = {
TodoPage.apply(model).renderIntoDOM(dom.document.getElementById("mount-node"))
}
}
empty, no click on button
first click on button
second click on button
In your first snippet there's a bug:
Here you've got a variable todos (inpure) which you're accessing in a pure context:
def mod(f: Seq[Todo] => Seq[Todo]): Callback = {
val newTodos = f(todos)
Callback(todos = newTodos)
Impurity should be in Callback. Even reading a variable outside of a Callback is unsafe, so it should be:
def mod(f: Seq[Todo] => Seq[Todo]): Callback =
Callback(todos = f(todos))
(See scalajs-react's Ref.scala an example of safely working with a variable.)
Secondly, with regards to your larger snippet, scalajs-react is very FP friendly but that's very unconventional way of trying to use it, and has some significant problems:
StateInterpreter.interpet isn't referentially-transparent; there's shared global state underlying that. Fails the FP test. Stops being a lawful natural transformation.
You're tracking two sets of identical state separately: the component state and the state in TodoModel (impure, fails the FP test). Not only is this approach redundant and runs the risk of the two states getting out-of-sync, but it also makes the component less reusable; imagine you decided to draw it twice on the same screen for the same data - they're going to go out of sync. Best to keep the component stateless and pure.
If you're going to transform a free structure into a component effect, it's best to transform it into a state monad, see here for an example.
It's really, really awesome that you're learning about free monads and scalajs-react. FP will make your entire program really, really easy to reason about and prevent confusing surprises in behaviour, but you've got to not cut any corners and make sure that you keep all of your code pure. Any impurity will render the entire stack all the way up to the entrypoint impure and remove those nice dependable FP properties from those layers. I'd suggest making everything as pure as possible using the points above as starting points, and then I think you'll find that the bug just disappears, or at least is very easy to then detect. Cheers
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
I am trying to build a simple RESTful service that performs CRUD operations on a database and returns JSON. I have a service adhering to an API like this
GET mydomain.com/predictions/some%20string
I use a DAO which contains the following method that I have created to retrieve the associated prediction:
def getPrediction(rawText: String): Prediction = {
val predictionAction = predictions.filter{_.rawText === rawText}.result
val header = predictionAction.head
val f = db.run(header)
f.onComplete{case pred => pred}
throw new Exception("Oops")
}
However, this can't be right, so I started reading about Option. I changed my code accordingly:
def getPrediction(rawText: String): Option[Prediction] = {
val predictionAction = predictions.filter{_.rawText === rawText}.result
val header = predictionAction.headOption
val f = db.run(header)
f.onSuccess{case pred => pred}
None
}
This still doesn't feel quite right. What is the best way to invoke these filters, return the results, and handle any uncertainty?
I think the best way to rewrite your code is like this:
def getPrediction(rawText: String): Future[Option[Prediction]] = {
db.run(users.filter(_.rawText === rawText).result.headOption)
}
In other words, return a Future instead of the plain result. This way, the database actions will execute asynchronously, which is the preferred way for both Play and Akka.
The client code will then work with the Future. Per instance, a Play action would be like:
def prediction = Action.async {
predictionDao.getPrediction("some string").map { pred =>
Ok(views.html.predictions.show(pred))
}.recover {
case ex =>
logger.error(ex)
BadRequest()
}
}
I'm using scala and slick here, and I have a baserepository which is responsible for doing the basic crud of my classes.
For a design decision, we do have updatedTime and createdTime columns all handled by the application, and not by triggers in database. Both of this fields are joda DataTime instances.
Those fields are defined in two traits called HasUpdatedAt, and HasCreatedAt, for the tables
trait HasCreatedAt {
val createdAt: Option[DateTime]
}
case class User(name:String,createdAt:Option[DateTime] = None) extends HasCreatedAt
I would like to know how can I use reflection to call the user copy method, to update the createdAt value during the database insertion method.
Edit after #vptron and #kevin-wright comments
I have a repo like this
trait BaseRepo[ID, R] {
def insert(r: R)(implicit session: Session): ID
}
I want to implement the insert just once, and there I want to createdAt to be updated, that's why I'm not using the copy method, otherwise I need to implement it everywhere I use the createdAt column.
This question was answered here to help other with this kind of problem.
I end up using this code to execute the copy method of my case classes using scala reflection.
import reflect._
import scala.reflect.runtime.universe._
import scala.reflect.runtime._
class Empty
val mirror = universe.runtimeMirror(getClass.getClassLoader)
// paramName is the parameter that I want to replacte the value
// paramValue is the new parameter value
def updateParam[R : ClassTag](r: R, paramName: String, paramValue: Any): R = {
val instanceMirror = mirror.reflect(r)
val decl = instanceMirror.symbol.asType.toType
val members = decl.members.map(method => transformMethod(method, paramName, paramValue, instanceMirror)).filter {
case _: Empty => false
case _ => true
}.toArray.reverse
val copyMethod = decl.declaration(newTermName("copy")).asMethod
val copyMethodInstance = instanceMirror.reflectMethod(copyMethod)
copyMethodInstance(members: _*).asInstanceOf[R]
}
def transformMethod(method: Symbol, paramName: String, paramValue: Any, instanceMirror: InstanceMirror) = {
val term = method.asTerm
if (term.isAccessor) {
if (term.name.toString == paramName) {
paramValue
} else instanceMirror.reflectField(term).get
} else new Empty
}
With this I can execute the copy method of my case classes, replacing a determined field value.
As comments have said, don't change a val using reflection. Would you that with a java final variable? It makes your code do really unexpected things. If you need to change the value of a val, don't use a val, use a var.
trait HasCreatedAt {
var createdAt: Option[DateTime] = None
}
case class User(name:String) extends HasCreatedAt
Although having a var in a case class may bring some unexpected behavior e.g. copy would not work as expected. This may lead to preferring not using a case class for this.
Another approach would be to make the insert method return an updated copy of the case class, e.g.:
trait HasCreatedAt {
val createdAt: Option[DateTime]
def withCreatedAt(dt:DateTime):this.type
}
case class User(name:String,createdAt:Option[DateTime] = None) extends HasCreatedAt {
def withCreatedAt(dt:DateTime) = this.copy(createdAt = Some(dt))
}
trait BaseRepo[ID, R <: HasCreatedAt] {
def insert(r: R)(implicit session: Session): (ID, R) = {
val id = ???//insert into db
(id, r.withCreatedAt(??? /*now*/))
}
}
EDIT:
Since I didn't answer your original question and you may know what you are doing I am adding a way to do this.
import scala.reflect.runtime.universe._
val user = User("aaa", None)
val m = runtimeMirror(getClass.getClassLoader)
val im = m.reflect(user)
val decl = im.symbol.asType.toType.declaration("createdAt":TermName).asTerm
val fm = im.reflectField(decl)
fm.set(??? /*now*/)
But again, please don't do this. Read this stackoveflow answer to get some insight into what it can cause (vals map to final fields).