I'm trying to build a generic router for a personal project by Binding.scala.
I've defined a PageState trait
sealed trait WhistState {
def text: String
def hash: String
def render: Binding[Node]
}
with a number of subclasses for each route type. I've then trying to create a router, which based on the hash chooses the correct class.
object Router {
val defaultState: WhistState = DefaultState("Games")
val allStates: Vector[WhistState] = Vector(defaultState)
val route: Route.Hash[WhistState] = Route.Hash[WhistState](defaultState)(new Route.Format[WhistState] {
override def unapply(hashText: String): Option[WhistState] = allStates.find(_.hash == window.location.hash)
override def apply(state: WhistState): String = state.hash
})
route.watch()
}
And then I'm my application classes im trying to use this
object Application {
import example.route.Router._
#dom
def render: Binding[Node] = {
for (hash <- route.state.bind.hash) yield println("hash: " + hash)
route.state.bind match {
case default: WhistState => println("default"); default.render.bind
case _ => println("none"); <div>NotFound</div>
}
}
def main(args: Array[String]): Unit = {
dom.render(document.querySelector("#content"), render)
}
}
I've would expect that changing the hash would force a reevaluation of route.state.bind match ... expression.
Any idea to why this doesn't work?
Best regards
route.state is only changed when unapply returns a Some(newState) and newState does not equal to the previous state.
In your case, unapply always returns Some(defaultState) or None. That's why route.state never changes.
Related
I wrote simple callback(handler) function which i pass to async api and i want to wait for result:
object Handlers {
val logger: Logger = Logger("Handlers")
implicit val cs: ContextShift[IO] =
IO.contextShift(ExecutionContext.Implicits.global)
class DefaultHandler[A] {
val response: IO[MVar[IO, A]] = MVar.empty[IO, A]
def onResult(obj: Any): Unit = {
obj match {
case obj: A =>
println(response.flatMap(_.tryPut(obj)).unsafeRunSync())
println(response.flatMap(_.isEmpty).unsafeRunSync())
case _ => logger.error("Wrong expected type")
}
}
def getResponse: A = {
response.flatMap(_.take).unsafeRunSync()
}
}
But for some reason both tryPut and isEmpty(when i'd manually call onResult method) returns true, therefore when i calling getResponse it sleeps forever.
This is the my test:
class HandlersTest extends FunSuite {
test("DefaultHandler.test") {
val handler = new DefaultHandler[Int]
handler.onResult(3)
val response = handler.getResponse
assert(response != 0)
}
}
Can somebody explain why tryPut returns true, but nothing puts. And what is the right way to use Mvar/channels in scala?
IO[X] means that you have the recipe to create some X. So on your example, yuo are putting in one MVar and then asking in another.
Here is how I would do it.
object Handlers {
trait DefaultHandler[A] {
def onResult(obj: Any): IO[Unit]
def getResponse: IO[A]
}
object DefaultHandler {
def apply[A : ClassTag]: IO[DefaultHandler[A]] =
MVar.empty[IO, A].map { response =>
new DefaultHandler[A] {
override def onResult(obj: Any): IO[Unit] = obj match {
case obj: A =>
for {
r1 <- response.tryPut(obj)
_ <- IO(println(r1))
r2 <- response.isEmpty
_ <- IO(println(r2))
} yield ()
case _ =>
IO(logger.error("Wrong expected type"))
}
override def getResponse: IO[A] =
response.take
}
}
}
}
The "unsafe" is sort of a hint, but every time you call unsafeRunSync, you should basically think of it as an entire new universe. Before you make the call, you can only describe instructions for what will happen, you can't actually change anything. During the call is when all the changes occur. Once the call completes, that universe is destroyed, and you can read the result but no longer change anything. What happens in one unsafeRunSync universe doesn't affect another.
You need to call it exactly once in your test code. That means your test code needs to look something like:
val test = for {
handler <- TestHandler.DefaultHandler[Int]
_ <- handler.onResult(3)
response <- handler.getResponse
} yield response
assert test.unsafeRunSync() == 3
Note this doesn't really buy you much over just using the MVar directly. I think you're trying to mix side effects inside IO and outside it, but that doesn't work. All the side effects need to be inside.
My use case has case classes something like
case class Address(name:String,pincode:String){
override def toString =name +"=" +pincode
}
case class Department(name:String){
override def toString =name
}
case class emp(address:Address,department:Department)
I want to create a DSL like below.Can anyone share the links about how to create a DSL and any suggestions to achieve the below.
emp.withAddress("abc","12222").withDepartment("HR")
Update:
Actual use case class may have more fields close to 20. I want to avoid redudancy of code
I created a DSL using reflection so that we don't need to add every field to it.
Disclamer: This DSL is extremely weakly typed and I did it just for fun. I don't really think this is a good approach in Scala.
scala> create an Employee where "homeAddress" is Address("a", "b") and "department" is Department("c") and that_s it
res0: Employee = Employee(a=b,null,c)
scala> create an Employee where "workAddress" is Address("w", "x") and "homeAddress" is Address("y", "z") and that_s it
res1: Employee = Employee(y=z,w=x,null)
scala> create a Customer where "address" is Address("a", "b") and "age" is 900 and that_s it
res0: Customer = Customer(a=b,900)
The last example is the equivalent of writing:
create.a(Customer).where("address").is(Address("a", "b")).and("age").is(900).and(that_s).it
A way of writing DSLs in Scala and avoid parentheses and the dot is by following this pattern:
object.method(parameter).method(parameter)...
Here is the source:
// DSL
object create {
def an(t: Employee.type) = new ModelDSL(Employee(null, null, null))
def a(t: Customer.type) = new ModelDSL(Customer(null, 0))
}
object that_s
class ModelDSL[T](model: T) {
def where(field: String): ValueDSL[ModelDSL2[T], Any] = new ValueDSL(value => {
val f = model.getClass.getDeclaredField(field)
f.setAccessible(true)
f.set(model, value)
new ModelDSL2[T](model)
})
def and(t: that_s.type) = new { def it = model }
}
class ModelDSL2[T](model: T) {
def and(field: String) = new ModelDSL(model).where(field)
def and(t: that_s.type) = new { def it = model }
}
class ValueDSL[T, V](callback: V => T) {
def is(value: V): T = callback(value)
}
// Models
case class Employee(homeAddress: Address, workAddress: Address, department: Department)
case class Customer(address: Address, age: Int)
case class Address(name: String, pincode: String) {
override def toString = name + "=" + pincode
}
case class Department(name: String) {
override def toString = name
}
I really don't think you need the builder pattern in Scala. Just give your case class reasonable defaults and use the copy method.
i.e.:
employee.copy(address = Address("abc","12222"),
department = Department("HR"))
You could also use an immutable builder:
case class EmployeeBuilder(address:Address = Address("", ""),department:Department = Department("")) {
def build = emp(address, department)
def withAddress(address: Address) = copy(address = address)
def withDepartment(department: Department) = copy(department = department)
}
object EmployeeBuilder {
def withAddress(address: Address) = EmployeeBuilder().copy(address = address)
def withDepartment(department: Department) = EmployeeBuilder().copy(department = department)
}
You could do
object emp {
def builder = new Builder(None, None)
case class Builder(address: Option[Address], department: Option[Department]) {
def withDepartment(name:String) = {
val dept = Department(name)
this.copy(department = Some(dept))
}
def withAddress(name:String, pincode:String) = {
val addr = Address(name, pincode)
this.copy(address = Some(addr))
}
def build = (address, department) match {
case (Some(a), Some(d)) => new emp(a, d)
case (None, _) => throw new IllegalStateException("Address not provided")
case _ => throw new IllegalStateException("Department not provided")
}
}
}
and use it as emp.builder.withAddress("abc","12222").withDepartment("HR").build().
You don't need optional fields, copy, or the builder pattern (exactly), if you are willing to have the build always take the arguments in a particular order:
case class emp(address:Address,department:Department, id: Long)
object emp {
def withAddress(name: String, pincode: String): WithDepartment =
new WithDepartment(Address(name, pincode))
final class WithDepartment(private val address: Address)
extends AnyVal {
def withDepartment(name: String): WithId =
new WithId(address, Department(name))
}
final class WithId(address: Address, department: Department) {
def withId(id: Long): emp = emp(address, department, id)
}
}
emp.withAddress("abc","12222").withDepartment("HR").withId(1)
The idea here is that each emp parameter gets its own class which provides a method to get you to the next class, until the final one gives you an emp object. It's like currying but at the type level. As you can see I've added an extra parameter just as an example of how to extend the pattern past the first two parameters.
The nice thing about this approach is that, even if you're part-way through the build, the type you have so far will guide you to the next step. So if you have a WithDepartment so far, you know that the next argument you need to supply is a department name.
If you want to avoid modifying the origin classes you can use implicit class, e.g.
implicit class EmpExtensions(emp: emp) {
def withAddress(name: String, pincode: String) {
//code omitted
}
// code omitted
}
then import EmpExtensions wherever you need these methods
I am broadcasting a value in Spark Streaming application . But I am not sure how to access that variable in a different class than the class where it was broadcasted.
My code looks as follows:
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreachRDD(rdd => {
val obj = AppObject1
rdd.filter(p => obj.apply(p))
rdd.count
}
}
object AppObject1: Boolean{
def apply(str: String){
AnotherObject.process(str)
}
}
object AnotherObject{
// I want to use broadcast variable in this object
val B = broadcastA.Value // compilation error here
def process(): Boolean{
//need to use B inside this method
}
}
Can anyone suggest how to access broadcast variable in this case?
There is nothing particularly Spark specific here ignoring possible serialization issues. If you want to use some object it has to be available in the current scope and you can achieve this the same way as usual:
you can define your helpers in a scope where broadcast is already defined:
{
...
val x = sc.broadcast(1)
object Foo {
def foo = x.value
}
...
}
you can use it as a constructor argument:
case class Foo(x: org.apache.spark.broadcast.Broadcast[Int]) {
def foo = x.value
}
...
Foo(sc.broadcast(1)).foo
method argument
case class Foo() {
def foo(x: org.apache.spark.broadcast.Broadcast[Int]) = x.value
}
...
Foo().foo(sc.broadcast(1))
or even mixed-in your helpers like this:
trait Foo {
val x: org.apache.spark.broadcast.Broadcast[Int]
def foo = x.value
}
object Main extends Foo {
val sc = new SparkContext("local", "test", new SparkConf())
val x = sc.broadcast(1)
def main(args: Array[String]) {
sc.parallelize(Seq(None)).map(_ => foo).first
sc.stop
}
}
Just a short take on performance considerations that were introduced earlier.
Options proposed by zero233 are indeed very elegant way of doing this kind of things in Scala. At the same time it is important to understand implications of using certain patters in distributed system.
It is not the best idea to use mixin approach / any logic that uses enclosing class state. Whenever you use a state of enclosing class within lambdas Spark will have to serialize outer object. This is not always true but you'd better off writing safer code than one day accidentally blow up the whole cluster.
Being aware of this, I would personally go for explicit argument passing to the methods as this would not result in outer class serialization (method argument approach).
you can use classes and pass the broadcast variable to classes
your psudo code should look like :
object AppMain{
def main(args: Array[String]){
//...
val broadcastA = sc.broadcast(a)
//..
lines.foreach(rdd => {
val obj = new AppObject1(broadcastA)
rdd.filter(p => obj.apply(p))
rdd.count
})
}
}
class AppObject1(bc : Broadcast[String]){
val anotherObject = new AnotherObject(bc)
def apply(str: String): Boolean ={
anotherObject.process(str)
}
}
class AnotherObject(bc : Broadcast[String]){
// I want to use broadcast variable in this object
def process(str : String): Boolean = {
val a = bc.value
true
//need to use B inside this method
}
}
Edit2
Okay, so maybe I should parse out two desires here.
I had in mind that when it came time to get to the setSendTimeout(0) part, I would be using something like implicitly[Socket].
new ZContext(1) {
createSocket(ZMQ.PUB).setSendTimeout(0).//RATS!
}
I also had in mind a more generic approach to it, that would be (in pseudo code terms):
This is how you wrap a reference of T at a point in time without copying it, so that moving forward, you can tease out state of the reference after potential state changes from the value of whatever expression used it.
If it could be thought of as a chain of map map map s from T to where ever it ended up, then it is easy to append / apply a value onto it - just map again...
This is the motivating example.
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = new GraphStageLogic(shape) {
logger.info("Initializing ZMQ context.")
val context = new ZContext(1)
logger.info(s"Binding PUB socket to ${endpoint}")
val socket = {
val s = context.createSocket(ZMQ.PUB)
s.setSendTimeOut(0)
s.bind(endpoint)
s
}
Look at socket down there. For some reason that feels uglier than it needs to be to me, but is a consequence of the fact that setters don't return stuff like setSendTimeOut.
I would normally try to improve it as follows:
new ZContext(1) {
createSocket(ZMQ.PUB).setSendTimeout(0).//RATS!
}
Here a version of #Dima's answer. Again the setup:
trait Instance {
def createPort(): Port
}
trait Port {
def makeSpecial(): Unit
def bindTo(address: Any): Unit
}
trait Provider {
def getTheInstance(i: Int): Instance
}
Now the trick:
implicit class InstanceOps(i: Instance) {
def withCreatePort(fun: (Unit => Port) => Any): Port = {
val res = i.createPort()
fun(_ => res)
res
}
}
And if you add an implicit modifier to the argument of the function passed into withCreatePort, you "import" the implicit conversion:
trait ConnectTest extends Provider {
getTheInstance(2).withCreatePort { implicit p =>
().makeSpecial().bindTo("foo")
}
}
This is potentially more dangerous, because you have an implicit conversion from Unit to Port, although it is locally encapsulated. This is generic because Connect is generic.
This trick is perhaps too clever and difficult to understand by some outside standing person reading your code.
Yes, you can create two wrappers, one giving you withCreatePort, the other giving you variants of the port method that return this:
trait Instance {
def createPort(): Port
}
trait Port {
def makeSpecial(): Unit
def bindTo(address: Any): Unit
}
class PortOps(p: Port) {
def makeSpecial() : this.type = { p.makeSpecial() ; this }
def bindTo(address: Any): this.type = { p.bindTo(address); this }
}
implicit class InstanceOps(i: Instance) {
def withCreatePort[A](fun: PortOps => A): A = fun(new PortOps(i.createPort()))
}
Example:
trait Provider {
def getTheInstance(i: Int): Instance
}
trait Plain extends Provider {
val instance = getTheInstance(2)
val port = instance.createPort()
port.makeSpecial()
port.bindTo("foo")
}
trait Rich extends Provider {
getTheInstance(2).withCreatePort { p =>
p.makeSpecial().bindTo("foo")
}
}
The question is if the effort is worth it. You can also experiment with import:
trait Import extends Provider {
val instance = getTheInstance(2)
val port = instance.createPort()
locally {
import port._
makeSpecial(); bindTo("foo")
}
}
I am not sure where you are going with this Zipped thingy ... But what you described in the beginning of your question (assuming that port in the end of that snippet is a typo, and you really meant to return instance) can be done with something like this:
object Taps {
implicit class Tap[T](t: T) extends Anyval {
def tap(f: T => Unit) = { f(t); t }
}
}
Then you can write:
import Taps._
val instance = getTheInstance(2).tap {
_.createPort
.makeSpecial
.bindTo(...)
}
Is this what you are looking for?
There is some data that I have pulled from a remote API, for which I use a Future-style interface. The data is structured as a linked-list. A relevant example data container is shown below.
case class Data(information: Int) {
def hasNext: Boolean = ??? // Implemented
def next: Future[Data] = ??? // Implemented
}
Now I'm interested in adding some functionality to the data class, such as map, foreach, reduce, etc. To do so I want to implement some form of IterableLike such that it inherets these methods.
Given below is the trait Data may extend, such that it gets this property.
trait AsyncIterable[+T]
extends IterableLike[Future[T], AsyncIterable[T]]
{
def hasNext : Boolean
def next : Future[T]
// How to implement?
override def iterator: Iterator[Future[T]] = ???
override protected[this] def newBuilder: mutable.Builder[Future[T], AsyncIterable[T]] = ???
override def seq: TraversableOnce[Future[T]] = ???
}
It should be a non-blocking implementation, which when acted on, starts requesting the next data from the remote data source.
It is then possible to do cool stuff such as
case class Data(information: Int) extends AsyncIterable[Data]
val data = Data(1) // And more, of course
// Asynchronously print all the information.
data.foreach(data => println(data.information))
It is also acceptable for the interface to be different. But the result should in some way represent asynchronous iteration over the collection. Preferably in a way that is familiar to developers, as it will be part of an (open source) library.
In production I would use one of following:
Akka Streams
Reactive Extensions
For private tests I would implement something similar to following.
(Explanations are below)
I have modified a little bit your Data:
abstract class AsyncIterator[T] extends Iterator[Future[T]] {
def hasNext: Boolean
def next(): Future[T]
}
For it we can implement this Iterable:
class AsyncIterable[T](sourceIterator: AsyncIterator[T])
extends IterableLike[Future[T], AsyncIterable[T]]
{
private def stream(): Stream[Future[T]] =
if(sourceIterator.hasNext) {sourceIterator.next #:: stream()} else {Stream.empty}
val asStream = stream()
override def iterator = asStream.iterator
override def seq = asStream.seq
override protected[this] def newBuilder = throw new UnsupportedOperationException()
}
And if see it in action using following code:
object Example extends App {
val source = "Hello World!";
val iterator1 = new DelayedIterator[Char](100L, source.toCharArray)
new AsyncIterable(iterator1).foreach(_.foreach(print)) //prints 1 char per 100 ms
pause(2000L)
val iterator2 = new DelayedIterator[String](100L, source.toCharArray.map(_.toString))
new AsyncIterable(iterator2).reduceLeft((fl: Future[String], fr) =>
for(l <- fl; r <- fr) yield {println(s"$l+$r"); l + r}) //prints 1 line per 100 ms
pause(2000L)
def pause(duration: Long) = {println("->"); Thread.sleep(duration); println("\n<-")}
}
class DelayedIterator[T](delay: Long, data: Seq[T]) extends AsyncIterator[T] {
private val dataIterator = data.iterator
private var nextTime = System.currentTimeMillis() + delay
override def hasNext = dataIterator.hasNext
override def next = {
val thisTime = math.max(System.currentTimeMillis(), nextTime)
val thisValue = dataIterator.next()
nextTime = thisTime + delay
Future {
val now = System.currentTimeMillis()
if(thisTime > now) Thread.sleep(thisTime - now) //Your implementation will be better
thisValue
}
}
}
Explanation
AsyncIterable uses Stream because it's calculated lazily and it's simple.
Pros:
simplicity
multiple calls to iterator and seq methods return same iterable with all items.
Cons:
could lead to memory overflow because stream keeps all prevously obtained values.
first value is eagerly gotten during creation of AsyncIterable
DelayedIterator is very simplistic implementation of AsyncIterator, don't blame me for quick and dirty code here.
It's still strange for me to see synchronous hasNext and asynchronous next()
Using Twitter Spool I've implemented a working example.
To implement spool I modified the example in the documentation.
import com.twitter.concurrent.Spool
import com.twitter.util.{Await, Return, Promise}
import scala.concurrent.{ExecutionContext, Future}
trait AsyncIterable[+T <: AsyncIterable[T]] { self : T =>
def hasNext : Boolean
def next : Future[T]
def spool(implicit ec: ExecutionContext) : Spool[T] = {
def fill(currentPage: Future[T], rest: Promise[Spool[T]]) {
currentPage foreach { cPage =>
if(hasNext) {
val nextSpool = new Promise[Spool[T]]
rest() = Return(cPage *:: nextSpool)
fill(next, nextSpool)
} else {
val emptySpool = new Promise[Spool[T]]
emptySpool() = Return(Spool.empty[T])
rest() = Return(cPage *:: emptySpool)
}
}
}
val rest = new Promise[Spool[T]]
if(hasNext) {
fill(next, rest)
} else {
rest() = Return(Spool.empty[T])
}
self *:: rest
}
}
Data is the same as before, and now we can use it.
// Cool stuff
implicit val ec = scala.concurrent.ExecutionContext.global
val data = Data(1) // And others
// Print all the information asynchronously
val fut = data.spool.foreach(data => println(data.information))
Await.ready(fut)
It will trow an exception on the second element, because the implementation of next was not provided.