For example I have A,B,C,D flowables emitting independently. For example where this maps to Foo:
Flowable<Foo> fooStream = Flowable.combineLatest(A,B,C,D -> Foo::new);
And it emits whenever A, B, C or D emits an update. a new Foo is emitted downstream.
Which combination of operators, if any, could I use for merging (A,B,C,D)'s latest result into Foo ONLY some select flowable (lets say A) emits?
There is exactly operator you need: withLatestFrom
A.withLatestFrom(B, C, D, Foo::new)
It will emit every time A emits, with latest values from B, C and D.
Keep in mind B, C and D must have value in them. So if for ex. D didn't emit anything at all, stream will get stuck.
Related
Looking into how Either is defined as a functor, I can see that
derive instance functorEither :: Functor (Either a)
which reads to me as "You can map an Either so long as you can map its element.
But either doesn't have just one element. How would this be implemented without derive? Here's what I've tried:
data Either a b = Left a | Right b
instance functorEither :: Functor (Either a)
where
map f (Right b) = Right $ f b
map _ a = a
Of course, the types don't work here:
The Right has this signature: map :: forall a b. (a -> b) -> f a -> f b
The Left however, isn't okay: map :: forall a b. (a -> b) -> f a -> f a
Part of my intuition is saying that Either a b isn't a functor, only Either a is a functor. Which is why map works over Right and ignores Left
That doesn't really give me any intuition for how this is implemented. I still need a way of matching both constructors, don't I?
On the other hand, I think an implementation of map that replaces the inner function with identity is technically law-abiding for functor? The law of composition is met if you just ignore it?
While your proposed definition of the Functor instance indeed fails to compile, it isn't for the reason you say. And it's also "essentially" correct, just not written in a way that will satisfy the compiler.
For convenience, here's your definition again:
data Either a b = Left a | Right b
instance functorEither :: Functor (Either a)
where
map f (Right b) = Right $ f b
map _ a = a
and here's the actual error that you get when trying to compile it:
Could not match type
a02
with type
b1
while trying to match type Either a0 a02
with type Either a0 b1
while checking that expression a
has type Either a0 b1
in value declaration functorEither
where a0 is a rigid type variable
bound at (line 0, column 0 - line 0, column 0)
b1 is a rigid type variable
bound at (line 0, column 0 - line 0, column 0)
a02 is a rigid type variable
bound at (line 0, column 0 - line 0, column 0)
I admit that's a little hard to interpret, if you're not expecting it. But it has to do with the fact that map for Either a needs to have type forall b c. (b -> c) -> Either a b -> Either a c. So the a on the left of map _ a = a has type Either a b, while the one on the right has type Either a c - these are different types (in general), since b and c can be anything, so you can't use the same variable, a, to denote a value of each type.
(This question, although about Haskell rather than Purescript, goes deeper into explanation of exactly this error.)
To fix it, as implied in the question above, you have to explicitly mention that the value you're mapping over is a Left value:
data Either a b = Left a | Right b
instance functorEither :: Functor (Either a)
where
map f (Right b) = Right $ f b
map _ (Left a) = Left a
which is fine because Left a can be interpreted on the left as of type Either a b and on the right as an Either a c.
As for what the instance "does": you are correct that "Either a b isn't a functor, only Either a is a functor" - because a functor must take one type variable, which Either a does but Either a b doesn't. And yes, because the type variable that actually "varies" between Either a b and Either a c is the one that is used in Right, map must only map over the Right values, and leave the Left ones alone - that's the only thing that will satisfy the types needed.
Either a b is often interpreted as representing the result of a computation, where Left values represent failure while Right ones represent success. In this sense it's a slightly "expanded" version of Maybe - the difference is that rather than failure being represented by a single value (Nothing), you get a piece of data (the a type in Either a b) which can tell you information about the error. But the Functor instance works identically to that for Maybe: it maps over any success, and leaves failures alone.
(But there's no logical reason why you can't "map over" the Left values as well. The Bifunctor class is an extension of Functor which can do exactly that.)
I'm trying to achieve something like this:
I am trying to create this flow using Flow.fromGraph
I can do join using Zip[B, C] which takes in 2 streams
I can do split in 2 ways:
using Broadcast[A](2)
using UnZip[(A,A)], preceded by a .map(a -> (a, a))
Both map(f1) and map(f2) are custom flows I'm obtaining from included libraries, so I can't really modify them, so please don't say .map(a => (f1(a), f2(a)))
What are the differences between the 2 cases, or are equivalent?The only thing I found different was Broadcast's ability to cancel only when all downstreams cancel (eagerCancel = false) which is its default behavior, as against UnZip (which does what broadcast does with eagerCancel = true)
Also, what happens in case of failures in either of the 2 paths? i.e. what is the impact if, for a specific element, f1 throws an error?
Additionally, say if we don't have an f2 function (so no map operation) and we want to emit (b,a) at the end, should f2 be replaced by an identity flow, or can it be skipped all together? (if latter, would you ever use an identity flow?)
val split = builder.add(BroadCast[A](2))
val join = builder.add(Zip[B, A])
val F1: Flow[A, B, NotUsed] = Flow[A].map(f1)
val I = Flow[A].map(identity)
split ~> F1 ~> join.in1
split ~> /* I ~> */ join.in0 // do i need the commented part?
Probably this helps with internal buffers/back-pressure ?
They are both Fanout operators; however
Unzip from the docs:
Takes a stream of two element tuples and unzips the two elements ino two different downstreams.
While Broadcast
Emit each incoming element each of n outputs.
Therefore we can conclude that Unzip is just a Broadcast with n = 2; but importantly if the elements are a tuple, Broadcast will just create n outputs of the same tuple. Unzip will create 2 outputs each for element A and B
I'm looking for the best solution to accumulate the last N number of messages in a Spark DStream. I'd also like to specify the number of messages to retain.
For example, given the following stream, I'd like to retain the last 3 elements:
Iteration New message Downstream
1 A [A]
2 B [A, B]
3 C [A, B, C]
4 D [B, C, D]
So far I'm looking at the following methods on DStream:
updateStateByKey: given that all messages have the same key I can do this. But looks a bit odd why this needs to know about the key at all.
mapWithState: the API in Scala is just too tedious for such a simple thing
window: doesn't seem to do this job, also it needs a time value for windowing instead of the last number of elements
Accumulators: not really used yet Accumulators in Spark docs
What's the best solution to achieve this?
mapWithState is exactly what you need, and it's definitely not too tedious:
case class Message(x: String)
def statefulTransformation(key: Int,
value: Option[Message],
state: State[mutable.MutableList[Message]]): Option[Message] = {
def updateState(value: Message): Message = {
val updatedList =
state
.getOption()
.map(list => if (list.size > 3) list.drop(1) :+ value else list :+ value)
.getOrElse(mutable.MutableList(value))
state.update(updatedList)
value
}
value.map(updateState)
}
And now all you need is:
val stateSpec = StateSpec.function(statefulTransformation _)
dStream.mapWithState(stateSpec)
Side note - I used mutable.MutableList for the constant time append.
I have 2 collections: a is a sequence of Scala objects of class C. b is a sequence of strings. C has a string field, name, that could possibly match an item in b. What I want is to loop through a and find all c.name that matches with one of the item in b. How do I do this in Scala?
Iterating through both a and b can get expensive because one loop nested inside another yields O(n^2) time. If b is sufficiently large, you probably want to make it into a Set first to bring this down to O(n).
val bSet = b.toSet;
a.filter(c => b.contains(c.name))
I would read this as "Apply the following filter to a: for each item c in a, include it in the result if and only if the name of c is in b."
Here's the equivalent for loop with yield.
for(c <- a if b.contains(c.name)) yield c.name
In Python
def cross(A, B):
"Cross product of elements in A and elements in B."
return [a+b for a in A for b in B]
returns an one-dimensional array if you call it with two arrays (or strings).
But in CoffeeScript
cross = (A, B) -> (a+b for a in A for b in B)
returns a two-dimensional array.
Do you think it's by design in CoffeeScript or is it a bug?
How do I flatten arrays in CoffeScript?
First I would say say that 2 array comprehensions in line is not a very maintainable pattern. So lets break it down a little.
cross = (A, B) ->
for a in A
for b in B
a+b
alert JSON.stringify(cross [1,2], [3,4])
What's happening here is that the inner creates a closure, which has its own comprehension collector. So it runs all the b's, then returns the results as an array which gets pushed onto the parent comprehension result collector. You are sort of expecting a return value from an inner loop, which is a bit funky.
Instead I would simply collect the results myself.
cross = (A, B) ->
results = []
for a in A
for b in B
results.push a + b
results
alert JSON.stringify(cross [1,2], [3,4])
Or if you still wanted to do some crazy comprehension magic:
cross = (A, B) ->
results = []
results = results.concat a+b for b in B for a in A
results
alert JSON.stringify(cross [1,2], [3,4])
Whether this is a bug in CS or not is a bit debatable, I suppose. But I would argue it's good practice to do more explicit comprehension result handling when dealing with nested iterators.
https://github.com/jashkenas/coffee-script/issues/1191