How can I compose an operator like replay that replays the last item(s) matching a predicate, and then all future items?
For example, if a hot observable emits A, B, C, D, E, F and my predicate matches vowels, then a subscriber joining between C and D should receive A, D, E, F.
I'm only an egg, but here's what I came up with to replay a single item:
public static <T> Observable<T> replay(final Observable<T> observable,
final Predicate<? super T> predicate) {
final AtomicReference<T> mLastMatch = new AtomicReference<>();
return observable.map(e -> {
if (predicate.test(e)) {
mLastMatch.set(e);
}
return e;
})
.startWith(Observable.defer(() -> {
final T t = mLastMatch.get();
return t == null ? Observable.empty() : Observable.just(t);
}));
}
Related
I'm working in an scenario where duplicated messages could arrive at a consumer (a KStream application). To use the typical case let's suppose it's an OrderCreatedEvent and the KStream has a logic that processes the order. The event has an order-id that would help me identify duplicated messages.
What I want to do is:
1) Add every order to a persistent state store
2) When processing the message in the KStream, query the state store to check if the message had already been received, not doing anything in that case.
val persistentKeyValueStore = Stores.persistentKeyValueStore("order-store")
val stateStore: Materialized<Int, Order, KeyValueStore<Bytes, ByteArray>> =
Materialized.`as`<Int, Order>(persistentKeyValueStore)
.withKeySerde(intSerde)
.withValueSerde(orderSerde)
val orderTable: KTable<Int, Order> = input.groupByKey(Serialized.with(intSerde, orderSerde))
.reduce({ _, y -> y }, stateStore)
var orderStream: KStream<Int, Order> = ...
orderStream.filter { XXX }
.map { key, value ->
processingLogic()
KeyValue(key, value)
}...
In the filter { XXX } bit I would like to query the state store check if the order id is there (let's assume the order is used as the key of the keyvaluestore), filtering out orders already processed (present in the state store).
My first question is: how can I query a state store in the KStream DSL, e.g. inside the filter operation.
Second question: in this case, how can I handle the arrival of a new (not previously processed message)? If the KTable persists the order to the state store BEFORE the orderStream KStream execution the message would already be in the store. They should be added only after the processing has completed.
How can I do this? It's likely I shouldn't be using a KTable for it but something like:
orderStream.filter { keystore.get(key) == null }
.map { key, value ->
processingLogic()
KeyValue(key, value)
}
.foreach { key, value ->
keystore.put(key, value);
}
Following Matthias' indications I implemented it like this:
DeduplicationTransformer
package com.codependent.outboxpattern.operations.stream
import com.codependent.outboxpattern.account.TransferEmitted
import org.apache.kafka.streams.KeyValue
import org.apache.kafka.streams.kstream.Transformer
import org.apache.kafka.streams.processor.ProcessorContext
import org.apache.kafka.streams.state.KeyValueStore
import org.slf4j.LoggerFactory
#Suppress("UNCHECKED_CAST")
class DeduplicationTransformer : Transformer<String, TransferEmitted, KeyValue<String, TransferEmitted>> {
private val logger = LoggerFactory.getLogger(javaClass)
private lateinit var dedupStore: KeyValueStore<String, String>
private lateinit var context: ProcessorContext
override fun init(context: ProcessorContext) {
this.context = context
dedupStore = context.getStateStore(DEDUP_STORE) as KeyValueStore<String, String>
}
override fun transform(key: String, value: TransferEmitted): KeyValue<String, TransferEmitted>? {
return if (isDuplicate(key)) {
logger.warn("****** Detected duplicated transfer {}", key)
null
} else {
logger.warn("****** Registering transfer {}", key)
dedupStore.put(key, key)
KeyValue(key, value)
}
}
private fun isDuplicate(key: String) = dedupStore[key] != null
override fun close() {
}
}
FraudKafkaStreamsConfiguration
const val DEDUP_STORE = "dedup-store"
#Suppress("UNCHECKED_CAST")
#EnableBinding(TransferKafkaStreamsProcessor::class)
class FraudKafkaStreamsConfiguration(private val fraudDetectionService: FraudDetectionService) {
private val logger = LoggerFactory.getLogger(javaClass)
#KafkaStreamsStateStore(name = DEDUP_STORE, type = KafkaStreamsStateStoreProperties.StoreType.KEYVALUE)
#StreamListener
#SendTo(value = ["outputKo", "outputOk"])
fun process(#Input("input") input: KStream<String, TransferEmitted>): Array<KStream<String, *>>? {
val fork: Array<KStream<String, *>> = input
.transform(TransformerSupplier { DeduplicationTransformer() }, DEDUP_STORE)
.branch(Predicate { _: String, value -> fraudDetectionService.isFraudulent(value) },
Predicate { _: String, value -> !fraudDetectionService.isFraudulent(value) }) as Array<KStream<String, *>>
...
Why is the following RxSwift code not compiling and how do I solve the problem? This line observer.onNext("test123") is the problem.
final class TestA<String>: ObservableType {
typealias E = String
private let _observable: Observable<String>
init() {
_observable = Observable<String>.create { observer -> Disposable in
print("mark 1")
observer.onNext("test123")
observer.onCompleted()
return Disposables.create()
}
}
func subscribe<O>(_ observer: O) -> Disposable where O : ObserverType, O.E == E {
return _observable.subscribe(observer)
}
}
let observable = TestA<String>()
print("mark 2")
observable.subscribe(onNext: { element in
print(element)
})
I am testing in the playground and get the following error:
Playground execution failed:
error: Introduction.xcplaygroundpage:25:26: error: cannot invoke 'onNext' > with an argument list of type '(String)'
observer.onNext("test123")
^
Introduction.xcplaygroundpage:25:26: note: expected an argument list of > type '(String)'
observer.onNext("test123")
^
One of the reasons behind this setup with the class is that I want to pass in the dependencies with constructor injection and use them in the create closure in order to avoid having to capture self. I also want to avoid having all those Observable.creates in the wild and have a more OOP approach.
The swift compiler was not helpful with this error...
The problem here is that, when declaring TestA, you override the name String to represent the generic parameter for TestA. It is then an error to sent a Swift.String as a parameter to an observer expecting a TestA.String, which could be anything.
You can fix the issue with removing the unused generic parameter (final class TestA: ObservableType { ...), or taking the value sent to onNext as a parameter to the init, depending on the use case.
final class TestA<Element>: ObservableType {
typealias E = Element
private let _observable: Observable<Element>
init(_ value: Element) {
_observable = Observable<Element>.create { observer -> Disposable in
print("mark 1")
observer.onNext(value)
observer.onCompleted()
return Disposables.create()
}
}
func subscribe<O>(_ observer: O) -> Disposable where O : ObserverType, O.E == E {
return _observable.subscribe(observer)
}
}
I have 2 stream which are being concatenated. If the first stream executes onError instead of onComplete, I shouldn't be concatenating the second stream.
example:
func updateEntity(entities: [Member]) -> Observable<[Result<Member>]> {
let remoteUpdate = remoteStore.update(entities: entities)
return remoteUpdate.concat(localStore.update(entities: entities))
}
I shouldn't be updating the localStore if remoteUpdate throws an error, onError is called in remoteUpdate
Update:
public override func update(entities: [PlaceAlert]) -> Observable<[Result<PlaceAlert>]> {
let remoteUpdate = remoteStore.update(entities: entities)
var entityPlaceHolder: [PlaceAlert] = entities
return remoteUpdate.catchError { _ in
entityPlaceHolder = []
return localStore.update(entities: entityPlaceHolder)
}.concat(localStore.update(entities: entityPlaceHolder))
}
Just tried improvising. would this make any difference? LocalUpdate with emptyArray does nothing if there is an error
concat is not an if/then statement. Both updates are getting called at the same scope, it's just that the result won't reflect the localStore.update response if the remoteStore errors.
catchError is what you want in this context:
func updateEntity(entities: [Member]) -> Observable<[Result<Member>]> {
let remoteUpdate = remoteStore.update(entities: entities)
return remoteUpdate.catchError { _ in
localStore.update(entities: entities)
}
}
I get the following error with this Swift code in an Xcode 7.3 Playground.
Code
/**
* Represents a future value of type U, from a computation that has not been done yet.
*/
public struct Promise<T, U> {
var futureValue: T -> U
public func then<T,U,V>(closure: U -> V) -> Promise<T, V> {
return Promise<T, V>(futureValue: {
input in
let value = futureValue(input)
return closure(value)
})
}
}
Error
error: cannot invoke value of type 'T -> U' with argument list '((T))'
let value = futureValue(input)
This looks like it should work, because the types match up in my definition. What could be causing this behavior?
In
public func then<T,U,V>(closure: U -> V) -> Promise<T, V> { ... }
the local generic placeholders T, U hide (or shadow) the
generic placeholders from the
public struct Promise<T, U> { ... }
type definition. Your method definition is equivalent to
public func then<T1,U1,V>(closure: U1 -> V) -> Promise<T1, V> {
return Promise<T1, V>(futureValue: {
input in
let value = futureValue(input)
return closure(value)
})
}
which then shows the more understandable error message
error: cannot invoke value of type 'T -> U' with argument list '((T1))'
The solution is simple: Just omit T and U from the generic placeholder
list in the method definition, these are inherited automatically from
struct Promise. Note also that the futureValue property must
be referenced via self inside the closure:
public func then<V>(closure: U -> V) -> Promise<T, V> {
return Promise<T, V>(futureValue: {
input in
let value = self.futureValue(input)
return closure(value)
})
}
what is the return type of this sort of bool function...... i know the return type is either true of false but this seems complicated when you have got like this..
bool mypredicate (int i, int j) {
return (i==j);
}
this bool function are used in a library function called equal...... another example is....
bool compare(int a, int b){
return a<b;
}
so what are the perspective here to return type of these bool function.when is goes true & false....
Your functions mypredicate and compare are merely thin wrappers over the binary operators == and <. Operators are like functions: they take a number of arguments of a given type, and return a result of a given type.
For example, imagine a function bool operator==(int a, int b) with the following specification:
if a equals b then return true
otherwise return false
And a function bool operator<(int a, int b) with the following specification:
if a is strictly lesser than b then return true
otherwise return false.
Then you could write:
bool mypredicate (int i, int j) {
return operator==(i, j);
}
bool compare(int a, int b){
return operator<(a, b);
}
For convenience, most programming languages allow you to use a shorter, functionnaly equivalent syntax: i == j and a < b.