I get an Unsupported Operation Exception when trying to modify a list from inside "apply" method of flatmap in RxJava2.
compositeDisposable.add(createObservable()
.flatMap(new Function<List<String>, ObservableSource<List<String>>>() {
#Override
public ObservableSource<List<String>> apply(List<String> s) throws Exception {
List<String> modiList = new ArrayList<String>();
modiList.addAll(s);
modiList.add("barber");
//s.add("barber") and return Observable.fromArray(s) thows error
return Observable.fromArray(modiList);
}
})
.subscribeWith(getObserver()));
However, If i create a new list it works fine as shown above.
Any insights about it ?
Below is my Observable creation logic :
String[] arr = {"hi", "hello", "bye"};
Observable<List<String>> observable;
observable = Observable.fromCallable(() -> Arrays.asList(arr));
As akarnokd as pointed out, mutating the value of the list in your flatMap is generally a bad idea, but your superficial problem is that implementation of the interface List returned by Arrays.asList does not implement addAll.
Related
How to compare List to Flux in non blocking way
Below the code in blocking way
public static void main(String[] args) {
List<String> all = List.of("A", "B", "C", "D");
Flux<String> valid = Flux.just("A", "D");
Map<Boolean, List<String>> collect = all.stream()
.collect(Collectors.groupingBy(t -> valid.collectList().block().contains(t)));
System.out.println(collect.get(Boolean.TRUE));
System.out.println(collect.get(Boolean.FALSE));
}
how to get it working in non-blocking way?
Above is an example of what i am trying to do in web application. I receive list of object which is List all. Then i query database which return Flux . Flux returned by database will be subset of List all. I need to prepare two lists. List of items which are present in Flux of valid and List of items which are not present in Flux of valid
EDIT:
I converted Flux to Mono and List to Mono,
public static void main(String[] args) {
Mono<List<String>> all = Mono.just(List.of("A", "B", "C", "D"));
Mono<List<String>> valid = Mono.just(List.of("A", "D"));
var exist = all.flatMap(a -> valid.map(v -> a.stream().collect(Collectors.groupingBy(v::contains))));
System.out.println(exist.block().get(Boolean.TRUE));
System.out.println(exist.block().get(Boolean.FALSE));
}
There is no straightforward way of achieve this in reactive programming without breaking some of its semantics.
If you reflect back on what reactive programming tris to achieve and your problem statement, you should notice that those won't play well that much together.
Reactive programming, as the name suggests, is about reacting to events which in your case would be valid items emitted from your datastore. In a typical situation, you should have been programming your statement to compute some assertions around the emitted valid items then emit these (or some other transformations downstream). Unfortunately, you won't be able to compute the all and valid items intersection and diversion without stopping at some point (otherwise how would you know that an item you assumed non-valid is not emitted at some point by the valid publisher).
Though, to achieve the desired behavior, you will lean on memory to buffer items then trigger your validations.
Retrieving valid items should be achievable using the filterWhen operator paired with the hasElement one:
Flux<String> validItems = Flux.fromIterable(all)
.filterWhen(valid::hasElement);
To retrieve the invalid items, you can collect all and validItems merged together then filter out elements that do appear more than once:
Flux<String> inValidItems = Flux.fromIterable(all)
.mergeWith(validItems)
.collectList()
.flatMapIterable(list -> list.stream().filter(item -> Collections.frequency(list, item) == 1).collect(Collectors.toList()));
Thanks to #roddy for his answer to my query here
Copy pasting from earlier to set the context :
here is my data structure :
public class Premium{
private Map<String,Map<String,String>> valuesMap = new HashMap<String,Map<String,String>>();
public Map<String, Map<String, String>> getValuesMap() {
return valuesMap;
}
}
Sample values that will be present inside this 'valuesMap' :
Map<String,String> m1= new HashMap<String,String>();
m1.put("death","100");
m1.put("income","50");
valuesMap.put("Male",m1);
valuesMap.put("Female",m2);
....
Thanks to #Roddy now I can extract the map 'm1' embedded within 'valuesMap' for "Male"
rule "rule#7 testing me 001 "
when
// below line extracts 'valuesMap' from Premium object
$pr:Premium($masterMap:valuesMap)
// now have a handle to the embedded map for 'Male'
Map( $male: this["Male"] ) from $masterMap
// defining an object in which I want to populate the value from map obtained for male
$rulesResponse:RulesResponse();
then
System.out.println("rule#7 map " + $map);
// this is where in below code it is failing
$rulesResponse.abc = $innerMap.get("income");
end
when I am trying to extract the string from map against key 'income' and assign it to the 'RulesResponse' object it fails with :
[Error: unable to resolve method using strict-mode: java.lang.Object.get(java.lang.String)]
[Near : {... nse.abc = $innerMap.get("income"); ....}]
The response object is a simple POJO with getter and setter for attribute : abc
public class RulesResponse {
private String abc = "";
public String getAbc() {
return abc;
}
public void setAbc(String abc) {
this.abc = abc;
}
If I try and assign a hard coded value - it works and also reflects after the rule is executed
// this works
$rulesResponse.abc = "hard coded value";
When you get this["Male"] out of the map, it's an Object, not anything typed. It's basically due to type erasure -- Map<String, ?>.
You can get "income" out by doing Map( $income: this["income"]) from $male. Of course, now $income will too also be an Object so you'll need to cast it again. Could be as simple as a (String)$income on the right-hand side, or a $incomeStr: String() from $income on the left.
rule "Example"
when
$pr: Premium( $masterMap: valuesMap != null )
Map( $male: this["Male"] != null ) from $masterMap
Map( $income: this["income"] != null ) from $male
$rulesResponse: RulesResponse()
then
$rulesResponse.abc = (String)$income; // cast as necessary
end
We lose the nested type identity because of type erasure -- you've got a Map<String, ?> which becomes Map<String, Object> in practice.
Strongly suggest using a properly structured POJO instead of a Map as a rule input. Even if your actual code uses these nested maps (bad practice!), you should leverage a transform before calling the rules -- not only will your rules be a lot simpler and easier to work with, but they'll also be much more performant.
Even converting that inner map into an object will make things easier:
class GenderValues {
String death;
String income;
}
class Premium {
Map<String, GenderValues> valuesByGender;
}
Best practice would be to omit the Map entirely.
I have to following scenario
case class A(name:String)
class Eq { def isMe(s:String) = s == "ME" }
val a = List(A("ME")).toDS
a.filter(l => new Eq().isMe(l.name))
Does this create a new object Eq every time for each data point on each executor ?
Nice one! I didn't know there is a different filter method for a typed dataset.
In order to answer your question, I will do some deep dive into Spark internals.
filter on a typed Dtaset has the following signature:
def filter(func: T => Boolean): Dataset[T]
Note that func is parameterized with T, hence Spark needs to deserialize both your object A along with the function.
TypedFilter Main$$$Lambda$, class A, [StructField(name,StringType,true)], newInstance(class A)
where Main$$$Lambda$ is a randomly generated function name
During optimization phase it might be eliminated by the EliminateSerialization rule if the following condition is met:
ds.map(...).filter(...) can be optimized by this rule to save extra deserialization, but ds.map(...).as[AnotherType].filter(...) can not be optimized.
If the rule is applicable TypedFilter is replaced by Filter.
The catch here is a Filter's condition. In fact, it is another special expression named Invoke where:
targetObject is the filter function Main$$$Lambda$
functionName is apply since it is a regular Scala function.
Spark eventually runs in one of these two modes - generate code or interpreter. Let's concentrate on the first one as it is the default.
Here is a simplified stack trace of the methods invocation that will generate the code
SparkPlan.execute
//https://github.com/apache/spark/blob/03e30063127fd71bef8a14553381e805fe5b6679/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala#L596
-> WholeStageCodegenExec.execute
[child: Filter]
-> child.execute
[condition Invoke]
-> Invoke.genCode
//https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala#L345
-> doGenCode
Simplified code after generation phase:
final class GeneratedIteratorForCodegenStage1 extends BufferedRowIterator {
private Object[] references;
private scala.collection.Iterator input;
private UnsafeRowWriter writer = new UnsafeRowWriter();
public GeneratedIteratorForCodegenStage1(Object[] references) {
this.references = references;
}
public void init(Iterator inputs) {
this.inputs = inputs;
}
protected void processNext() throws IOException {
while (input.hasNext() && !stopEarly()) {
InternalRow row = input.next();
do {
//Create A object
UTF8String value = row.getUTF8String(0));
A a = new A(value.toString)
//Filter by A's value
result = (scala.Function1) references[0].apply(a);
if (!result) continue;
writer.write(0, value)
append((writer.getRow());
}
if (shouldStop()) return;
}
}
}
We can see that projection is constructed with an array of objects passed in references variable. But where and how many times the references variable is instantiated?
It is created during WholeStageCodegenExec and instantiated only once per partition.
And this leads us to the answer that, however, filter function will be created only once per partition and not per data point, the Eq and A classes will be created per data point.
If you are curious about where it has been added to the code context:
It happens here
where javaType is scala.function1 .
and value is the implementation - Main$$$Lambda$
I am trying to write a Play Framework asynchronous Action for the following URL:
POST /users/:userId/items
My database calls all return Future[...], where ... is Option[A] for find methods and Option[Id] for create methods.
I would like to check for the existence of the userId before trying to create the new item. I have a method Users.findById(userId) that returns a Future[Option[User]]. The result is Some(User) if the user exists and None if not. Items.create() also returns a Future[Option[itemId]].
I am trying to compose something using for:
for {
user <- Users.findById(userId)
if user.isDefined
} yield {
Items.create(...) map { itemId => Ok(itemId) } getOrElse NotFound
}
I would like to return Ok(itemId) if the item is successfully created. I'm not sure how to handle the error case. I would like to return NotFound if either the userId is invalid or the item cannot be created (maybe a field conflicts with a unique value already in the database).
I'm not sure what to put after the for structure. I tried getOrElse, but that does not compile, since Future does not have a getOrElse method.
Ideally, I can handle URLs containing several ids to check, e.g.:
PUT /users/:userId/foo/:fooId/bar/:barId
and confirm that userId, fooId, and barId are all valid before doing the update. All of those calls (Users.findById, Foo.findById, and Bar.findById) will return Future[Option[A]].
It's that double-nesting (Future of Option) that seems to get people every time. Things become a lot easier if you can flatten stuff out first.
In this case, Future already has a way of representing an error condition, it can wrap an Exception as well as a success value, that's something you can use...
// making this a Singleton avoids the cost of building a stack trace,
// which only happens when an Exception is constructed (not when it's thrown)
object NotFoundException extends RuntimeException("Empty Option")
// The map operation will trap any thrown exception and fail the Future
def squish[T](x: Future[Option[T]]) =
x map { _.getOrElse(throw NotFoundException) }
It's now a lot easier to use those squished results in a comprehension:
val result = for {
user <- squish(Users findById userId)
itemId <- squish(Items.create(user, ...))
} yield {
Ok(itemId)
} recover {
case NotFoundException => NotFound
}
Which will, of course, evaluate to a Future. This is async programming, after all :)
Any exceptions other than NotFoundException will still be exposed.
Is there a way to have an observable sequence to resume execution with the next element in the sequence if an error occurs?
From this post it looks like you need to specify a new observable sequence in Catch() to resume execution, but what if you needed to just continue processing with the next element in the sequence instead? Is there a way to achieve this?
UPDATE:
The scenario is as follows:
I have a bunch of elements that I need to process. The processing is made up of a bunch of steps. I have
decomposed the steps into tasks that I would like to compose.
I followed the guidelines for ToObservable() posted here
to convert by tasks to an observables for composition.
so basically I'm doing somethng like so -
foreach(element in collection)
{
var result = from aResult in DoAAsync(element).ToObservable()
from bResult in DoBAsync(aResult).ToObservable()
from cResult in DoCAsync(bResult).ToObservable()
select cResult;
result.subscribe( register on next and error handlers here)
}
or I could something like this:
var result =
from element in collection.ToObservable()
from aResult in DoAAsync(element).ToObservable()
from bResult in DoBAsync(aResult).ToObservable()
from cResult in DoCAsync(bResult).ToObservable()
select cResult;
What is the best way here to continue processing other elements even if let's say the processing of
one of the elements throws an exception. I would like to be able to log the error and move on ideally.
Both James & Richard made some good points, but I don't think they have given you the best method for solving your problem.
James suggested using .Catch(Observable.Never<Unit>()). He was wrong when he said that "will ... allow the stream to continue" because once you hit an exception the stream must end - that is what Richard pointed out when he mentioned the contract between observers and observables.
Also, using Never in this way will cause your observables to never complete.
The short answer is that .Catch(Observable.Empty<Unit>()) is the correct way to change a sequence from one that ends with an error to one that ends with completion.
You've hit on the right idea of using SelectMany to process each value of the source collection so that you can catch each exception, but you're left with a couple of issues.
You're using tasks (TPL) just to turn a function call into an observable. This forces your observable to use task pool threads which means that the SelectMany statement will likely produce values in a non-deterministic order.
Also you hide the actual calls to process your data making refactoring and maintenance harder.
I think you're better off creating an extension method that allows the exceptions to be skipped. Here it is:
public static IObservable<R> SelectAndSkipOnException<T, R>(
this IObservable<T> source, Func<T, R> selector)
{
return
source
.Select(t =>
Observable.Start(() => selector(t)).Catch(Observable.Empty<R>()))
.Merge();
}
With this method you can now simply do this:
var result =
collection.ToObservable()
.SelectAndSkipOnException(t =>
{
var a = DoA(t);
var b = DoB(a);
var c = DoC(b);
return c;
});
This code is much simpler, but it hides the exception(s). If you want to hang on to the exceptions while letting your sequence continue then you need to do some extra funkiness. Adding a couple of overloads to the Materialize extension method works to keep the errors.
public static IObservable<Notification<R>> Materialize<T, R>(
this IObservable<T> source, Func<T, R> selector)
{
return source.Select(t => Notification.CreateOnNext(t)).Materialize(selector);
}
public static IObservable<Notification<R>> Materialize<T, R>(
this IObservable<Notification<T>> source, Func<T, R> selector)
{
Func<Notification<T>, Notification<R>> f = nt =>
{
if (nt.Kind == NotificationKind.OnNext)
{
try
{
return Notification.CreateOnNext<R>(selector(nt.Value));
}
catch (Exception ex)
{
ex.Data["Value"] = nt.Value;
ex.Data["Selector"] = selector;
return Notification.CreateOnError<R>(ex);
}
}
else
{
if (nt.Kind == NotificationKind.OnError)
{
return Notification.CreateOnError<R>(nt.Exception);
}
else
{
return Notification.CreateOnCompleted<R>();
}
}
};
return source.Select(nt => f(nt));
}
These methods allow you to write this:
var result =
collection
.ToObservable()
.Materialize(t =>
{
var a = DoA(t);
var b = DoB(a);
var c = DoC(b);
return c;
})
.Do(nt =>
{
if (nt.Kind == NotificationKind.OnError)
{
/* Process the error in `nt.Exception` */
}
})
.Where(nt => nt.Kind != NotificationKind.OnError)
.Dematerialize();
You can even chain these Materialize methods and use ex.Data["Value"] & ex.Data["Selector"] to get the value and selector function that threw the error out.
I hope this helps.
The contract between IObservable and IObserver is OnNext*(OnCompelted|OnError)? which is upheld by all operators, even if not by the source.
Your only choice is to re-subscribe to the source using Retry, but if the source returns the IObservable instance for every description you won't see any new values.
Could you supply more information on your scenario? Maybe there is another way of looking at it.
Edit: Based on your updated feedback, it sounds like you just need Catch:
var result =
from element in collection.ToObservable()
from aResult in DoAAsync(element).ToObservable().Log().Catch(Observable.Empty<TA>())
from bResult in DoBAsync(aResult).ToObservable().Log().Catch(Observable.Empty<TB>())
from cResult in DoCAsync(bResult).ToObservable().Log().Catch(Observable.Empty<TC>())
select cResult;
This replaces an error with an Empty which would not trigger the next sequence (since it uses SelectMany under the hood.