I am very confused about how actually the StreamController is implemented in dart.
So tell me if this code cause any memory leaks.
class Backend{
final streams = <int,StreamController>{};
final ws;
StreamHandler(this.ws){
listenToWebSocket();
}
listenToWebSocket(){
ws.listen((e){
streams[e['index']].add(e['data']);
});
}
getStream(int index){
var s = StreamController();
streams[index] = s;
return s.stream;
}
}
Backend listens to the websocket and passes its data to the stream appropriate index.
class StreamListenerHandler{
Map<int,StreamListener> listeners = {};
addListener(int index){
map[index] = listeners;
}
}
class StreamListener{
final int index;
StreamListener(this.index){
startListening();
}
startListening(){
getIt<Backend>().getStream().listen((data){
//do stuff
});
}
}
StreamListener wants the websocket data. StreamListenerHandler stores the listeners.
(for those who don't know getIt package just accesses the global singleton type thing.)
so what happens if...
void main(){
var handler = StreamListenerHandler();
handler.addListener(5);
// after some time...
handler.addListener(5);
}
here on the first call of addListener a StreamListener is created and it recieves a fresh stream from the Backend.
here is my expectation about the second call...
the StreamListenerHandler replaces the current StreamListener with a new one. Then the new StreamListener calls Backend and gets a fresh stream. The old streamController has no references so it's disposed. For the old StreamListener, the reference in the StreamListenerHandler is lost and also as the old streamController is gone the listen callback is also worthless, so it will be disposed.
So I am closing nothing just removing the references. Will this cause memory wastage?
does the garbage collector calls close on the controller or just disposing the object is enough?
(I am asking all this because all over the internet people are saying you should close the streams. I don't like the "should". I want to do it only if it "required")
Related
i have just recently stated working with riverpod state mangement in flutter.
i have issue related to comunicate between to state providers.
here is my sample code:
class SomeClass_ONE extends stateNotifer <SomeState> {
SomeClass_ONE({required this.somevalue}):super(null);
final SomeCustomClass somevalue;
void methodOne(SomeState newstatevalue){
state = newstatevalue;
}
}
final someClassOneProvider =
StateNotifierProvider<SomeClass_ONE,SomeState>.
((ref)=>SomeClass_ONE(somevalue: SomeCustomClass()));
now i have another state provider class as below
class SomeClass_Two extends stateNotifer <SomeStateTwo> {
SomeClass_ONE({required this.somevalue}):super(null);
final SomeCustomClass somevalue;
void methodtwo(SomeState newstatevalue){
state = newstatevalue;
}
}
final someClassTwoProvider =
StateNotifierProvider<SomeClass_Two,SomeStateTwo>
((ref)=>someClassTwoProvider(somevalue: SomeCustomClass()));
now what i want to achhive is that on methodOne execution i have to listen that state cahnge and have to trigger methodTow and have to upate secondproviders state as well.
so how can i achive this without using Ref in class cunstroctors?
i have tried with ref.listner to trigger and have passed Ref in both class constructors. but as per some condition i can't use Ref directly in constructors as per some guideline followed by seniors.
You can pass a Ref ref object to the methodtwo method and then call the necessary methods from other StateNotifierProvider. In any case, to refer to other methods of other classes, you need to have a Ref object.
Try to use watch provided by StateNotifierProvider
Try this code:
class SomeClass_ONE extends stateNotifer <SomeState> {
SomeClass_ONE({required this.somevalue}):super(null);
final SomeCustomClass somevalue;
void methodOne(SomeState newstatevalue){
state = newstatevalue;
// Listen to the changes in the state of the first provider and call the methodtwo of the second provider
someClassTwoProvider.watch((_) => _.methodtwo(newstatevalue));
}
}
I want to fuse the inputs of several Android sensors and expose the output as an observable (or at least something that can be subscribed to) that supports multiple simultaneous observers. What's the idiomatic way to approach this? Is there a class in the standard library that would make a good starting point?
I was thinking of wrapping a PublishSubject in an object with delegates for one or more subscribe methods that test hasObservers to activate the sensors, and wrap the returned Disposable in a proxy that tests hasObservers to deactivate them. Something like this, although this already has some obvious problems:
public class SensorSubject<T> {
private final PublishSubject<T> mSubject = PublishSubject.create();
public Disposable subscribe(final Consumer<? super T> consumer) {
final Disposable d = mSubject.subscribe(consumer);
if(mSubject.hasObservers()) {
// activate sensors
}
return new Disposable() {
#Override
public void dispose() {
// possible race conditions!
if(!isDisposed()) {
d.dispose();
if(!mSubject.hasObservers()) {
// deactivate sensors
}
}
}
#Override
public boolean isDisposed() {
return d.isDisposed();
}
};
}
}
The idiomatic way to do that in RxJava would be to use hot observable.
Cold observables do some action when someone subscribes to them and emit all items to that subscriber. So it's 1 to 1 relation.
Hot observable do some action and emits items independently on individual subscription. So if you subscribe too late, you might not get some values that were emitted earlier. This is 1 to many relation, aka multicast - which is what you want.
Usual way to do it is Flowable.publish() which makes Flowable multicast, but requires calling connect() method to start emitting values.
In your case you can also call refCount() which adds your desired functionality - it subscribes to source Flowable when there is at least one subscription and unsubscribes when everyone unsubsribed.
Because publish().refCount() is pretty popular combination, there is a shortcut for them - share(). And as far as I understand this is exactly what you want.
Edit by asker: This code incorporates this answer and David Karnok's comment in the form of a Dagger 2 provider method. SimpleMatrix is from EJML. This seems to be doing what I asked for.
#Provides
#Singleton
#Named(MAGNETOMETER)
public Observable<SimpleMatrix> magnetometer(final SensorManager sensorManager) {
final PublishSubject<SimpleMatrix> ps = PublishSubject.create();
final Sensor sensor = sensorManager.getDefaultSensor(TYPE_MAGNETIC_FIELD);
final SensorEventListener listener = new SensorEventAdapter() {
#Override
public void onSensorChanged(final SensorEvent event) {
ps.onNext(new SimpleMatrix(1, 3, true, event.values));
}
};
return ps.doOnSubscribe(s -> {
sensorManager.registerListener(listener, sensor, SENSOR_DELAY_NORMAL);
}).doOnDispose(() -> {
sensorManager.unregisterListener(listener);
}).share();
}
The Intro To RX book describes the return value on OnSubscribe as IDisposible and notes that subscriptions should be disposed of when OnError and OnCompleted are called.
An interesting thing to consider is that when a sequence completes or
errors, you should still dispose of your subscription.
From Intro to RX: Lifetime Management, OnError and OnCompleted
Why is this?
For reference, this is the class I'm currently working on. I'm probably going to submit it to code review at some point.
using System;
using System.Reactive;
using System.Reactive.Linq;
using System.Reactive.Subjects;
/// <summary>
/// Provides a timeout mechanism that will not timeout if it is signalled often enough
/// </summary>
internal class TrafficTimeout
{
private readonly Action onTimeout;
private object signalLock = new object();
private IObserver<Unit> signals;
/// <summary>
/// Initialises a new instance of the <see cref="TrafficTimeout"/> class.
/// </summary>
/// <param name="timeout">The duration to wait after receiving signals before timing out.</param>
/// <param name="onTimeout">The <see cref="Action"/> to perform when the the timeout duration expires.</param>
public TrafficTimeout(TimeSpan timeout, Action onTimeout)
{
// Subscribe to a throttled observable to trigger the expirey
var messageQueue = new BehaviorSubject<Unit>(Unit.Default);
IDisposable subscription = null;
subscription = messageQueue.Throttle(timeout).Subscribe(
p =>
{
messageQueue.OnCompleted();
messageQueue.Dispose();
});
this.signals = messageQueue.AsObserver();
this.onTimeout = onTimeout;
}
/// <summary>
/// Signals that traffic has been received.
/// </summary>
public void Signal()
{
lock (this.signalLock)
{
this.signals.OnNext(Unit.Default);
}
}
}
The disposable returned by the Subscribe extension methods is returned solely to allow you to manually unsubscribe from the observable before the observable naturally ends.
If the observable completes - with either OnCompleted or OnError - then the subscription is already disposed for you.
Try this code:
var xs = Observable.Create<int>(o =>
{
var d = Observable.Return(1).Subscribe(o);
return Disposable.Create(() =>
{
Console.WriteLine("Disposed!");
d.Dispose();
});
});
var subscription = xs.Subscribe(x => Console.WriteLine(x));
If you run the above you'll see that "Disposed!" is written to the console when the observable completes without you needing call .Dispose() on the subscription.
One important thing to note: the garbage collector never calls .Dispose() on observable subscriptions, so you must dispose of your subscriptions if they have not (or may not have) naturally ended before your subscription goes out of scope.
Take this, for example:
var wc = new WebClient();
var ds = Observable
.FromEventPattern<
DownloadStringCompletedEventHandler,
DownloadStringCompletedEventArgs>(
h => wc.DownloadStringCompleted += h,
h => wc.DownloadStringCompleted -= h);
var subscription =
ds.Subscribe(d =>
Console.WriteLine(d.EventArgs.Result));
The ds observable will only attach to the event handler when it has a subscription and will only detach when the observable completes or the subscription is disposed of. Since it is an event handler the observable will never complete because it is waiting for more events, and hence disposing is the only way to detach from the event (for the above example).
When you have a FromEventPattern observable that you know will only ever return one value then it is wise to add the .Take(1) extension method before subscribing to allow the event handler to automatically detach and then you don't need to manually dispose of the subscription.
Like so:
var ds = Observable
.FromEventPattern<
DownloadStringCompletedEventHandler,
DownloadStringCompletedEventArgs>(
h => wc.DownloadStringCompleted += h,
h => wc.DownloadStringCompleted -= h)
.Take(1);
I hope this helps.
Firstly, here's an example of the trouble you can run into:
void Main()
{
Console.WriteLine(GC.GetTotalMemory(true));
for (int i = 0; i < 1000; i++)
{
DumbSubscription();
Console.WriteLine(GC.GetTotalMemory(true));
}
Console.WriteLine(GC.GetTotalMemory(true));
}
public void DumbSubscription()
{
Observable.Interval(TimeSpan.FromMilliseconds(50))
.Subscribe(i => {});
}
You will see your memory usage go up forever. Active Rx subscriptions do not get garbage collected and this observable is infinite. Therefore, if you increase the loop limit, or add a delay, and you'll simply have more wasted memory: Nothing will help you except for disposing of those subscriptions.
However, let's say we change the definition of DumbSubscription to this:
public void DumbSubscription()
{
Observable.Interval(TimeSpan.FromMilliseconds(50))
.Take(1)
.Subscribe(i => {});
}
The .Take(1) addition means that the observable will complete after one interval, so it's no longer infinite. You'll see that your memory usage stabilizes: Subscriptions tend to properly dispose of themselves upon completion or exception.
However, that doesn't change the fact that, like any other IDisposable, it's a best practice to call Dispose (either manually or via using) to make sure resources are properly disposed of. Additionally, if you tweak your observable, you can easily run into the memory leak problem pointed out at the beginning.
My requirement is to start a long running process to tag all the products that are expired. This is run every night at 1:00 AM. The customers may be accessing some of the products on the website, so they have instances around the time when the job is run. The others are in the persistent media, not yet having instances because the customers are not accessing them.
Where should I hook up the logic to read the latest state of an actor from a persistent media and create a brand new actor? Should I have that call in the Prestart override method? If so, how can I tell the ProductActor that a new actor being created.
Or should I send a message to the ProductActor like LoadMeFromAzureTable which will load the state from the persistent media after an actor being created?
There are different ways to do it depending on what you need, as opposed to there being precisely one "right" answer.
You could use a Persistent Actor to recover state from a durable store automatically on startup (or in case of crash, to recover). Or, if you don't want to use that module (still in beta as of July 2015), you could do it yourself one of two ways:
1) You could load your state in PreStart, but I'd only go with this if you can make the operation async via your database client and use the PipeTo pattern to send the results back to yourself incrementally. But if you need to have ALL the state resident in memory before you start doing work, then you need to...
2) Make a finite state machine using behavior switching. Start in a gated state, send yourself a message to load your data, and stash everything that comes in. Then switch to a receiving state and unstash all messages when your state is done loading. This is the approach I prefer.
Example (just mocking the DB load with a Task):
public class ProductActor : ReceiveActor, IWithUnboundedStash
{
public IStash Stash { get; set; }
public ProductActor()
{
// begin in gated state
BecomeLoading();
}
private void BecomeLoading()
{
Become(Loading);
LoadInitialState();
}
private void Loading()
{
Receive<DoneLoading>(done =>
{
BecomeReady();
});
// stash any messages that come in until we're done loading
ReceiveAny(o =>
{
Stash.Stash();
});
}
private void LoadInitialState()
{
// load your state here async & send back to self via PipeTo
Task.Run(() =>
{
// database loading task here
return new Object();
}).ContinueWith(tr =>
{
// do whatever (e.g. error handling)
return new DoneLoading();
}).PipeTo(Self);
}
private void BecomeReady()
{
Become(Ready);
// our state is ready! put all those stashed messages back in the mailbox
Stash.UnstashAll();
}
private void Ready()
{
// handle those unstashed + new messages...
ReceiveAny(o =>
{
// do whatever you need to do...
});
}
}
/// <summary>
/// Marker interface.
/// </summary>
public class DoneLoading {}
I am writing a server in netty, in which I need to make a call to memcached. I am using spymemcached and can easily do the synchronous memcached call. I would like this memcached call to be async. Is that possible? The examples provided with netty do not seem to be helpful.
I tried using callbacks: created a ExecutorService pool in my Handler and submitted a callback worker to this pool. Like this:
public class MyHandler extends ChannelInboundMessageHandlerAdapter<MyPOJO> implements CallbackInterface{
...
private static ExecutorService pool = Executors.newFixedThreadPool(20);
#Override
public void messageReceived(ChannelHandlerContext ctx, MyPOJO pojo) {
...
CallingbackWorker worker = new CallingbackWorker(key, this);
pool.submit(worker);
...
}
public void myCallback() {
//get response
this.ctx.nextOutboundMessageBuf().add(response);
}
}
CallingbackWorker looks like:
public class CallingbackWorker implements Callable {
public CallingbackWorker(String key, CallbackInterface c) {
this.c = c;
this.key = key;
}
public Object call() {
//get value from key
c.myCallback(value);
}
However, when I do this, this.ctx.nextOutboundMessageBuf() in myCallback gets stuck.
So, overall, my question is: how to do async memcached calls in Netty?
There are two problems here: a small-ish issue with the way you're trying to code this, and a bigger one with many libraries that provide async service calls, but no good way to take full advantage of them in an async framework like Netty. That forces users into suboptimal hacks like this one, or a less-bad, but still not ideal approach I'll get to in a moment.
First the coding problem. The issue is that you're trying to call a ChannelHandlerContext method from a thread other than the one associated with your handler, which is not allowed. That's pretty easy to fix, as shown below. You could code it a few other ways, but this is probably the most straightforward:
private static ExecutorService pool = Executors.newFixedThreadPool(20);
public void channelRead(final ChannelHandlerContext ctx, final Object msg) {
//...
final GetFuture<String> future = memcachedClient().getAsync("foo", stringTranscoder());
// first wait for the response on a pool thread
pool.execute(new Runnable() {
public void run() {
String value;
Exception err;
try {
value = future.get(3, TimeUnit.SECONDS); // or whatever timeout you want
err = null;
} catch (Exception e) {
err = e;
value = null;
}
// put results into final variables; compiler won't let us do it directly above
final fValue = value;
final fErr = err;
// now process the result on the ChannelHandler's thread
ctx.executor().execute(new Runnable() {
public void run() {
handleResult(fValue, fErr);
}
});
}
});
// note that we drop through to here right after calling pool.execute() and
// return, freeing up the handler thread while we wait on the pool thread.
}
private void handleResult(String value, Exception err) {
// handle it
}
That will work, and might be sufficient for your application. But you've got a fixed-sized thread pool, so if you're ever going to handle much more than 20 concurrent connections, that will become a bottleneck. You could increase the pool size, or use an unbounded one, but at that point, you might as well be running under Tomcat, as memory consumption and context-switching overhead start to become issues, and you lose the scalabilty that was the attraction of Netty in the first place!
And the thing is, Spymemcached is NIO-based, event-driven, and uses just one thread for all its work, yet provides no way to fully take advantage of its event-driven nature. I expect they'll fix that before too long, just as Netty 4 and Cassandra have recently by providing callback (listener) methods on Future objects.
Meanwhile, being in the same boat as you, I researched the alternatives, and not being too happy with what I found, I wrote (yesterday) a Future tracker class that can poll up to thousands of Futures at a configurable rate, and call you back on the thread (Executor) of your choice when they complete. It uses just one thread to do this. I've put it up on GitHub if you'd like to try it out, but be warned that it's still wet, as they say. I've tested it a lot in the past day, and even with 10000 concurrent mock Future objects, polling once a millisecond, its CPU utilization is negligible, though it starts to go up beyond 10000. Using it, the example above looks like this:
// in some globally-accessible class:
public static final ForeignFutureTracker FFT = new ForeignFutureTracker(1, TimeUnit.MILLISECONDS);
// in a handler class:
public void channelRead(final ChannelHandlerContext ctx, final Object msg) {
// ...
final GetFuture<String> future = memcachedClient().getAsync("foo", stringTranscoder());
// add a listener for the Future, with a timeout in 2 seconds, and pass
// the Executor for the current context so the callback will run
// on the same thread.
Global.FFT.addListener(future, 2, TimeUnit.SECONDS, ctx.executor(),
new ForeignFutureListener<String,GetFuture<String>>() {
public void operationSuccess(String value) {
// do something ...
ctx.fireChannelRead(someval);
}
public void operationTimeout(GetFuture<String> f) {
// do something ...
}
public void operationFailure(Exception e) {
// do something ...
}
});
}
You don't want more than one or two FFT instances active at any time, or they could become a drain on CPU. But a single instance can handle thousands of outstanding Futures; about the only reason to have a second one would be to handle higher-latency calls, like S3, at a slower polling rate, say 10-20 milliseconds.
One drawback of the polling approach is that it adds a small amount of latency. For example, polling once a millisecond, on average it will add 500 microseconds to the response time. That won't be an issue for most applications, and I think is more than offset by the memory and CPU savings over the thread pool approach.
I expect within a year or so this will be a non-issue, as more async clients provide callback mechanisms, letting you fully leverage NIO and the event-driven model.