how to sync firebase and local data model in flutter - flutter

Pretty positive I'm just totally overthinking this or approaching it from an illogical angle.
I'm separating my logic from my ui where button presses call a method located in the userModel which has a change notifier (getting passed into MyApp with a provider). I'm trying to implement firebase but have never called firebase directly from the ui (always just had the requests in the ui code, never used a model).
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/material.dart';
import 'package:tuple/tuple.dart';
//TODO: firebase has been installed now I need to figure out how to implement it
// TODO: add firebase read and write
class UserModel with ChangeNotifier {
//index = session number
List session = [];
//create an instance of firebase (this might need to go higher in the tree)
var db = FirebaseFirestore.instance;
//TODO:
//convert incoming firebase to JSON
//convert outgoing json to firebase
//track session
// takes in current session data and adds the new chunk
// already recorded (new item in list but time, breaks etc. are adding from the last one)
// IF ADDING TO EXISTING, ALL PARAMETRS MUST BE SET
addSessionChunk(
{required String intention,
int workTime = 0,
String progress = "null",
int breakTime = 0}) {
session.add({
"intention": intention,
"workTime": workTime,
"progress": progress,
"breakTime": breakTime,
});
//firebase update?
}
//TODO: when returning to a previous intention, add to the numbers that were
//TODO: currently only works for 1 call per chunk (no going back to the same intention)
//get previous data from this intention for returning to a task (do
//these update functions updadate the LAST CHUNK in the session
updateChunkWorkTime({required int workTime}) {
//this later)
session.last["workTime"] = workTime;
}
//takes in inputed progress and updates the latest chunk with it
updateChunkProgress({required String progress}) {
session.last["progress"] = progress;
}
//takes inputed breaktime and updates the lastest chunk with it
updateChunkBreakTime({required int breakTime}) {
session.last["breakTime"] = breakTime;
}
//returns tuple of the total time spent working and breaking in the current session
calculateSessionTimeTotal() {
int totalWorkTime = 0;
int totalBreakTime = 0;
for (var chunk in session) {
totalWorkTime += chunk["workTime"] as int;
totalBreakTime += chunk["breakTime"] as int;
}
return Tuple2(totalWorkTime, totalBreakTime);
}
//firebase functions
pushDataUp() {
db.collection("sessions").doc().set({
"currentSession": session,
"total": calculateSessionTimeTotal().toString()
});
}
pullDataDown() {}
}
You can see at the bottom there I started to try and come up with a way to sync the local data state with firebase but am confused. Seems weird for the user to send their data up to firebase then back down into the model which is already holding that data?
Whats the best approach to local model and cloud database interaction? Any guidance in the right direction is greatly appreciated.

What seems weird to you, is actually a quite well defined patterns known as command query responsibility segregation, and is the basic pattern behind most modern UI frameworks. By separating the command (the writing of the data here) from the query (the reading of the data here) each remains simpler, and the total app becomes much easier to reason about.
With Firestore in Flutter, this usually translates into:
The user takes some action.
Your code writes to the database.
Your onSnapshot listener gets triggered with the updated data.
Your code updates the data model/state with the new data.
Which then renders the updated UI.
All of this happens pretty instantly, as Firebase actually handles it locally before even sending the data to the server and handles any exception that may occur during the synchronization with the server.

Related

Flutter Future timeouts not always working correctly

Hey I need some help here for How to use timeouts in flutter correctly. First of all to explain what the main goal is:
I want to recive data from my Firebase RealTime Database but need to secure this request api call with an time out of 15 sec. So after 15 sec my timeout should throw an exception that will return to the Users frontend the alert for reasons of time out.
So I used the simple way to call timeouts on future functions:
This functions should only check if on some firebase node an ID is existing or not:
Inside this class where I have declared this functions I also have an instance which called : timeoutControl this is a class which contains a duration and some reasons for the exceptions.
Future<bool> isUserCheckedIn(String oid, String maybeCheckedInUserIdentifier, String onGateId) async {
try {
databaseReference = _firebaseDatabase.ref("Boarding").child(oid).child(onGateId);
final snapshot = await databaseReference.get().timeout(Duration(seconds: timeoutControl.durationForTimeOutInSec), onTimeout: () => timeoutControl.onEppTimeoutForTask());
if(snapshot.hasChild(maybeCheckedInUserIdentifier)) {
return true;
}
else {
return false;
}
}
catch (exception) {
return false;
}
}
The TimeOutClass where the instance timeoutControl comes from:
class CustomTimeouts {
int durationForTimeOutInSec = 15; // The seconds for how long to try until we throw an timeout exception
CustomTimeouts();
// TODO: Implement the exception reasons here later ...
onEppTimeoutForUpload() {
throw Exception("Some reason ...");
}
onEppTimeoutForTask() {
throw Exception("Some reason ...");
}
onEppTimeoutForDownload() {
throw Exception("Some reason ...");
}
}
So as you can see for example I tried to use this implementation above. This works fine ... sometimes I need to fight with un explain able things -_-. Let me try to introduce what in somecases are the problem:
Inside the frontend class make this call:
bool isUserCheckedIn = await service.isUserCheckedIn(placeIdentifier, userId, gateId);
Map<String, dynamic> data = {"gateIdActive" : isUserCheckedIn};
/*
The response here is an Custom transaction handler which contains an error or an returned param
etc. so this isn't relevant for you ...
*/
_gateService.updateGate(placeIdentifier, gateId, data).then((response) {
if(response.hasError()) {
setState(() {
EppDialog.showErrorToast(response.getErrorMessage()); // Shows an error message
isSendButtonDiabled = false; /*Reset buttons state*/
});
}
else {
// Create an gate process here ...
createGateEntrys(); // <-- If the closures update was successful we also handle some
// other data inside the RTDB for other reasons here ...
}
});
IMPORTANT to know for you guys is that I am gonna use the returned "boolean" value from this function call to update some other data which will be pushed and uploaded into another RTDB other node location for other reasons. And if this was also successful the application is going on to update some entrys also inside the RTDB -->createGateEntrys()<-- This function is called as the last one and is also marked as an async function and called with its closures context and no await statement.
The Data inside my Firebase RTDB:
"GateCheckIns" / "4mrithabdaofgnL39238nH" (The place identifier) / "NFdxcfadaies45a" (The Gate Identifier)/ "nHz2mhagadzadzgadHjoeua334" : 1 (as top of the key some users id who is checked in)
So on real devices this works always without any problems... But the case of an real device or simulator could not be the reason why I'am faceing with this problem now. Sometimes inside the Simulator this Function returns always false no matter if the currentUsers Identifier is inside the this child nodes or not. Therefore I realized the timeout is always called immediately so right after 1-2 sec because the exception was always one of these I was calling from my CustomTimeouts class and the function which throws the exception inside the .timeout(duration, onTimeout: () => ...) call. I couldn't figure it out because as I said on real devices I was not faceing with this problem.
Hope I was able to explain the problem it's a little bit complicated I know but for me is important that someone could explain me for what should I pay attention to if I am useing timeouts in this style etc.
( This is my first question here on StackOverFlow :) )

RN Web + Firebase: snapshot listeners unsubscribe in unmount vs in global unmount (using context)

In short: which is most memory + cost efficient way to use Firestore snapshot listeners, unmount them always at screen unmount or have the unsubscribe function in context and unmount when whole site "unmounts"?
Lets say in home screen I use snapshot listener for collection "events" which has 100 documents. Now I navigate through the site and return to home screen 2 more times during using the site. In this case which is better memory and cost efficiently wise (is there also other things to consider) and is there drawbacks?
to mount and unmount the listener on each mount and unmount of the home screen.
to mount on home screen and to unmount in whole site "unmount" (for example using window.addEventListener('beforeunload', handleSiteClose).
The usage of first is probably familiar with most but usage of the second could be done with something like this:
-Saving listener unsubscribe function in context with collection name as key:
const { listenerHolder, setListenerHolder } = DataContext();
useEffect(() => {
const newListeners = anyDeepCopyFunction(listenerHolder);
const collection = 'events';
if (listenerHolder[collection] === undefined) {
//listenerBaseComponent would be function to establish listener and return unsubscribe function
const unSub = listenerBaseComponent();
if (unSub)
newListeners[collection] = unSub;
}
if (Object.entries(newListeners).length !== Object.entries(listenerHolder).length) {
setListenerHolder(newListeners);
}
}, []);
-Unmounting all listeners (in component that holds inside of it all screens and is unmounted only when whole site is closed):
const { listenerHolder, setListenerHolder } = DataContext();
const handleTabClosing = () => {
Object.entries(listenerHolder).forEach(item => {
const [key, value] = item;
if (typeof value === 'function')
value();
});
setListenerHolder({});
}
useEffect(() => {
window.addEventListener('beforeunload', handleTabClosing)
return () => {
window.removeEventListener('beforeunload', handleTabClosing)
}
})
In both cases the home screen is showing most recent from "events" collection, but in my understanding...
-The first approach creates listener 3 times to collection "events" and so 3 x 100 read operations are done.
-The second approach creates listener 1 time to collection "events" and so 1 x 100 read operations are done.
If storing the unsubscribe function to context is possible and all listener unsubscribtions are handled at once in site unmount or in logout, doesn't this make using it this way super easy, more maintainable and more cost efficient? If I would need to see data from "events" collection in any other screen I would not have to do get call / create a new listener, because I would always have latest data from "events" when site is used. Just check if there is (in this case) collection name as key in global state "listenerHolder", and if there is, there would be most up to date data always for events.
Since there wasn't information from others about this use case I made some testing myself jumping from this "homescreen" to another screen back and forth multiple times. "Homescreen" has about 150 items and second screen 65.
The results are from Firebase, Cloud Firestore usage tab:
This is the result of reads from that jumping: 654(1.52pm-1.53pm) + 597(1.53pm-1.54pm) = 1251 reads
Now I tested the same jumping back and forth when using global context listeners: 61(1.59pm-2.00pm) + 165(2.00pm-2.01pm) = 226 reads
So using listeners in global context will result significantly less reads. This is depending how many times new listeners (in normal use case) would need to be recreated.
I have not yet tested well enough memory usage comparing these two cases. But if I test it, I will add the results here for others to benefit.

What impact does changing a IReliableQueue to a IReliableConcurrentQueue have in an existing deployment?

I am working in a Service Fabric application that uses IReliableQueue. For the uses cases of this system, the IReliableConcurrentQueue makes sense to use and some local testing (i.e. basically by just changing the code to use IReliableConcurrentQueue instead of IReliableQueue - queue name does not change) shows great performance improvements. However, I am worried about the impact of changing this in a production system (i.e. upgrading). I can't find any docs or online questions (unless I just missed them) about these considerations. For example, in this system, the existing IReliableQueue will almost always have items. So what happens to that data when I upgrade the SF application? Will it be available to dequeue in the IReliableConcurrentQueue? Or would data be lost? I know I can "just try it" but wanted to see if someone out there had done the same or could offer pointers to existing resources. Thanks!
Sorry for a late answer (that you probably don't need anymore but still).
When we calling GetOrAddAsync method on IReliableStateManager we aren't retrieving the interface to store values - we actually creating an instance of reliable collection. This basically means that type of the interface we specify is very important.
Taking this into account if we do this:
Service v. 1.0
// Somewhere in RunAsync for example
await this.StateManager.GetOrAddAsync<IReliableQueue<long>>("MyCollection")
Then doing this in the next version:
Service v. 1.1
// Somewhere in RunAsync for example
await this.StateManager.GetOrAddAsync<IReliableConcurrentQueue<long>>("MyCollection")
will throw an exception:
Returned reliable object of type Microsoft.ServiceFabric.Data.Collections.DistributedQueue`1[System.Int64] cannot be casted to requested type Microsoft.ServiceFabric.Data.Collections.IReliableConcurrentQueue`1[System.Int64]
and then:
System.ExecutionEngineException: 'Exception of type 'System.ExecutionEngineException' was thrown.'
The above exception looks like a bug so I have filled one.
UPDATE 2019.06.28
It turned out that appearance of System.ExecutionEngineException isn't a bug but rather an undocumented behavior of Environment.FailFast method in combination with Visual Studio debugger.
Please see my comment to the above issue.
This is what would happen.
There are plenty ways to overcome this.
Here is the most obvious one:
Example
var migrate = false; // This flag indicates whether the migration was already done.
var migrateValues = new List<long>();
var applicationFlags = await this.StateManager
.GetOrAddAsync<IReliableDictionary<string, bool>>("application-flags");
using (var transaction = this.StateManager.CreateTransaction())
{
var flag = await applicationFlags
.TryGetValueAsync(transaction, "queue-to-concurrent-queue-migration");
if (!flag.HasValue || !flag.Value)
{
var queue = await this.StateManager
.GetOrAddAsync<IReliableQueue<long>>("value-collection");
for (;;)
{
var c = await queue.TryDequeueAsync(transaction);
if (!c.HasValue)
{
break;
}
migrateValues.Add(c.Value);
}
migrate = true;
}
}
if (migrate)
{
await this.StateManager.RemoveAsync("value-collection");
using (var transaction = this.StateManager.CreateTransaction())
{
var concurrentQueue = await this.StateManager
.GetOrAddAsync<IReliableConcurrentQueue<long>>("value-collection");
foreach (var i in migrateValues)
{
await concurrentQueue.EnqueueAsync(transaction, i);
}
await applicationFlags.AddOrUpdateAsync(
transaction,
"queue-to-concurrent-queue-migration",
true,
(s, b) => true);
}
await transaction.CommitAsync();
}
Please note that this code is just an illustrative example and should be properly tested before applying it to real life application.

RXJS : Idiomatic way to create an observable stream from a paged interface

I have paged interface. Given a starting point a request will produce a list of results and a continuation indicator.
I've created an observable that is built by constructing and flat mapping an observable that reads the page. The result of this observable contains both the data for the page and a value to continue with. I pluck the data and flat map it to the subscriber. Producing a stream of values.
To handle the paging I've created a subject for the next page values. It's seeded with an initial value then each time I receive a response with a valid next page I push to the pages subject and trigger another read until such time as there is no more to read.
Is there a more idiomatic way of doing this?
function records(start = 'LATEST', limit = 1000) {
let pages = new rx.Subject();
this.connect(start)
.subscribe(page => pages.onNext(page));
let records = pages
.flatMap(page => {
return this.read(page, limit)
.doOnNext(result => {
let next = result.next;
if (next === undefined) {
pages.onCompleted();
} else {
pages.onNext(next);
}
});
})
.pluck('data')
.flatMap(data => data);
return records;
}
That's a reasonable way to do it. It has a couple of potential flaws in it (that may or may not impact you depending upon your use case):
You provide no way to observe any errors that occur in this.connect(start)
Your observable is effectively hot. If the caller does not immediately subscribe to the observable (perhaps they store it and subscribe later), then they'll miss the completion of this.connect(start) and the observable will appear to never produce anything.
You provide no way to unsubscribe from the initial connect call if the caller changes its mind and unsubscribes early. Not a real big deal, but usually when one constructs an observable, one should try to chain the disposables together so it call cleans up properly if the caller unsubscribes.
Here's a modified version:
It passes errors from this.connect to the observer.
It uses Observable.create to create a cold observable that only starts is business when the caller actually subscribes so there is no chance of missing the initial page value and stalling the stream.
It combines the this.connect subscription disposable with the overall subscription disposable
Code:
function records(start = 'LATEST', limit = 1000) {
return Rx.Observable.create(observer => {
let pages = new Rx.Subject();
let connectSub = new Rx.SingleAssignmentDisposable();
let resultsSub = new Rx.SingleAssignmentDisposable();
let sub = new Rx.CompositeDisposable(connectSub, resultsSub);
// Make sure we subscribe to pages before we issue this.connect()
// just in case this.connect() finishes synchronously (possible if it caches values or something?)
let results = pages
.flatMap(page => this.read(page, limit))
.doOnNext(r => this.next !== undefined ? pages.onNext(this.next) : pages.onCompleted())
.flatMap(r => r.data);
resultsSub.setDisposable(results.subscribe(observer));
// now query the first page
connectSub.setDisposable(this.connect(start)
.subscribe(p => pages.onNext(p), e => observer.onError(e)));
return sub;
});
}
Note: I've not used the ES6 syntax before, so hopefully I didn't mess anything up here.

RxJava - Somewhat complicated API requests, together with details of progress

To start of, I've only been playing around with RxJava/RxAndroid for a couple of days.
In psuedocode I'm trying to accomplish something like this when loading data from the Guild Wars 2 API:
refreshDatabase {
getAllIds - single API call (50.000+ ids)
chunk ids into chunks of 100 each
for each chunk
getItems - single API call which fetches 100 items
update ui on the progress (# chunks out of # chunks done)
loop through all items when all chunks are fetched
determine if the item is something we're after
save it in our own database
update ui on progress (# items out of # items done)
return bool telling if successful or not
}
I've read through a lot of tutorials and documentation about it, but I'm not really grasping how to code this.
I've got finished and working logic for all of it if I want the sorting/saving to the database to run directly after the chunk has been fetched, but then I'm unable to update the UI with the details.
Since this is an operation that takes quite a while on the first run I'd like the progress to be pretty detailed rather than saving .5-2 seconds of loading time.
Any tips on how I get make it something like what I put in the psuedocode?
Divide an conquer.
You actually have 2 or 3 different operations, so don't try to do this in one function.
First of all, you should use the WorkManager from AndroidX Jetpack to perform the sync the local DB, and Room for reading and writing to a local SQLite database.
Starting with fetching all relevant items and storing them in your DB (not doing UI udpates here):
class MyRepository(private val api: MyApi, private val dao:MyItemDao) {
fun isItemInteresting(item: Item): Boolean {
return true
}
fun fetchAllInterestingItems(): Single<List<Item>> {
return api.getAllIds()
.flatMapIterable { it }
.buffer(100)
.flatMap { api.getItems(it) }
.flatMapIterable { it }
.filter { isItemInteresting(it) }
.toList()
}
fun updateDatabase(items: List<Item>): Completable {
return dao.storeItems(items)
}
}
class SyncWorker(
context: Context,
params: WorkerParameters,
private val repository: MyRepository
) : RxWorker(context, params) {
override fun createWork(): Single<Result> {
return repository.fetchAllInterestingItems()
.flatMapCompletable { repository.updateDatabase(it) }
.subscribeOn(Schedulers.io())
.toSingle { Result.success()}
}
}
This will fetch all items, check which are relevant, and store those in your database (MyApi would be a Retrofit interface and MyItemDao a Room Dao interface).
Reporting progress to the UI can be done through the progress reporting API in WorkManager (version 2.3.0, currently in beta).