Is there are good way to find out what the total number of records to be synced will be before the records are actually synced via the datastore? This is refering to at the start of time when I am going to sync the datastore with what's in the cloud (so the downstream sync). I'm wanting to create an actual progress indicator for the user (since it takes about a minute for ~1500 records to sync), and don't want to just put up a CircleProgressIndicator().
All I'm currently able to do is:
hubSubscription = Amplify.Hub.listen([HubChannel.DataStore], (msg) {
if (msg.eventName == "ready") {
getAllDevicesInDataStore().then((value) => stopListeningToHub());
}
if (kDebugMode) {
if (msg.eventName == "modelSynced") {
final syncedModelPayload = msg.payload as ModelSyncedEvent;
print(
'Model: ${syncedModelPayload.modelName}, Delta? ${syncedModelPayload.isDeltaSync}');
print(
'${syncedModelPayload.added}, ${syncedModelPayload.updated}, ${syncedModelPayload.deleted}');
}
}
});
I can implement a CircleProgressIndicator() while this is happening, but I want something more definitive.
Related
Pretty positive I'm just totally overthinking this or approaching it from an illogical angle.
I'm separating my logic from my ui where button presses call a method located in the userModel which has a change notifier (getting passed into MyApp with a provider). I'm trying to implement firebase but have never called firebase directly from the ui (always just had the requests in the ui code, never used a model).
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:flutter/material.dart';
import 'package:tuple/tuple.dart';
//TODO: firebase has been installed now I need to figure out how to implement it
// TODO: add firebase read and write
class UserModel with ChangeNotifier {
//index = session number
List session = [];
//create an instance of firebase (this might need to go higher in the tree)
var db = FirebaseFirestore.instance;
//TODO:
//convert incoming firebase to JSON
//convert outgoing json to firebase
//track session
// takes in current session data and adds the new chunk
// already recorded (new item in list but time, breaks etc. are adding from the last one)
// IF ADDING TO EXISTING, ALL PARAMETRS MUST BE SET
addSessionChunk(
{required String intention,
int workTime = 0,
String progress = "null",
int breakTime = 0}) {
session.add({
"intention": intention,
"workTime": workTime,
"progress": progress,
"breakTime": breakTime,
});
//firebase update?
}
//TODO: when returning to a previous intention, add to the numbers that were
//TODO: currently only works for 1 call per chunk (no going back to the same intention)
//get previous data from this intention for returning to a task (do
//these update functions updadate the LAST CHUNK in the session
updateChunkWorkTime({required int workTime}) {
//this later)
session.last["workTime"] = workTime;
}
//takes in inputed progress and updates the latest chunk with it
updateChunkProgress({required String progress}) {
session.last["progress"] = progress;
}
//takes inputed breaktime and updates the lastest chunk with it
updateChunkBreakTime({required int breakTime}) {
session.last["breakTime"] = breakTime;
}
//returns tuple of the total time spent working and breaking in the current session
calculateSessionTimeTotal() {
int totalWorkTime = 0;
int totalBreakTime = 0;
for (var chunk in session) {
totalWorkTime += chunk["workTime"] as int;
totalBreakTime += chunk["breakTime"] as int;
}
return Tuple2(totalWorkTime, totalBreakTime);
}
//firebase functions
pushDataUp() {
db.collection("sessions").doc().set({
"currentSession": session,
"total": calculateSessionTimeTotal().toString()
});
}
pullDataDown() {}
}
You can see at the bottom there I started to try and come up with a way to sync the local data state with firebase but am confused. Seems weird for the user to send their data up to firebase then back down into the model which is already holding that data?
Whats the best approach to local model and cloud database interaction? Any guidance in the right direction is greatly appreciated.
What seems weird to you, is actually a quite well defined patterns known as command query responsibility segregation, and is the basic pattern behind most modern UI frameworks. By separating the command (the writing of the data here) from the query (the reading of the data here) each remains simpler, and the total app becomes much easier to reason about.
With Firestore in Flutter, this usually translates into:
The user takes some action.
Your code writes to the database.
Your onSnapshot listener gets triggered with the updated data.
Your code updates the data model/state with the new data.
Which then renders the updated UI.
All of this happens pretty instantly, as Firebase actually handles it locally before even sending the data to the server and handles any exception that may occur during the synchronization with the server.
hopefully someone can point me in the right direction on this, I'm trying to delete log streams from cloud watch and have the following function, which deletes the events of the stream, but not the stream itself. I don't want a massive page of empty logs as I'm looping through these to check for an event, and eventually the loop is going to take ages due to al the empty stream logs
private async Task DeleteLog(IAmazonCloudWatchLogs client, GetLogEventsRequest eventsRequest)
{
var request = new DeleteLogStreamRequest
{
LogGroupName = eventsRequest.LogGroupName,
LogStreamName = eventsRequest.LogStreamName,
};
var response = await client.DeleteLogStreamAsync(request);
if (response.HttpStatusCode == System.Net.HttpStatusCode.OK)
{
Console.WriteLine($"Successfully deleted CloudWatch log stream, {eventsRequest.LogStreamName}.");
}
}
In short: which is most memory + cost efficient way to use Firestore snapshot listeners, unmount them always at screen unmount or have the unsubscribe function in context and unmount when whole site "unmounts"?
Lets say in home screen I use snapshot listener for collection "events" which has 100 documents. Now I navigate through the site and return to home screen 2 more times during using the site. In this case which is better memory and cost efficiently wise (is there also other things to consider) and is there drawbacks?
to mount and unmount the listener on each mount and unmount of the home screen.
to mount on home screen and to unmount in whole site "unmount" (for example using window.addEventListener('beforeunload', handleSiteClose).
The usage of first is probably familiar with most but usage of the second could be done with something like this:
-Saving listener unsubscribe function in context with collection name as key:
const { listenerHolder, setListenerHolder } = DataContext();
useEffect(() => {
const newListeners = anyDeepCopyFunction(listenerHolder);
const collection = 'events';
if (listenerHolder[collection] === undefined) {
//listenerBaseComponent would be function to establish listener and return unsubscribe function
const unSub = listenerBaseComponent();
if (unSub)
newListeners[collection] = unSub;
}
if (Object.entries(newListeners).length !== Object.entries(listenerHolder).length) {
setListenerHolder(newListeners);
}
}, []);
-Unmounting all listeners (in component that holds inside of it all screens and is unmounted only when whole site is closed):
const { listenerHolder, setListenerHolder } = DataContext();
const handleTabClosing = () => {
Object.entries(listenerHolder).forEach(item => {
const [key, value] = item;
if (typeof value === 'function')
value();
});
setListenerHolder({});
}
useEffect(() => {
window.addEventListener('beforeunload', handleTabClosing)
return () => {
window.removeEventListener('beforeunload', handleTabClosing)
}
})
In both cases the home screen is showing most recent from "events" collection, but in my understanding...
-The first approach creates listener 3 times to collection "events" and so 3 x 100 read operations are done.
-The second approach creates listener 1 time to collection "events" and so 1 x 100 read operations are done.
If storing the unsubscribe function to context is possible and all listener unsubscribtions are handled at once in site unmount or in logout, doesn't this make using it this way super easy, more maintainable and more cost efficient? If I would need to see data from "events" collection in any other screen I would not have to do get call / create a new listener, because I would always have latest data from "events" when site is used. Just check if there is (in this case) collection name as key in global state "listenerHolder", and if there is, there would be most up to date data always for events.
Since there wasn't information from others about this use case I made some testing myself jumping from this "homescreen" to another screen back and forth multiple times. "Homescreen" has about 150 items and second screen 65.
The results are from Firebase, Cloud Firestore usage tab:
This is the result of reads from that jumping: 654(1.52pm-1.53pm) + 597(1.53pm-1.54pm) = 1251 reads
Now I tested the same jumping back and forth when using global context listeners: 61(1.59pm-2.00pm) + 165(2.00pm-2.01pm) = 226 reads
So using listeners in global context will result significantly less reads. This is depending how many times new listeners (in normal use case) would need to be recreated.
I have not yet tested well enough memory usage comparing these two cases. But if I test it, I will add the results here for others to benefit.
My situation is the following:
I developed a game, where people can save their progress via google cloud. I'm releasing big content-updates, resulting in many people returning to my game at the same time, trying to get their savegame.
This overload causing my customers to get stucked in the savegame-loading process - not able to start playing with their progress.
Here's an updated 4 days screenshot of the cloud-api-dashboard
(And here's the old "12 hours" screenshot of the cloud-api-dashboard)
more informations about the Project:
The game keeps using the "save in cloud"-function in the background on some stages of the game to provide players with the functionality to play on two diffrent devices.
I'm using Unity 2019.3.9f1 and the Asset "Easy Mobile Pro 2.17.3" for the Game-Service-Feature.
The "Google Play Games Plugin" has the version "0.10.12" and can be found on github
more informations about the Cloud-Situation:
The OAuth "user type" is "External" (and can't be changed)
The OAuth user cap display shows "0/100" for the user-cap
And The OAuth rate limits is displaying this for the token-grant-rate (highest "Average Token Grant Rate" is 3,33 of 10.000 as limit)
All used quotas are within the limit. The project reaches
1/10 of "queries per day" (1.000.000.000 max) and
1/2 of "queries per 100 sec" (20.000 max).
more informations about the Error-Trace in the cloud-API:
On my search for a better Error-Log I tried to find “Cloud Logging”-tools in the “Google Cloud Platform”-console. But every section i tried won’t display anything:
“Logging” (Operations tool) is empty
“Cloud Logging API” says: “no data available for the selected time frame.”
“Cloud Debugger API” says: “no data available for the selected time frame.”
I can't find a more detailed variant of the errors as this (the "Metrics"-Section in the "Google Drive API"):
Is there anything I miss to get a better insight?
more informations about the Core-Code
As I mentioned, I’m using “EasyMobilePro”, so I have one “SaveGame”-Var and 8 calls for google and apple as well. I already contacted their support: They assured me that those calls are unchangeable & kind of rock solid (so it can’t be caused from their code) and I should try to contact google if the problem is not on my side.
The 5 calls from EasyMobile for cloudsave are:
bool “GameServices.IsInitialized()”
void “GameServices.OpenWithAutomaticConflictResolution”
void “GameServices.WriteSavedGameData”
void “GameServices.ReadSavedGameData”
void “GameServices.DeleteSavedGame”
The 3 calls from EasyMobile for cloud-login are:
void “GameServices.Init()”
delegate “GameServices.UserLoginSucceeded”
delegate “GameServices.UserLoginFailed”
The Process, that causes the Issue:
I call “GameService.Init()”, the user logs in (no problem)
On that “LoginSuccess”-Callback I call my Function “HandleFirstCloudOpening”:
//This Method is Called, after the player Pressed "Save/ Load" on the StartScreen
//The button is disabled imidiately (and will be re-enabled if an error/fail happens)
public void TryCallUserLogin() {
if (!IsLoginInit) {
EasyMobile.GameServices.UserLoginFailed += HandleLoginFail;
EasyMobile.GameServices.UserLoginSucceeded += HandleFirstCloudOpening;
IsLoginInit = true;
}
if (!IsGameServiceInitialized) {
EasyMobile.GameServices.Init();
} else { //This "else" is only be called, if the "Init" was successfull, but the player don't have a connected savegame
HandleFirstCloudOpening();
}
}
private void HandleLoginFail() {
//(...) Show ErrorPopup, let the player try to login again
}
private void HandleFirstCloudOpening() {
if (currentSaveState != CloudSaveState.NONE) {
CloudStateConflictDebug(CloudSaveState.OPENING);
return;
}
currentSaveState = CloudSaveState.OPENING;
EasyMobile.GameServices.SavedGames.OpenWithAutomaticConflictResolution(cloudSaveNameReference, UseFirstTimeOpenedSavedGame);
}
private void UseFirstTimeOpenedSavedGame(EasyMobile.SavedGame _savedGame, string _error) {
currentSaveState = CloudSaveState.NONE;
if (string.IsNullOrEmpty(_error)) {
cloudSaveGame = _savedGame;
ReadDataFromCloud(cloudSaveGame);
} else {
ErrorPopupWithCloseButton("cloud_open", "failed with error: " + _error);
}
}
private void ReadDataFromCloud(EasyMobile.SavedGame _savedGame) {
if (_savedGame.IsOpen) {
currentSaveState = CloudSaveState.LOADING;
EasyMobile.GameServices.SavedGames.ReadSavedGameData(_savedGame, UseSucessfullLoadedCloudSaveGame);
} else { //backup function if the fresh-opened savegame is "closed" for some reason (can happen later while "saving" ingame)
HandleFirstCloudOpening();
}
}
private void UseSucessfullLoadedCloudSaveGame(EasyMobile.SavedGame _game, byte[] _cloudData, string error) {
if (!string.IsNullOrEmpty(error)) {
ErrorPopupWithCloseButton("cloud_read", "Reading saved game data failed: " + error);
return;
}
if (_cloudData.Length > 0) {
//A function, that converts the saved bytes to my useable Savegame-Data
//having a "try&catch": if it fails, it useses the callback with the param "null"
SaveGameToByteConverter.LoadFromBytes<CoreSaveData>(_cloudData, UseSucessfullConvertedSavegameData);
} else {
//this will "fail", causing the use of the callback with the param "null"
SaveGameToByteConverter.LoadFromBytes<CoreSaveData>(null, UseSucessfullConvertedSavegameData);
}
}
private void UseSucessfullConvertedSavegameData(CoreSaveData _convertedSaveGame) {
//Has a Loaded & normal SaveGame in his cloud
if (_convertedSaveGame != null) {
//Loaded Save matches verify-conditions
if (CheckLoadedSaveIsVerified(_convertedSaveGame)) {
OverrideGameSaveDatawithLoaded(_convertedSaveGame);
ReloadCurrentScene();
return;
} else { //This happens if the cloud-save doesn't pass my verification-process
ErrorPopupWithCloseButton("cloud_loadedSave", "Couldn't find a compatible Savegame!");
return;
}
} else { //User uses Cloud-save for the frist Time or has an unusable savegame and gets a "new" (lost his old data)
TrySaveGameToCloud((bool _saved) => {
SaveAllGameFilesLocally();
});
}
}
I shrunk the code by removing most of my “if error happens, do XY”, since there are many and they would extend the reprex. If necessary I can provide a more detailed (but more complicated) code.
current conclusion
I can't find any issue on my side, that wouldn't have been fixed with a "restart of the game" or woudln't been covered by an error-popup for the user. It's like they are queued because of the amount of users and need to wait way too long for a response. Some users told us they had to wait & tried "x hours" (it's variable from 2h to 36h) and then they passed to play with their progress (so it worked). But some players mentioned they couldn't play again on the next day (same problem). Like their "access-token" only holds for a day?
Edit-History:
(1) updated the first dash-board-picture to match the ongoing situation
(1) added "more informations about the cloud-situation"
(1) can't find a more detailed error-log
(2) removed most pictures as displayables (kept the links)
(2) added "more informations about the Error-Trace in the cloud-API"
(2) added "more informations about the Core-Code" and a Reprex
(2) added "current conclusion"
I have reason to believe that some ServerValue.increment() commands are not executing.
In my App, when the user submits, two commands are executed:
Future<void> _submit() async {
alimentoBloc.descontarAlimento(foodId, quantity);
salidaAlimentoBloc.crearSalidaAlimento(salidaAlimento);
}
The first command updates the amount of inventory left in the warehouse (using ServerValue.increment)...
Future<bool> descontarAlimento(String foodId, int quantity) async {
try {
dbRef.child('inventory/$foodId/quantity')
.set(ServerValue.increment(-quantity));
} catch (e) {
print(e);
}
return true;
}
The second command makes a food output register, where it records the quantity, type of food and other key data.
Future<bool> crearSalidaAlimento(SalidaAlimentoModel salidaAlimento) async {
try {
dbRef.child('output')
.push().set(salidaAlimento.toJson());
} catch (e) {
print(e);
}
return true;
}
After several reviews, I have noticed that the increase command is not executed sometimes, and then the inventory does not correspond to what it should be.
Then, I would like to do something similar to a transaction, this is: If neither of the two commands is executed, do not execute either of the two.
Is it possible to do a batch of commands in Firebase Realtime without losing the offline functionalities?
You can do a multi-path update to perform both writes transactionally:
var id = dbRef.push().key;
Map<String, dynamic> updates = {
"inventory/$foodId/quantity": ServerValue.increment(-quantity),
"output/$id": salidaAlimento.toJson()
}
dbRef.update(updates);
With the above, either both writes are completed, or neither of them is.
While you're offline, the client will fire local events based on its best guess for the current value of the server (which is gonna be 0 if it never read the value), and it will then send all pending changes to the server when it reconnects. For a quick test, see https://jsbin.com/wuhuyih/2/edit?js,console
You can't use a transaction while the device is offline.
They need to check the current value on the database and that is not possible while offline. If you want to make sure that they succeed you would need to check if a connection is awailable or not.