I am trying to get the download urls from 300 files inside the firebase storage :
items.forEach((element) {
setUrls(element.id);
});
void setUrls(String id) async {
String imageurl = await storageRef.child('id').getDownloadURL();
saveUrl(imageurl, id);
}
but it takes a lot of time to get all the files urls, how can i speed it up ?
Tried to call method to excute all 300 files at the same time, but still very slow, it gets one file at a time.
You could use the .listAll() method which buffers results in memory. You can even do this recursively if you need to go additional layers. Then call forEach like you did. Honestly not sure what kind of time this would save but it might help.
I would create a separate Firestore collection which stores just the id and URL of each image. Be sure to call getDownloadURL and save the result when images are added or changed. Then create a custom index to increase read speed. Just query that collection and loop the results.
Related
Is there a way to get all the URLs of the files located in a specific firebase storage library?
Currently using the following method:
List<String> URLs = [];
final ListResult _files = await _fireStorage.ref('groups/').list();
for (var element in _files.items) {
String addable = await element.getDownloadURL();
URLs.add(addable);
}
MY problem is that I have lots of files located in the groups folder and this async call takes like a ton of time to fetch for all individual elements. Is there any API call to get the links in bulk? Or should I just store the links in a firebase document and fetch them from there?
Thanks in advance!
In order to successfully return the downloadURL the client has to ping the storage bucket. Firebase then has to check the security rules to ensure that the operation is allowed, then verify that the object exists, and then return it's download URL to you. Unfortunately there is currently no way of doing this process in bulk.
It's quite common to store the downloadURL in a Firestore document so you can avoid the additional overhead and retrieve the downloadURL directly. But remember, downloadURLs are public and do not adhere to security rules. So make sure the document you're storing them in has proper rules placed on it.
Remember to perform a clean up operation and remove the reference in Firestore if you every delete the file.
To reduce the number of reads it is a general technique to maintain timestamp of last edits in documents and comparing timestamp to load only modified documents.
Here is an example from firebase docs:
db.collection('groups')
.where('participants', 'array-contains', 'user123')
.where('lastUpdated', '>', lastFetchTimestamp)
.orderBy('lastUpdated', 'desc')
.limit(25)
They claim this would reduce the reads.
I tried implementing the use-case, I have a document as shown below:
I have sections in my app where I use scorecards to list top scorers, My query is as follows
private void loadFriendScores(UserScorecard scorecard) {
Query friendScoreQuery=scorecardRef.whereIn("uid", scorecard.getFriendsList())
.whereGreaterThan("lastActive", scorecard.getLastActive()).limit(5);
FirestoreRecyclerOptions<UserScorecard> friends = new FirestoreRecyclerOptions
.Builder<UserScorecard>()
.setQuery(friendScoreQuery, UserScorecard.class)
.setLifecycleOwner(getViewLifecycleOwner())
.build();
TopScoresAdapter friendsAdapter = new TopScoresAdapter(friends, getContext(), this);
binding.topScorersFriendsRcv.setAdapter(friendsAdapter);
binding.topScorersFriendsRcv.setLayoutManager(new LinearLayoutManager(getContext()));
}
I assumed the query to load all modified changes along with others (from cache):
The screen on android is as follows:
While I expected it to load all of my friendlist (as I understood from docs).
I suppose they did not mention that we need to fetch the cached list, there is a way to do a cached request in firestore.
but I'm not sure if this is reliable perhaps the cache will be cleaned and the last request would be empty ,
then, you should save the last response using the localstorage library
#react-native-async-storage/async-storage
I'm struggling myself with the costs issue. The reads are way higher then 50 reads and I'm not sure how to count them properly. so I upvoted the issue
I want to store documents that I download from firestore. Each document has a unique id so I was thinking of a way I can store that document in json format or in database table (so I can access its fields) and if that id is not present in the memory than just simply download it from firestore and save it. I also want these documents to be deleted if they are not used/called for a long time. I found this cache manager but was unable to understand how to use it.
you can download the content by using any of the flutter downloadmanager and keep the path of the downloaded content in sqllite and load the data from local storage rather than url if the data is already saved.
Following pseudo-code may help you:
if(isDownloaded(id)){
//show from local;
} else {
// show from remote
}
you can manually check whether the resources are being used or not by keeping a timestam of it and update the timestamp only when it is shown. and you can run a service which looks for the unused resources and delete it from storage and the databse too.
I have 100 documents
Document id
0001
0002
0003
....
....
0100
and if we load 5 documents with id 001,002,004,005,006
then firestore charge for 5 document reads and then we again load(call the read operation query) documents with id 004,005,006,007,008,001,002
then firestore will charge for 7 document reads
here on first time we already loaded document with ids 001,002,004,005,006 and in second time or refresh time we are loading documents already loaded and some new documents
Here we need to avoid multiple times reading document from server and read it from cash and need to avoid the firestore over document read charges How to do it?
Firestore have cash loading option but it will only load from cash and not from server here what we need is load exiting data from cash and load remaning data form server.Here now what firestore doing is it will load from server and if it is failed then it will read from cash that is ok but i need in revise order
Now what happening is if non internet all data load faster with out showing progress and if there is internet it will take few sec to load and it will shows loader When we do it without fireabase our app will shows loading only one time then first it will shows the data from sqlite then when ever the api call resoppnce reached we will update in ui, so users will not face any loader but with firestore user need to wait for a progress bar to finish
From a first glance it seems that you may use firebase firestore caching for this use case. This can be done easily for example in JS:
documentRef.get({ source: 'server' }) // or 'cache'
this will indeed reduce costs however it may read always from your local cache and never reach the server for new changes in your document. This might be what you want but it seems practical only if your documents (immutable) and never change. so you will be able to read new documents but if they change you might not see the changes. Please, read more about this here
A better suggestion is to change your app logic. So, rather than reading the documents this way:
001,002,004,005,006
004,005,006,007,008,001,002
it's better to read them in a paginated way like this:
001,002,003, 004,005,006
007,008,009,010,011,012
You can achieve that easily by using the concept of pagination in Firestore:
var first = db.collection("cities")
.orderBy("population")
.limit(25);
return first.get().then(function (documentSnapshots) {
// Get the last visible document
var lastVisible = documentSnapshots.docs[documentSnapshots.docs.length-1];
console.log("last", lastVisible);
// Construct a new query starting at this document,
// get the next 25 cities.
var next = db.collection("cities")
.orderBy("population")
.startAfter(lastVisible)
.limit(25);
});
check Firestore documentation for more details
I am retrieving a lot of documents from Firestore, it can take a lot of time depending on the network connection of the user.
I would like to display a progres bar.
I didn't find any documentation on this
Here is my code :
final databaseReference = Firestore.instance;
databaseReference.collection("XXX").getDocuments().then((QuerySnapshot snapshot) {
Map<String, Transac.Transaction> map = new Map<String, Transac.Transaction>();
snapshot.documents.forEach((f) {
//doing something
});
I would like to have a percentage of the data loaded. For example :
Loading (58%)
Thank you.
Firestore doesn't expose any progress indicator for the documents within a query. By the time your then callback gets called, all documents have been loaded on the client and are present in the QuerySnapshot.
The usual way to deal with this is to show a "spinner" instead of a progress bar, and then hide that spinner as the first code inside your then callback.
What you can do instead is just call getDocument() for each document individually, and track their individual completions. You should track them in the order that you issued them, as the requests are pipelined over a single connection. This would give you a rough sense of progress, if you're requesting lot of documents.