I'm writing a Flutter web/mobile calendar application / todo list, the main feature of which is a long list of items (appointments, tasks, and the like).
Much of the time, the list won't be longer than a few hundred items. But it should be able to scale to thousands of items, as many as the user wants. Not too hard to do should the user make many repeating items. I'm using ReorderableListView.builder to virtualize the list. The part I'm struggling with is when to load the list data and how to store it.
Important: When the user picks a day, the list can jump to somewhere in the middle... and can scroll back to the top of the list.
The easiest thing to do would be to just load all data up to the users position. But this means storing a potentially very large list in memory at best, and in the web application it means requesting way more data than is really needed.
A good summary of the problem might be: Jumping to particular day is more challenging than jumping to a known index on the list. It's not easy to know what an item's index would be in a fully constructed version of the list without fully constructing the list up to that item. Yes, you can get items at particular date, but what if you wanted to get fifty items before a particular date, and fifty items after a particular date (useful for keeping scrolling smooth)? There could be a huge gap, or there could be a whole ton of items all clustered on one day.
A complication is that not all items in the list are items in the database, for example day headers. The day headers need to behave like regular items and not be attached to other items when it comes to the reordering drag animation, yet storing them as records in the database feels wrong. Here's an image of the web application to give you some idea:
THIS ANSWER IS MY OWN WORK IN PROGRESS. OPEN TO BETTER ANSWERS, CORRECTIONS.
I like the answer here (Flutter: Display content from paginated API with dynamic ListView) and would like to do something like it.
I would do this both for web app, where there's http bottlenecks. I would also do this for the mobile app, even when all data is in the database. Why keep the entire list in memory when you don't have to? (I have very little mobile development experience, so please correct me if I'm wrong)
Problem:
Jumping to particular day is more challenging than jumping to known index on the list. It's not easy to know what an item's index would be in a fully constructed version of the list without fully constructing the list up to that item.
Solution I'm leaning toward:
ReferencesList + keyValueStorage solution
Store the item ids in order as well as key value pairs of all items by id as a json list in NoSql. This list will include references to items, with day headings included and represented by something like 'dh-2021-05-21' or its epoch time. This list will be very lightweight, just a string per item, so you don't need to worry about holding it all in memory. Item data can be pulled out of storage whenever it's built. (Should this be a problem in Sembast or hive? Hive, and here's why: https://bendelonlee.medium.com/hive-or-sembast-for-nosql-storage-in-flutter-web-fe3711146a0 )
When a date is jumped to, run a binary search on this list to find its exact position and scroll to that position. You can easily preload, say 30 items before and 50 items after that position.
other solutions:
SplayTreeMap + QuerySQLByDate Solution:
When jumping, since you don't know the index, insert it into a new SplayTreeMap at an arbitrarily high index, say 100 * number_of_list_items_in_database just to be safe. Run two queries, on ascending from scrolled to date, and one descending from it. Cache these queries and return them in paged form. Should the user reach the beginning of the list, simply prevent them from scrolling further up the list manually with a ScrollController or something like it.
Related
What is the maximum number of items that a Dart List can store? I cant find any resources regarding the maximum length of a list. Dart API only mentioned growable list, but I am curious about the maximum number of elements a list can hold.
En, to your question, there's no limitation on how many items can be added to a list but you do have a restriction at hardware device which consumes memory to actually load the list it self.
That's why there are two main types of lists, with out a builder and the other with builder.
Using a list with out a builder, will load all elements on the list at once, which is perfect when you know in advance you won't have many, for example top ten articles, a drawer etc..
Using a list with builder, is basically used to prevent loading (in this lame example) 1000 records at once, rather load them on demand while you start scrolling and dismissing the items in memory when you scroll enough.
Think about this, if you have 10 items it should not be a problem using a normal list, but imagine you have 1000.
First, 1000 won't fit on the screen, each of them will consume a bit of memory leading to battery consumption and stuttering depending on performance. That's why when you know in advance you will have lots of items, using a list builder is recommended, flutter will dismiss previous elements of the list and load new ones while scrolling preventing the case I mention of using a normal list.
So, about your question i'm not sure if there is a limit in dart list items count.
But, what we should do if there an limit !! in this case we must use if statement the check the list length if it reach the limit then store the data in second list.
the idea is to split your data in this case to multiple of lists.
So the problem seems very simple but every way I approach the solution it seems to be a poor implementation approach with either duplicated content or messy data.
The Problem
I want to provide an option for “overwrites” per user on “default” items. Basically I have a mongodb database with a collection containing items with the following information:
ID
Name
Icon
Description
There is a set of 20-30 items in this collection, which each user using the app views.
Most users will be happy to see the default but if a user wishes to say change the icon for an item or the name then how do I handle this “overwrite” on the ”default” item for just that single user.
Possible solutions
My thoughts are to implement one of the following options but all just seem a little wrong (I have provided my thoughts on this):
for each “overwriten” item add a duplicated item to the collection with the changes and a user_id field to link the user - this seems like a little bit of duplicated content as the user might only change the icon and not the name/description. Also if the name is changed int eh future on the default item how do you handle this and also how do you understand that this item must replace one of the “defaults” for just that user. I worry it will be a little bit of a performance issue too when looking up the items and then replacing the changed item
having all the items duplicated per user in the same collection - very much duplication of content but might be the best performing option but could cause issues in the future if new “default” items need to be added or default options need changing
collection per user - same as the previous. This options seems all kinds of wrong but maybe I’m just new to this and it is actually the best option.
collection containing overwrites - this seems like a good idea but equally a bad one due to looking up and comparing. If everything is changed then why not just have all new items rather than effectively a find a replace.
Reason for wanting to get this right
Maybe I’m over thinking this but it seems like I will face this issue a lot and I think I need to get it right to remove future issues with performance, data management and updates to default items.
I'm right now experimenting with a nodejs based experimental app, where I will be putting in a list of books and it will be posted on a forum automatically every x minutes.
Now my question is about order of these things posted.
I use mongodb (not sure if this changes the question or not) and I just add a new entry for every item to be posted. Normally, things are posted in the exact order I add them.
However, for the web interface of this experimental thing, I made a re-ordering interaction where I can simply drag and drop elements to reorder them.
My question is: how can I reflect this change to the database?
Or more in general terms, how can I order stuff in general, in databases?
For instance if I drag the 1000th item to 1st order, everything below needs to be edited (in db) between 1 and 1000 the entries. This does not seem like a valid and proper solution to me.
Any enlightenment is appreciated.
An elegant way might be lexicographic sorting. Introduce a String attribute for each item. Make the initial length of the values large enough to accomodate the estimated number of items. E.g., if you expect 1000 items, let the keys be baa, bab, bac, ... bba, bbb, bbc, ...
Then, when an item is moved from where it is to another place between two items, assign a value to the sorting attribute of the moved item that is somewhere equidistant (lexicographically) to those items. So to move an item between dei and dej, give it the value deim. To move an item between fadd and fado, give it the value fadi.
Keys starting with a were initially not used to leave space for elements that get dragged before the first one. Never use the key a, as it will be impossible to move an element before this one.
Of course, the characters used may vary according to the sort order provided by the database.
This solution should work fine as long as elements are not reordered extremely frequently. In a worst case scenario, this may lead to longer and longer attribute values. But if the movements are somewhat equally distributed, the length of values should stay reasonable.
I've a ItemsControl which shows a list of days with some data.
I want this control so be able to scroll infinitely in two directions.
With ISupportIncrementalLoading I could add more days when the end is reached.
But how can I achieve this when the first element is reached?
Does somebody now a collection class which returns the postion of the currently first displayed item?
Any hints?
One possibility is IObserveableVector, per this page on MSDN: Load, store, and display large sets of data efficiently
IObserveableVector allows you to access data in a random access order, which would allow you to load data at both ends of the list.
I have not used it, nor have I seen many examples on how to use it.
I'm working on some basic logging/history functionality for a Core Data iPhone app. I want to maintain a maximum number of history items.
My general plan is to ignore the maximum when adding a new item and enforce it whenever I need to fetch all the items anyway (e.g. for searching or browsing the history). Alternatively, I could do it when adding a new item: fetch the current items, add the new one, and delete the oldest one if we're at the maximum. The second way seems less efficient, since I would be fetching all the items when I otherwise wouldn't need to.
So, the questions:
Which way is better? Is there an even better way to do this that I'm not considering?
How many items would be a reasonable maximum? The history is used for text field autocompletion, so more items means better usability, unless the number of items is so huge that it's slowing stuff down.
Thanks!
Whichever method is easier to implement is the right one. You shouldn't bother with a more efficient/more complicated implementation unless it proves it's needed.
If these objects are in a to-many relationship of some kind, I'd use the relationship to manage the maximum number. (Override add<Whatever>Object: and delete the extraneous items then).
If you're just fetching them, then that's really your only opportunity to filter them out. If you're using an NSArrayController, you might be able to implement a subclass that detects when new objects are added and chops off the extra ones.
If the items are added manually by the user, then you can safely use the method of cleaning up later. With text data, a user won't enter more a few hundred items at most and text data takes up very little room. If the items are added by software, you have to check every so many entries or risk spill over.
You might not want to spend a lot of time on this. Autocomplete is not that big, usually just a few hundred entries. I would right it the simplest way, with clean up later, and then fiddle with it only if you hit a definite performance bottleneck.
Remember, premature optimization is the root of all programming evil. That and the dweebs in marketing.