I'm right now experimenting with a nodejs based experimental app, where I will be putting in a list of books and it will be posted on a forum automatically every x minutes.
Now my question is about order of these things posted.
I use mongodb (not sure if this changes the question or not) and I just add a new entry for every item to be posted. Normally, things are posted in the exact order I add them.
However, for the web interface of this experimental thing, I made a re-ordering interaction where I can simply drag and drop elements to reorder them.
My question is: how can I reflect this change to the database?
Or more in general terms, how can I order stuff in general, in databases?
For instance if I drag the 1000th item to 1st order, everything below needs to be edited (in db) between 1 and 1000 the entries. This does not seem like a valid and proper solution to me.
Any enlightenment is appreciated.
An elegant way might be lexicographic sorting. Introduce a String attribute for each item. Make the initial length of the values large enough to accomodate the estimated number of items. E.g., if you expect 1000 items, let the keys be baa, bab, bac, ... bba, bbb, bbc, ...
Then, when an item is moved from where it is to another place between two items, assign a value to the sorting attribute of the moved item that is somewhere equidistant (lexicographically) to those items. So to move an item between dei and dej, give it the value deim. To move an item between fadd and fado, give it the value fadi.
Keys starting with a were initially not used to leave space for elements that get dragged before the first one. Never use the key a, as it will be impossible to move an element before this one.
Of course, the characters used may vary according to the sort order provided by the database.
This solution should work fine as long as elements are not reordered extremely frequently. In a worst case scenario, this may lead to longer and longer attribute values. But if the movements are somewhat equally distributed, the length of values should stay reasonable.
Related
This feels like it should be a common requirement, but I'm not sure how best to implement the requirement.
I'm going to make up a really simple example. It's similar enough to what I actually need, without getting over-complicated.
Imagine we have a table called transport and it has the following columns:
type
model_name
size
number_of_wheels
fuel
maximum_passenger_count
We want to store all sorts of different types of transportation in this table, but not every type will have values in every column. Imagine the commonality is a lot higher, as this is a bit of fake example.
Here's a few examples of how this might work in practice:
type = cycle, we ban fuel, as it's not relevant for a cycle
type = bus, all columns are valid
type = sledge, we ban number_of_wheels, as sledges don't have wheels, we also ban fuel
type = car, all columns are valid
I want my UI to show a grid with a list of all the rows in the transport table. Users can edit the data directly in the grid and add new rows. When they add a new row, they're forced to pick the transport type in a dropdown before it appears in the grid. They then complete the details. All the values are optional, apart from the ones we explicitly don't want to record a value for, where we expect to not see anything at all.
I can see a number of ways to implement this, but none of them seems like a complete solution:
I could put this logic into the UI, enabling/ disabling grid cells based on type. But there's nothing to stop someone directly inserting data into the "wrong" columns in the backend of the database or via the API, which would then come through into the UI unless I added a rule to also mask out values in disabled cells. Making changes to which columns are restricted per transport type would be very difficult
I could put this logic into the API, raising an error if someone enters data into a cell that should be disallowed. This closes one gap for insertion to the database via the API, but SQL scripts would still allow entry into the "wrong" column. Also, the user experience would suck, as users would have to guess which columns to complete and which to leave blank. It would still be difficult to make changes to which columns are allowed/ restricted
I could add a trigger to the database, maybe to set the values to NULL if they shouldn't be allowed, but this seems clunky and users would not understand what was happening
I could add a generated column, but this doesn't help if I sometimes need to set a value and sometimes don't
I could just allow the unnecessary data to be stored in the database, then hide it by using a view to report it back. It doesn't seem great, as users would still see data disappearing from the UI with no explanation
I could add a second table, storing a matrix of which values are allowed and which are restricted by type. The API, UI and database could all implement this list using different mechanisms - this comes with the advantage of having a single place to make changes that will immediately be reflected across the entire system, but it's a lot of work and I have lots of these tables that work the same way
I'm writing a Flutter web/mobile calendar application / todo list, the main feature of which is a long list of items (appointments, tasks, and the like).
Much of the time, the list won't be longer than a few hundred items. But it should be able to scale to thousands of items, as many as the user wants. Not too hard to do should the user make many repeating items. I'm using ReorderableListView.builder to virtualize the list. The part I'm struggling with is when to load the list data and how to store it.
Important: When the user picks a day, the list can jump to somewhere in the middle... and can scroll back to the top of the list.
The easiest thing to do would be to just load all data up to the users position. But this means storing a potentially very large list in memory at best, and in the web application it means requesting way more data than is really needed.
A good summary of the problem might be: Jumping to particular day is more challenging than jumping to a known index on the list. It's not easy to know what an item's index would be in a fully constructed version of the list without fully constructing the list up to that item. Yes, you can get items at particular date, but what if you wanted to get fifty items before a particular date, and fifty items after a particular date (useful for keeping scrolling smooth)? There could be a huge gap, or there could be a whole ton of items all clustered on one day.
A complication is that not all items in the list are items in the database, for example day headers. The day headers need to behave like regular items and not be attached to other items when it comes to the reordering drag animation, yet storing them as records in the database feels wrong. Here's an image of the web application to give you some idea:
THIS ANSWER IS MY OWN WORK IN PROGRESS. OPEN TO BETTER ANSWERS, CORRECTIONS.
I like the answer here (Flutter: Display content from paginated API with dynamic ListView) and would like to do something like it.
I would do this both for web app, where there's http bottlenecks. I would also do this for the mobile app, even when all data is in the database. Why keep the entire list in memory when you don't have to? (I have very little mobile development experience, so please correct me if I'm wrong)
Problem:
Jumping to particular day is more challenging than jumping to known index on the list. It's not easy to know what an item's index would be in a fully constructed version of the list without fully constructing the list up to that item.
Solution I'm leaning toward:
ReferencesList + keyValueStorage solution
Store the item ids in order as well as key value pairs of all items by id as a json list in NoSql. This list will include references to items, with day headings included and represented by something like 'dh-2021-05-21' or its epoch time. This list will be very lightweight, just a string per item, so you don't need to worry about holding it all in memory. Item data can be pulled out of storage whenever it's built. (Should this be a problem in Sembast or hive? Hive, and here's why: https://bendelonlee.medium.com/hive-or-sembast-for-nosql-storage-in-flutter-web-fe3711146a0 )
When a date is jumped to, run a binary search on this list to find its exact position and scroll to that position. You can easily preload, say 30 items before and 50 items after that position.
other solutions:
SplayTreeMap + QuerySQLByDate Solution:
When jumping, since you don't know the index, insert it into a new SplayTreeMap at an arbitrarily high index, say 100 * number_of_list_items_in_database just to be safe. Run two queries, on ascending from scrolled to date, and one descending from it. Cache these queries and return them in paged form. Should the user reach the beginning of the list, simply prevent them from scrolling further up the list manually with a ScrollController or something like it.
I have a database of million of Objects (simply say lot of objects). Everyday i will present to my users 3 selected objects, and like with tinder they can swipe left to say they don't like or swipe right to say they like it.
I select each objects based on their location (more closest to the user are selected first) and also based on few user settings.
I m under mongoDB.
now the problem, how to implement the database in the way it's can provide fastly everyday a selection of object to show to the end user (and skip all the object he already swipe).
Well, considering you have made your choice of using MongoDB, you will have to maintain multiple collections. One is your main collection, and you will have to maintain user specific collections which hold user data, say the document ids the user has swiped. Then, when you want to fetch data, you might want to do a setDifference aggregation. SetDifference does this:
Takes two sets and returns an array containing the elements that only
exist in the first set; i.e. performs a relative complement of the
second set relative to the first.
Now how performant this is would depend on the size of your sets and the overall scale.
EDIT
I agree with your comment that this is not a scalable solution.
Solution 2:
One solution I could think of is to use a graph based solution, like Neo4j. You could represent all your 1M objects and all your user objects as nodes and have relationships between users and objects that he has swiped. Your query would be to return a list of all objects the user is not connected to.
You cannot shard a graph, which brings up scaling challenges. Graph based solutions require that the entire graph be in memory. So the feasibility of this solution depends on you.
Solution 3:
Use MySQL. Have 2 tables, one being the objects table and the other being (uid-viewed_object) mapping. A join would solve your problem. Joins work well for the longest time, till you hit a scale. So I don't think is a bad starting point.
Solution 4:
Use Bloom filters. Your problem eventually boils down to a set membership problem. Give a set of ids, check if its part of another set. A Bloom filter is a probabilistic data structure which answers set membership. They are super small and super efficient. But ya, its probabilistic though, false negatives will never happen, but false positives can. So thats a trade off. Check out this for how its used : http://blog.vawter.com/2016/03/17/Using-Bloomfilters-to-Avoid-Repetition/
Ill update the answer if I can think of something else.
I have about 20,000 records (coming from an SQLite db) that I need to present to the user for possible choices, so conventional picker control is out of question.
Another possibility could be an indexed UITableView where user could check the desired value, however keeping all the 20K records in the memory doesn’t seem like a good idea.
how should I go about implementing UI for it? EDIT: is it possible to do something like auto-complete combo-box?
I'd look at some sort of nested UI i.e. a UITableView that has just A, B, C etc to start with and when the user presses on a row show another table view with all the results starting with A.
There would need to be a query that got the number of results that started with A, B, C etc so you only showed letters in the first table that actually has results but then each query in the second table would be very simple - all results starting with 'A' etc.
You'd never have to load all 20,000 results into memory at the same time :)
However, you should probably make sure that you have an index on the field that you're querying, otherwise your queries are all going to be rather slow :(
The other solution is to use a search box at the top of a table view - the user types in letters and each time the list is reduced to only results starting with that letter. That's a pretty simple query to implement :)
However, you would then still have the problem of what to do if the user hasnt typed anything in - do you show a message asking them to type or do you show all 20,000 results in an enormous list?
Are you aware of UIPickerView's "components"? 20,000 choices might be pushing it, but it is certainly one way (that is familiar to users) to narrow down choices by an order of magnitude with each spin.
If you find yourself constrained in one dimension, you could implement master-slave picker views.
I'm working on some basic logging/history functionality for a Core Data iPhone app. I want to maintain a maximum number of history items.
My general plan is to ignore the maximum when adding a new item and enforce it whenever I need to fetch all the items anyway (e.g. for searching or browsing the history). Alternatively, I could do it when adding a new item: fetch the current items, add the new one, and delete the oldest one if we're at the maximum. The second way seems less efficient, since I would be fetching all the items when I otherwise wouldn't need to.
So, the questions:
Which way is better? Is there an even better way to do this that I'm not considering?
How many items would be a reasonable maximum? The history is used for text field autocompletion, so more items means better usability, unless the number of items is so huge that it's slowing stuff down.
Thanks!
Whichever method is easier to implement is the right one. You shouldn't bother with a more efficient/more complicated implementation unless it proves it's needed.
If these objects are in a to-many relationship of some kind, I'd use the relationship to manage the maximum number. (Override add<Whatever>Object: and delete the extraneous items then).
If you're just fetching them, then that's really your only opportunity to filter them out. If you're using an NSArrayController, you might be able to implement a subclass that detects when new objects are added and chops off the extra ones.
If the items are added manually by the user, then you can safely use the method of cleaning up later. With text data, a user won't enter more a few hundred items at most and text data takes up very little room. If the items are added by software, you have to check every so many entries or risk spill over.
You might not want to spend a lot of time on this. Autocomplete is not that big, usually just a few hundred entries. I would right it the simplest way, with clean up later, and then fiddle with it only if you hit a definite performance bottleneck.
Remember, premature optimization is the root of all programming evil. That and the dweebs in marketing.