What is the correct way of storing large lists in PostgreSQL?
I currently have a "user" table, where the "followers" column stores a list of the followers that that user has. This list is stored in JSON, and every time the server wants to add a new user to that list, it retrieves it from the database, appends the new user, and then replaces the old list with the new list.
The problem is that these lists tend to get quite lengthy, which might affect performance. Is it possible to simply append to the list directly via SQL without retrieving it and rewriting it later?
Use a separate table for followers. The table should have at least two columns: userid and followerid. And it's good practice to have a primary key for this table as well, so let's give it a "ufid".
You can do a select to get all the elements and compute the JSON string if your application needs it. But do not work with JSON or any other string representation of the list, as it defeats the purpose of a relational database.
To add a new follower, simply add a new record to the follower table with the userid; deleting and update are also done on the record level without working with the "other records".
If followers is a list of integers which are primary keys to their accounts, make it an integer array int[]. If they are usernames or other words, go with a string array character varying[].
To append to an array column you can do this:
UPDATE the_table SET followers = followers || new_follower WHERE id = user;
Related
We are attempting to merge multiple datasets created in in filmmaker pro.
These datasets have multiple tables, and each entry within each table has a local ID that is used to relate entries between tables. The local ID values for all the entries were serially generated values, but some of the ID values are repeated between the different datasets, though the indicated records are non equivalent.
How can the ID values be updated in the data that is being imported to remove these overlaps without destroying the relationships that depend on them?
If you have access to the original database, you can try to migrate the ID's over to UUID or something unique before exporting. This has to be done manually, either cut/paste by hand or by a script.
Such a script will have do the following:
Loop through the parent records
For each record go to the related records
Generate an UUID with the get(UUID) function and put it in a variable
Replace the parent ID in the related record with this variable
Return to the parent record and replace the record ID with the variable.
Move to the next record.
Repeat until all records have been updated.
I retrieve an ordered list of items from a table of items in a Sqlite Database. How can I swap the id so the order of two items in the Sqlite database table?.
The id shouldn't determine position or ordering. It should be an immutable identifier.
If you need to represent order in a database you need to create another orderNumber column. A couple options are (1) either have values that span a range or (2) have a pointer to next (like a linked list).
For ranges: Spanning a range helps you avoid rewriting the orderNumber column for all items after the insert point. For example, in the range, insert first gets 1, insert 2nd gets max range, insert 3rd between first and second gets mid-range number - if you reposition you have to assign mid-points of the items it's between. One downside is if the list gets enough churn (minimized by a large span) you may have to rebalance the ranges. The pro of this solution is you can get the ordered list just by ordering by this column in the sql statement.
For linked list: If the database has a next column that points to the id that's after it in order, you need to update a couple rows to insert something. Upside is it's simple. Downside is you can't order in the sql statement - you're relying on the code getting the list to sort it.
One other variation is you could pull the ordered list data out of that table altogether. For example, you could have an ordered list table that has listid, itemid, orderedNumber. That allows you to have one or multiple logical ordered lists of the items in that table it references.
Some other references:
How to store ordered items which often change position in DB
Best way to save a ordered List to the Database while keeping the ordering
https://dba.stackexchange.com/questions/5683/how-to-design-a-database-for-storing-a-sorted-list
I have a form with checkboxes and I need to know what the best way to submit them to the database is. I have the following table setup:
roles users user_roles
----- ----- ----------
id id user_id
role_id
I have a page where you can edit a user and assign them different roles via checkbox, then those checkboxes are saved in the user_roles table. Since editing a user's roles can involve either deleting rows or adding rows, this is how I currently handle it:
my $form_vals = (1=>1,2=>2); #submitted by user
my $db_vals = (3=>3); #gotten out of db
So I have these two hashes and I will compare the keys in $form_vals with the keys in $db_vals, then I see that I have two extra values that are not present in the database so I add them. And vice versa I find which values are no longer selected on the form by comparing the keys in $db_vals with the keys in $form_vals and then I delete those rows from the database. My question is, does anyone know of a better/easier way to do this? It's never really seemed obvious to me how to handle checkboxes and I'd like to know what best practice is. Thanks!
I wouldn't say that this has much to do with check boxes per se.
Basically what you have is two array of arrays, [ (uid, rid), (uid, rid) ], and you want to make array1 (the one in your database) a copy of array2 (the user input from the checkboxes). You could have a multi select or a comma separated string, and the case would be the same. You have a user id, and you want that user to have only the roles supplied.
Two ways to achieve that would be to either
Put both arrays in one hash each, do foreach key on the submitted, if not present in the database one do insert. Then do the same for the database hash and delete those not present in the submitted hash
Delete everything from the member_role table and insert what's submitted.
You really have to know everything in the database and everything submitted and check twice if you don't want to delete everything and do a fresh insert. You can of course make a function doing this for you, hiding the ugliness a bit. Think about how you'd do if it was just two arrays and no database was around.
I am obtaining a json array from a url and inserting data into a table. Since the contents of the url are subject to change, I want to make a second connection to a url and check for updates and insert new records in y table using sqlite3.
The issues that I face are:
1) My table doesn't have a primary key
2) The url lists the changes on the same day. Hence, if I run my app multiple times, when I insert values in my database, I get duplicate entries. I want to keep a check for the day duplicated entries that should be removed. The problem can be solved by adding a constraint, but since the url itself has duplicated values, I find it difficult.
The only way I can see you can do it if you have no primary key or something you can use that is unique to each record, is when you get your new data in you go through the new entries where for each one you check if the exact same data exists in the database already. If it doesn't then you add it, if it does then you skip over it.
You could even do something like create a unique key yourself for each entry which is a concatenation of each column of the table. That way you can quickly do the check for if the entry already exists in the database.
I see two possibilities depending on your setup:
You have a column setup as UNIQUE (this can be through a PRIMARY KEY or not). In this case, you can use the ON CONFLICT clause:
http://www.sqlite.org/lang_conflict.html
If you find this construct a little confusing, you can instead use "INSERT OR REPLACE" or "INSERT OR IGNORE" as described here:
http://www.sqlite.org/lang_insert.html
You do not have a column setup as UNIQUE. In this case, you will need to SELECT first to verify for duplicate data, and based on the result INSERT, UPDATE, or do nothing.
A more common & robust way to handle this is to associate a timestamp with each data item on the server. When your app interrogates the server it provides the timestamp corresponding to the last time it synced. The server then queries its database and returns all values that are timestamped later than the timestamp provided by the app. Then it also returns a new timestamp value for the app to store, to use on the next sync.
Is it possible to invalidate memcahced entries based on values.
In my app, I am assigning user in different groups and I store this mapping in memcached.
key = userID
value = groupID
So multiple userIds map to one groupID.
When I delete a particular group then I want to remove all entries in memcache store which have value as the groupID of the deleted group.
So Essentially I want to delete the entries having particular values. How do I do it.
You cannot get key's by value in memcached. What you could do though is have a key called groupID that has a comma separated userID's. If you want to see who is part of a group then you could get the key called groupID and parse out the userID's. Then if you have a key for each userID you could delete them with the parsed userID's. You could use memcached's append function too to just append the userID's to a groupID key when a new user registers for you system.
I wrote a blog post on maintaining a set a while back that may do what you're looking for.
It's essentially what mikewied is suggesting, but with more words and code samples.