I am going to create an app (iOS and Android) that will save data to the users device (text, images, files, etc) and will stay on the users device until they decide to send it to the server. I can do it either with a sqlite database or using ionic storage but I don't know what the best practice would be
For simplicity I will only present two types of items that will be stored - notes and records
notes structure
notes = {
description: this.description,
'otherText': this.otherText,
fileOrImage1: this.imageOrFileURL1,
fileOrImage2: this.imageOrFileURL2,
..... Unlimited number of fileOrImageURL'S here
};
records structure
records = {
name: this.name,
'description': this.description,
// NOTE: These files and images are different than the ones above. They will be in separate components
fileOrImage1: this.imageOrFileURL1,
fileOrImage2: this.imageOrFileURL2,
..... Unlimited number of fileOrImageURL'S here
}
The user will first store the data on the device and it will only get uploaded when the user sends it to the server. Once its uploaded it gets deleted.
There can be many notes and records, lets say 25 each. Should I use Ionic Storage or something like a sqlite database? If I use ionic storage I will need to create a unique ID for each note and record and save it.
I am willing to change my approach if anybody has a better way. I'm still in the planning stage
I used the sqlite database for an app I did with Ionic, the reason for my choice was that I could then easily query the data, as with any database.
Related
I want to store documents that I download from firestore. Each document has a unique id so I was thinking of a way I can store that document in json format or in database table (so I can access its fields) and if that id is not present in the memory than just simply download it from firestore and save it. I also want these documents to be deleted if they are not used/called for a long time. I found this cache manager but was unable to understand how to use it.
you can download the content by using any of the flutter downloadmanager and keep the path of the downloaded content in sqllite and load the data from local storage rather than url if the data is already saved.
Following pseudo-code may help you:
if(isDownloaded(id)){
//show from local;
} else {
// show from remote
}
you can manually check whether the resources are being used or not by keeping a timestam of it and update the timestamp only when it is shown. and you can run a service which looks for the unused resources and delete it from storage and the databse too.
I have an API for synchronizing contacts from the user's phone to our database. The controller essentially iterates the data sent in the request body and if it passes validation a new contact is saved:
const contact = new Contact({ phoneNumber, name, surname, owner });
await contact.save();
Having a DB with 100 IOPS and considering the average user has around 300 contacts, when the server is busy this API takes a lot of time.
Since the frontend client is made in a way that a contact ID is necessary for other operations (edit, delete), I was thinking about changing the data structure to subdocuments, and instead of saving each Contact as a separate document, the idea is to save one document with many contacts inside:
const userContacts = new mongoose.Schema({
owner: //the id of the contacts owner,
contacts: [new mongoose.Schema({
name: { type: String },
phone: { type: String }
})]
});
This way I have to do just one save. But since Mongo has to generate an ID for each subdocument, is this really that much faster than the original approach?
Summary
This really depends on your exact usage scenarios:
are contacts often updated?
what is the max / average quantity of contacts per user
are they ever partially loaded, or are they always fetched all together?
But for a fairly common collection such as contacts, I would not recommend storing them in subdocuments.
Instead you should be able to use insertMany for your initial sync scenario.
Explanation
Storing as subdocuments makes a bulk-write easier will make querying and updating contacts slower and more awkward than as regular documents.
For example, if I have 100 contacts, and I want to view and edit 1 of them, it needs to load the full 100 contacts. I can make the change via a partial update using $set or $update, so the update will be OK. But when I add a new contact, I will have to add a new contact subDocument to you Contacts document. This makes it a growing document, meaning your database will suffer from fragmentation which can slow things down a lot (see this answer)
You will have to use aggregate with $ projection or $unwind to search through contacts in MongoDB. If you want to apply a specific sort order, this too would have to be done via aggregate or in code.
Matching via projection can also lead to problems with duplicate contacts being difficult to find.
And this won't scale. What if you get users with 1000s of contacts later? Then this single document will grow large and querying it will become very slow.
Alternatives
If your contacts for sync are in the 100s, you might get away with a splitting them into groups of ~50-100 and calling insertMany for each batch.
If they grow into the thousands, then I would suggest uploading all contacts, saving them as JSON / CSV files to disk, then slowly processing these in the background in batches.
I am solving this problem:
I am building an IMGUR clone, where users can upload images and there is a 'latest uploads' page that shows the last 1000 uploaded images.
users can upload pictures as soon as they sign up, but
until the user verifies their email address, their uploads do not show up in 'latest uploads'
as soon as the user verified their email, their images start showing up.
if a user is banned, their images do not show up in 'latest uploads'
Originally I had Images contain a User ref, I would select the last 1000 images populating the User. I would then iterate over the returned collection discarding images owned by banned or non-verified users. This is broken when the last 1000 images were uploaded by unverified users.
I am considering using an array of inner Image documents on the User object, but that is not ideal either because a User might own a lot of Images and I do not always want to load them when I load the User object.
I am open to any solution
I would do the following based on what knowledge I have of your application:
There are two entities that should exist in two different collections: user and uploads.
The uploads collection will be very large, so we want to make sure we can index and shard the collection to handle the scale and performance required by your queries. With that said, some key elements in uploads are:
uploads=
{
_id:uploadId
user:{id:userId, emailverified:true, banned:false}
ts:uploadTime
.
.
.
}
possible indexes:
i. {ts:1,banned:1,"user.emailverified":1,"user.banned":1} (this index should be multi-purpose)
ii. {"user.id":1,ts:1}
Note that I store some redundant data to optimize your latest 1000 query. The cost is that in the rare case where emailverified and banned have to be updated, you need to run an update on your user collection as well as your uploads collection (this one will require multi:true).
query:
db.uploads.find({ts:{$gt:sometime},banned:false,emailverified:true}.sort({ts:-1}).limit(1000)
I'm trying to find the best way the save data obtained from JSON.
The website which hosts the data is: "JSON data".
Since I will be using the data in places where I won't have a connection to the internet, I want to save this data on the iPhone itself, with an ability to update when I do have an internet connection.
I'll want to display this data in a table view, and I'll need to be able to filter/search this data. This search will either be on the City, or on the store ID ("no:" in the data). Clicking the row will show a detail view of the store.
I was thinking of storing the data in an SQL table. I'm however unsure of the best way to update the data, and I don't know how to filter the data on two different columns(City/ID)?
Also, if you know a better approach I'd love to hear it!
Your data appears to be a table of addresses, with some sort of "detail" records associated with each address. This is a classic master/detail database. Normally you'd have one table with a record for each address, and assign some sort of unique ID to each address. Then have a second table that's keyed by unique ID to contain all of the detail records.
I don't understand one thing about Cassandra. Say, I have similar website to Facebook, where people can share, like, comment, upload images and so on.
Now, let's say, I want to get all of the things my friends did:
Username1 liked you comment
username 2 updated his profile picture
And so on.
So after a lot of reading, I guess I would need to do is create new Column Family for each single thing, for example: user_likes user_comments, user_shares. Basically, anything you can think off, and even after I do that, I would still need to create secondary indexes for most of the columns just so I could search for data? And even so how would I know which users are my friends? Would I need to first get all of my friends id's and then search through all of those Column Families for each user id?
EDIT
Ok so i did some more reading and now i understand things a little bit better, but i still can't really figure out how to structure my tables, so i will set a bounty and i want to get a clear example of how my tables should look like if i want to store and retrieve data in this kind of order:
All
Likes
Comments
Favourites
Downloads
Shares
Messages
So let's say i want to retrieve ten last uploaded files of all my friends or the people i follow, this is how it would look like:
John uploaded song AC/DC - Back in Black 10 mins ago
And every thing like comments and shares would be similar to that...
Now probably the biggest challenge would be to retrieve 10 last things of all categories together, so the list would be a mix of all the things...
Now i don't need an answer with a fully detailed tables, i just need some really clear example of how would i structure and retrieve data like i would do in mysql with joins
With sql, you structure your tables to normalize your data, and use indexes and joins to query. With cassandra, you can't do that, so you structure your tables to serve your queries, which requires denormalization.
You want to query items which your friends uploaded, one way to do this is t have a single table per user, and write to this table whenever a friend of that user uploads something.
friendUploads { #columm family
userid { #column
timestamp-upload-id : null #key : no value
}
}
as an example,
friendUploads {
userA {
12313-upload5 : null
12512-upload6 : null
13512-upload8 : null
}
}
friendUploads {
userB {
11313-upload3 : null
12512-upload6 : null
}
}
Note that upload 6 is duplicated to two different columns, as whoever did upload6 is a friend of both User A and user B.
Now to query the friends upload display of a friend, do a getSlice with a limit of 10 on the userid column. This will return you the first 10 items, sorted by key.
To put newest items first, use a reverse comparator that sorts larger timestamps before smaller timestamps.
The drawback to this code is that when User A uploads a song, you have to do N writes to update the friendUploads columns, where N is the number of people who are friends of user A.
For the value associated with each timestamp-upload-id key, you can store enough information to display the results (probably in a json blob), or you can store nothing, and fetch the upload information using the uploadid.
To avoid duplicating writes, you can use a structure like,
userUploads { #columm family
userid { #column
timestamp-upload-id : null #key : no value
}
}
This stores the uploads for a particular user. Now when want to display the uploads of User B's friends, you have to do N queries, one for each friend of User B, and merge the result in your application. This is slower to query, but faster to write.
Most likely, if users can have thousands of friends, you would use the first scheme, and do more writes rather than more queries, as you can do the writes in the background after the user uploads, but the queries have to happen while the user is waiting.
As an example of denormalization, look at how many writes twitter rainbird does when a single click occurs. Each write is used to support a single query.
In some regards, you "can" treat noSQL as a relational store. In others, you can denormalize to make things faster. For instance, PlayOrm's #OneToMany stored the many like so
user1 -> friend.user23, friend.user25, friend.user56, friend.user87
This is the wide row approach so when you find your user, you have all the foreign keys to his friends. Each row can be different lengths. You may also have a reverse reference stored as well so the user might have references to the people that marked him as a friend but he did not mark them back(let's call it buddy) so you might have
user1 -> friend.user23, friend.user25, buddy.user29, buddy.user37
Notice that if designed right, you may NOT need to "search" for the data. That said, with PlayOrm, you can still do Scalable SQL and do joins(you just have to figure out how to partition your tables so it can scale to trillions of rows).
A row can have millions of columns in it or it could have just 10. We are actually in the process of updating alot of the documentation in PlayOrm and the noSQL patterns this month so if you keep an eye on that, you can also learn more about general noSQL there as well.
Dean
Think of each DB query as of request to the service running on another machine. Your goal is to minimize number of these requests (because each request requires network roundtrip).
Here comes the main difference from RDBMS paradigm: In SQL you would typically use joins and secondary indexes. In cassandra joins aren't possible, since related data would reside on different servers. Things like materialized views are used in cassandra for the same purpose (to fetch all related data with single query).
I'd recommend to read this article:
http://maxgrinev.com/2010/07/12/do-you-really-need-sql-to-do-it-all-in-cassandra/
And to look into twissandra sample project https://github.com/twissandra/twissandra
This is nice collection of optimization technics for the kind of projects you described.