not quite sure if this question is good to ask on Stackoverflow or not.
Im currently creating a webpage that would use Mongodb logic, and redis. ( with node.js )
when user are on a page, the backend will get asked to get their user details every 5 seconds.
when retrieving this that frequently, should i get it / store it in the redis, or the mongoDB? need it for some sort of caching.
the reason is that its every 5 sec, is because it could be some changes to it that needs to be reflected backend.
each user would have their details as username, password, money, and 25 other values.
How should i approach this, to make less heavy if i were just using MongoDB alone?
example:
function calledEvery5Sec(userid) {
// get from Mongodb...
}
or
function CalledEverySec(userid) {
// get from redis if its avaliable there, else load from mongodb).
}
Use TTL index for the fields you want to store for a short period of time
For that using $out or $merge create a new tmp collection and fetch details from it .
Can you post sample document.
Related
I have 100 documents
Document id
0001
0002
0003
....
....
0100
and if we load 5 documents with id 001,002,004,005,006
then firestore charge for 5 document reads and then we again load(call the read operation query) documents with id 004,005,006,007,008,001,002
then firestore will charge for 7 document reads
here on first time we already loaded document with ids 001,002,004,005,006 and in second time or refresh time we are loading documents already loaded and some new documents
Here we need to avoid multiple times reading document from server and read it from cash and need to avoid the firestore over document read charges How to do it?
Firestore have cash loading option but it will only load from cash and not from server here what we need is load exiting data from cash and load remaning data form server.Here now what firestore doing is it will load from server and if it is failed then it will read from cash that is ok but i need in revise order
Now what happening is if non internet all data load faster with out showing progress and if there is internet it will take few sec to load and it will shows loader When we do it without fireabase our app will shows loading only one time then first it will shows the data from sqlite then when ever the api call resoppnce reached we will update in ui, so users will not face any loader but with firestore user need to wait for a progress bar to finish
From a first glance it seems that you may use firebase firestore caching for this use case. This can be done easily for example in JS:
documentRef.get({ source: 'server' }) // or 'cache'
this will indeed reduce costs however it may read always from your local cache and never reach the server for new changes in your document. This might be what you want but it seems practical only if your documents (immutable) and never change. so you will be able to read new documents but if they change you might not see the changes. Please, read more about this here
A better suggestion is to change your app logic. So, rather than reading the documents this way:
001,002,004,005,006
004,005,006,007,008,001,002
it's better to read them in a paginated way like this:
001,002,003, 004,005,006
007,008,009,010,011,012
You can achieve that easily by using the concept of pagination in Firestore:
var first = db.collection("cities")
.orderBy("population")
.limit(25);
return first.get().then(function (documentSnapshots) {
// Get the last visible document
var lastVisible = documentSnapshots.docs[documentSnapshots.docs.length-1];
console.log("last", lastVisible);
// Construct a new query starting at this document,
// get the next 25 cities.
var next = db.collection("cities")
.orderBy("population")
.startAfter(lastVisible)
.limit(25);
});
check Firestore documentation for more details
I am new to NoSQL and I'm trying to figure out a good way to represent my data. I have a series of workers that need to request vacations via mobile app.
When I try to write a Firebase query with Flutter, I can do this:
Firestore.instance
.collection("ferie_permessi")
.document("worker1#test.com")
.snapshot();
It works but there are two main errors:
If I try to create another collection called "Woker info" I cannot use worker1#test.com as document ID as it already esists;
I have to sort data client side because firestore doesn't give me the possibility (with this setup I've made).
I'm quite sure that this structure isn't good at all. Each worker needs to have 2 lists: vacations and other. What is wrong?
My guess is that I should move worker1#test.com together with vacations and other so that I can make a query of this kind:
Firestore.instance
.collection("ferie_permessi")
.where("user", "==", "worker1#test.com)
.snapshot();
But now the id? Is an automatic one good?
I had a chance to recently explore creating an app using a firebase-firestore. A couple of things will help here:
Yes, the autogenerated id is good since it is unique, for example, you can have a collections vacation_requests, users you can then use that user_id as a document in vaccation_requests -> user_id -> vacations, instead of using email as a document key.
Or
You can do it like this collections users, vacation_requests, and requests.
store user details in users.
store requests in requests with from and to dates.
store reference of User and Request in vaccation_requests.
Hope this helps!
Our Java app saves its configurations in a MongoDB collections. When the app starts it reads all the configurations from MongoDB and caches them in Maps. We would like to use the change stream API to be able also to watch for updates of the configurations collections.
So, upon app startup, first we would like to get all configurations, and from now on - watch for any further change.
Is there an easy way to execute the following atomically:
A find() that retrieves all configurations (documents)
Start a watch() that will send all further updates
By atomically I mean - without potentially missing any update (between 1 and 2 someone could update the collection with new configuration).
To make sure I lose no update notifications, I found that I can use watch().startAtOperationTime(serverTime) (for MongoDB of 4.0 or later), as follows.
Query the MongoDB server for its current time, using command such as Document hostInfoDoc = mongoTemplate.executeCommand(new Document("hostInfo", 1))
Query for all interesting documents: List<C> configList = mongoTemplate.findAll(clazz);
Extract the server time from hostInfoDoc: BsonTimestamp serverTime = (BsonTimestamp) hostInfoDoc.get("operationTime");
Start the change stream configured with the saved server time ChangeStreamIterable<Document> changes = eventCollection.watch().startAtOperationTime(serverTime);
Since 1 ends before 2 starts, we know that the documents that were returned by 2 were at least same or fresher than the ones on that server time. And any updates that happened on or after this server time will be sent to us by the change stream (I don't care to run again redundant updates, because I use map as cache, so extra add/remove won't make a difference, as long as the last action arrives).
I think I could also use watch().resumeAfter(_idOfLastAddedDoc) (didn't try). I did not use this approach because of the following scenario: the collection is empty, and the first document is added after getting all (none) documents, and before starting the watch(). In that scenario I don't have previous document _id to use as resume token.
Update
Instead of using "hostInfo" for getting the server time, which couldn't be used in our production, I ended using "dbStats" like that:
Document dbStats= mongoOperations.executeCommand(new Document("dbStats", 1));
BsonTimestamp serverTime = (BsonTimestamp) dbStats.get("operationTime");
Transactions in Cloud Firestore support getting a document using transaction.getDocument, but even though there is a .getDocuments method, there doesn’t seem to be a .getDocuments for getting multiple documents that works with a transaction.
I have a Yelp-like app using a Cloud Firestore database with the following structure:
- Places to rate are called spots.
- Each spot has a document in the spots collection (identified by a unique documentID).
- Each spot can have a reviews collection containing all reviews for that spot.
- Each review is identified by its own unique documentID, and each review document contains a rating of the spot.
Below is an image of my Cloud Firestore setup with some data.
I’ve tried to create a transaction getting data for all of the reviews in a spot, with the hope that I could then make an updated calculation of average review & save this back out to a property of the spot document. I've tried using:
let db = Firestore.firestore()
db.runTransaction({ (transaction, errorPointer) -> Any? in
let ref = db.collection("spots").document(self.documentID).collection("reviews")
guard let document = try? transaction.getDocuments(ref) else {
print("*** ERROR trying to get document for ref = \(ref)")
return nil
}
…
Xcode states:
Value of type ‘Transaction’ has no member ‘getDocuments’.
There is a getDocument, which that one can use to get a single document (see https://firebase.google.com/docs/firestore/manage-data/transactions).
Is it possible to get a collection of documents in a transaction? I wanted to do this because each place I'm rating (spot) has an averageRating, and whenever there's a change to one of the ratings, I want to call a function that:
- starts a transaction (done)
- reads in all of the current reviews for that spot (can't get to work)
- calculates the new averageRating
- updates the spot with the new averageRating value.
I know Google's FriendlyEats uses a technique where each change is applied to the current average rating value, but I'd prefer to make a precise re-calculation with each change to keep numerical precision (even if it's a bit more expensive w/an additional query).
Thanks for advice.
No. Client libraries do not allow you to make queries inside of transactions. You can only request specific documents inside of a query. You could do something really hacky, like run the query outside of the transaction, then request every individual document inside the transaction, but I would not recommend that.
What might be better is to run this on the server side. Like, say, with a Cloud Function, which does allow you to run queries inside transactions. More importantly, you no longer have to trust the client to update the average review score for a restaurant, which is a Bad Thing.
That said, I still might recommend using a Cloud Function that does some of the same logic that Friendly Eats does, where you say something along the lines of New average = Old average + new review / (Total number of reviews) It'll make sure you're not performing excessive reads if your app gets really popular.
I'm pretty confused concerning this hip thing called NoSQL, especially CloudantDB by Bluemix. As you know, this DB doesn't store the values chronologically. It's the programmer's task to sort the entries in case he wants the data to.. well.. be sorted.
What I try to achive is to simply get the last let's say 100 values a sensor has sent to Watson IoT (which saves everything in the connected CloudantDB) in an ORDERED way. In the end it would be nice to show them in a D3.css style kind of graph but that's another task. I first need the values in an ordered array.
What I tried so far: I used curl to get the data via PHP from https://averylongID-bluemix.cloudant.com/iotp_orgID_iotdb_2018-01-25/_all_docs?limit=20&include_docs=true';
What I get is an unsorted array of 20 row entries with random timestamps. The last 20 entries in the DB. But not in terms of timestamps.
My question is now: Do you know of a way to get the "last" 20 entries? Sorted by timestamp? I did a POST request with a JSON string where I wanted the data to be sorted by the timestamp, but that doesn't work, maybe because of the ISO timestamp string.
Do I really have to write a javascript or PHP script to get ALL the database entries and then look for the 20 or 100 last entries by parsing the timestamp, sorting the array again and then get the (now really) last entries? I can't believe that.
Many thanks in advance!
I finally found out how to get the data in a nice ordered way. The key is to use the _design api together with the _view api.
So a curl request with the following URL / attributes and a query string did the job:
https://alphanumerical_something-bluemix.cloudant.com/iotp_orgID_iotdb_2018-01-25/_design/iotp/_view/by-date?limit=120&q=name:%27timestamp%27
The curl result gets me the first (in terms of time) 120 entries. I just have to find out how to get the last entries, but that's already a pretty good result. I can now pass the data on to a nice JS chart and display it.
One option may be to include the timestamp as part of the ID. The _all_docs query returns documents in order by id.
If that approach does not work for you, you could look at creating a secondary index based on the timestamp field. One type of index is Cloudant Query:
https://console.bluemix.net/docs/services/Cloudant/api/cloudant_query.html#query
Cloudant query allows you to specify a sort argument:
https://console.bluemix.net/docs/services/Cloudant/api/cloudant_query.html#sort-syntax
Another approach that may be useful for you is the _changes api:
https://console.bluemix.net/docs/services/Cloudant/api/database.html#get-changes
The changes API allows you to receive a continuous feed of changes in your database. You could feed these changes into a D3 chart for example.