I have a node-red code that works in the following way:
It receives a message (json form) and saves it to cloudant DB
Then I can make an http call where I can see all the contents of the DB
This is all good, but the problem is that when it saves it to cloudant, it gives it a random _id, so the order of the documents in the DB isn't the same as the order they came in, but random.
Is there a way to maybe set the _id while saving in node red? Or is there another solution?
I just want that when I call the http it shows it in the order that it came in (last to first, or first to last, doesn't matter).
You can set the _id with a function node or a change node before passing it to the Cloudant out node.
But if you just want them in the order they arrived then add the timestamp field and make the query node use a view that sorts the documents by the timestamp
Related
I am trying to paginate data with Firestore, and would be ordering data on columns where duplicates are expected, if the pagination happens to be among those values its expected that it won't work correctly.
I can work around this issue by using StartAfter based on Document ID which will be always be unique.
One way I can accomplish this is passing id of the last document to server side rest api request. This would require two steps, i.e. to fetch the DocumentSnapshot using the DocumentId and constructing the query based on it
var lastSnapshot = fetchSnapshot(id);
citiesRef.OrderBy("Population").StartAfter(lastSnapshot);
Other approach is to persist the DocumentId in the document while creation.This would require two steps each time when the document is created, one to create and the other to update immediately with Id generated (As I don't see a way to persist DocumentId during creation itself)
citiesRef.OrderBy("Population").StartAfter(lastId);
Which one of these is a good approach to follow, either to fetch DocumentSnapshot and not to persist id into the document, or perform two operations by persisting the DocumentId in the first place and using it as key for StartAfter.
Decided to go with Option 1, instead of persisting Document ID in the Document itself as in Option 2.
Our Java app saves its configurations in a MongoDB collections. When the app starts it reads all the configurations from MongoDB and caches them in Maps. We would like to use the change stream API to be able also to watch for updates of the configurations collections.
So, upon app startup, first we would like to get all configurations, and from now on - watch for any further change.
Is there an easy way to execute the following atomically:
A find() that retrieves all configurations (documents)
Start a watch() that will send all further updates
By atomically I mean - without potentially missing any update (between 1 and 2 someone could update the collection with new configuration).
To make sure I lose no update notifications, I found that I can use watch().startAtOperationTime(serverTime) (for MongoDB of 4.0 or later), as follows.
Query the MongoDB server for its current time, using command such as Document hostInfoDoc = mongoTemplate.executeCommand(new Document("hostInfo", 1))
Query for all interesting documents: List<C> configList = mongoTemplate.findAll(clazz);
Extract the server time from hostInfoDoc: BsonTimestamp serverTime = (BsonTimestamp) hostInfoDoc.get("operationTime");
Start the change stream configured with the saved server time ChangeStreamIterable<Document> changes = eventCollection.watch().startAtOperationTime(serverTime);
Since 1 ends before 2 starts, we know that the documents that were returned by 2 were at least same or fresher than the ones on that server time. And any updates that happened on or after this server time will be sent to us by the change stream (I don't care to run again redundant updates, because I use map as cache, so extra add/remove won't make a difference, as long as the last action arrives).
I think I could also use watch().resumeAfter(_idOfLastAddedDoc) (didn't try). I did not use this approach because of the following scenario: the collection is empty, and the first document is added after getting all (none) documents, and before starting the watch(). In that scenario I don't have previous document _id to use as resume token.
Update
Instead of using "hostInfo" for getting the server time, which couldn't be used in our production, I ended using "dbStats" like that:
Document dbStats= mongoOperations.executeCommand(new Document("dbStats", 1));
BsonTimestamp serverTime = (BsonTimestamp) dbStats.get("operationTime");
I'm pretty confused concerning this hip thing called NoSQL, especially CloudantDB by Bluemix. As you know, this DB doesn't store the values chronologically. It's the programmer's task to sort the entries in case he wants the data to.. well.. be sorted.
What I try to achive is to simply get the last let's say 100 values a sensor has sent to Watson IoT (which saves everything in the connected CloudantDB) in an ORDERED way. In the end it would be nice to show them in a D3.css style kind of graph but that's another task. I first need the values in an ordered array.
What I tried so far: I used curl to get the data via PHP from https://averylongID-bluemix.cloudant.com/iotp_orgID_iotdb_2018-01-25/_all_docs?limit=20&include_docs=true';
What I get is an unsorted array of 20 row entries with random timestamps. The last 20 entries in the DB. But not in terms of timestamps.
My question is now: Do you know of a way to get the "last" 20 entries? Sorted by timestamp? I did a POST request with a JSON string where I wanted the data to be sorted by the timestamp, but that doesn't work, maybe because of the ISO timestamp string.
Do I really have to write a javascript or PHP script to get ALL the database entries and then look for the 20 or 100 last entries by parsing the timestamp, sorting the array again and then get the (now really) last entries? I can't believe that.
Many thanks in advance!
I finally found out how to get the data in a nice ordered way. The key is to use the _design api together with the _view api.
So a curl request with the following URL / attributes and a query string did the job:
https://alphanumerical_something-bluemix.cloudant.com/iotp_orgID_iotdb_2018-01-25/_design/iotp/_view/by-date?limit=120&q=name:%27timestamp%27
The curl result gets me the first (in terms of time) 120 entries. I just have to find out how to get the last entries, but that's already a pretty good result. I can now pass the data on to a nice JS chart and display it.
One option may be to include the timestamp as part of the ID. The _all_docs query returns documents in order by id.
If that approach does not work for you, you could look at creating a secondary index based on the timestamp field. One type of index is Cloudant Query:
https://console.bluemix.net/docs/services/Cloudant/api/cloudant_query.html#query
Cloudant query allows you to specify a sort argument:
https://console.bluemix.net/docs/services/Cloudant/api/cloudant_query.html#sort-syntax
Another approach that may be useful for you is the _changes api:
https://console.bluemix.net/docs/services/Cloudant/api/database.html#get-changes
The changes API allows you to receive a continuous feed of changes in your database. You could feed these changes into a D3 chart for example.
I am replicating a collection (I only have access to mongo shell on the server). In the current collection all documents have a field called jsonURL. The value of this field is a url http://www.something.com/api/abc.json. I want to copy each document from oldCollection to newCollection, but I want also want to fetch data from that url and add that to each new document created.
I last time heard that XMLHTTPRequest was on mongo's list, but as a low priority feature (I can understand why). And as I found nothing in the documentation, I am guessing its still in the queue. I am hoping I can get something in forEach(function(eachDoc){});
Do I have any other way of achieving this. Thanks.
I have an instance of Raven Db at localhost:8081. I made sure to change raven's config file to allow anonymous access. I created a database named AT. Inside AT I have a collection named Admins. Inside of Admins I have two documents. I'm trying to retrieve some data via Rest using RestClient. I try to hit the db using:
http://localhost:8081/docs/admins/7cb95e9a (last bit is the id of the document I want).
and
http://localhost:8081/docs/at/admins/7cb95e9a.
With both I receive a 404. I'm not sure what I'm missing here. Can someone point me in the right direction?
The URL has the following format:
http://localhost:8081/databases/{{database-name}}/docs/{{document-id}}.
Collection is a virtual thing. get a document only by its ID, there no nothing on collection here. The document ID can be anything you set, but if you let RavenDB to generate it, it will probably be admins/1.