I've made a database of sensor data, which in essence stores Double values paired with Date timestamps.
I've tried querying them with what seems to be the right syntax, but it returns just [].
The API is made in MongoDB Stitch, and I'm calling it through Postman.
If I make my API return all documents, it works fine.
This is an example of a Document in my database:
_id:5caf026c8bc97c06677967d6
time:"2019-04-11T09:01:32Z"
val:102.85
topic:"pressure/stjernelaks/sensor0"
isotime:2019-04-11T09:01:32.000+00:00
Edit:
This is how the document looks when returned as Array through the API:
"_id": {
"$oid": "5caf18078bc97c066779724a"
},
"time": "2019-04-11T10:33:43Z",
"val": {
"$numberDouble": "102.88"
},
"topic": "pressure/stjernelaks/sensor0",
"isotime": {
"$date": {
"$numberLong": "1554978823000"
}
}
},
And this is the query I'm making:
https://eu-west-1.aws.webhooks.mongodb-stitch.com/api/client/v2.0/app/semapres-charts-dsioa/service/get-chart-data/incoming_webhook/get-all-between?arg2=1554970464&arg1=1554980464
Alternatively I've used the arguments 2019-04-11T07:01:32.000+00:00 and 2019-04-11T09:01:32.000+00:00. It's the exact same format as in the database.
It's created by LocalDateTime with DateTimeFormatter.ISO_DATE_TIME in Java, and my MongoDB says the data field type is Date, not String.
This is my webhook in all it's simplicity:
exports = function(payload) {
const {arg1, arg2} = payload.query;
const contentTypes = payload.headers["application/json"];
const body = payload.body;
const mongodb = context.services.get("mongodb-atlas");
const mycollection = mongodb.db("pressure").collection("stjernelaks");
return mycollection.find({
isotime: {
'$gte': new Date(arg1),
'$lt': new Date(arg2)
}
}).toArray();
};
I can't figure out why it doesn't return any documents, so any help would be great.
Update:
I tried modifying my query to return documents with "val" inside a range. Also there was the same problem. Just as with the time and date string, I can use my webhook and hardcode the arguments, and it works. And I can make my webhook return the arguments, it reads them back to me without incident. But it just won't return any documents when it's making a query based on my arguments! It's bizarre!
Use $lte instead of $lt
mycollection.find({
isotime: {
'$gte': new Date("2019-04-11T07:01:32.000+00:00"),
'$lte': new Date("2019-04-11T09:01:32.000+00:00")
}
})
Related
I have a mongo collection where a field is supposed to point to another document's id in the same collection, but instead it is pointing to its "number". I need to perform an update on them but I'm having some problems on forming the query. Could you help me?
The structure of the document is like this:
{
"_id": "269410e2-cebf-40f1-a81f-fdce34185cdc",
"number": 1471,
"alternativeLocationId": "9871",
"locationType": "DUMMY"
},
{
"_id": "2945b24a-b82f-45a9-ad06-a884379b5597",
"number": 9871,
"locationType": "MAIN"
}
So as asked, I'd need to make document with number 1471 "alternativeLocationId" to be "2945b24a-b82f-45a9-ad06-a884379b5597" instead of 9871 (Note that the referenced documents are not locationType "DUMMY" nor have this alternativeLocationId field).
The query I've done so far goes like this, but when executed its not doing any changes:
db.location.find({alternativeLocationId: {$exists:true}}).forEach(
function (loc) {
var correctLocation = db.location.findOne({number: loc.alternativeLocationId});
db.location.update(
{_id: loc._id},
{$set: {alternativeLocationId: correctLocation._id} }
);
}
);
As mentioned by user20042973 the issue was the type mismatch between alternativeLocationId and locationNumber, after converting the value to int when looking for it, it works perfectly.
I'm trying to use expression languge to generate ISO string in Nifi getMongo Query field using following query,
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
But i'm getting invalid JSON error error as double quotes are not escaped. When we try to escape it using \ operator, nifi is not evaluating the expression language. Is there any method or workaround to get this working ?
Thanks in advance
GetMongo processor of nifi requires your query to be in extended json format of mongo.So you can use query of below format to query mongo based on datetime:
{"bday":{"$gt":{"$date":"2014-01-01T05:00:00.000Z"}, "$lt" :{"$date":"2019-01-
01T05:00:00.000Z"}}}
I used your not changed expression in UpdateAttribute processor to evaluate new flowFile attribute.
your expression:
{
"remindmeDate": {
"$gte": "${now():format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}",
"$lte": "${now():toNumber():plus(359999):format("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",'GMT')}"
}
}
the result:
{
"remindmeDate": {
"$gte": "2017-06-16T07:38:04.811Z",
"$lte": "2017-06-16T07:44:04.810Z"
}
}
and this is a correct json object.
Finally I found that GetMongo.Query property does not support nifi expression language (nifi 1.2.0 and 1.3.0). Just hover the question mark near parameter.
It means no way to build dynamic query (
Seems need to register an issue... https://issues.apache.org/jira/browse/NIFI-4082
But it's possible to specify current and relative date in mongo query language. something like this:
{
"remindmeDate": {
"$gte": new Date(),
"$lte": new Date(ISODate().getTime() + 359999)
}
}
Nifi's getMongo Query field doesnt support EL. So i created a stored function in MongoDB for my dynamic query and called it from Nifi.
{
"_id" : "reminderDateGMT",
"value" : function (reminderDateGMT) {
var reminder = new Date(reminderDateGMT)
var fromDate = new Date();
var toDate = new Date(new Date().getTime()+(1000 * 60 * 60));
if ((reminder >= fromDate) && (reminder <=toDate )) {
return true;
} else {
return false;
}
}
}
In nifi GetMongo Query,
{
"$where": "reminderDateGMT(this.reminderDateGMT)"
}
I think you may be able to use the unescapeJson expression language function to handle this. You have to provide valid JSON (escaped quotes) for the field level (PropertyDescriptor in NiFi parlance) validation, but the expression language string expects unescaped JSON during expression parsing, so the unescapeJson function removes the escapes first and then format receives a properly quoted string.
{
"remindmeDate": {
"$gte": "${now():format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}",
"$lte": "${now():toNumber():plus(359999):format(\"yyyy-MM-dd'T'HH:mm:ss.SSS'Z'\":unescapeJson(),'GMT')}"
}
}
I had a similar discussion on the mailing list, and here is the solution I found that works:
Mongo console:
db.system.js.save({
"_id": "lastFiveMinutes",
"value": function() {
return new Date(ISODate().getTime() - (1000 * 60 * 5));
}
});
db.loadServerScripts();
Query field:
{
"$where": "obj.ts >= lastFiveMinutes()"
}
Note: you probably want to set this on a timer in the scheduling property.
I know this is pretty old post, but I spent lot many hours and found a solution which worked for me.
Use UpdateAttribute Processor and created two attributes to calculate the date range, I need to fetch the mongo documents :
startDate: "${now():format('yyyy-MM-dd')}"
endDate : "${now():toNumber():plus(86400000):format('yyyy-MM-dd')}"
enter image description here
After that pass these attributes to GetMongo processor:
Query : {"createdDate":{"$gte":ISODate(${startDate}), "$lt":ISODate(${endDate})}}
I'm trying to use some nested arrays in mongoDB but after several hours fighting with the mongo docs and several other SO questions. I'm thinking I might be doing something terribly wrong. The code below is supposed to update the confirms array in the confirms array but it won't ever do anything but add new entries. Any help would be appreciated.
Events.update({_id: eventId, "confirms.person": personId}, {
$set: {
"confirms.$.person": personId,
"confirms.$.confirmed": isConfirmed,
"confirms.$.timestamp": new Date()
}
});
And here is a sample of the data I'm trying to operate on, it is currently WRONG because a personId should be unique.
{
"_id": "RnE4PaPSZ9FC9MAAQ",
"eventTitle": "Epic lan PARTY!!!!!",
"eventDate": "31/07/2015",
"confirms": [{
"person": "jjoqekYYaA6n8nYvs",
"confirmed": true,
"timestamp": ISODate("2015-07-25T17:15:28.212Z")
}, {
"person": "jjoqekYYaA6n8nYvs",
"confirmed": true,
"timestamp": ISODate("2015-07-25T17:16:50.485Z")
}
}]
}
Just in case it might turn out to be relevant, i'm working in meteor js.
To update the element of an array which is the variable of a Collection item you could do this:
Get the array:
var arr = Events.findOne({_id: eventId}).confirms;
Do your operation
Update old array with modified array:
Events.update({_id: eventId}, {$set: {confirms: arr}});
In case of problems with step 2: you could iterate through the array checking if this.person === personId then set this.confirmed = isConfirmed & this.timestamp = new Date.
There might be ways of operating on arrays within Mongo but this one works for sure.
I'm working on a Meteor application and I have data for a weekly timetable of the format -
events:
event:
day: "Monday"
time: "9am"
location: "A"
event:
day: "Monday"
time: "10am"
location: "B"
There are numerous entries for each day. Can I run a query that will return an object of the format -
day: "Monday"
events:
event:
time: "9am"
location: "A"
event:
time: "10am"
location: "B"
I could store the object in the second format but prefer the first for ease of deleting and updating individual events.
I also want to return them ordered by day of week if there's a nice way to do that.
Several options:
You can use an aggregation command but be warned, you will loose reactivity: it means that except when you reload your template, you will not get external updates. You would also need to use a package to add the aggregation command to Meteor in order to achieve that.
My personal favorite: you don't need to aggregate (and loose reactivity) to achieve your data transformation. You can use a simple Collection.find() query and extend/reduce/modify it using a clever mix of cursor.Observe() and conditional modifications. Have a look at this answer, it did the trick for me (I needed a sum with black listing of some fields, but you can easily adapt it to your group/sorting case) : https://stackoverflow.com/a/30813050/3793161. Comment if you need more details on this
If you plan to have several servers, be warned that each server will have to observe so it may lead to an unnecessary load. So my third solution is either use collection hooks or methods to update an additional field containing every event for each day/user (whatever you need).See #David Weldon answer about that here: https://stackoverflow.com/a/31190896/3793161. In you case, it would probably mean to re-think your database structure to fit your needs (i.e. adding more fields so you ca update them on insert.
EDIT Here are some thoughts on your comment:
If you stick to what you described in the question, you would need seven documents, one per day, with an events field where you put all the events. My second solution is good if you need to rework a collection structure before sending it. However, in your case, you just need an object week with 7 sub-objects matching the days of the week.
I advise you to possible approaches:
use the aggregation in a method, as described by #chridam. Be warned that you will not be able to directly get a sorted array, as stated in mongo $group documentation
$group does not order its output documents
So you need to sort them (by day and by hour within each day) using, for example _.sortBy() before you return your method result. By the way, if you want to know what is going on in your method call, clientside, here is how you should write the call:
Meteor.call("getGroupedDailyEvents", userId, function(error, result){
if(error){
console.log("error", error);
}
if(result){
//do whatever you need
}
});
Make the data sorting client-side. You are looking for an overkill solution because, afaik, you don't need to filter any data to keep it from the user, and you are going to send the data anyway (just with another structure). This is much easier to make a simple helper in your template like this:
Template.displaySchedule.helpers({
"monday_events": function() {
return _.sortBy (events.find({day:"Monday"}).fetch(), "time")
},
//add other days
);
It assumes the format of your time field is sortable this way. If not, you just need to create a function to sort them accordingly to their formats or change the original format into something better suited.
The rest (HTML) would just be to iterate on Monday events using a {{#each monday_events}}
To achieve the desired result, use the aggregation framework where the $group pipeline operator groups all the input documents and apply the accumulator expression $push to the group to get the events array.
Your pipeline would look like this:
var Events = new Mongo.Collection('events');
var pipeline = [
{
"$group": {
"_id": "$day",
"events": {
"$push": {
"time": "$time"
"location": "$location"
}
}
}
},
{
"$project": {
"_id": 0, "day": "$_id", "events": 1
}
}
];
var result = Events.aggregate(pipeline);
console.log(result)
You can add the meteorhacks:aggregate package to implement the aggregation in Meteor:
Add to your app with
meteor add meteorhacks:aggregate
Since this package exposes .aggregate method on Mongo.Collection instances, you can define a method that gets the aggregated result array. For example
if (Meteor.isServer) {
var Events = new Mongo.Collection('events');
Meteor.methods({
getGroupedDailyEvents: function () {
var pipeline = [
{
"$group": {
"_id": "$day",
"events": {
"$push": {
"time": "$time"
"location": "$location"
}
}
}
},
{
"$project": {
"_id": 0, "day": "$_id", "events": 1
}
}
];
var result = Events.aggregate(pipeline);
return result;
}
});
}
if (Meteor.isClient) {
Meteor.call('getGroupedDailyEvents', logResult);
function logResult(err, res) {
console.log("Result: ", res)
}
}
I have a mongo collection in which the documents have a field that is an array. I want to be able to publish everything in the documents except for the elements in the array that were created more than a day ago. I suspect the answer will be somewhat similar to this question.
Meteor publication: Hiding certain fields in an array document field?
Instead of limiting fields in the array, I just want to limit the elements in the array being published.
Thanks in advance for any responses!
EDIT
Here is an example document:
{
_id: 123456,
name: "Unit 1",
createdAt: (datetime object),
settings: *some stuff*,
packets: [
{
_id: 32412312,
temperature: 70,
createdAt: *datetime object from today*
},
{
_id: 32412312,
temperature: 70,
createdAt: *datetime from yesterday*
}
]
}
I want to get everything in this document except for the part of the array that was created more than 24 hours ago. I know I can accomplish this by moving the packets into their own collection and tying them together with keys as in a relational database but if what I am asking were possible, this would be simpler with less code.
You could do something like this in your publish method:
Meteor.publish("pubName", function() {
var collection = Collection.find().fetch(); //change this to return your data
_.each(collection, function(collectionItem) {
_.each(collectionItem.packets, function(packet, index) {
var deadline = Date.now() - 86400000 //should equal 24 hrs ago
if (packet.createdAt < deadline) {
collectionItem.packets.splice(index, 1);
}
}
}
return collection;
}
Though you might be better off storing the last 24 hours worth of packets as a separate array in your document. Would probably be less taxing on the server, not sure.
Also, code above is untested. Good luck.
you can use the $elemMatch projection
http://docs.mongodb.org/manual/reference/operator/projection/elemMatch/
So in your case, it would be
var today = new Date();
var yesterday = new Date(today);
yesterday.setDate(today.getDate() - 1);
collection.find({}, //find anything or specifc
{
fields: {
'packets': {
$elemMatch: {$gt : {'createdAt' : yesterday /* or some new Date() */}}
}
}
});
However, $elemMatch only returns the FIRST element matching your condition. To return more than 1 element, you need to use the aggregation framework, which will be more efficient than _.each or forEach, particularly if you have a large array to loop through.
collection.rawCollection().aggregate([
{
$match: {}
},
{
$redact: {
$cond: {
if : {$or: [{$gt: ["$createdAt",yesterday]},"$packets"]},
then: "$$DESCEND",
else: "$$PRUNE"
}
}
}], function (error, result ){
});
You specify the $match in a way similar to find({}). Then all the documents that match your conditions get pipped into the $redact which is specified by the $cond.
$redact scans the document from top level to bottom. At the top level, you have _id, name, createdAt, settings, packets; hence {$or: [***,"$packets"]}
The presence of $packets in the $or allows the $redact to scan the second level which contain the _id, temperature and createdAt; hence {$gt: ["$createdAt",yesterday]}
This is async, you can use Meteor.wrapAsync to wrap around the function.
Hope this help