Mongo 3.2 query timeseries value at specific time - mongodb

I have some timeseries data stored in Mongo with one document per account, like so:
{
"account_number": 123,
"times": [
datetime(2017, 1, 2, 12, 34, 56),
datetime(2017, 3, 4, 17, 18, 19),
datetime(2017, 3, 11, 0, 1, 11),
]
"values": [
1,
10,
9001,
]
}
So, to be clear in the above representation account 123 has a value of 1 from 2017-01-02 12:34:56 until it changes to 10 on 2017-03-04 17:18:19, which then changes to 9001 at 2017-03-11, 00:01:11.
There are many accounts and each account's data is all different (could be at different times and could have more or fewer value changes than other accounts).
I'd like to query for each users value at a given time, e.g. "What was each users value at 2017-01-30 02:03:04? Would return 1 for the above account as it was set to 1 before the given time and did not change until after the given time.
It looks like $zip would be useful but thats only available in Mongo 3.4 and I'm using 3.2 and have no plans to upgrade soon.
Edit:
I can get a small part of the way there using:
> db.account_data.aggregate([{$unwind: '$times'}, {$unwind: '$values'}])
which returns something like:
{"account_number": 123, "times": datetime(2017, 1, 2, 12, 34, 56), "values": 1},
{"account_number": 123, "times": datetime(2017, 1, 2, 12, 34, 56), "values": 10},
#...
which isn't quite right as it is returning the cross product of times/values

This is possible using only 3.2 features. I tested with the Mingo library
var mingo = require('mingo')
var data = [{
"account_number": 123,
"times": [
new Date("2017-01-02T12:34:56"),
new Date("2017-03-04T17:18:19"),
new Date("2017-03-11T00:01:11")
],
"values": [1, 10, 9001]
}]
var maxDate = new Date("2017-01-30T02:03:04")
// 1. filter dates down to those less or equal to the maxDate
// 2. take the size of the filtered date array
// 3. subtract 1 from the size to get the index of the corresponding value
// 4. lookup the value by index in the "values" array into new "valueAtDate" field
// 5. project the extra fields
var result = mingo.aggregate(data, [{
$project: {
valueAtDate: {
$arrayElemAt: [
"$values",
{ $subtract: [ { $size: { $filter: { input: "$times", as: "time", cond: { $lte: [ "$$time", maxDate ] }} } }, 1 ] }
]
},
values: 1,
times: 1
}
}])
console.log(result)
// Outputs
[ { valueAtDate: 1,
values: [ 1, 10, 9001 ],
times:
[ 2017-01-02T12:34:56.000Z,
2017-03-04T17:18:19.000Z,
2017-03-11T00:01:11.000Z ] } ]

Not sure how to do the same with MongoDb 3.2, however from 3.4 you can do the following query:
db.test.aggregate([
{
$project:
{
index: { $indexOfArray: [ "$times", "2017,3,11,0,1,11" ] },
values: true
}
},
{
$project: {
resultValue: { $arrayElemAt: [ "$values", "$index" ] }
}
}])

Related

MongoDB update array if condition matches

I'm looking for a MongoDB aggregation pipeline which updates an array with a conditional statement.
My data looks like the following:
{
"_id": 1,
"locations": [
{
"controllerID": 1,
"timestamp": 1234
},
{
"controllerID": 2,
"timestamp": 2342
}
]
},...
Potential new entry:
{
"controllerID": 2,
"timestamp": //will be set automatically
}
At first I want to match the _id (not a problem) and then push the new entry to the locations array if the element with the newest/latest timestamp has a different controllerID.
When pushing a new location object the timestamp will be set automatically.
Example 1
Input:
{
"controllerID": 2,
}
Expected Result:
{
"_id": 1,
"locations": [
{
"controllerID": 1,
"timestamp": 1234
},
{
"controllerID": 2,
"timestamp": 2342
}//noting is added because the newset entry in the array has the same controllerID
]
},
Example 2
Input:
{
"controllerID": 1,
}
Expected Result:
{
"_id": 1,
"locations": [
{
"controllerID": 1,
"timestamp": 1234
},
{
"controllerID": 2,
"timestamp": 2342
},
{//added because the controllerID is different to te last element
"controllerID": 1,
"timestamp": 4356
}
]
},
Thanks in advance!
Here's a solution.
var candidate = 2;
rc=db.foo.update({}, // add matching criteria here; for now, match ALL
[
[
// We cannot say "if condition then set fld = X else do nothing".
// We must say "set fld to something based on condition."
// The common pattern becomes:
// "Set fld to (if condition then X else fld)"
// in other words, set the fld to *itself*
//
// Note the use of the dot operator on the $locations field.
// Also, not sure about what sort of new timestamp is desired so let's
// just throw in an ISODate() for now.
{$set: {'locations': {$cond: [
{$ne:[candidate, {$last:'$locations.controllerID'}]}, // IF not same as candidate...
{$concatArrays: ['$locations',
// $concatArrays wants arrays, not objects, so we must wrap our new
// object with [] to make an array of 1:
[ {controllerId:candidate,timestamp:new ISODate() } ]
]}, // THEN concat a new entry to end of existing locations
'$locations' // ELSE just set back to existing locations
]}
}}
],
{multi:true}
);
The engine is "smart enough" to realize that setting a field to itself will not trigger a modification so the approach is performant and will not rewrite the entire set of matched objects; this can be seen in the output of the update() call, e.g.:
printjson(rc);
{ "nMatched" : 1002, "nUpserted" : 0, "nModified" : 1 }

Extract last value from an array of objects using monogdb query language?

I'm new to Mongo. By new I mean couple of hours new.
Basically I have this document structure:
{
_id: ObjectId("614513461af3bf569fdc420e"),
item: 'postcard',
status: 'A',
size: { h: 10, w: 15.25, uom: 'cm' },
instock: [ { warehouse: 'B', qty: 15 }, { warehouse: 'C', qty: 35 } ]
}
I would like if possible to extract particular field (i.e. its value) from instock's last element. In this case I just need to extract 35 i.e. qty field.
I have managed to do this:
db.offer.find( { _id: ObjectId("614513461af3bf569fdc420e") }, { instock: 1, _id: 0} )
Which results in :
{ instock: [ { warehouse: 'B', qty: 15 }, { warehouse: 'C', qty: 35 } ] }
I don't know how to reach to last object in array and than its qty field and everything needs to be as single query.
Aggregate solution
(requires MongoDB 5, else query would be a little bigger)
Query
filter for the _id with the $match stage
get last element of $instock, and then field qty
project to keep only the above part
*we do it like we would do it in a programming language, get last element, and get a field value.
Test code here
db.collection.aggregate([
{"$match": {"_id": ObjectId("614513461af3bf569fdc420e")}},
{
"$project": {
"_id": 0,
"qty": {"$getField": {"field": "qty","input": {"$last": "$instock"}}}
}
}
])

How to add every other columns together in Mongo?

I've been cracking my head over the addition of every 'other' columns together during aggregation in Mongo.
A sample of my data:
[
{'item': 'X',
'USA': 3,
'CAN': 1,
'CHN': 1,
'IDN': 1,
:
:
:
},
{'item': 'R',
'USA': 2,
'CAN': 2,
'CHN': 1,
'IDN': 2,
:
:
:
}
]
At the aggregate stage, I would like to have a new field called 'OTHER', which is the resultant of the summation of all the fields that are not specified.
My desired result is this:
[
{'item': 'X',
'NAM': 79,
'IDN': 51,
'OTHER': 32
},
{'item': 'R',
'NAM': 42,
'IDN': 11,
'OTHER': 20
}
]
So far, the closest I could get is using this:
mycoll.aggregate([
{'$addFields':{
'NAM': {'$add':[{'$ifNull':['$CAN', 0]},{'$ifNull':['$USA', 0]}]},
'INDIA': {'$ifNull':['$IDN', 0]},
'OTHER': /* $add all the fields that are not $USA, $CAN, $IDN*/
}},
])
Mongo gurus, please enlighten this poor soul. Deeply appreciate it. Thanks!
In general the idea is converting your document to an array so we could iterate over it while ignoring unwanted fields.
{
'$addFields': {
'NAM': {'$add': [{'$ifNull': ['$CAN', 0]}, {'$ifNull': ['$USA', 0]}]},
'INDIA': {'$ifNull': ['$IDN', 0]},
"OTHER": {
$reduce:
{
input: {"$objectToArray": "$$ROOT"},
initialValue: {sum: 0},
in: {
sum: {
$cond: {
if: {$in: ["$$this.k", ['_id', "item", "CAN", "USA", "IDN"]]},
then: "$$value.sum",
else: {$add: ["$$value.sum", "$$this.v"]}
}
}
}
}
}
}
}
Obivously you should also add any other fields that you have in your document that you do not want to sum up / are not of type number.

Trying to aggregate based on substring matches in mongodb 3.2

Let's say my collection has documents with ExpName field and Rname field. Expname are all of the type - exp_1, exp_2 etc. Rname is a character string with 4 dashes for example. "As-34rt-d3r5-4453f-er4"
I need to aggregate based on experiment name and removing the text between the last two dashes. In the example I gave above that would be "As-34rt-d3r5"
question 1) how do i incorporate this in one table?
question 2) i solved this in a dirty fashion for one exp, because it seemed like the number of characters was almost the same, so I could just take the first 13 characters which seemed like it was the the substring omitting the last two dashes. Is there a correct way to do this if the text was not so uniform?
db.getCollection('rest01').aggregate(
{$match : {ExpName : "exp_1"}},
{$group: {_id :"$ExpName",_id : {$substr : ["$RName", 0,13]}, total: { $sum:1 }}
})
Ideally I would like to have a result that says Expname, Rnamesubstring, count. This code snippet was for exp_1 one alone. Is it even possible to get it all in one result?
Here is how you could do that:
db.getCollection('rest01').aggregate({
$project: {
"ExpName": 1,
"splitRName": { $split: [ "$RName", "-" ] } // add an array with the constituents of your dash-delimited string id as a new field "splitRName"
}
}, {
$group: {
_id: { // our group id shall be made up of both...
"ExpName": "$ExpName", // ...the "ExpName" field...
"Rnamesubstring": { // and some parts of the "RName" field
$concat:
[
{ $arrayElemAt: [ "$splitRName", 0 ] },
"-",
{ $arrayElemAt: [ "$splitRName", 1 ] },
"-",
{ $arrayElemAt: [ "$splitRName", 2 ] }
]
}
},
total: { $sum: 1 }
}
})
In case you want to do it in MongoDB v3.2 (as stated in your comment), here is something that is not exactly pretty but works:
db.getCollection('rest01').aggregate({
$group: {
_id: { // our group id shall be made up of both...
"ExpName": "$ExpName", // ...the "ExpName" field...
"Rnamesubstring": {
$substr:
[
"$RName",
0,
{
$ifNull:
[
{
$arrayElemAt:
[{
$filter: {
input: {
$map: {
input: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 /* add numbers as required */ ],
as: "index",
in: {
$cond: {
if: { $eq: [ "-", { $substr: [ "$RName", "$$index", 1 ] } ] }, // if the string we look at is a dash...
then: "$$index", // ...then let's remember it
else: null // ...otherwise ignore it
}
}
}
},
as: "item",
cond: { $ne: [ null, "$$item" ] } // get rid of all null values
}
},
2 ] // we want the position of the third dash in the string (only)
},
1000 // in case of a malformed RName (wrong number of dashes or completely missing) we want the entire substring
]
}
]
}
},
total: { $sum: 1 }
}
})
Update 2: You seem to be having some data related issues as per your comments (so either missing RName values or improperly structured ones, i.e. without the required number of sections with dashes in between). I have updated the above statement for v3.2 to deal with these rows. You may want to find out, though, which rows actually cause this behaviour. They can be easily identified using the following statement:
db.getCollection('rest01').aggregate({
$project: {
_id: 1,
RName: 1,
"Rnamesubstring": {
$arrayElemAt:
[{
$filter: {
input: {
$map: {
input: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30 /* add numbers as required */ ],
as: "index",
in: {
$cond: {
if: { $eq: [ "-", { $substr: [ "$RName", "$$index", 1 ] } ] }, // if the string we look at is a dash...
then: "$$index", // ...then let's remember it
else: null // ...otherwise ignore it
}
}
}
},
as: "item",
cond: { $ne: [ null, "$$item" ] } // get rid of all null values
}
},
2 ] // we want the position of the third dash in the string (only)
}
}
}, {
$match: { "Rnamesubstring": { $exists:false } }
})

mongo equivalent of sql query

i need to build a mongo query to get results from a collection which has the same structure as the following sql.
click for picture of table structure
my sql query:
SELECT * FROM (
SELECT
db.date,
db.points,
db.type,
db.name,
db.rank,
YEARWEEK( db.date ) AS year_week
FROM _MyDatabase db
WHERE
db.personId = 100 AND
db.date BETWEEN '2012-10-01' AND '2015-09-30'
ORDER BY
YEARWEEK( db.date ),
db.type,
db.points DESC
) x
GROUP BY
x.year_week DESC,
x.type;
the result looks like this
date points type name rank year_week
-------------------------------------------------
23.10.2014 2000 1 Fish 2 201442
12.10.2014 2500 1 Fish 2 201441
16.10.2014 800 2 Fish 2 201441
i have tried different group / aggregate queries so far, but i couldn't get a similar result. hopefully one of you has more mongo experience than i and can give me a hint on how to solve this.
You would want something like this:
var start = new Date(2012, 9, 1),
end = new Date(2015, 8, 30),
pipeline = [
{
"$match": {
"personId": 100,
"date": { "$gte": start, "$lte": end }
}
},
{
"$project": {
"date": 1, "points": 1, "type": 1, "name": 1, "rank": 1,
"year_week": { "$week": "$date" }
}
},
{
"$sort": {
"year_week": 1,
"type": 1,
"points": -1
}
},
{
"$group": {
"_id": {
"year_week": "$year_week",
"type": "$type"
}
}
}
];
db.getCollection("_MyDatabase").aggregate(pipeline);