We are trying out MongoDB atlas APIs for one of our projects. However, when we try to filter the data based on a datetime field, it returns zero documents.
API: https://data.mongodb-api.com/app/data-dfmng/endpoint/data/beta/action/find
METHOD: POST
PAYLOAD:
{
"collection": "trade",
"database": "sm",
"dataSource": "Cluster0",
"filter": {
"DATE1": "2022-06-02T00:00:00Z"
},
"sort": {
"SYMBOL": -1
},
"projection": {
"DATE1": 1,
"SYMBOL": 1,
"AVG_PRICE": 1
}
}
What could be wrong here?
Date should be in extended JSON format https://www.mongodb.com/docs/atlas/api/data-api/#date Unix timestamp in millis for 2022-06-02T00:00:00Z is 1654128000000 :
"filter": {
"DATE1": {"$date": {"$numberLong": "1654128000000"}}
},
Related
My MongoDB compass document looks like this
{
"_id": "123456789",
"code": "x123",
"title": "cool",
"createdDate":2022-07-06T08:04:52.156+00:00
"expiryDate":2023-12-31T00:00:00.000+00:00
}
I tried to create a mongo DB script in my pipeline to update the "expirydate" field to a specific date value "9999-12-31T00:00:00.000Z" format. My update script looks like this as I am trying to update it via my pipeline. The createdDate field took the current date correctly.
{
"command": "updateOne",
"datasource_name": "Austrailia",
"collection_name": "Sydney",
"query": {
"code": "x123"
},
"options": {
"upsert": true,
},
"update": {
"$currentDate": {
"createdDate": {
"$type": "date"
}
},
"$set": {
"title": "hot",
"expiryDate": {
"$date": "9999-12-31T00:00:00.000"
}
}
}
}
The script is failing as it is throwing errors -
{MongoError: Unknown modifier: expiryDate. Expected a valid modifier}
The dollar ($) prefixed field \'$date\' in \'expiryDate.$date\' is not valid for storage.
What would be the correct query syntax to update the date field "expiryDate" to the value specified above in the same format here?
I am using mongodb compass and i am trying to insert below data, it is giving error. Any help appreceiated.
/**
* Paste one or more documents here
*/
{
"_id": {
"$oid": "621f567ceff392db081a4135"
},
"CompanyID": "620d2d9efc8cec9c94f26284",
"GeoLevelName": "All India",
"IsActive": 1,
"CreatedUser": "string",
"CreateDate": "2022-02-28T14:27:05.757Z",
"LastModifyDate": "2022-02-28T14:27:05.757Z",
"LastModifyUser": "string"
"GeoLevelMain": [{
"GeoLevelID": "621cdce8b876f1ec17b1cec9",
"GeoLevelValue": "Maharastra"
},{
"GeoLevelID": "621cdce8b876f1ec17b1cec9",
"GeoLevelValue": "Maharastra"
}],
"GeographyID": "621cde14b876f1ec17b1cece",
"DBID": "620f658d6dee6848caf53832",
"Division": {
"DivisionID": "6215d68d9e4786b2f7ab80a0",
"DivisionName": "DivisionName"
}
}
I'm trying to execute a query like:
{array.0.property: {$ne: null}}.
It return nothing even if all documents have this property different from null.
After some tests i noticed that it work using $elemMatch, but i need to query only for the first element of the array.
The first element is to be considered as "Master" where all query should search.
I can't change document "schema".
Anyone know ho to solve this problem?
I'm using Mongodb 3.6.8.
Thanks in advice.
Example query:
db.getCollection('tasks').find({'details.0.code': {$ne: null}});
Example documents:
{
"name": "test",
"date": 2018-07-17 06:30:00.000Z,
.....,
"details": [
{
"code": '123',
"description": 'something',
"resolutionYear": 2018
},
{
"code": null,
"description": 'secondary',
"resolutionYear": 2019
}
]
},
{
"name": "exam",
"date": 2018-09-20 09:00:00.000Z,
.....,
"details": [
{
"code": null,
"description": 'exam',
"resolutionYear": null
}
]
}
I got a big array with data in the following format:
{
"application": "myapp",
"buildSystem": {
"counter": 2361.1,
"hostname": "host.com",
"jobName": "job_name",
"label": "2361",
"systemType": "sys"
},
"creationTime": 1517420374748,
"id": "123",
"stack": "OTHER",
"testStatus": "PASSED",
"testSuites": [
{
"errors": 0,
"failures": 0,
"hostname": "some_host",
"properties": [
{
"name": "some_name",
"value": "UnicodeLittle"
},
<MANY MORE PROPERTIES>,
{
"name": "sun",
"value": ""
}
],
"skipped": 0,
"systemError": "",
"systemOut": "",
"testCases": [
{
"classname": "IdTest",
"name": "has correct representation",
"status": "PASSED",
"time": "0.001"
},
<MANY MORE TEST CASES>,
{
"classname": "IdTest",
"name": "normalized values",
"status": "PASSED",
"time": "0.001"
}
],
"tests": 8,
"time": 0.005,
"timestamp": "2018-01-31T17:35:15",
"title": "IdTest"
}
<MANY MORE TEST SUITES >,
]}
Where I can distinct three main structures with big data: TestSuites, Properties, and TestCases. My task is to sum all times from each TestSuite so that I can get the total duration of the test. Since the properties and TestCases are huge, the query cannot complete. I would like to select only the "time" value from TestSuites, but it kind of conflicts with the "time" of TestCases in my query:
db.my_tests.find(
{
application: application,
creationTime:{
$gte: start_date.valueOf(),
$lte: end_date.valueOf()
}
},
{
application: 1,
creationTime: 1,
buildSystem: 1,
"testSuites.time": 1,
_id:1
}
)
Is it possible to project only the "time" properties from TestSuites without loading the whole schema? I already tried testSuites: 1, testSuites.$.time: 1 without success. Please notice that TestSuites is an array of one element with a dictionary.
I already checked this similar post without success:
Mongodb update the specific element from subarray
Following code prints duration of each TestSuite:
query = db.my_collection.aggregate(
[
{$match: {
application: application,
creationTime:{
$gte: start_date.valueOf(),
$lte: end_date.valueOf()
}
}
},
{ $project :
{ duration: { $sum: "$testSuites.time"}}
}
]
).forEach(function(doc)
{
print(doc._id)
print(doc.duration)
}
)
Is it possible to project only the "time" properties from TestSuites
without loading the whole schema? I already tried testSuites: 1,
testSuites.$.time
Answering to your problem of prejecting only the time property of the testSuites document you can simply try projecting it with "testSuites.time" : 1 (you need to add the quotes for the dot notation property references).
My task is to sum all times from each TestSuite so that I can get the
total duration of the test. Since the properties and TestCases are
huge, the query cannot complete
As for your task, i suggest you try out the mongodb's aggregation framework for your calculations documents tranformations. The aggregations framework option {allowDiskUse : true} will also help you if you are proccessing "large" documents.
Consider the following Elasticsearch (v5.4) object (an "award" doc type):
{
"name": "Gold 1000",
"date": "2017-06-01T16:43:00.000+00:00",
"recipient": {
"name": "James Conroy",
"date_of_birth": "1991-05-30"
}
}
The mapping type for both award.date and award.recipient.date_of_birth is "date".
I want to perform a range aggregation to get a list of the age ranges of the recipients of this award ("Under 18", "18-24", "24-30", "30+"), at the time of their award. I tried the following aggregation query:
{
"size": 0,
"query": {"match_all": {}},
"aggs": {
"recipients": {
"nested": {
"path": "recipient"
},
"aggs": {
"age_ranges": {
"range": {
"script": {
"inline": "doc['date'].date - doc['recipient.date_of_birth'].date"
},
"keyed": true,
"ranges": [{
"key": "Under 18",
"from": 0,
"to": 18
}, {
"key": "18-24",
"from": 18,
"to": 24
}, {
"key": "24-30",
"from": 24,
"to": 30
}, {
"key": "30+",
"from": 30,
"to": 100
}]
}
}
}
}
}
}
Problem 1
But I get the following error due to the comparison of dates in the script portion:
Cannot apply [-] operation to types [org.joda.time.DateTime] and [org.joda.time.MutableDateTime].
The DateTime object is the award.date field, and the MutableDateTime object is the award.recipient.date_of_birth field. I've tried doing something like doc['recipient.date_of_birth'].date.toDateTime() (which doesn't work despite the Joda docs claiming that MutableDateTime has this method inherited from a parent class). I've also tried doing something further like this:
"script": "ChronoUnit.YEARS.between(doc['date'].date, doc['recipient.date_of_birth'].date)"
Which sadly also doesn't work :(
Problem 2
I notice if I do this:
"aggs": {
"recipients": {
"nested": {
"path": "recipient"
},
"aggs": {
"award_years": {
"terms": {
"script": {
"inline": "doc['date'].date.year"
}
}
}
}
}
}
I get 1970 with a doc_count that happens to equal the total number of docs in ES. This leads me to believe that accessing a property outside of the nested object simply does not work and gives me back some default like the epoch datetime. And if I do the opposite (aggregating dates of birth without nesting), I get the exact same thing for all the dates of birth instead (1970, epoch datetime). So how can I compare those two dates?
I am racking my brain here, and I feel like there's some clever solution that is just beyond my current expertise with Elasticsearch. Help!
If you want to set up a quick environment for this to help me out, here is some curl goodness:
curl -XDELETE http://localhost:9200/joelinux
curl -XPUT http://localhost:9200/joelinux -d "{\"mappings\": {\"award\": {\"properties\": {\"name\": {\"type\": \"string\"}, \"date\": {\"type\": \"date\", \"format\": \"yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ\"}, \"recipient\": {\"type\": \"nested\", \"properties\": {\"name\": {\"type\": \"string\"}, \"date_of_birth\": {\"type\": \"date\", \"format\": \"yyyy-MM-dd\"}}}}}}}"
curl -XPUT http://localhost:9200/joelinux/award/1 -d '{"name": "Gold 1000", "date": "2016-06-01T16:43:00.000000+00:00", "recipient": {"name": "James Conroy", "date_of_birth": "1991-05-30"}}'
curl -XPUT http://localhost:9200/joelinux/award/2 -d '{"name": "Gold 1000", "date": "2017-02-28T13:36:00.000000+00:00", "recipient": {"name": "Martin McNealy", "date_of_birth": "1983-01-20"}}'
That should give you a "joelinux" index with two "award" docs to test this out ("James Conroy" and "Martin McNealy"). Thanks in advance!
Unfortunately, you can't access nested and non-nested fields within the same context. As a workaround, you can change your mapping to automatically copy date from nested document to root context using copy_to option:
{
"mappings": {
"award": {
"properties": {
"name": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"date": {
"type": "date"
},
"date_of_birth": {
"type": "date" // will be automatically filled when indexing documents
},
"recipient": {
"properties": {
"name": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"date_of_birth": {
"type": "date",
"copy_to": "date_of_birth" // copy value to root document
}
},
"type": "nested"
}
}
}
}
}
After that you can access date of birth using path date, though the calculations to get number of years between dates are slightly tricky:
Period.between(LocalDate.ofEpochDay(doc['date_of_birth'].date.getMillis() / 86400000L), LocalDate.ofEpochDay(doc['date'].date.getMillis() / 86400000L)).getYears()
Here I convert original JodaTime date objects to system.time.LocalDate objects:
Get number of milliseconds from 1970-01-01
Convert to number of days from 1970-01-01 by dividing it to 86400000L (number of ms in one day)
Convert to LocalDate object
Create date-based Period object from two dates
Get number of years between two dates.
So, the final aggregation query looks like this:
{
"size": 0,
"query": {
"match_all": {}
},
"aggs": {
"age_ranges": {
"range": {
"script": {
"inline": "Period.between(LocalDate.ofEpochDay(doc['date_of_birth'].date.getMillis() / 86400000L), LocalDate.ofEpochDay(doc['date'].date.getMillis() / 86400000L)).getYears()"
},
"keyed": true,
"ranges": [
{
"key": "Under 18",
"from": 0,
"to": 18
},
{
"key": "18-24",
"from": 18,
"to": 24
},
{
"key": "24-30",
"from": 24,
"to": 30
},
{
"key": "30+",
"from": 30,
"to": 100
}
]
}
}
}
}