Are there some good examples where I want to aggregate the following using Mongo's Aggregation Pipeline in scala.
Want to get the sum PriceAmt.
Any good URL with some great Scala examples would be of great help..
Following is the JSON that is being retrieved from MongoDB.
{
"_id": ObjectId("5269723bd516ec3a69f3639e"),
"Id": ObjectId("5269723ad516ec3a69f3639d"),
"results": [
{
"Id": "526971f5b5b8b9148404623a",
"pricingResult": {
"TxId": 0,
"PriceId": "Large_Batch_1",
"Errors": [ ],
"Disposition": [
{
"GroupId": 1,
"PriceAmt": 20,
"Status": "Priced Successfully",
"ReasonCode": 0,
"PricingMethodologyId": 2,
"Lines": [{ "Id": 1 }]
}
]
}
},
{
"ClaimId": "526971f4b5b8b91484046229",
"pricingResult": {
"TxId": 0,
"PriceId": "Large_Batch_1",
"Errors": [ ],
"Disposition": [
{
"GroupId": 1,
"PriceAmt": 20,
"Status": "Priced Successfully",
"ReasonCode": 0,
"PricingMethodologyId": 2,
"Lines": [{ "Id": 1 }]
}
]
}
}
]
}
I do not think that this is a valid question, but if you are looking for examples of aggregation framework and scala take a look at this links:
https://bhudgeons.telegr.am/blog_posts/how-to-use-the-new-mongodb-aggregation-framew
http://mongoblog.tumblr.com/post/44108207769/mongodb-aggregation-using-scala-1-2
and after this you can try to replicate some results with scala using examples from mongodb
Casbah, the official Scala driver, support the aggregation framework.
Please take a look at the documentation.
Related
I'm very new to mongodb. I'm trying to do an aggregation pipeline for lookup kinda like SQL left join. Given the following document's schema:
Mongo playground: https://mongoplayground.net/p/yAIwH5V2yv8
Characters:
{
"_id": 1,
"account_id": 1,
"world_id": 0,
"name": "hello"
}
Inventories:
{
"_id": 7,
"character_id": 2,
"type": "EQUIPPED"
}
Items:
{
"_id": 1,
"inventory_id": 7
}
I want to query for characters and look up inventories as well as items in inventories. I was able to achieve this however I would like to separate the inventories field in characters result document.
Current result:
{
"_id": 2,
"account_id": 1,
"world_id": 0,
"name": "hello",
"inventories: [
{
"_id": 1,
"character_id": "2",
"type": "EQUIPPED",
"items: [...]
}
]
}
What I want is based on the type of inventory I want it to be a separate field of the resulted character document something like this:
{
"_id": 2,
"account_id": 1,
"world_id": 0,
"name": "hello",
"equippedInventory: {
"_id": 1,
"character_id": "2",
"type": "EQUIPPED",
"items: [...]
},
"equipInventory: {
"_id": 2,
"character_id": "2",
"type": "EQUIP",
"items: [...]
},
}
Also, is my pipeline the best way to achieve this?
I got a collection of 10000 ca. docs, where each doc has the following format:
{
"_id": {
"$oid": "631edc6e207c89b932a70a26"
},
"name": "Ethereum",
"auditInfoList": [
{
"coinId": "1027",
"auditor": "Fairyproof",
"auditStatus": 2,
"reportUrl": "https://www.fairyproof.com/report/Covalent"
}
],
"circulatingSupply": 122335921.0615,
"cmcRank": 2,
"dateAdded": "2015-08-07T00:00:00.000Z",
"id": 1027,
"isActive": 1,
"isAudited": true,
"lastUpdated": 1662969360,
"marketPairCount": 6085,
"quotes": [
{
"name": "USD",
"price": 1737.1982544180462,
"volume24h": 14326453277.535921,
"marketCap": 212521748520.66168,
"percentChange1h": 0.62330307,
"percentChange24h": -1.08847937,
"percentChange7d": 10.96517745,
"lastUpdated": 1662966780,
"percentChange30d": -13.49374496,
"percentChange60d": 58.25153862,
"percentChange90d": 42.27475921,
"fullyDilluttedMarketCap": 212521748520.66,
"marketCapByTotalSupply": 212521748520.66168,
"dominance": 20.0725,
"turnover": 0.0674117,
"ytdPriceChangePercentage": -53.9168
}
],
"selfReportedCirculatingSupply": 0,
"slug": "ethereum",
"symbol": "ETH",
"tags": [
"mineable",
"pow",
"smart-contracts",
"ethereum-ecosystem",
"coinbase-ventures-portfolio",
"three-arrows-capital-portfolio",
"polychain-capital-portfolio",
"binance-labs-portfolio",
"blockchain-capital-portfolio",
"boostvc-portfolio",
"cms-holdings-portfolio",
"dcg-portfolio",
"dragonfly-capital-portfolio",
"electric-capital-portfolio",
"fabric-ventures-portfolio",
"framework-ventures-portfolio",
"hashkey-capital-portfolio",
"kenetic-capital-portfolio",
"huobi-capital-portfolio",
"alameda-research-portfolio",
"a16z-portfolio",
"1confirmation-portfolio",
"winklevoss-capital-portfolio",
"usv-portfolio",
"placeholder-ventures-portfolio",
"pantera-capital-portfolio",
"multicoin-capital-portfolio",
"paradigm-portfolio",
"injective-ecosystem"
],
"totalSupply": 122335921.0615
}
Im pulling updated version of it and, to aviod duplicates, im doing the following by using 'update_one'
for doc in new_doc_list:
CRYPTO_TEMPORARY_LIST.update_one(
{ "name" : doc['name']},
{ "$set": {
"lastUpdated": doc['lastUpdated']
}
},
upsert=True)
The problem is it's too slow.
I'm trying to figure out how to improve speed by using update_many but can't figure out how to set it up.
I Basically want to update every document x name. Completely change the doc and not the "lastUpdated" field would b even better.
Thanks guys <3
I am running a simple query using mongoDB compass using filter and project and I'm having a behaviour I can't explain.
Here is my filter:
{
"$and": [
{
"_id": ObjectId('611ee5ee6b93815ee436969e')
},
{
"type": "article"
}
]
}
I get the following result:
{
"_id": {
"$oid": "611ee5ee6b93815ee436969e"
},
"type": "article",
"history": [],
"liked": [],
"parentId": "61105f00cc11ec10406fd1c4",
"permissionList": [],
"title": "Test",
"wikiId": "610de623fbfa1e58cdba9d2c"
}
As expected I get all the fields, in particular type and wikiId
However if i add the following projection:
{
"_id": 1,
"wikiId": 1,
"parentId": 1,
"title": 1,
"type": 1,
"permissionList": 1,
"liked": 1,
"history": 1
}
I would expect the same result, however i get:
{
"_id": {
"$oid": "611ee5ee6b93815ee436969e"
},
"type": "article",
"history": [],
"liked": [],
"parentId": "61105f00cc11ec10406fd1c4",
"permissionList": [],
"title": "Test"
}
This time i don't have the field wikiId, however it was requested in the projection.
And what's bug me is that if I do this projection instead:
{
"_id": 1,
"wikiId": 1,
"parentId": 1,
"title": 1,
"permissionList": 1,
"liked": 1,
"history": 1
}
Then I got the wikiId field as expected in the result again.
Anyone can provide me an insight of what is going on with those queries and where i'm mistaken.
Edit 1: The fact I want to use a projection is that depending on the type field I can use different document with different field.
In my Java code I'm using #BsonDiscriminator(key = "type") but when I explicitly want a kind of document I'm creating the appropriate projection to be sure. However in this case I just wanted to simplify the issue I'm facing to the simplest.
Thanks
in compass by default filters are having and condition so your filter can be refactored as below:
{ "_id": ObjectId('611ee5ee6b93815ee436969e'), "type": "article" }
And regarding projection you can just simply mention fields to Exclude or include like below:
{
"_id": 1,
"wikiId": 1,
"parentId": 1,
"title": 1,
"type": 1,
"permissionList": 1,
"liked": 1,
"history": 1
}
if you need all fields then there is no need to mention anything by default in compass you get all fields, but lets say you want all but few fields(_id,wikiId) then use below projection:
{
"_id": 0,
"wikiId": 0,
}
Also in compass you can try clicking find again or refresh button to see as sometimes it does not reflects current filter as as we change filter it fetches data so just hit refresh or find and see it should work.
I'm using loopback 3 to build a backend with mongoDB.
So i have 2 models: Object and Attachment. Object have a relation Embeds2Many to Attachment.
Objects look like that in mongoDB
[
{
"fieldA": "valueA1",
"attachments": [
{
"id": 1,
"url": "abc.com/image1"
},
{
"id": 2,
"url": "abc.com/image2"
}
]
},
{
"fieldA": "valueA2",
"attachments": [
{
"id": 4,
"url": "abc.com/image4"
},
{
"id": 5,
"url": "abc.com/image5"
}
]
}
]
The question is: how can i get Objects with attachments.id=4 over the RestAPI?
I have try with the where and include filter. But it didn't work. It look like, that this function is not implemented in loopback3, right?
I have found the solution. It only works on Mongodb, Cloudant and Memory database.
{
"filter": {
"where": {
"attachments.id": 4
}
}
}
I got a big array with data in the following format:
{
"application": "myapp",
"buildSystem": {
"counter": 2361.1,
"hostname": "host.com",
"jobName": "job_name",
"label": "2361",
"systemType": "sys"
},
"creationTime": 1517420374748,
"id": "123",
"stack": "OTHER",
"testStatus": "PASSED",
"testSuites": [
{
"errors": 0,
"failures": 0,
"hostname": "some_host",
"properties": [
{
"name": "some_name",
"value": "UnicodeLittle"
},
<MANY MORE PROPERTIES>,
{
"name": "sun",
"value": ""
}
],
"skipped": 0,
"systemError": "",
"systemOut": "",
"testCases": [
{
"classname": "IdTest",
"name": "has correct representation",
"status": "PASSED",
"time": "0.001"
},
<MANY MORE TEST CASES>,
{
"classname": "IdTest",
"name": "normalized values",
"status": "PASSED",
"time": "0.001"
}
],
"tests": 8,
"time": 0.005,
"timestamp": "2018-01-31T17:35:15",
"title": "IdTest"
}
<MANY MORE TEST SUITES >,
]}
Where I can distinct three main structures with big data: TestSuites, Properties, and TestCases. My task is to sum all times from each TestSuite so that I can get the total duration of the test. Since the properties and TestCases are huge, the query cannot complete. I would like to select only the "time" value from TestSuites, but it kind of conflicts with the "time" of TestCases in my query:
db.my_tests.find(
{
application: application,
creationTime:{
$gte: start_date.valueOf(),
$lte: end_date.valueOf()
}
},
{
application: 1,
creationTime: 1,
buildSystem: 1,
"testSuites.time": 1,
_id:1
}
)
Is it possible to project only the "time" properties from TestSuites without loading the whole schema? I already tried testSuites: 1, testSuites.$.time: 1 without success. Please notice that TestSuites is an array of one element with a dictionary.
I already checked this similar post without success:
Mongodb update the specific element from subarray
Following code prints duration of each TestSuite:
query = db.my_collection.aggregate(
[
{$match: {
application: application,
creationTime:{
$gte: start_date.valueOf(),
$lte: end_date.valueOf()
}
}
},
{ $project :
{ duration: { $sum: "$testSuites.time"}}
}
]
).forEach(function(doc)
{
print(doc._id)
print(doc.duration)
}
)
Is it possible to project only the "time" properties from TestSuites
without loading the whole schema? I already tried testSuites: 1,
testSuites.$.time
Answering to your problem of prejecting only the time property of the testSuites document you can simply try projecting it with "testSuites.time" : 1 (you need to add the quotes for the dot notation property references).
My task is to sum all times from each TestSuite so that I can get the
total duration of the test. Since the properties and TestCases are
huge, the query cannot complete
As for your task, i suggest you try out the mongodb's aggregation framework for your calculations documents tranformations. The aggregations framework option {allowDiskUse : true} will also help you if you are proccessing "large" documents.