I want to find products and for each product attach deals to it. A deal is a product from same collection, yet based on some common properties.
So as per my requirement pipeline should return documents, for each document find other products those aren't same as current, but have equal detail.duration. But even though I've many docs with same duration, deals are always []. Could you please figure out the issue with my pipeline?
Following is the aggregation pipeline I'm running:
I've added filter _id $in just for clarity based on shown documents below. This isn't a part of real pipeline $match query.
db.products
.aggregate([
{
$match: {
_id: {
$in: [
ObjectId("6210fa8746bee3fcbd0ad062"),
ObjectId("6210fa7c46bee3fcbd0acc21"),
],
},
"detail.duration": { $gt: 0 },
},
},
{
$lookup: {
from: "products",
let: { id: "$_id", duration: "$detail.duration" },
as: "deals",
pipeline: [
{
$match: {
_id: { $ne: "$id" },
"detail.duration": "$duration",
},
},
{ $project: { detail: 1 } },
{ $limit: 1 },
],
},
},
{ $limit: 2 },
{ $project: { deals: 1 } },
])
.pretty();
This was the result:
{ "_id" : ObjectId("6210fa7c46bee3fcbd0acc21"), "deals" : [ ] }
{ "_id" : ObjectId("6210fa8746bee3fcbd0ad062"), "deals" : [ ] }
Following are two example documents in the collection:
{
"_id" : ObjectId("6210fa8746bee3fcbd0ad062"),
"book" : "https://wegotrip.com/en/paris-d3/muse-d-orsay-and-musee-de-l-orangerie-combined-tour-ticket-p1117/?SUB_ID=336264",
"address" : "Rue de Lille, 62bis",
"countryName" : "France",
"cityName" : "Paris",
"location" : {
"lang" : 48.859886,
"lat" : 2.3254821,
"country" : ObjectId("6210fa7746bee3fcbd0aca20"),
"city" : ObjectId("6210fa7746bee3fcbd0aca1c"),
"location" : "Rue de Lille, 62bis",
"_id" : ObjectId("6210fa8746bee3fcbd0ad063")
},
"includes" : [
{
"value" : "Skip-the-line ticket to Orsay Museum",
"included" : true
},
{
"value" : "Skip-the-line ticket to the Musée de l'Orangerie",
"included" : true
},
{
"value" : "Detailed description of the Nymphéas from Claude Monet",
"included" : true
},
{
"value" : "Interesting stories of many great artists and their lives",
"included" : true
},
{
"value" : "An easy walkthrough of the Musée d'Orsay and the Musée de l'Orangerie and their great collection",
"included" : true
},
{
"value" : "Headphones — you should bring your own",
"included" : false
}
],
"price" : {
"priceConcession" : null,
"priceChild" : null,
"price" : 57,
"currency" : ObjectId("6210fa7746bee3fcbd0aca2f"),
"_id" : ObjectId("6210fa8746bee3fcbd0ad064")
},
"detail" : {
"isPass" : false,
"features" : [
{
"key" : "audio_guide",
"value" : "Audio Guide"
}
],
"highlights" : [
"Admire the masterpieces by Monet, Renoir, Degas, Cézanne, and many more",
"Discover one of the finest collections of Impressionist art in the world",
"Visit the Nymphéas by Monet, one of the greatest pieces of Impressionism",
"Explore the Guillaume and Walter collection and find out what makes it unique"
],
"details" : [ ],
"images" : [
{
"id" : 7270,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/dsc04800/01d0770dcc0cac4c6de0f6eae70742f6.jpg",
"full" : "https://app.wegotrip.com/media/store/1117/dsc04800.jpg"
},
{
"id" : 7269,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/nympheasannees30salle1parisiennephotorogerviolet/e1270aef1c01391290df71d1f83c8abc.jpg",
"full" : "https://app.wegotrip.com/media/store/1117/nympheasannees30salle1parisiennephotorogerviolet.jpg"
},
{
"id" : 7268,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/ob1f7c80dsc02414-large/7712cb29e133ee3acb4b2bffbc2ac654.jpg",
"full" : "https://app.wegotrip.com/media/store/1117/ob1f7c80dsc02414-large.jpg"
},
{
"id" : 7267,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/tuileriesgardensb16dsc00678talrg/47430ab8a257e3ccd2337d7a0d750c57.jpg",
"full" : "https://app.wegotrip.com/media/store/1117/tuileriesgardensb16dsc00678talrg.jpg"
},
{
"id" : 7266,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/009/54223ef27aac5cd94fe5c20893abf2de.jpg",
"full" : "https://app.wegotrip.com/media/store/1117/009.jpg"
},
{
"id" : 7264,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/monet-morning-with-weeping-willow/09bf842cc9a9d7eade8d0739f704699f.jpg",
"full" : "https://app.wegotrip.com/media/store/1117/monet-morning-with-weeping-willow.jpg"
}
],
"duration" : 2,
"_id" : ObjectId("6210fa8746bee3fcbd0ad065")
},
"availability" : null,
"subcategory" : [
{
"id" : 6,
"title" : "Sightseeing Tickets & Passes",
"slug" : "sightseeing-tickets-passes"
}
],
"category" : [
{
"id" : 6,
"title" : "Sightseeing Tickets & Passes",
"slug" : "sightseeing-tickets-passes"
}
],
"type" : "Audio Guide",
"description" : "Visit the famous Musee d'Orsay and Musée de l'Orangerie in Paris with this combined self-guided tour! \r\n\r\nNavigate through the maze of exhibition rooms with mobile app and see a collection of works by the Impressionists and Expressionists – Seurat, Cezanne, Gaugin, Monet, Renoir, Manet, Van Gogh, Degas; sculptors like Rodin, Pompon and others. Check out a mini-version of the Statue of Liberty! \r\n\r\nExplore the Nymphéas paintings by Claude Monet, that is called \"the Sistine chapel of Impressionism\". Admire the great works of Picasso, Soutine, Rousseau, Matisse and many others part of the Paul Guillaume and Jean Walter collection. Learn about the style and private life of the artists.\r\n\r\nThe audio-guide will provide you with all the information on the cultural significance of these paintings. Walking through rooms you will understand how revolutionary for those times Manet’s, Cezanne’s and Degas’ creation really was casting doubts on conservative, academic conceptions of 'true art' and offering new techniques and ideas.",
"thumbnail" : "https://app.wegotrip.com/media/CACHE/images/store/1117/013/c0b8cce52cb61ab1f30872e6e93385b4.jpg",
"name" : "Musée d'Orsay/Musée de l'Orangerie Combined Admission Ticket & Audio Tour",
"attractionDescription" : "",
"attractionName" : "Musée d'Orsay & Musée de l'Orangerie",
"attraction" : ObjectId("6210fa8746bee3fcbd0ad056"),
"provider" : {
"rating" : {
"count" : 0,
"average" : null,
"_id" : ObjectId("6210fa8746bee3fcbd0ad067")
},
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/1117/013/c0b8cce52cb61ab1f30872e6e93385b4.jpg",
"slug" : "muse-d-orsay-and-musee-de-l-orangerie-combined-tour-ticket",
"id" : "1117",
"key" : "1",
"_id" : ObjectId("6210fa8746bee3fcbd0ad066")
},
"__v" : 0
}
{
"_id" : ObjectId("6210fa7c46bee3fcbd0acc21"),
"book" : "https://wegotrip.com/en/barcelona-d1/the-dali-museum-in-figueres-p3/?SUB_ID=336264",
"address" : "Pujada del Castell, 43",
"countryName" : "Spain",
"cityName" : "Barcelona",
"location" : {
"lang" : 42.26829425831263,
"lat" : 2.95884132385254,
"country" : ObjectId("6210fa7746bee3fcbd0aca3e"),
"city" : ObjectId("6210fa7746bee3fcbd0aca3a"),
"location" : "Pujada del Castell, 43",
"_id" : ObjectId("6210fa7c46bee3fcbd0acc22")
},
"includes" : [
{
"value" : "Recommendations of places to visit to understand the life of Dali better",
"included" : true
},
{
"value" : "Skip-the-line ticket to Dali Theatre-Museum",
"included" : true
},
{
"value" : "Headphones — you should bring your own",
"included" : false
}
],
"price" : {
"priceConcession" : null,
"priceChild" : null,
"price" : 33,
"currency" : ObjectId("6210fa7746bee3fcbd0aca2f"),
"_id" : ObjectId("6210fa7c46bee3fcbd0acc23")
},
"detail" : {
"isPass" : false,
"features" : [
{
"key" : "audio_guide",
"value" : "Audio Guide"
}
],
"highlights" : [
"Discover Dali's surrealism starting with the building of the museum — it's definitely one of a kind",
"Inside the museum you'll find the most famous and controversial works of the artist",
"Our tour will provide you with insights and exiting facts about Dali's works"
],
"details" : [ ],
"images" : [
{
"id" : 6916,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/figueres-oleguer2/032b55c27bb2cd119bdc7fe6c4b86491.jpeg",
"full" : "https://app.wegotrip.com/media/store/3/figueres-oleguer2.jpeg"
},
{
"id" : 6915,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/sky-monument-statue-golden-museum-yellow-1156442-pxherecom/28c645449a9f45ec1e8ede7b7ffbe30f.jpg",
"full" : "https://app.wegotrip.com/media/store/3/sky-monument-statue-golden-museum-yellow-1156442-pxherecom.jpg"
},
{
"id" : 6914,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/architecture-window-museum-landmark-surrealism-catalonia-800928-pxherecom/43691ba6aecc2ee084c300c150e32a03.jpg",
"full" : "https://app.wegotrip.com/media/store/3/architecture-window-museum-landmark-surrealism-catalonia-800928-pxherecom.jpg"
},
{
"id" : 831,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/figueres-oleguers3k6yoz/b9c3093c79cf50e621e022706af59ad6.jpg",
"full" : "https://app.wegotrip.com/media/store/3/figueres-oleguers3k6yoz.jpg"
},
{
"id" : 832,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/shutterstock82210018/2a2450d4f75edf4549d36f2286b6f19b.jpg",
"full" : "https://app.wegotrip.com/media/store/3/shutterstock82210018.jpg"
},
{
"id" : 833,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/dali-museum-8983261920/aa0d93e475c7b7388bee88ff14f8d795.jpg",
"full" : "https://app.wegotrip.com/media/store/3/dali-museum-8983261920.jpg"
},
{
"id" : 834,
"description" : "",
"cover" : false,
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/3/shutterstock196896461/74fc427d0a27f0aa199ed24f4c51bcc5.jpg",
"full" : "https://app.wegotrip.com/media/store/3/shutterstock196896461.jpg"
}
],
"duration" : 2,
"_id" : ObjectId("6210fa7c46bee3fcbd0acc24")
},
"availability" : null,
"subcategory" : [
{
"id" : 3,
"title" : "Theme Tours",
"slug" : "theme-tours"
},
{
"id" : 1,
"title" : "Culture & History",
"slug" : "culture-and-history"
},
{
"id" : 6,
"title" : "Sightseeing Tickets & Passes",
"slug" : "sightseeing-tickets-passes"
}
],
"category" : [
{
"id" : 3,
"title" : "Theme Tours",
"slug" : "theme-tours"
},
{
"id" : 1,
"title" : "Culture & History",
"slug" : "culture-and-history"
},
{
"id" : 6,
"title" : "Sightseeing Tickets & Passes",
"slug" : "sightseeing-tickets-passes"
}
],
"type" : "Audio Guide",
"description" : "The Dalí Theatre and Museum is a museum of the artist Salvador Dalí in his home town of Figueres, in Catalonia, Spain. Dalí is buried there in a crypt below the stage. \r\n\r\nImmerse yourself in an exciting journey through the world of the genius of surrealism. Reveal the meaning of his ambiguous creations and learn the history of the artist's life. Enjoy the unique world of Dali in this excursion.",
"thumbnail" : "https://app.wegotrip.com/media/CACHE/images/store/001_Ispaniya_Figeras_Teatr-01/783c3a10c34eb40c29f14f704cd9c8d1.jpeg",
"name" : "The Dali Theatre-Museum: Skip-the-Line & Audio Tour",
"attractionDescription" : "",
"attractionName" : "Dali Theatre and Museum",
"attraction" : ObjectId("6210fa7c46bee3fcbd0acc15"),
"provider" : {
"rating" : {
"count" : 0,
"average" : null,
"_id" : ObjectId("6210fa7c46bee3fcbd0acc26")
},
"preview" : "https://app.wegotrip.com/media/CACHE/images/store/001_Ispaniya_Figeras_Teatr-01/783c3a10c34eb40c29f14f704cd9c8d1.jpeg",
"slug" : "the-dali-museum-in-figueres",
"id" : "3",
"key" : "1",
"_id" : ObjectId("6210fa7c46bee3fcbd0acc25")
},
"__v" : 0
}
Both of the above have detail.duration set to 2 and as per query, these 2 should have each other considered as a deal and found in result docs, but query returns deals: [], an empty array. I'm unable to figure out the problem.
From $match (Restrictions)
The $match query syntax is identical to the read operation query syntax; i.e. $match does not accept raw aggregation expressions. To include aggregation expression in $match, use a $expr query expression.
And you need to use $$ to get the variable value.
let
To reference variables in pipeline stages, use the "$$" syntax.
Change the $match stage in the pipeline as:
{
$match: {
_id: {
$ne: "$$id"
},
$expr: {
$eq: [
"$detail.duration",
"$$duration"
]
}
}
}
Sample Mongo Playground
I need to upload data to an existing model. This has to be done on daily basis. I guess some changes needs to be done in the index file and i am not able to figure out. I tried pushing the data with the same model name but the parent data was removed.
Any help would be appreciated.
Here is the ingestion json file :
{
"type" : "index",
"spec" : {
"dataSchema" : {
"dataSource" : "mksales",
"parser" : {
"type" : "string",
"parseSpec" : {
"format" : "json",
"dimensionsSpec" : {
"dimensions" : ["Address",
"City",
"Contract Name",
"Contract Sub Type",
"Contract Type",
"Customer Name",
"Domain",
"Nation",
"Contract Start End Date",
"Zip",
"Sales Rep Name"
]
},
"timestampSpec" : {
"format" : "auto",
"column" : "time"
}
}
},
"metricsSpec" : [
{ "type" : "count", "name" : "count", "type" : "count" },
{"name" : "Price","type" : "doubleSum","fieldName" : "Price"},
{"name" : "Sales","type" : "doubleSum","fieldName" : "Sales"},
{"name" : "Units","type" : "longSum","fieldName" : "Units"}],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "day",
"queryGranularity" : "none",
"intervals" : ["2000-12-01T00:00:00Z/2030-06-30T00:00:00Z"],
"rollup" : true
}
},
"ioConfig" : {
"type" : "index",
"firehose" : {
"type" : "local",
"baseDir" : "mksales/",
"filter" : "mksales.json"
},
"appendToExisting" : false
},
"tuningConfig" : {
"type" : "index",
"targetPartitionSize" : 10000000,
"maxRowsInMemory" : 40000,
"forceExtendableShardSpecs" : true
}
}
}
There are 2 ways using which you can append/update the data to an existing segment.
Reindexing and Delta Ingestion
You need to reindex your data every time new data comes in a particular segment.(In your case its day) For the reindexing you need to give all the files having data for that day.
For Delta Ingestion you need to use inputSpec type="multi"
You can refer the documentation link for more details - http://druid.io/docs/latest/ingestion/update-existing-data.html
I have the following document, I need to search for multiple items from the embedded collection"items".
Here's an example of a single SKU
db.sku.findOne()
{
"_id" : NumberLong(1192),
"description" : "Uploaded via CSV",
"items" : [
{
"_id" : NumberLong(2),
"category" : DBRef("category", NumberLong(1)),
"description" : "840 tag visual",
"name" : "840 Visual Mini Round",
"version" : NumberLong(0)
},
{
"_id" : NumberLong(7),
"category" : DBRef("category", NumberLong(2)),
"description" : "Maxi",
"name" : "Maxi",
"version" : NumberLong(0)
},
{
"_id" : NumberLong(11),
"category" : DBRef("category", NumberLong(3)),
"description" : "Button",
"name" : "Button",
"version" : NumberLong(0)
},
{
"_id" : NumberLong(16),
"category" : DBRef("category", NumberLong(4)),
"customizationFields" : [
{
"_class" : "CustomizationField",
"_id" : NumberLong(1),
"displayText" : "Custom Print 1",
"fieldName" : "customPrint1",
"listOrder" : 1,
"maxInputLength" : 12,
"required" : false,
"version" : NumberLong(0)
},
{
"_class" : "CustomizationField",
"_id" : NumberLong(2),
"displayText" : "Custom Print 2",
"fieldName" : "customPrint2",
"listOrder" : 2,
"maxInputLength" : 17,
"required" : false,
"version" : NumberLong(0)
}
],
"description" : "2 custom lines of farm print",
"name" : "Custom 2",
"version" : NumberLong(2)
},
{
"_id" : NumberLong(20),
"category" : DBRef("category", NumberLong(5)),
"description" : "Color Red",
"name" : "Red",
"version" : NumberLong(0)
}
],
"skuCode" : "NF-USDA-XC2/SM-BC-R",
"version" : 0,
"webCowOptions" : "840miniwithcust2"
}
There are repeat items.id throughout the embedded collection. Each Sku is made up of multiple items, all combinations are unique, but one item will be part of many Skus.
I'm struggling with the query structure to get what I'm looking for.
Here are a few things I have tried:
db.sku.find({'items._id':2},{'items._id':7})
That one only returns items with the id of 7
db.sku.find({items:{$all:[{_id:5}]}})
That one doesn't return anything, but it came up when looking for solutions. I found about it in the MongoDB manual
Here's an example of a expected result:
sku:{ "_id" : NumberLong(1013),
"items" : [ { "_id" : NumberLong(5) },
{ "_id" : NumberLong(7) },
{ "_id" : NumberLong(12) },
{ "_id" : NumberLong(16) },
{ "_id" :NumberLong(2) } ] },
sku:
{ "_id" : NumberLong(1014),
"items" : [ { "_id" : NumberLong(5) },
{ "_id" : NumberLong(7) },
{ "_id" : NumberLong(2) },
{ "_id" : NumberLong(16) },
{ "_id" :NumberLong(24) } ] },
sku:
{ "_id" : NumberLong(1015),
"items" : [ { "_id" : NumberLong(5) },
{ "_id" : NumberLong(7) },
{ "_id" : NumberLong(12) },
{ "_id" : NumberLong(2) },
{ "_id" :NumberLong(5) } ] }
Each Sku that comes back has both a item of id:7, and id:2, with any other items they have.
To further clarify, my purpose is to determine how many remaining combinations exist after entering the first couple of items.
Basically a customer will start specifying items, and we'll weed it down to the remaining valid combinations. So Sku.items[0].id=5 can only be combined with items[1].id=7 or items[1].id=10 …. Then items[1].id=7 can only be combined with items[2].id=20 … and so forth
The goal was to simplify my rules for purchase, and drive it all from the Sku codes. I don't know if I dug a deeper hole instead.
Thank you,
On the part of extracting the sku with item IDs 2 and 7, when I recall correctly, you have to use $elemMatch:
db.sku.find({'items' :{ '$all' :[{ '$elemMatch':{ '_id' : 2 }},{'$elemMatch': { '_id' : 7 }}]}} )
which selects all sku where there is each an item with _id 2 and 7.
You can use aggregation pipelines
db.sku.aggregate([
{"$unwind": "$sku.items"},
{"$group": {"_id": "$_id", "items": {"$addToSet":{"_id": "$items._id"}}}},
{"$match": {"items._id": {$all:[2,7]}}}
])
I'm still new to MongoDB and non-relational databases in general so if the answer to my question is "Dude, you're thinking about your data in the wrong way", please let me know.
That being said, here's what I'm after. I have the following data:
{
"_id" : ObjectId("549b26b370b452eeb6acecd3"),
"user" : "54530e03c575dc86d61d22f8",
"workHistory" : [
{
"description" : "",
"endDate" : null,
"name" : "My Company",
"skills" : [
{
"_id" : ObjectId("549b29c970b452eeb6acecd4")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecd5")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecda")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecdb")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecdd")
},
{
"_id" : ObjectId("549b29c970b452eeb6acece9")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecea")
},
{
"_id" : ObjectId("549b995b70b452eeb6acecf9")
},
{
"_id" : ObjectId("549b999470b452eeb6acecfa")
},
{
"_id" : ObjectId("549b9ab670b452eeb6acecfb")
}
],
"startDate" : "2013-10-01"
},
{
"description" : "",
"endDate" : "2013-10-01",
"name" : "Another Company",
"skills" : [
{
"_id" : ObjectId("549b29c970b452eeb6acecd4")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecd5")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecdb")
},
{
"_id" : ObjectId("549b29c970b452eeb6acecdd")
},
{
"_id" : ObjectId("549b29c970b452eeb6acece1")
},
{
"_id" : ObjectId("549b29c970b452eeb6acece9")
},
{
"_id" : ObjectId("549b995b70b452eeb6acecf9")
}
],
"startDate" : "2012-04-01"
},
.....
What I am trying to do is resolve that data I can list the actual skill names under "skills" instead of just object ID's. For reference here's the skills collection:
/* 0 */
{
"_id" : ObjectId("549b29c970b452eeb6acecd4"),
"name" : "CSS",
"description" : ""
}
/* 1 */
{
"_id" : ObjectId("549b29c970b452eeb6acecd5"),
"name" : "HTML5",
"description" : ""
}
/* 2 */
{
"_id" : ObjectId("549b29c970b452eeb6acecd6"),
"name" : "Ruby",
"description" : ""
}
/* 3 */
{
"_id" : ObjectId("549b29c970b452eeb6acecd7"),
"name" : "Ruby on Rails",
"description" : ""
}
I've tried looking into using MapReduce to do some kind of join, but I can barely wrap my head around it and all the examples I'm finding don't have data this complex.
What's the best approach for getting my desired result? Side note: I'm not 100% sure that simply embedding the skills in the "skills" property is the right approach. I'm trying to keep them separate because the skills may be listed in other areas of my application separate from work history.