mongodb select all fields which is not null or empty string - mongodb

I have mongodb document which has data like this:
{
"_id": "ObjectId(\"5e09db41d4ccee7f500f7326\")",
"time": "ISODate(\"2019-12-30T09:10:53Z\")",
"syslog_fac": 23,
"syslog_sever": 6,
"syslog_tag": "",
"procid": "",
"pid": "-",
"status": "Allowed",
"logtype": "Firewall Logs",
"date": "2019-12-30",
"Module": "01UM",
"Desc": "Theuserloggedout.(UserName",
"Severity_Level": "6",
"Vsys": "public",
"SourceIP": "10.10.0.57",
"ParentGroup": "/netgrup.com.tr",
"LogonTime": "2019/12/3014:07:20",
"LogoutTime": "2019/12/3014:10:53",
"ObversePackets": "0",
"ObverseBytes": "0",
"ReversePackets": "0",
"ReverseBytes": "0",
"LogoutReason": "AnIPaddressconflictoccurred",
"VSys": "",
"PolicyName": "",
"Country": "",
"DestinationIP": "",
"SourcePort": "",
"DestinationPort": "",
"SourceZone": "",
"DestinationZone": "",
"User": "",
"Protocol": "",
"ApplicationName": "",
"Profile": "",
"Type": "",
"Category": "",
"SubCategory": "",
"Page": "",
"Host": "",
"Referer": "",
"Item": "",
"Action": "",
"level": "",
"SourceNatIP": "",
"SourceNatPort": "",
"BeginTime": "",
"EndTime": "",
"SendPkts": 0,
"SendBytes": 0,
"RcvPkts": 0,
"RcvBytes": 0,
"SourceVpnID": "",
"DestinationVpnID": "",
"CloseReason": "",
"Task": "",
"Ip": "",
"VpnName": "",
"AuthenticationMethod": "",
"Command": "",
"user-name": "",
"attack-type": "",
"interface": "",
"packet-count": "",
"rate-number": "",
"file-name": "",
"file-type": "",
"direction": "",
"detect-type": "",
"virus-name": "",
"signature-id": "",
"event-number": "",
"target": "",
"role": "",
"user": ""
}
As you see there are a lot of fields which have an empty string or 0 , so what I want to do is to select only columns that have value except 0 or empty string.
this is my code that I am using to execute query from laravel but return nothings , when I apply that code in mongo directly I get correct data , and return exactly what I want , but return nothings from laravel ,
$data = Logss::raw(function($collection)use ($_id)
{
return $collection->aggregate([
[ '$match' => ['_id'=>'ObjectId(5e09dc63715d622be622c639)']],
[
'$replaceRoot'=> [
'newRoot'=>[
'$arrayToObject'=>[
'$filter'=> [
'input'=> [ '$objectToArray'=>'$$ROOT' ],
'cond'=> [
'$and'=> [
[ '$ne'=> [ '$$this.v', "" ] ],
[ '$ne'=>[ '$$this.v', 0 ] ]
]
]
]
]
]
]
]
]);
});

You can run below query:
db.collection.aggregate([
{ $match: { _id: ObjectId(...) } },
{
$replaceRoot: {
newRoot: {
$arrayToObject: {
$filter: {
input: { $objectToArray: "$$ROOT" },
cond: {
$and: [
{ $ne: [ "$$this.v", "" ] },
{ $ne: [ "$$this.v", 0 ] }
]
}
}
}
}
}
}
])
The idea is pretty simple: you start with $objectToArray to get all the fields as an array. Then you can run $filter to remove all zeros and whitespaces and convert that array back to an object $arrayToObject promoting it to the root level ($replaceRoot).
Mongo Playground

Related

$lookup aggregation for nested array object

I have category collection which has subcategory embedded object. I need to write aggregation script to first lookup from this collection to storelisting collection. Then add the storelisting fields to the corresponding subcategories.
category
{
"_id": "",
"categoryName": "",
"subCategories": [
{
"subCategoryId": "",
"subCategoryName": "",
"storeListingIds": [
"1",
"2"
]
},
{
"subCategoryId": "",
"subCategoryName": "",
"storeListingIds": [
"3","4","5"
]
}
],
"order": 2,
"createdAt": {
"$date": "2020-12-01T22:26:11.669Z"
},
"updatedAt": {
"$date": "2021-04-27T17:17:25.442Z"
},
"_class": ""
}
storelisting
{
"_id": "1",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "2",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "3",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "4",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "5",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
}
I need to lookup form above collection to storeListing collection. I want all the fields of storeListing also.
Result should be look like this:
{
"_id": "",
"categoryName": "",
"subCategories": [
{
"subCategoryId": "",
"subCategoryName": "",
"storeListingIds": [
"1",
"2"
],
"storeListings":[
{
"_id": "1",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "2",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
}
]
},
{
"subCategoryId": "",
"subCategoryName": "",
"storeListingIds": [
"3","4","5"
],
"storeListings":[
{
"_id": "3",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "4",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
},
{
"_id": "5",
"storeListingName": "",
"storeListingUrl": "",
"catalogueIds": [
""
],
"_class": ""
}
]
}
],
"order": 2,
"createdAt": {
"$date": "2020-12-01T22:26:11.669Z"
},
"updatedAt": {
"$date": "2021-04-27T17:17:25.442Z"
},
"_class": ""
}
Explanation
We perform $lookup for nested storeListingId and store the result in the tmp field
We apply filtering and merge subCategories object with storeListings values
Note: The reason why we use $mergeObjects is to avoid naming every field in subCategories
$map: {
input: "$subCategories",
as: "subCat",
in: {
subCategoryId:"$$subCat.subCategoryId",
subCategoryName:"$$subCat.subCategoryName",
storeListingIds: "$$subCat.storeListingIds",
storeListening: {
$filter: {
input: "$tmp",
cond: {
$in: [ "$$this._id", "$$subCat.storeListingIds" ]
}
}
}
}
}
Try this one:
db.category.aggregate([
{
$lookup: {
from: "storelisting",
localField: "subCategories.storeListingIds",
foreignField: "_id",
as: "tmp"
}
},
{
$addFields: {
tmp: "$$REMOVE",
subCategories: {
$map: {
input: "$subCategories",
as: "subCat",
in: {
"$mergeObjects": [
"$$subCat",
{
storeListings: {
$filter: {
input: "$tmp",
cond: {
$in: [ "$$this._id", "$$subCat.storeListingIds" ]
}
}
}
}
]
}
}
}
}
}
])
MongoPlayground

Clean docs in collection by comparing them with reference

I have a lot of complex JSON objects which are placed in the collection. For example:
{
"name": "Mike",
"price": "444",
"distance": 881,
"someFiend": 123,
"lots": [
{
"aa": "111",
"bb": "222"
},
{
"xx": "000"
}
],
"apps": [
{
"app": 1
},
{
"app": 2
}
]
}
I only want to project only those fields which are present in the following reference document:
{
"name": "",
"price": "",
"lots": [
{
"aa": "",
"bb": ""
}
]
}
Expected output:
{
"name": "Mike",
"price": "444",
"lots": [
{
"aa": "111",
"bb": "222"
}
]
}
Is there any way to iterate all documents in the collection and then filter out fields that are not present in the reference doc?

Extract ? number of JSON array objects into rows

So I am trying to pivot some large Json arrays into a row by row format (postgres db). So imagine
{“searchResults”:[{“id”:“89705”,“score”:42,“title”:“Foo.”,“properties”:{…
Currently the most results we have in an array is about 300~ id's and explicitly saying;
Data::Json->'searchResults'->0->'tags'->0->>'label' as "Tag - Result 1",
...
Data::Json->'searchResults'->300->'tags'->0->>'label' as "Tag - Result
1",
Ideal Output
Array, ID , Score, Title
----
1 89705, 42, foo
1 89706, 34, bar
2 90003, 54, thing
2 98594, 53, that
(so 1,2 represent different rows in the initial table that both contain ??? amounts of objects the JSON data array)
Expanded JSON
{
"searchResults": [
{
"id": "897096",
"score": 42,
"title": "foo.",
"properties": {
"#type": "blah",
},
"publishedDate": "2018-06-30T10:20:20.555040Z",
"comments": [
{
"content": "",
"owner": {
"firstName": "",
"id": 0,
"lastName": ""
},
"id": 0,
"contentType": "",
"documentPk": 0,
"workflowStep": 0,
"order": 0
}
],
"tags": [
{
"tag": 783,
"label": "NO",
"iconClass": "",
"subGroup": "",
"exclude": false
},
{
"tag": 786,
"label": "Different name",
"iconClass": "",
"subGroup": "",
"exclude": false
}
],
"reviewTags": [
{
"tag": 2,
"label": "Accept",
"iconClass": "",
"subGroup": "",
"exclude": false
}
],
"original": {
..."names": [
{
"full_name": "This name"
}
],
"Entry Type": "Organization",
"Last Updated": "2018/05/03",
"Hit Category": "N/A",
"Aliases": [
"Olaj",
"hbhbhb"
]
},
"snippet": "",
"url": "",
"source": "_"
},
{
"id": "879057",
"score": 36,
"title": "name of company",
"properties": {
"#type": "",
"category": "SOE",
"type": "Organization",
"country": "Korea, Republic Of",
"subcategory": ""
},
"publishedDate": "2018-05-31T10:20:20.559714Z",
"comments": [
{
"content": "",
"owner": {
"firstName": "",
"id": 0,
"lastName": ""
},
"id": 0,
"contentType": "",
"documentPk": 0,
"workflowStep": 0,
"order": 0
}
],
"tags": [
{
"tag": 783,
"label": "NO",
"iconClass": "",
"subGroup": "",
"exclude": false
},
{
"tag": 786,
"label": "Different name",
"iconClass": "",
"subGroup": "",
"exclude": false
}
Any advise on what my options are here ?
Thanks #a_horse_with_no_name this worked perfectly.
it's unclear to me how the JSON continues in the array, but it seems you are looking for something like this:
with testdata(props) as (
values (
'{"searchResults":
[
{"id":"89705","score":42,"title":"Foo"},
{"id":"89706","score":34, "title":"bar"}
]}'::jsonb
)
)
select x.idx,
x.val ->> 'id' as id,
x.val ->> 'score' as score,
x.val ->> 'title' as title
from testdata, jsonb_array_elements(props -> 'searchResults') with ordinality as x (val, idx);
returns
idx | id | score | title
----+-------+-------+------
1 | 89705 | 42 | Foo
2 | 89706 | 34 | bar

Not picking Index - when querying Object -> array -> Object in N1QL

Index not picking when using N1QL to match all the customers belonging to a country
Note : the bucket has around 10 lack records, the document is fetching after 50 seconds,
query
select * from `user-prof` WHERE ANY item IN devices.location SATISFIES item.country IN ["US"]
index
CREATE INDEX id_userProfile ON `user-prof` ( ALL ARRAY v.country FOR v IN devices.location END )
Document
[
{
"customers": {
"devices": {
"user": "u1",
"custid": "14CE5534CCE",
"token": "4D5BE85896833148D696A1397C",
"guest": {
"lastActive": null,
"searches": [
]
},
"latt": "6655059908",
"locale": "en-US",
"location": [
{
"city": "Afc",
"country": "IN"
},
{
"city": "Newyork",
"country": "US"
},
{
"city": null,
"country": null
}
],
"long": "4.21787927806",
"notify": {
"Stats": false
},
"tier": null,
"tmz": "EU",
"version": "0.1"
}
}
},
{
"customers": {
"devices": {
"user": "u2",
"custid": "64CE5534CC1E",
"token": "6D5BE85896833148D696A1397C",
"guest": {
"lastActive": null,
"searches": [
]
},
"latt": "6655059908",
"locale": "en-US",
"location": [
{
"city": "Texas",
"country": "US"
},
{
"city": null,
"country": null
}
],
"long": "4.21787927806",
"notify": {
"Stats": false
},
"tier": null,
"tmz": "EU",
"version": "0.1"
}
}
}
You are using Pre 4.6.2 The variable in the ANY clause needs to be same as the variable in the CREATE INDEX (i.e item ,v )
https://developer.couchbase.com/documentation/server/current/n1ql/n1ql-language-reference/indexing-arrays.html

Json parsing wrong in IOS

Hi i have some problem parsing the json in IOS. This is the json data.
{
"app_info": [
{
"app_name": "haka",
"sync_protocol_version": "1"
}
],
"updates": [
{
"timestamp": "Sat Apr 21 13:04:08 IST 2012",
"people": [
{
"personal_info": [
{
"first_name": "phlox",
"last_name": "",
"employee_id": "010",
"gender": "-",
"marital_status": "-",
"nationality": "Denobulan",
"dob": "re-23",
"photo": "http://c.cc/users/010/profile/image"
}
],
"contact_details": [
{
"address": [
{
"street": "#1, this way",
"city": "tank",
"state": "sick bay",
"zip": "0978",
"country": "Enterprise"
}
],
"telephone": [
{
"work": "010",
"mobile": "010",
"home": "010"
}
],
"email": [
{
"work": "phlox#nx-10.ent",
"personal": ""
}
]
}
],
"emergency": [
{
"contact": [
{
"name": "-",
"relationship": "",
"telephone": [
{
"work": "",
"home": "",
"mobile": ""
}
]
}
],
"blood_group": ""
}
],
"categorization": [
{
"designation": "",
"department": "",
"location": "",
"joining_date": ""
}
]
},
{
"personal_info": [
{
"first_name": "",
"last_name": "",
"employee_id": "",
"gender": "",
"marital_status": "",
"nationality": "",
"dob": "",
"photo": ""
}
],
"contact_details": [
{
"address": [
{
"street": "",
"city": "",
"state": "",
"zip": "",
"country": ""
}
],
"telephone": [
{
"work": "",
"mobile": "",
"home": ""
}
],
"email": [
{
"work": "",
"personal": ""
}
]
}
],
"emergency": [
{
"contact": [
{
"name": "",
"relationship": "",
"telephone": [
{
"work": "",
"home": "",
"mobile": ""
}
]
}
],
"blood_group": ""
}
],
"categorization": [
{
"designation": "",
"department": "",
"location": "",
"joining_date": ""
}
]
}
],
"messages": [
{
"sender": "Archer<admin#nx-10.ent>",
"sender_role": "admin",
"message_type": "broadcast",
"message": "parking space up for grabs",
"message_recipients": "all",
"reply_permitted": "0"
}
],
"events": [
{
"creator": "Travis<ensign#nx-01.ent>",
"event_title": "",
"event_description": "",
"event_time_start": "",
"event_time_end": "",
"location": "",
"invitees": [
{
"id": "020",
"acceptance": "1"
}
]
}
],
"settings": [
{
"sync_frequency": "0"
}
]
}
]}
This is a valid json format. I have checked it with http://jsonlint.com/ and using http://braincast.nl/samples/jsoneditor/
see the structure of json value.According to structure the people tag should return count of 2, as it has 2 objects, but it is returning only 1 while parsing.
I am completely out of options. Please help guys.
This is the code i am using for parsing
NSString *textPAth = [[NSBundle mainBundle] pathForResource:#"sync" ofType:#"json"];
NSError *error;
NSString *content = [NSString stringWithContentsOfFile:textPAth encoding:NSUTF8StringEncoding error:&error];
NSArray *jsonArray = [[content JSONValue] retain];
NSLog(#"JSON ARRAY %# AND COUNT %d",[[jsonArray valueForKey:#"updates"] valueForKey:#"people"],[[[jsonArray valueForKey:#"updates"] valueForKey:#"people"] count]);
I'm not sure why every dictionary is wrapped with an array, when there is only 1 entry (unnecessary), but if you want to parse it you'd want something like this:
NSDictionary *jsonDict = [[content JSONValue] retain];
NSArray *updatesArray = [jsonDict objectForKey:#"updates"];
NSDictionary *updatesDict = [updatesArray objectAtIndex:0];
NSArray *peopleArray = [updatesDict objectForKey:#"people"];
NSLog(#"People count: %i", [peopleArray count]);
for (NSDictionary *peopleDict in peopleArray) {
//do something
}
As a side note, if this is your JSON, you'll want to remove the [arrays] for entries that are not actually arrays. While you can still parse it, it means you'll have to read the array first then get the dictionary from the objectAtIndex:0, which is totally inefficient and pointless.
Could it be that every one of your objects is wrapped in an array and giving unexpected results?