Let's say I have a column with this jsonb data :
{
"indicators": [
{
"year": 2019,
"indicatorsByYear": [
{
"value": 3120,
"code": "Nb_h"
},
{
"value": 18,
"code": "S_ppa"
},
{
"value": 95,
"code": "T_occ"
}
]
},
{
"year": 2020,
"indicatorsByYear": [
{
"value": 300,
"code": "Nb_h"
},
{
"value": 18,
"code": "S_ppa"
},
{
"value": 55,
"code": "T_occ"
}
]
}
],
"dataProvidedByUser": false
}
The idea is to migrate this column to a simplified object like this :
{
"indicatorsByYear": {
"2019": [
{
"value": 3120,
"code": "Nb_h"
},
{
"value": 18,
"code": "S_ppa"
},
{
"value": 95,
"code": "T_occ"
}
],
"2020": [
{
"value": 300,
"code": "Nb_h"
},
{
"value": 18,
"code": "S_ppa"
},
{
"value": 55,
"code": "T_occ"
}
]
},
"dataProvidedByUser": false
}
How can I transform the indicators array to map object with year as key and indicatorsByYear as value.
For info, the maximum number of years that I can have is 11 years (from 2010 to 2020), some columns have all the years others only some.
My attempts with something like that without success
update site
SET data = data
|| jsonb_build_object('indicatorsByYear',
jsonb_build_object(
data -> 'indicators' ->> 'year',
data -> 'indicators' ->> 'indicatorsByYear'
))
Any help would be very much appreciated! Thanks in advance.
data -> 'indicators' is an array, whose elements you need to consider individually and then aggregate back together into an object. You can use jsonb_array_elements and jsonb_object_agg respectively for this.
Also, you'll want to remove the old indicators key from the data column.
UPDATE site
SET data = jsonb_set(
data - 'indicators',
'{indicatorsByYear}',
(
SELECT jsonb_object_agg(el ->> 'year', el -> 'indicatorsByYear')
FROM jsonb_array_elements(data -> 'indicators') el
)
);
Related
The problem I am facing is that I want to develop an autocomplete search bar using Mean Stack like the one in this site, but when I type, for example, 'ag' it's not returning the right location that should be 'Aguascalientes'.
I have two different search indexes set up and a different query for each.
First Index:
{
"mappings": {
"dynamic": false,
"fields": {
"name": {
"foldDiacritics": false,
"maxGrams": 7,
"minGrams": 3,
"tokenization": "edgeGram",
"type": "autocomplete"
},
"searchName": {
"foldDiacritics": false,
"maxGrams": 7,
"minGrams": 3,
"tokenization": "edgeGram",
"type": "autocomplete"
}
}
}
}
First Query:
[
{
$search: {
index: "autocomplete2",
compound: {
must: [
{
text: {
query: search,
path: "searchName",
fuzzy: {
maxEdits: 2,
},
},
},
],
},
},
},
{
$limit: 10,
},
]
The first ones are not returning any document at all. But the second example is:
{
"mappings": {
"dynamic": false,
"fields": {
"name": {
"analyzer": "lucene.standard",
"type": "string"
},
"searchName": {
"analyzer": "lucene.standard",
"type": "string"
}
}
}
}
Query:
[
{
$search: {
index: 'default',
compound: {
must: [
{
text: {
query: search,
path: 'name',
fuzzy: {
maxEdits: 1,
},
},
},
{
text: {
query: search,
path: 'searchName',
fuzzy: {
maxEdits: 1,
},
},
},
],
},
},
},
{
$limit: 5,
},
]
The second example is only returning documents if the search term 'aguascalient' but is not returning any document if the search term is shorter like the site. Maybe it has something to do with the fuzzy edits but if I set it up to greater than 2 I get an error.
Also the order is not right, it returns first the CITY and second the STATE but I need the STATE first because the search term is more similar than the city. Let me explain, search field for STATE is only 'Aguascalientes' but search field cities is 'Aguascalientes Aguascalientes' so I don't know why is not working properly. Maybe in that case I should give weights accordingly but I'm not sure if it's the right approach to solve this.
My data structure:
{
"_id": "638d0ffc34ad076c6bd12cb6",
"depth": 2,
"label": "CITY",
"location_id": "V1-C-247",
"name": "Aguascalientes",
"parent": "Aguascalientes",
"fullName": "Aguascalientes, Aguascalientes",
"parentId": "V1-B-61",
"searchName": "Aguascalientes Aguascalientes",
}
{
"_id": "638d0ffc34ad076c6bd12cb6",
"depth": 1,
"label": "STATE",
"location_id": "V1-C-248",
"name": "Aguascalientes",
"parent": null,
"fullName": "Aguascalientes",
"parentId": null,
"searchName": "Aguascalientes",
}
For the first index + query setup:
First, you are indexing the name field but are not searching on it. I will remove it from the code snippets for readability, but you can add it back to your index definition if you find you need to search on it.
There are two problems with the this index + query setup if you want to return results with a query for "ag". You have searchName defined as a field mapping of type autocomplete, but you also need to use the autocomplete operator in your query:
[
{
$search: {
index: "autocomplete2",
compound: {
must: [
{
autocomplete: {
query: search,
path: "searchName",
},
},
],
},
},
},
{
$limit: 10,
},
]
Second, in your index definition field mapping for searchName, you have minGram set to 3 and maxGram set to 7. Based on the documentation for the autocomplete field mapping, this means that your data will be tokenized into sequences of character lengths between 3 to 7, using the selected tokenization strategy. Since you have selected edgeGram, the tokens generated by the text "Aguascalientes" will be tokenized starting from the left edge, resulting in tokens "agu", "agua", "aguas", "aguasc", "aguasca". Since the search term "ag" does not match any of the tokens, nothing is returned. So, you must change the minGram to 2 to get the token "ag":
{
"mappings": {
"dynamic": false,
"fields": {
"searchName": {
"foldDiacritics": false,
"maxGrams": 7,
"minGrams": 2,
"tokenization": "edgeGram",
"type": "autocomplete"
}
}
}
}
Finally, if you want the document with an exact match to return over a partial match, ie. "Aguascalientes" should return before "Aguascalientes Aguascalientes", you need to implement exact matching. Here is a MongoDB blog post outlining a few options.
One option that I tried: In the index, use a keyword analyzer on the "searchName" field typed as a string data type. In the query, use the text operator nested in a should clause so that exact matches will return higher than other results.
Index:
{
"mappings": {
"dynamic": false,
"fields": {
"searchName": [
{
"foldDiacritics": false,
"maxGrams": 7,
"type": "autocomplete"
},
{
"analyzer": "lucene.keyword",
"searchAnalyzer": "lucene.keyword",
"type": "string"
}
]
}
}
}
Query:
[
{
$search: {
compound: {
must: [
{
autocomplete: {
query: search,
path: "searchName"
}
}
],
should:[
{
text: {
query: search,
path: "searchName"
}
}
],
},
},
},
]
I have a map, with multiple layers, some of the layers have an events field which may contain 0 or more event listings inside of it.
Some of events are nested deeper into the map, while others are closer to the top layer.
Here is what the graphql query result looks like:
{
"data": {
"users": [
{
"id": 16,
"friends": [
{
"senderId": 16,
"receiverId": 17,
"userByReceiverid": {
"id": 17,
"events": [],
"friends": [
{
"receiverId": 14,
"userByReceiverid": {
"id": 14,
"events": [
{
"id": 3,
"photoUrl": "none",
"name": "hello",
"date": "1982-06-27",
"startTime": "01:00:00+00",
"endTime": "02:00:00+00",
"fee": "$2.00",
"maxNumber": 10,
"ageRestriction": "none",
"about": "amazing",
"allowShare": false,
"private": false,
"timestamp": "2021-06-26T17:57:13.224383+00:00",
"senderId": 14
}
]
}
},
{
"receiverId": 20,
"userByReceiverid": {
"id": 20,
"events": []
}
}
]
}
}
],
"friendsByReceiverid": [
{
"senderId": 20,
"receiverId": 16,
"user": {
"id": 20,
"events": [],
"friendsByReceiverid": [
{
"receiverId": 20,
"user": {
"id": 14,
"events": [
{
"id": 3,
"photoUrl": "none",
"name": "hello",
"date": "1982-06-27",
"startTime": "01:00:00+00",
"endTime": "02:00:00+00",
"fee": "$2.00",
"maxNumber": 10,
"ageRestriction": "none",
"about": "amazing",
"allowShare": false,
"private": false,
"timestamp": "2021-06-26T17:57:13.224383+00:00",
"senderId": 14
}
]
}
},
{
"receiverId": 20,
"user": {
"id": 17,
"events": []
}
}
]
}
}
]
}
]
}
}
I want to search the Map and make a list that contains all the events that are unique, so no duplicates if possible.
How would I go about pulling out only the events field from a map at any given point?
Essentially what you need is a function (visitObject) that goes through each key of the a Map and recursively calls itself till all properties have been checked.
The function should also check for the possible types it can operate on and handle the data so you can call the visitObject function with the right arguments.
I believe the right data structure here would be a Set. In dart as Set is defined as A collection of objects in which each object can occur only once.
However Set a set falls back to the dart method of equality which is the == operator, so you would either need to use a calls for Events or use a package like BuiltValue which will generate the == operator for your value types.
Here is an example that can help you get started, I have left equality checking for you to determine what best and how you want to apply it.
final Set<Map<String, dynamic>> items = {};
void main() {
visitObject(data);
}
void visitObject(object) {
if (object is List) {
for(dynamic item in object) {
visitObject(item);
}
} else if (object is Map) {
if (object.containsKey('events')) {
var _events = object['events'];
if (_events is List && _events.isNotEmpty) {
_events.forEach((e) {
if (!items.contains(e)) {
items.add(e);
}
});
}
}
for (dynamic item in object.values) {
visitObject(item);
}
}
}
Additional Reading for equality:
==Operator
hashCode
BuiltValue
I am basically from JS background and now trying my hands on flutter framework.
I am stuck in formatting my data I received from API.
Data Received an array of objects
[
{
"id": 1,
"name": "name1",
"type": "Both",
"count": 4
},
{
"id": 2,
"name": "name2",
"type": "Both",
"count": 6
},
{
"id": 3,
"name": "name1",
"type": "Both",
"count": 2
},
{
"id": 4,
"name": "name3",
"type": "Both",
"count": 8
},
......
]
My Requirement is I will have to group the data based on the name
{
name1: [
{
"id": 1, "name":"name1", "type": "Both", "count": 4
},
{
"id": 3, "name":"name1", "type": "Both", "count": 2
}
],
name2: [
{
"id": 2, "name":"name2", "type": "Both", "count": 6
}
],
.....
}
In dart, I have managed to do that grouping using groupBy from collections.dart package.
The problem is I cannot loop through the grouped data. I need to access the count for some manipulation for each grouped name.
Below is the code snipped which I tried
Map data;
List partySnapData;
Map newMap;
Future _getTopParties() async {
final response =await http.get(API_URL + '/getPartySnapShot');
data =json.decode(response.body);
partySnapData = data['resultObj'];
setState(() {
newMap = groupBy(partySnapData, (obj) => obj['party_name']);
for(var v in newMap.values) {
print(v);
}
});
}
The print(v) actually gave me the value mapped for the map E.G
[
{ "id": 1, "name":"name1", "type": "Both", "count": 4 },
{ "id": 3, "name":"name1", "type": "Both", "count": 2 }
],
.....,
......
Now, how do I loop or iterate through the array, here the v, so that i can access the elements inside?
I found a solution to this. I am posting that, incase it helps someone.
I wanted to loop through the v to generate objects.
newMap = groupBy(partySnapData, (obj) => obj['party_name']);
for(var v in newMap.values) {
print(v);
//below is the solution
v.asMap().forEach((i, value) {
print('index=$i, value=$value');
}
}
Ignoring your specific case but answering the question in case others come across this question like I did while searching for How to iterate through map values in dart / flutter.
There is a simpler and a better approach.
Here's the best approach (because this way you can break the for loop after finding the right entry and doing what you had to do to avoid unnecessary iterations).
for (MapEntry<type, type> item in myMap.entries) {
//Here each item has a key and a value proterties.
//Don't forget to use break; to end the loop when you are done and avoid unnecessary iterations.
}
And here is the easier:
myMap.forEach((key, value) {
//Here you have key and value for each item in your map but you can't break the loop
//to avoid unnecessary iterations.
//This approach is only useful when you need to perform changes in all Map items.
});
UPDATED
List<Map> list = [
{
"id": 1,
"name": "name1",
"type": "Both",
"count": 4
},
{
"id": 2,
"name": "name2",
"type": "Both",
"count": 6
},
{
"id": 3,
"name": "name1",
"type": "Both",
"count": 2
},
{
"id": 4,
"name": "name3",
"type": "Both",
"count": 8
}];
Map sorted={};
#override
void initState(){
super.initState();
for(Map m in list){
if(sorted[m['name'].toString()]==null)
sorted[m['name'].toString()]=[];
sorted[m['name'].toString()].add(m);
}
}
#override
Widget build(BuildContext context) {
return Scaffold(
body:Container(
padding:EdgeInsets.all(20),
color:Colors.white,
alignment:Alignment.center,
child:Column(
mainAxisSize:MainAxisSize.min,
children: <Widget>[
Text('Original\n $list',
style:TextStyle(color:Colors.black)),
SizedBox(height:15),
Text('Sorted\n $sorted',
style:TextStyle(color:Colors.black)),
//EDIT
SizedBox(height:15),
ListView(
shrinkWrap:true,
children: sorted.entries.map<Widget>((value){
return Column(
mainAxisSize:MainAxisSize.min,
crossAxisAlignment:CrossAxisAlignment.start,
children:[
Text(value.key.toString(),style:TextStyle(color:Colors.black,
fontWeight:FontWeight.bold)),
Column(
mainAxisSize:MainAxisSize.min,
children:value.value.map<Widget>((subValue){
return Text(subValue['count'].toString(),style:TextStyle(color:Colors.black));
}).toList(),
),
]
);
}).toList(),
)
],
)
)
);
}
I've crereated an index on elasticsearch same as bellow:
"settings" : {
"number_of_shards": 1,
"number_of_replicas": 0,
"analysis": {
"filter": {
"trigrams_filter": {
"type": "ngram",
"min_gram": 3,
"max_gram": 3
}
},
"analyzer": {
"trigrams": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"trigrams_filter"
]
}
}
}
},
"mappings": {
"issue": {
"properties": {
"description": {
"type": "string",
"analyzer": "trigrams"
}
}
}
}
My test items are bellow:
"alici onay verdi basarili satisiniz gerceklesti diyor ama hesabima para transferi gerceklesmemis"
"otomatik onay işlemi gecikmiş"
"************* nolu iade islemi urun kargoya verilmedi zamaninda iade islemlerinde urun erorr hata veriyor"
I've test this index with bellow query:
GET issue/_search
{
"query": {
"match": {
"description":{
"query": "otomatik onay istemi zamaninda gerceklesmemis"
}
}
}
}
And result:
{
....
"hits": {
....
"max_score": 2.3507352,
"hits": [
{
....
"_score": 2.3507352,
"_source": {
"issue_id": "*******",
"description": "alici onay verdi basarili satisiniz gerceklesti diyor ama hesabima para transferi gerceklesmemis"
}
}
]
}
}
But same data on postgresql with bellow SQL response another result:
SELECT
public.tbl_issue_descriptions_big.description,
similarity(description, 'otomatik onay islemi zamaninda gerceklesmemis') AS sml
FROM
public.tbl_issue_descriptions_big
WHERE
description %'otomatik onay islemi zamaninda gerceklesmemis'
ORDER BY
sml DESC
LIMIT 10
Result is:
description | sml
======================================================|======
otomatik onay islemi gecikmis |0,351852
Why is this difference caused?
I dont know enough about postgres to give a qualified answer there (as this also depends on the documents that are indexed and if they scoring formulas are exactly the same, which I doubt), but Elasticsearch has an explain API and an explain parameter in the search, that help you to find out why a certain document was scored this way.
I'm trying to post data to a mapquest table and I've done everything according to their API
Here is my json:
{
"tableName": "mqap.jZ3uoAVablablabla_reported_points",
"append": true,
"returnResults": true,
"rows": [
[
{
"mqap_geography": "POINT (-123.3141509 48.4647675)",
"date": "2016-03-13T02:19:09.674Z",
"email": "anna#gmail.com",
"latitude": "48.47542496514236",
"longitude": "-123.37663650512695",
"message": "ss",
"user": "a"
}
]
]
}
Here is a screenshot of the table
And this is the response I get in Postman
{
"failures": [
{
"reason": "Either no latitude/longitude was provided, or the other information was insufficient for geocoding at or near line #0 of input.",
"row": ""
}
],
"append": true,
"returnResults": true,
"totalRows": 1,
"tableName": "mqap.jblablabla_reported_points",
"tableSize": {
"raw": 8192,
"readable": "8 KB"
},
"data": {
"rows": []
},
"failureCount": 1,
"rowCount": 0,
"info": {
"statusCode": 0,
"copyright": {
"text": "© 2015 MapQuest, Inc.",
"imageUrl": "http://api.mqcdn.com/res/mqlogo.gif",
"imageAltText": "© 2015 MapQuest, Inc."
},
"messages": []
}
}
What am I doing wrong?
The geography column should be in the table by default when the table is created: mqap_geography. This column is populated depending on information in other columns depending on their data type: Geography, [Latitude, Longitude], [Street, City, StateProvince, PostalCode, County, Country], and FullAddress. The Latitude, Longitude columns in the table above are type String so they won't populate the Geography column.
Two things I was missing:
The database NEEDS to have a geography object. So I added a latlong Geography object that I didn't end up using.
The format of the JSON should've been the following:
{
"tableName": "mqap.jZ3uoAVablablabla_reported_points",
"append": true,
"returnResults": true,
"rows": [
[
{"name":"user","value":user},
{"name":"email","value":email},
{"name":"date","value":date},
{"name":"message","value":message},
{"name":"latitude","value":latitude},
{"name":"longitude","value"longitude},
{"name":"latlong","value":"POINT(" + parseFloat(longitude) + " " + parseFloat(latitude) +")"}
]
]
}