Please excuse me this newbie question ....
If during creation POST, if the user sends an id as part of the param how do i inform the connector to ignore it.
I have follow json config for the model ...
{
"name": "Activity",
"plural": "activities",
"base": "PersistedModel",
"idInjection": true,
"options": {
"validateUpsert": true
},
"properties": {
"action" : { "type" : "string" },
"status" : { "type" : "date" },
},
"validations": [],
"relations": {},
"acls": [],
"methods": {}
}
I was expecting mongodb connector to know that I want it it to create an ID because I have following param
idInjection": true,
For some reason when "id" param is send with the call, the connector does not create a record new ObjectId instead tries to created an record with send id
I suspect I am doing something wrong, bu fail to find in the documentation to rectify this behavior.
See this, under _id Field. If you send an "_id" field, MongoDB doesn't create it.
NB. MongoDB only cares about "_id" field, not "id" fields, so I'm guessing you have a type there, or have somehow changed this special name in the configuration.
Related
I am trying to use post aggregation. I have used aggregation to count the number of rows which match the given filter. Following is the post aggregation query:
{
"queryType": "groupBy",
"dataSource": "datasrc1",
"intervals": ["2020-09-16T21:15/2020-09-16T22:30"],
"pagingSpec":{ "threshold":100},
"dimensions": ["city", "zip_code", "country"],
"filter": {
"fields": [
{
"type": "selector",
"dimension": "bankId",
"value": "<bank id>"
}
]
},
"granularity": "all",
"aggregations": [
{ "type": "count", "name": "row"}
],
"postAggregations": [
{ "type": "arithmetic",
"name": "sum_rows",
"fn": "+",
"fields": [
{ "type": "fieldAccess", "fieldName": "row" }
]
}
]
}
If I remove the post aggregation part, it returns me result like:
[ {
"version" : "v1",
"timestamp" : "2020-09-16T21:15:00.000Z",
"event" : {
"city": "Sunnyvale",
"zip_code": "94085",
"country": "US",
"row" : 1
}
}, {
"version" : "v1",
"timestamp" : "2020-09-16T21:15:00.000Z",
"event" : {
"city": "Sunnyvale",
"zip_code": "94080",
"country": "US",
"row" : 1
}
}
If I add the post aggregations part, I get parser exception:
{
"error" : "Unknown exception",
"errorMessage" : "Instantiation of [simple type, class io.druid.query.aggregation.post.ArithmeticPostAggregator] value failed: Illegal number of fields[
%s], must be > 1 (through reference chain: java.util.ArrayList[0])",
"errorClass" : "com.fasterxml.jackson.databind.JsonMappingException",
"host" : null
}
I want to add all the rows (column 'row') in the response we are getting for aggregation query; and put the output in "sum_rows".
I don't understand what I am missing in post_aggregations. Any help is appreciated.
Confess that I spend most of my time in the SQL API not in the native API (!!) but I think your issue is that you're only supplying one field to your post aggregator. See these examples:
https://druid.apache.org/docs/latest/querying/post-aggregations.html#example-usage
If you need sum of rows, perhaps you need a normal aggregator to sum the row count?
The error message says the ArithmeticPostAggregator requires 2 arguments; the example code has only one. There's an example of this post aggregator at the bottom of this answer.
However...the example query doesn't have multiple numeric aggregations to perform arithmetic post-aggregation against. Maybe the goal is to "combine" the two output rows into one?
...To change the two-row result into only one with the total row count (for all database records matching the query filter and interval), removing zip_code from the dimension list would be one way.
Removing zip_code from dimensions would produce one result like this:
[
{
"version" : "v1",
"timestamp" : "2020-09-16T21:15:00.000Z",
"event" : {
"city": "Sunnyvale",
"country": "US",
"row" : 2
}
]
As you can see, by submitting a groupBy query with aggregations, Druid will do this aggregation for you dynamically (based on the dimension values in the database at the time the query is run) without needing post aggregations.
Example arithmetic post aggregator:
{
"type": "arithmetic",
"name": "my_output_sum",
"fn": "+",
"fields": [
{"fieldName": "input_addend_1", "type":"fieldAccess"},
{"fieldName": "input_addend_2", "type":"fieldAccess"}
]
}
I'm trying to import edges from a CSV-file into OrientDB. The vertices are stored in a separate file and already imported via ETL into OrientDB.
So my situation is similar to OrientDB import edges only using ETL tool and OrientDB ETL loading CSV with vertices in one file and edges in another.
Update
Friend.csv
"id","client_id","first_name","last_name"
"0","0","John-0","Doe"
"1","1","John-1","Doe"
"2","2","John-2","Doe"
...
The "id" field is removed by the Friend-Importer, but the "client_id" is stored. The idea is to have a known client-side generated id for searching etc.
PeindingFriendship.csv
"friendship_id","client_id","from","to"
"0","0-1","1","0"
"2","0-15","15","0"
"3","0-16","16","0"
...
The "friendship_id" and "client_id" should be imported as attributes of the "PendingFriendship" edge. "from" is a "client_id" of a Friend. "to" is a "client_id" of another Friend.
For "client_id" exists a unique Index on both Friend and PendingFriendship.
My ETL configuration looks like this
...
"extractor": {
"csv": {
}
},
"transformers": [
{
"command": {
"command": "CREATE EDGE PendingFriendship FROM (SELECT FROM Friend WHERE client_id = '${input.from}') TO (SELECT FROM Friend WHERE client_id = '${input.to}') SET client_id = '${input.client_id}'",
"output": "edge"
}
},
{
"field": {
"fieldName": "from",
"expression": "remove"
}
},
{
"field": {
"fieldName": "to",
"operation": "remove"
}
},
{
"field": {
"fieldName": "friendship_id",
"expression": "remove"
}
},
{
"field": {
"fieldName": "client_id",
"operation": "remove"
}
},
{
"field": {
"fieldName": "#class",
"value": "PendingFriendship"
}
}
],
...
The issue with this configuration is that it creates two edge entries. One is the expected "PendingFriendship" edge. The second one is an empty "PendingFriendship" edge, with all the fields I removed as attributes with empty values.
The import fails, at the second row/document, because another empty "PendingFriendship" cannot be inserted because it violates a uniqueness constraint.
How can I avoid the creation of the unnecessary empty "PendingFriendship".
What is the best way to import edges into OrientDB? All the examples in the documentation use CSV files where vertices and edges are in one file, but this is not the case for me.
I also had a look into the Edge-Transformer, but it returns a Vertex not an Edge!
Created PendingFriendships
After some time I found a way (workaround) to import the above data into OrientDB. Instead of using the ETL Tool I wrote simple ruby scripts which call the HTTP API of OrientDB using the Batch endpoint.
Steps:
Import the Friends.
Use the response to create a mapping of client_ids to #rids.
Parse the PeindingFriendship.csv and build batch requests.
Each Friendships is created by its own command.
The mapping from 2. is used to insert the #rids into the command from 4.
Send the batch requests in junks of 1000 commands.
Example Batch-Request body:
{
"transaction" : true,
"operations" : [
{
"type" : "cmd",
"language" : "sql",
"command" : "create edge PendingFriendship from #27:178 to #27:179 set client_id='4711'"
}
]
}
This isn't the answer to the question I asked, but it solves the higher goal of importing data into OrientDB, for me. Therefore I leave it open for the community to mark this question as solved or not.
I am using ibm graph in bluemix and new to this.
I created a graph named 'test' using the GUI provided by bluemix and uploaded the sample data 'Music Festival' provided by ibm in that graph.
Now I am trying to query all the vertices having label 'attendee' using below query.
def gt = graph.traversal();
gt.V().hasLabel("attendee");
But I am getting error as
Error: Error encountered evaluating script def gt = graph.traversal();gt.V().hasLabel("attendee"); with reason com.thinkaurelius.titan.core.TitanException: Could not find a suitable index to answer graph query and graph scans are disabled: [(~label = attendee)]:VERTEX
Not sure what I am doing wrong.
Can somebody tell where am i going wrong?
How can i get rid of this error and get the expected output?
Thanks
#Radhika, Your Gremlin query is a valid Gremlin query. However, some vendors (such as IBM Graph and Titan) chose to only allow users to start their queries with a query that is indexed.This is to make sure you get the performance of your queries. Calling hasLabel() by itself will give you the Could not find a suitable index... error as you can't create indexes for labels. What you need to do is follow this step with a step that uses a indexed property as in this query :
graph.traversal();gt.V().hasLabel("band").has("genre","pop");
An index for genre has been created in the schema for the sample music festival data as you can see below
{
"propertyKeys": [
{ "name": "name", "dataType": "String", "cardinality": "SINGLE" },
{ "name": "gender", "dataType": "String", "cardinality": "SINGLE" },
{ "name": "age", "dataType": "Integer", "cardinality": "SINGLE" },
{ "name": "genre", "dataType": "String", "cardinality": "SINGLE" },
{ "name": "monthly_listeners", "dataType": "String", "cardinality": "SINGLE" },
{ "name":"date","dataType":"String","cardinality":"SINGLE" },
{ "name":"time","dataType":"String","cardinality":"SINGLE" }
],
"vertexLabels": [
{ "name": "attendee" },
{ "name": "band" },
{ "name": "venue" }
],
"edgeLabels": [
{ "name": "bought_ticket", "multiplicity": "MULTI" },
{ "name":"advertised_to","multiplicity":"MULTI" },
{ "name":"performing_at","multiplicity":"MULTI" }
],
"vertexIndexes": [
{ "name": "vByName", "propertyKeys": ["name"], "composite": true, "unique": false },
{ "name": "vByGender", "propertyKeys": ["gender"], "composite": true, "unique": false },
{ "name": "vByGenre", "propertyKeys": ["genre"], "composite": true, "unique": false}
],
"edgeIndexes" :[
{ "name": "eByBoughtTicket", "propertyKeys": ["time"], "composite": true, "unique": false }
]
That's why the above query works and you need to do the same.
If you don't have a schema, create one. You can model it after the
one above or follow the API
doc
Create an (Vertex/Label) index for the properties that you'll start
your traversals from. In this example, Name, Gender and Genre for
vertex properties and name for the edge properties.
Call the schema
endpoint
to add your schema to your graph
It's recommended to create your schema before adding any data to
your graph so that you don't have to reindex later. That'll save you
a lot of time.
Once you create your schema, you can't modify what you created
already, but you can add new properties/indexes later on.
Look at the following code samples for Java and Nodejs for the exact code to use.
I hope that helps
I'm using Fiware Orion Context Broker version 0.20.
I notice that when I create a context subscription, my provided endpoint immediately gets notified about changes to the corresponding context elements that happened before I created the subscription.
To clarify: (note: I used these steps with a clean database)
I started the accumulator from the test package /usr/share/contextBroker/tests/accumulator-server.py 1028 /accumulate on
Created a context element using http://localhost:1026/v1/updateContext
{
"contextElements": [
{
"type": "Room",
"isPattern": "false",
"id": "Room1",
"attributes": [
{
"name": "temperature",
"value": "20"
}
]
}
],
"updateAction": "APPEND"
}
Then I created the subscription using http://localhost:1026/v1/subscribeContext
{
"entities": [
{
"type": "Room",
"isPattern": "true",
"id": ".*"
}
],
"attributes": [
"temperature"
],
"reference": "http://localhost:1028/accumulate",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"temperature"
]
}
],
"throttling": "PT5S"
}
I immediately receive the following content in the accumulator
POST http://localhost:1028/accumulate
Content-Length: 472
User-Agent: orion/0.20.0 libcurl/7.19.7
Host: localhost:1028
Accept: application/xml, application/json
Content-Type: application/json
{
"subscriptionId" : "55521671985dc3976b879780",
"originator" : "localhost",
"contextResponses" : [
{
"contextElement" : {
"type" : "Room",
"isPattern" : "false",
"id" : "Room1",
"attributes" : [
{
"name" : "temperature",
"type" : "",
"value" : "20"
}
]
},
"statusCode" : {
"code" : "200",
"reasonPhrase" : "OK"
}
}
]
}
Furthermore, if I create multiple contextElements before adding the subscription, they are all part to the contextResponses in the notification.
For my use case, this behavior is undesirable. The subscriptions are very dynamic (they come and go often throughout the lifecycle of the application) and I do not want the entire history every time I create a subscription. I only want to be notified about changes starting from the moment T created the subscription. (Not a history)
Did I overlook something in the documentation and can I resolve this by changing the contents of the subscription request? If not, is this generally accepted behavior for the context broker or just a plain bug?
It is the expected behaviour, as described in the manual:
You may wonder why accumulator-server.py is getting this message if you don't actually do any update. This is because the Orion Context Broker considers the transition from "non existing subscription" to "subscribed" as a change.
We understand that for some uses cases this is not convenient. However, behaving in the opossite way ruins another uses cases which need to know the "inicial state" before starting getting notifications corresponding to actual changes. The best solution to make everybody happy is to make this configurable, so each client can chose what it prefers. This feature is currently in our roadmap (see this issue in github.com).
While this gets implemented in Orion, in your case maybe a possible workaround is just ignore the first received nofitication belonging to a subscription (you can identify the subscription to which one notification belongs by the subscriptionId field in the notification payload). All the following notifications beloning to that subscription will correspond to actual changes.
EDIT: the posibility of avoiding initial notification has been finally implemented at Orion. Details are at this section of the documentation. It is now in the master branch (so if you use fiware/orion:latest docker you will get it) and will be include in next Orion version (2.2.0).
I created a CloudFormation template and I wanted to create IAM user, to do that I used this JSON string:
"CFNUser" : {
"Type" : "AWS::IAM::User",
"Properties" : {
"LoginProfile": {
"Password": { "Ref" : "AdminPassword" }
}
}
},
Then for group I used this:
"CFNUserGroup" : {
"Type" : "AWS::IAM::Group"
},
After creating the stack, I got the following:
user name - IAMUsers-CFNUser-E1BT342YK7G6
group name - IAMUsers-CFNUserGroup-1UBUBRYALTIMI
So my question is, how can I set the user name here? same goes for the group name?
After talking with one of the AWS support, at this time of writing, it is not possible to specify your own username and group name in IAM using CloudFormation template :-(
Maybe there's a reason why they do not allow user to do this...anyway it's good thing that I have answer to this question and I will be glad if someone find this useful.
Amazon has added support from 20th July 2016.
https://aws.amazon.com/about-aws/whats-new/2016/07/aws-cloudformation-adds-support-for-aws-iot-and-additional-updates/
{
"Type": "AWS::IAM::User",
"Properties": {
"Groups": [ String, ... ],
"LoginProfile": LoginProfile Type,
"ManagedPolicyArns": [ String, ... ],
"Path": String,
"Policies": [ Policies, ... ],
"UserName": String
}
}
For groups, it's a GroupName property:
"CFNUserGroup" : {
"Type" : "AWS::IAM::Group",
"Properties": {
"GroupName": "My_CFN_User_Group"
}
}