I want to create a Map variable in the SDK - AWS API - for which I need to write a JSON Schema Input / Output Model.
Help me write JSON Schema syntax such that I can achieve my objective
To just make one Map variable -
{
"type" : "object",
"required" : [ "answers" ],
"properties" : {
"answers" : {
"type" : "object",
"additionalProperties": { "type": "string" }
}
},
"title" : "test"
}
to Create a Map of Map use:
"answers": {
"type" :"object",
"additionalProperties" :
{ "type" : "object",
"additionalProperties" : {"type" : "string"}
}
}
Related
I am trying read an avro file. But it says the below error using the below code.
val expected_avro=spark.read.format("avro").load("avro_file_path")
Error:
Found recursive reference in Avro schema, which can not be processed by Spark:
Below is the sample data:
{
"type" : "record",
"name" : "ABC",
"namespace" : "com.abc.xyzlibrary",
"doc" : "Description Sample",
"fields" : [ {
"name" : "id",
"type" : "int",
"default" : 0
},
{
"name" : "location",
"type" : "string",
"default" : ""
}]
}
Is it because we have name twice (one at the root level and one inside the fields section). Any leads on how to get this fixed?
Thanks in advance.
I'm new to Nifi and in my case have to consume a JSON topic from Kafka
and would like to convert it into a CSV file where I need to select only few scalar and some nested fields.
I need to do the following things:
Consume topic - Done
Json to CSV
Include header in the CSV file
Merge into single file (if its split)
Give a proper filename with date
Following the below link:
https://community.hortonworks.com/articles/64069/converting-a-large-json-file-into-csv.html
But not sure if this is the right approach and also don't know how to make a single file.
Im using NiFi (1.8) and schema is stored in Confluent Schema Registry
Json Sample:
{
"type" : "record",
"name" : "Customer",
"namespace" : "namespace1"
"fields" : [ {
"name" : "header",
"type" : {
"type" : "record",
"name" : "CustomerDetails",
"namespace" : "namespace1"
"fields" : [ {
"name" : "Id",
"type" : "string"
}, {
"name" : "name",
"type" : "string"
}, {
"name" : "age",
"type" : [ "null", "int" ],
"default" : null
}, {
"name" : "comm",
"type" : [ "null", "int" ],
"default" : null
} ]
},
"doc" : ""
}, {
"name" : "data",
"type" : {
"type" : "record",
"name" : "CustomerData"
"fields" : [ {
"name" : "tags",
"type" : {
"type" : "map",
"values" : "string"
}
}, {
"name" : "data",
"type" : [ "null", "bytes" ]
"default" : null
} ]
},
"doc" : ""
} ]
}
Please guide me.
Try ConvertRecord with a JsonTreeReader and CSVRecordSetWriter. The writer can be configured to include the header line, and you won't need to split/merge. UpdateAttribute can be used to set the filename attribute which is the associated filename for the flow file (used by PutFile, e.g.).
If ConvertRecord doesn't give you the output you want, can you please elaborate on the difference between what it gives you and what you expect?
I have setup druid and was able to run the tutorial at:Tutorial: Loading a file. I was also able to execute native json queries and get the results as described at : http://druid.io/docs/latest/tutorials/tutorial-query.html The druid setup is working fine.
I now want to ingest additional data from a Java program into this datasource. Is it possible to send data into druid using tranquility from a java program for a datasource created using batch load?
I tried the example program at : https://github.com/druid-io/tranquility/blob/master/core/src/test/java/com/metamx/tranquility/example/JavaExample.java
But this program just keeps running and doesn't show any output. how can druid be setup to accept data using tranquility core APIs?
Following are the ingestion specs and config file for tranquility:
wikipedia-index.json
{
"type" : "index",
"spec" : {
"dataSchema" : {
"dataSource" : "wikipedia",
"parser" : {
"type" : "string",
"parseSpec" : {
"format" : "json",
"dimensionsSpec" : {
"dimensions" : [
"channel",
"cityName",
"comment",
"countryIsoCode",
"countryName",
"isAnonymous",
"isMinor",
"isNew",
"isRobot",
"isUnpatrolled",
"metroCode",
"namespace",
"page",
"regionIsoCode",
"regionName",
"user",
{ "name": "added", "type": "long" },
{ "name": "deleted", "type": "long" },
{ "name": "delta", "type": "long" }
]
},
"timestampSpec": {
"column": "time",
"format": "iso"
}
}
},
"metricsSpec" : [],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "day",
"queryGranularity" : "none",
"intervals" : ["2015-09-12/2015-09-13"],
"rollup" : false
}
},
"ioConfig" : {
"type" : "index",
"firehose" : {
"type" : "local",
"baseDir" : "quickstart/",
"filter" : "wikiticker-2015-09-12-sampled.json.gz"
},
"appendToExisting" : false
},
"tuningConfig" : {
"type" : "index",
"targetPartitionSize" : 5000000,
"maxRowsInMemory" : 25000,
"forceExtendableShardSpecs" : true
}
}
}
example.json (tranquility config):
{
"dataSources" : [
{
"spec" : {
"dataSchema" : {
"dataSource" : "wikipedia",
"metricsSpec" : [
{ "type" : "count", "name" : "count" }
],
"granularitySpec" : {
"segmentGranularity" : "hour",
"queryGranularity" : "none",
"type" : "uniform"
},
"parser" : {
"type" : "string",
"parseSpec" : {
"format" : "json",
"timestampSpec" : { "column": "time", "format": "iso" },
"dimensionsSpec" : {
"dimensions" : ["channel",
"cityName",
"comment",
"countryIsoCode",
"countryName",
"isAnonymous",
"isMinor",
"isNew",
"isRobot",
"isUnpatrolled",
"metroCode",
"namespace",
"page",
"regionIsoCode",
"regionName",
"user",
{ "name": "added", "type": "long" },
{ "name": "deleted", "type": "long" },
{ "name": "delta", "type": "long" }]
}
}
}
},
"tuningConfig" : {
"type" : "realtime",
"windowPeriod" : "PT10M",
"intermediatePersistPeriod" : "PT10M",
"maxRowsInMemory" : "100000"
}
},
"properties" : {
"task.partitions" : "1",
"task.replicants" : "1"
}
}
],
"properties" : {
"zookeeper.connect" : "localhost"
}
}
I did not find any example on setting up a datasource on druid which accepts continuously accepts data from a java program. I don't want to use Kafka. Any pointers on this would be greatly appreciated.
You need to create the data files with the additional data first and than run the ingestion task with new fields, You can't edit the same record in druid, It overwrites to new record.
I can successfully create templates and tasks using the rest api.
How do i create a task using a template via rest api?
Which endpoint should i use?
Okay found out how:
Basically just use the same task rest endpoint and do a post and pass in the json.
In the json you can specify templateId and the vars like below.
{
"status": "disabled"
,"id": "test_task4"
,"template-id": "generic_mean_alert"
,"vars" : {
"measurement": {"type" : "string", "value" : "cpu" },
"where_filter": {"type": "lambda", "value": "\"cpu\" == 'cpu-total'"},
"groups": {"type": "list", "value": [{"type":"string", "value":"host"},{"type":"string", "value":"dc"}]},
"field": {"type" : "string", "value" : "usage_idle" },
"warn": {"type" : "lambda", "value" : "\"mean\" < 30.0" },
"crit": {"type" : "lambda", "value" : "\"mean\" < 10.0" },
"window": {"type" : "duration", "value" : "1m" },
"slack_channel": {"type" : "string", "value" : "#alerts_testing" }
}
,"dbrps": [ { "db": "test","rp": "autogen" } ]
,"type": "stream"
}
http://yoururl/kapacitor/v1/tasks
Hi I hope someone can help tell me what I am doing wrong. I am writing a CF template that just adds a VPN Gateway to a VPC. No need to update routing tables etc.
I am using the below template but I get an error that I can't quite see the problem, I thought an extra pair of eyes might help!
:
Template validation error: Template format error: Every Resources object must contain a Type member.
Template:
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "CF Just add a VPN Gateway to a VPC ",
"Parameters" : {
"targetVPCtoAttachGatewayTo" : {
"Description" : "VPC ID to attach VPN Gateway",
"Default" : "vpc-xxxxx",
"Type": "AWS::EC2::VPC::Id"
}
},
"Resources" : {
"VPNGateway" : {
"Type" : "AWS::EC2::VPNGateway",
"Properties" : {
"Type" : "ipsec.1",
"Tags" : [
{"Key": "Name", "Value": {"Fn::Join": ["",["Virtual Private Gateway for ", { "Ref": "targetVPCtoAttachGatewayTo"} ] ]}}]
}
},
"AttachVpnGateway" : {
"Type" : "AWS::EC2::VPCGatewayAttachment",
"DependsOn" : "VPNGateway",
"Properties" : {
"VpcId" : { "Ref" : "targetVPCtoAttachGatewayTo" },
"VpnGatewayId" : { "Ref" : "VPNGateway" }
}
},
"Outputs" : {
}}}
Resolved the issue, curly brackets in the wrong place. Working template below.
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "CF Just add a VPN Gateway to a VPC ",
"Parameters" : {
"targetVPCtoAttachGatewayTo" : {
"Description" : "VPC ID to attach VPN Gateway",
"Default" : "vpc-xxxxx",
"Type": "AWS::EC2::VPC::Id"
}
},
"Resources" : {
"VPNGateway" : {
"Type" : "AWS::EC2::VPNGateway",
"Properties" : {
"Type" : "ipsec.1",
"Tags" : [
{"Key": "Name", "Value": {"Fn::Join": ["",["Virtual Private Gateway for ", { "Ref": "targetVPCtoAttachGatewayTo"} ] ]}}]
}
},
"AttachVpnGateway" : {
"Type" : "AWS::EC2::VPCGatewayAttachment",
"DependsOn" : "VPNGateway",
"Properties" : {
"VpcId" : { "Ref" : "targetVPCtoAttachGatewayTo" },
"VpnGatewayId" : { "Ref" : "VPNGateway" }
}
}
},
"Outputs" : {
}}