I have this template that works as expected and creates a S3 bucket and a few athena queries.
https://datameetgeobk.s3.amazonaws.com/cftemplates/athena_saved_query_blog_partition_parquet.tpl
The question is:
How do I replace S3 bucketname "athenadata162a" with the bucket name created in the first step.
external_location = 's3://athenadata162a/optimized-data11/'
The code that created a new bucket looks like this...
"myS3Bucket" : {
"Type" : "AWS::S3::Bucket"
},
As suggested, changed the code to yaml and still getting this error:
Template validation error: Template error: One or more Fn::Sub
intrinsic functions don't specify expected arguments. Specify a string
as first argument, and an optional second argument to specify a
mapping of values to replace in the string
The code:
Resources:
myS3Bucket:
Type: 'AWS::S3::Bucket'
createTableASParquetPartioned:
Type: 'AWS::Athena::NamedQuery'
Properties:
Database: default
Description: Create table as parquet partitioned
Name: Blog2createTableAS
QueryString: !Sub
- |-
CREATE table new_parquet11 WITH (format='PARQUET',
parquet_compression='SNAPPY', partitioned_by=array['year'],
external_location = 's3://myS3Bucket/optimized-data11/') AS SELECT
id, date, element, datavalue, mflag, qflag, sflag, obstime,
substr("date",1,4) AS year FROM original_csv WHERE
cast(substr("date",1,4) AS bigint) >= 2015 AND cast(substr("date",1,4)
AS bigint) <= 2019
Typically you use Fn::Sub to substitute variables in some input string.
In your case in YAML this would be something like:
QueryString: !Sub
- |-
CREATE table new_parquet11
WITH (format='PARQUET',
parquet_compression='SNAPPY',
partitioned_by=array['year'],
external_location = 's3://${myS3Bucket}/optimized-data11/')
...
Sorry I will not provide JSON syntax as it is too horrible for multiline values.
You can do external_location = 's3://${myS3Bucket}/optimized-data11/'.
Related
In my CRUD Rest Service I do an insert into a DB and want to respond to the caller with the created new record. I am looking for a nice way to convert the map to json.
I am running on ballerina 0.991.0 and using a postgreSQL.
The return of the Update ("INSERT ...") is a map.
I tried with convert and stamp but i did not work for me.
import ballerinax/jdbc;
...
jdbc:Client certificateDB = new({
url: "jdbc:postgresql://localhost:5432/certificatedb",
username: "USER",
password: "PASS",
poolOptions: { maximumPoolSize: 5 },
dbOptions: { useSSL: false }
}); ...
var ret = certificateDB->update("INSERT INTO certificates(certificate, typ, scope_) VALUES (?, ?, ?)", certificate, typ, scope_);
// here is the data, it is map<anydata>
ret.generatedKeys
map should know which data type it is, right?
then it should be easy to convert it to json like this:
{"certificate":"{certificate:
"-----BEGIN
CERTIFICATE-----\nMIIFJjCCA...tox36A7HFmlYDQ1ozh+tLI=\n-----END
CERTIFICATE-----", typ: "mqttCertificate", scope_: "QARC", id_:
223}"}
Right now i do a foreach an build the json manually. Quite ugly. Maybe somebody has some tips to do this in a nice way.
It cannot be excluded that it is due to my lack of programming skills :-)
The return value of JDBC update remote function is sql:UpdateResult|error.
The sql:UpdateResult is a record with two fields. (Refer https://ballerina.io/learn/api-docs/ballerina/sql.html#UpdateResult)
UpdatedRowCount of type int- The number of rows which got affected/updated due to the given statement execution
generatedKeys of type map - This contains a map of auto generated column values due to the update operation (only if the corresponding table has auto generated columns). The data is given as key value pairs of column name and column value. So this map contains only the auto generated column values.
But your requirement is to get the entire row which is inserted by the given update function. It can’t be returned with the update operation if self. To get that you have to execute the jdbc select operation with the matching criteria. The select operation will return a table or an error. That table can be converted to a json easily using convert() function.
For example: Lets say the certificates table has a auto generated primary key column name ‘cert_id’. Then you can retrieve that id value using below code.
int generatedID = <int>updateRet.generatedKeys.CERT_ID;
Then use that generated id to query the data.
var ret = certificateDB->select(“SELECT certificate, typ, scope_ FROM certificates where id = ?”, (), generatedID);
json convertedJson = {};
if (ret is table<record {}>) {
var jsonConversionResult = json.convert(ret);
if (jsonConversionResult is json) {
convertedJson = jsonConversionResult;
}
}
Refer the example https://ballerina.io/learn/by-example/jdbc-client-crud-operations.html for more details.?
I'm currently using the pg-promise library to insert multiple values into a database in the format:
const cs = new pgp.helpers.ColumnSet(['booking_id', {name:'timeslot', cast:'timestamp'}], {table: 'booking'});
// data input values:
const values = [];
bookings.forEach(slot => {
values.push({booking_id: booking_id, timeslot: slot});
});
Where I need timeslot to be a timestamp. However it comes into the API as value like
1515586500.0
Using the above cast property my query gets resolved like so
insert into "booking"("booking_id","timeslot") values(1,'1515586500.0'::timestamp)
however this throws an error of cannot cast type numeric to timestamp without time zone
If I use the to_timestamp function however this works how I need it to e.g
insert into "booking"("booking_id","timeslot") values(1,to_timestamp('1515586500.0'));
Is there any way I can get pg-promise to use to_timestamp rather than the ::timestamp notation?
Change the column definition to this one:
{
name: 'timeslot',
mod: ':raw',
init: c => pgp.as.format('to_timestamp($1)', c.value)
}
or
{
name: 'timeslot',
mod: ':raw',
init: c => pgp.as.format('to_timestamp(${value})', c)
}
...as per the Column type documentation.
Or you can use Custom Type Formatting on the type, to self-format automatically.
Also, you do not need to remap values to suit the ColumnSet object, you use ColumnSet object to fit the data instead. So if the value for column timeslot is in property slot, you just use prop: 'slot' within your column definition to change where the value is coming from.
I am trying to create a proper, REST API, and document it with Swagger (2.0).
So, I have an API call that is a query, ie, it makes no changes and doesn't create anything (idempotent and safe). But it requires passing in a complex JSON parameter (list of items, 2 or 3 sets of addresses, etc). So I'm doing a GET with a parameter thats URL encoded JSON. That seems like the proper way to do it.
I see so often API's like this where they do it as a POST for this reason, but that's an incorrect use of the POST verb.
I'm seeing lots of swagger API's that do this...
I can't figure out if there's a way to do a proper rest API with Swagger, using a JSON parameter. You can define the parameter as a string, of course, and pass your encoded JSON into it, but then the swagger tooling doesn't understand that there's a schema/definition for it.
Is swagger not able to properly document this kind of call?
OpenAPI 2.0 (Swagger 2.0)
OpenAPI 2.0 does not support objects in query strings, it only supports primitive values and arrays of primitives. The most you can do is define your parameter as type: string, add an example of a JSON value, and use description to document the JSON object structure.
swagger: '2.0'
...
paths:
/something:
get:
parameters:
- in: query
name: params
required: true
description: A JSON object with the `id` and `name` properties
type: string
example: '{"id":4,"name":"foo"}'
OpenAPI 3.x
JSON in query string can be described using OpenAPI 3.x. In OAS 3, query parameters can be primitives, arrays as well as objects, and you can specify how these parameters should be serialized – flattened into key=value pairs, encoded as a JSON string, and so on.
For query parameters that contain a JSON string, use the content keyword to define a schema for the JSON data:
openapi: 3.0.1
...
paths:
/something:
get:
parameters:
- in: query
name: params
required: true
# Parameter is an object that should be serialized as JSON
content:
application/json:
schema:
type: object
properties:
id:
type: integer
name:
type: string
This corresponds to the following GET request (before URL encoding):
GET /something?params={"id":4,"name":"foo"}
or after URL encoding:
GET /something?params=%7B%22id%3A4%2C%22name%22%3A%22foo%22%7D
Note for Swagger UI users:
Parameters with content are supported in Swagger UI 3.23.8+ and Swagger Editor 3.6.34+.
Workaround for earlier versions of UI/Editor:
Define the parameter as just type: string and add an example of the JSON data. You lose the ability to describe the JSON schema for the query string, but "try it out" will work.
parameters:
- in: query
name: params
required: true
schema:
type: string # <-------
example: '{"id":4,"name":"foo"}' # <-------
For .Net and Swashbuckle (tested on 3.0)
I have a generic class JsonModelBinder that implements IModelBinder interface. The class is used like this:
public IActionResult SomeAction(
[FromRoute] int id,
[FromQuery][ModelBinder(BinderType = typeof(JsonModelBinder<SomeModel>))] SomeModelquery query) => {}
I have created Operation filter that does the following:
Removes parameters created by Swashbuckle from properties of my model
Add query parameter of type string
As a result in Swagger I have a text field where I can insert json and test requests
public class JsonModelBinderOperationFilter : IOperationFilter
{
public void Apply(Operation operation, OperationFilterContext context)
{
if (operation.Parameters == null || context.ApiDescription.HttpMethod != HttpMethod.Get.ToString())
return;
//Find json parameters
var jsonGetParameters = context.ApiDescription.ActionDescriptor.Parameters.Cast<ControllerParameterDescriptor>()
.Where(p => p.ParameterInfo.CustomAttributes.Any(c => c.AttributeType == typeof(ModelBinderAttribute) && c.NamedArguments.Any(IsJsonModelBinderType))).ToArray();
if (jsonGetParameters.Length > 0)
{
//Select parameters names created by Swagger from json parameters
var removeParamNames = new HashSet<string>(context.ApiDescription.ParameterDescriptions.Where(d => jsonGetParameters.Any(p => p.Name == d.ParameterDescriptor.Name)).Select(p => p.Name));
//Create new Swagger parameters from json parameters
var newParams = jsonGetParameters.Select(p => new NonBodyParameter()
{
In = "query",
Name = p.Name,
Type = "string",
Description = "Json representation of " + p.ParameterType.Name
});
//Remove wrong parameters and add new parameters
operation.Parameters = operation.Parameters.Where(p => p.In != "query" || !removeParamNames.Contains(p.Name)).Concat(newParams).ToList();
}
}
private static bool IsJsonModelBinderType(CustomAttributeNamedArgument arg)
{
var t = arg.TypedValue.Value as Type;
return t != null && t.GetGenericTypeDefinition().IsAssignableFrom(typeof(JsonModelBinder<>));
}
}
Notes:
I use IsAssignableFrom because I have classes derived from JsonModelBinder. You can omit it if you don't inherit
You can also omit GetGenericTypeDefinition if your binder is not generic
This solution doesn't check for parameter name collision, though you should never have it if the API made with common sense
I created a table with following syntax:
create table poll(poll_id string primary key,
poll_type_id integer,
poll_rating array(object as (rating_id integer,fk_user_id string, israted_image1 integer, israted_image2 integer, updatedDate timestamp, createdDate timestamp )),
poll_question string,
poll_image1 string,
poll_image2 string
)
And I inserted a record without "poll_rating" field which is actually an array of objects fields.
Now when I try to update a poll_rating with the following commands:
update poll set poll_rating = [{"rating_id":1,"fk_user_id":-1,"israted_image1":1,"israted_image2":0,"createddate":1400067339.0496}] where poll_id = "f748771d7c2e4616b1865f37b7913707";
I'm getting an error message like this:
"SQLParseException[line 1:31: no viable alternative at input '[']; nested: ParsingException[line 1:31: no viable alternative at input '[']; nested: NoViableAltException;"
Can anyone tell me why I get this error when I try to update the array of objects fields.
Defining arrays and objects directly in SQL statement is currently not supported by our SQL parser, please use parameter substitution using placeholders instead as described here:
https://crate.io/docs/current/sql/rest.html
Example using curl is as below:
curl -sSXPOST '127.0.0.1:4200/_sql?pretty' -d#- <<- EOF
{"stmt": "update poll set poll_rating = ? where poll_id = ?",
"args": [ [{"rating_id":1,"fk_user_id":-1,"israted_image1":1,"israted_image2":0,"createddate":1400067339.0496}], "f748771d7c2e4616b1865f37b7913707" ]
}
EOF
I have the following (Grails) domain object:
class Country {
Integer id
char country_abbr
String country_name
static mapping = {
version false
id name: 'id'
table 'country'
id generator:'identity', column:'id'
}
static constraints = {
}}
The 'country_abbr' field within the 'country table' has type: character(2). However, whenever I am setting the domain object's data type (for 'country_abbr') to String, initialization is failing with the following exception
org.hibernate.HibernateException: Wrong column type in mydb.country for column country_abbr. Found: bpchar, expected: varchar(255)
On the other hand, leaving this type as a Java char would only retrieve the first character. Any ideas how may I map to this type? Also, what is bpchar exactly?
Thanks
Just to make this question answered. The solution is to change the country_abbr mapping:
country_abbr columnDefinition: 'char'
Reference here.