Envoy External Authorization with OPA - evaluate fail with large JSON body - kubernetes

I have k8s pod running 3 containers: my app, opa, envoy
All my setup follow this guide: https://www.openpolicyagent.org/docs/latest/envoy-authorization/
Everything went well until I have 15kb JSON body.
Checking the OPA container log I see in request.http.body - only about half of JSON there.
{
"decision_id": "",
"error": {},
"input": {
"attributes": {
"destination": {
"address": {
"Address": {
"SocketAddress": {
"PortSpecifier": {
"PortValue": 8000
},
"address": "10.244.8.102"
}
}
}
},
"request": {
"http": {
"body": "only half of JSON body come here",
"headers": {
":authority": "api-service.com",
":method": "PUT",
":path": "/api",
"accept": "application/json",
"content-length": "14822",
"content-type": "application/json",
"x-envoy-decorator-operation": "....",
"x-envoy-internal": "true",
"x-forwarded-for": "10.244.6.0",
"x-forwarded-proto": "https",
"x-istio-attributes": "..."
},
"host": "....com",
"id": "12114967460600931537",
"method": "PUT",
"path": "/api",
"size": 14822
}
},
"source": {
"address": {
"Address": {
"SocketAddress": {
"PortSpecifier": {
"PortValue": 34670
},
"address": "10.244.3.164"
}
}
}
}
},
"parsed_path": [
"api"
],
"parsed_query": {}
},
"level": "info",
"msg": "Decision Log",
"query": "data.app.allow",
"type": "openpolicyagent.org/decision_logs"
}
I tried increase with_request_body.
http_filters:
- name: envoy.ext_authz
config:
with_request_body:
max_request_bytes: 819200
allow_partial_message: true
failure_mode_allow: false
Is there any other thing I missed?
Thanks a lot for your help

Are there any errors in the Envoy logs ?
What is the data that you are trying to send ? Does it need to be part of OPA's input document or can you leverage OPA's bundle feature.

I finally make it works by increasing max_request_bytes.
name: envoy.ext_authz
config:
with_request_body:
max_request_bytes: 819200
I configured this before in configmap but forgot to restart the pod. Just redeploy everything with new max_request_bytes - it's ok now
Reference: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/buffer/v3/buffer.proto.html?highlight=max_request_bytes
Thank you all

Related

Getting started with KrakenD

I need some beginner help to KrakenD. I am running it on Ubuntu. The config is provided below.
I am able to reach the /healthz API without problem.
My challenge is that the /hello path returns error 500. I want this path to redirect to a Quarkus app that runs at http://getting-started36-getting-going.apps.bamboutos.hostname.us/.
Why is this not working? If I modify the /hello backend and use a fake host, I get the exacts ame result. This suggests that KrakendD is not even trying to connect to the backend.
In logs, KrakendD is saying:
Error #01: invalid character 'H' looking for beginning of value
kraken.json:
{
"version": 2,
"port": 9080,
"extra_config": {
"github_com/devopsfaith/krakend-gologging": {
"level": "DEBUG",
"prefix": "[KRAKEND]",
"syslog": false,
"stdout": true,
"format": "default"
}
},
"timeout": "3000ms",
"cache_ttl": "300s",
"output_encoding": "json",
"name": "KrakenD API Gateway Service",
"endpoints": [
{
"endpoint": "/healthz",
"extra_config": {
"github.com/devopsfaith/krakend/proxy": {
"static": {
"data": { "status": "OK"},
"strategy": "always"
}
}
},
"backend": [
{
"url_pattern": "/",
"host": ["http://fake-backend"]
}
]
},
{
"endpoint": "/hello",
"extra_config": {},
"backend": [
{
"url_pattern": "/hello",
"method": "GET",
"host": [
"http://getting-started36-getting-going.apps.bamboutos.hostname.us/"
]
}
]
}
]
}
What am I missing?
add "encoding": "string" to the backend section.
"backend": [
{
"url_pattern": "/hello",
"method": "GET",
"encoding": "string" ,
"host": [
"http://getting-started36-getting-going.apps.bamboutos.hostname.us/"
]
}
]

AWS CloudFormation. Calling synchronously Step Function from API Gateway v1

I am trying to synchronously execute AWS Step Function via API Gateway. The problem is that with API Gateway V1 I have to use OpenAPI syntax (i.e. swagger) in order to specify integrationSubtype parameter, but something just doesn't work. Here is CloudFormation template I am using:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters": {
"restApiName": {
"Type": "String",
"Default": "stepApi"
}
},
"Resources": {
"MyStepFunction": {
"Type": "AWS::StepFunctions::StateMachine",
"Properties": {
"StateMachineName": "HelloWorld-StateMachine",
"StateMachineType": "EXPRESS",
"DefinitionString": "{\"Comment\": \"A Hello World example of the Amazon States Language using Pass states\", \"StartAt\": \"Hello\", \"States\": {\"Hello\": { \"Type\": \"Pass\", \"Result\": \"Hello\", \"Next\": \"World\" }, \"World\": { \"Type\": \"Pass\", \"Result\": \"World\", \"End\": true } } }",
"RoleArn": {
"Fn::GetAtt": [
"StepFunctionRole",
"Arn"
]
}
}
},
"StepFuncGateway": {
"Type": "AWS::ApiGateway::RestApi",
"Properties": {
"Name": {
"Ref": "restApiName"
},
"Body": {
"openapi": "3.0.1",
"info": {
"title": "processFormExample",
"version": "2020-11-06 15:32:29UTC"
},
"paths": {
"/step": {
"post": {
"responses": {
"200": {
"description": "Pet updated.",
"content": {
"application/json": {},
"application/xml": {}
}
},
"405": {
"description": "Method Not Allowed",
"content": {
"application/json": {},
"application/xml": {}
}
}
},
"parameters": [
],
"x-amazon-apigateway-integration": {
"integrationSubtype": "StepFunctions-StartSyncExecution",
"credentials": {
"Fn::GetAtt": [
"APIGatewayRole",
"Arn"
]
},
"RequestTemplates": {
"application/json": {
"Fn::Join": [
"",
[
"#set( $body = $util.escapeJavaScript($input.json('$')) ) \n\n{\"input\": \"$body\",\"name\": \"$context.requestId\",\"stateMachineArn\":\"",
{
"Ref": "MyStepFunction"
},
"\"}"
]
]
}
},
"httpMethod": "POST",
"payloadFormatVersion": "1.0",
"passthroughBehavior": "NEVER",
"type": "AWS_PROXY",
"connectionType": "INTERNET"
}
}
}
},
"x-amazon-apigateway-cors": {
"allowMethods": [
"*"
],
"maxAge": 0,
"allowCredentials": false,
"allowOrigins": [
"*"
]
}
}
},
"DependsOn": [
"APIGatewayRole",
"MyStepFunction"
]
},
"APIGatewayRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"apigateway.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs",
"arn:aws:iam::aws:policy/AWSStepFunctionsFullAccess"
]
}
},
"StepFunctionRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "states.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
},
"Path": "/",
"ManagedPolicyArns": [
"arn:aws:iam::aws:policy/service-role/AWSLambdaRole"
]
}
}
},
"Outputs": {
"HelloWorldApi": {
"Description": "Sync WF API endpoint",
"Value": {
"Fn::Sub": "https://${StepFuncGateway}.execute-api.${AWS::Region}.amazonaws.com/step"
}
}
}
}
The error I am seeing is following:
Errors found during import: Unable to put integration on 'POST' for
resource at path '/step': Invalid integration URI specified (Service:
AmazonApiGateway; Status Code: 400; Error Code: BadRequestException;
Request ID: 0c74acf9-147f-4561-9f4f-e457096c5533; Proxy: null)
I am out of ideas. Please help me to fix it.
UPDATE:
I had to add following code into x-amazon-apigateway-integration section and change type to AWS:
"uri": {
"Fn::Join": [
"",
[
"arn:aws:apigateway:",
{
"Ref": "AWS::Region"
},
":states:action/StartSyncExecution"
]
]
},
Another thing I had to fix is RequestTemplates, it should start with lower case r. After mentioned change the stack was deployed correctly, but now I have throttling problem to solve.
x-amazon-apigateway-integration is missing the uri property.
From the Amazon Developer Guide, the URI property is defined as:
The endpoint URI of the backend. For integrations of the aws type,
this is an ARN value. For the HTTP integration, this is the URL of the
HTTP endpoint including the https or http scheme.
For example:
"x-amazon-apigateway-integration": {
"type": "AWS_PROXY",
"httpMethod": "POST",
"uri": "http://petstore.execute-api.us-west-1.amazonaws.com/petstore/pets",
"payloadFormatVersion": 1.0,
"otherPropterties": "go here"
}
Amazon has additional information on URI definitions here. (Copied for convienience)
For HTTP or HTTP_PROXY integrations, the URI must be a fully formed, encoded HTTP(S) URL according to the RFC-3986 specification, for either standard integration, where connectionType is not VPC_LINK, or private integration, where connectionType is VPC_LINK. For a private HTTP integration, the URI is not used for routing.
For AWS or AWS_PROXY integrations, the URI is of the form arn:aws:apigateway:{region}:{subdomain.service|service}:path|action/{service_api}. Here, {Region} is the API Gateway region (e.g., us-east-1); {service} is the name of the integrated AWS service (e.g., s3); and {subdomain} is a designated subdomain supported by certain AWS service for fast host-name lookup. action can be used for an AWS service action-based API, using an Action={name}&{p1}={v1}&p2={v2}... query string. The ensuing {service_api} refers to a supported action {name} plus any required input parameters. Alternatively, path can be used for an AWS service path-based API. The ensuing service_api refers to the path to an AWS service resource, including the region of the integrated AWS service, if applicable. For example, for integration with the S3 API of GetObject, the uri can be either arn:aws:apigateway:us-west-2:s3:action/GetObject&Bucket={bucket}&Key={key} or arn:aws:apigateway:us-west-2:s3:path/{bucket}/{key}

Kubernetes - Job scheduling API

I am trying to schedule Jobs in Kubernetes.
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
The job can be created using the below command.
$ kubectl create -f ./cronjob.yaml
cronjob "hello" created
Is there any REST API using which the Job can be created from a Java client?
Thanks
The respective REST endpoint is described in the official API reference. You will find the CronJob resource in the batch/v1beta1 API group. To create a new CronJob resource, you'll need a POST call to the /apis/batch/v1beta1/namespaces/{namespace}/cronjobs URL.
A respective HTTP request might look something like this:
POST /apis/batch/v1beta1/namespaces/default/cronjobs HTTP/1.1
Content-Type: application/json
Content-Length: ...
Authorization: ...
[other headers]
{
"metadata": {
"name": "some-cron"
},
"spec": {
...
}
}
There are also older versions of the same resource, for example in the batch/v2alpha1 API group. As a rule of thumb, I'd recommend using the newest API version available to you. Especially, do not rely on alpha APIs in production; they tend to deprecate pretty quickly between releases.
To create a batch/v1beta1 CronJob using the Java client, have a look at the createNamespacedCronJob method of the io.kubernetes.client.openapi.apis.BatchV1beta1Api class.
HTTP Request
POST /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
You can take a look here for API overview: cronjob-v1beta1-batch
Add CronJob object in request Body.
{
"apiVersion": "batch/v1beta1",
"kind": "CronJob",
"metadata": {
"name": "hello"
},
"spec": {
"schedule": "*/1 * * * *",
"jobTemplate": {
}
}
}
Check here its spec: writing-a-cron-job-spec
Part of swagger.json
"post": {
"description": "create a CronJob",
"consumes": [
"*/*"
],
"produces": [
"application/json",
"application/yaml",
"application/vnd.kubernetes.protobuf"
],
"schemes": [
"https"
],
"tags": [
"batch_v1beta1"
],
"operationId": "createBatchV1beta1NamespacedCronJob",
"parameters": [
{
"name": "body",
"in": "body",
"required": true,
"schema": {
"$ref": "#/definitions/io.k8s.api.batch.v1beta1.CronJob"
}
}
],
"responses": {
"200": {
"description": "OK",
"schema": {
"$ref": "#/definitions/io.k8s.api.batch.v1beta1.CronJob"
}
},
"201": {
"description": "Created",
"schema": {
"$ref": "#/definitions/io.k8s.api.batch.v1beta1.CronJob"
}
},
"202": {
"description": "Accepted",
"schema": {
"$ref": "#/definitions/io.k8s.api.batch.v1beta1.CronJob"
}
},
"401": {
"description": "Unauthorized"
}
},
"x-kubernetes-action": "post",
"x-kubernetes-group-version-kind": {
"group": "batch",
"kind": "CronJob",
"version": "v1beta1"
}
}
See full swagger.json

Problems to POST chaincode (smart contract) to hyperledger-fabric using API

I've deployed the hyperledger-fabric service on Bluemix and obtained the credentials from there, one line looks like this:
{"enrollId":"user_type1_0","enrollSecret":"XXXXX","group":"group1","affiliation":"0001","username":"user_type1_0","secret":"XXXXX"}
I post the following to the "registrar" REST endpoint:
Secret: { "enrollId": "user_type1_0", "enrollSecret": "xxxxx" }
I get this response:
{ "OK": "Login successful for user 'user_type1_0'." }
Then I try to register some chaincode using POSTing the following to the chaincode REST endpoint:
QuerySpec {
"jsonrpc": "2.0",
"method": "deploy",
"params": {
"type": 1,
"chaincodeID": {
"path": "https://github.com/ibm-blockchain/learn-chaincode/finished"
},
"ctorMsg": {
"function": "init",
"args": [
"hi there"
]
},
"secureContext": "user_type1_0_xxxxx"
},
"id": 1 }
I get this reponse:
{ "jsonrpc": "2.0", "error": {
"code": -32000,
"message": "Registration missing",
"data": "User not logged in. Use the '/registrar' endpoint to obtain a security token." }, "id": 1 }
Any idea?
Fabric expects that you will provide EnrolmentID as a security context but you are trying to use "ID+Pass".
Can you try to run your deploy command with another SecurityContext value ?
QuerySpec { "jsonrpc": "2.0", "method": "deploy", "params": { "type": 1, "chaincodeID": { "path": "https://github.com/ibm-blockchain/learn-chaincode/finished" }, "ctorMsg": { "function": "init", "args": [ "hi there" ] }, "secureContext": "user_type1_0" }, "id": 1 }

Simple REST API Call from logic app - Azure

First of all I want you to know that I am new to Azure.
Recently, I am trying to work on Azure Logic App.
My motive is to make a simple REST API Call from the HTTP API (from Microsoft) and mail the response JSON via Office 365 connector.
Here is my code:
{
..
.
.
"triggers": {
"http": {
"recurrence": {
"frequency": "Day",
"interval": 1
},
"type": "Http",
"inputs": {
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"uri": "http://xxx/wcf/myrestservice.svc/is_online"
}
}
},
"actions": {
"office365connector": {
"type": "ApiApp",
"inputs": {
"apiVersion": "2015-01-14",
"host": {
"id": "/subscriptions/xxx/resourcegroups/resourcegroup1/providers/Microsoft.AppService/apiapps/office365connector",
"gateway": "https://xxx.azurewebsites.net"
},
"operation": "SendMail",
"parameters": {
"message": {
"To": "xxx#example.com",
"Subject": "My Service Status",
"Importance": "High",
"Body": "Hi #{triggers().outputs.body.Is_OnlineResult}"
}
},
"authentication": {
"type": "Raw",
"scheme": "Zumo",
"parameter": "#parameters('/subscriptions/xxx/resourcegroups/resourcegroup1/providers/Microsoft.AppService/apiapps/office365connector/token')"
}
},
"conditions": []
}
},
"outputs": {}
}
I am wondering, how could I get the response of the HTTP call?
Then I want to send the same in the mail body.
Please correct me if I am going in wrong direction. Any response from you will be very helpful to me.
Manish!
Have you tried using "Content"?
#triggers().outputs.body.Content