I'm using Envoy as the gateway of my backend and I'm trying to use its JWT mechanism.
Here is what I've done:
Generate a pair of prv & pub keys with the command openssl:
openssl genrsa -out keyprv.pem 512
openssl rsa -in keyprv.pem -pubout -out keypub.crt
Let's say that the private key is PRVKEY and the public key is PUBKEY.
Generate a token with Python3. Here is my code:
#!/usr/bin/env python3
import jwt
from datetime import datetime
from datetime import timedelta
due_date = datetime.now() + timedelta(minutes=30)
expiry = int(due_date.timestamp())
payload = {"iss": "ImIss", "sub": "ImSub", "user_id": 12345, "exp": expiry}
private_key=b"-----BEGIN RSA PRIVATE KEY-----\nPRVKEY\n-----END RSA PRIVATE KEY-----"
jwt_token = jwt.encode(payload, private_key, algorithm="RS256", headers={"alg": "RS256", "typ": "JWT"})
print(jwt_token)
After executing the Python3 script, I get a JWT token aaa.bbb.ccc.
Configure my Envoy to verify the JWT token above. Here is the config file of Envoy:
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 9989
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
remote_jwt_provider:
issuer: "ImIss"
local_jwks:
inline_string: |
{
"keys": [
{
"kty": "RSA",
"kid": "my-key-id",
"use": "sig",
"n": "PUBKEY", # is this correct?
"e": "AQAB",
}
]
}
forward: true
from_params:
- jwt_token
rules:
- match:
prefix: "/"
requires:
provider_name: remote_jwt_provider
- name: envoy.filters.http.router
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: cluster_example
clusters:
- name: auth_service
connect_timeout: 30s
type: STATIC
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: auth_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 172.16.0.44
port_value: 8081
- name: cluster_example
connect_timeout: 30s
type: STATIC
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: cluster_example
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 172.16.0.226
port_value: 11111
But the config file would generate an Envoy error: Jwks RSA [n] or [e] field is missing or has a parse error. It seems that the characters / and = in the PUBKEY cause this error. So I'm thinking I may need to encode the public key with base64. So I use the output of base64.b64encode(PUBKEY) in the config file of Envoy. Now Envoy won't generate any error but it can't verify the token: Jwt verification fails.
Related
I'm new to google envoy and following this documentation : https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/jwt_authn/v3/config.proto#envoy-v3-api-msg-extensions-filters-http-jwt-authn-v3-jwtheader
Im using envoy with docker with this simple docker file :
FROM envoyproxy/envoy:v1.24-latest COPY envoy.yaml /etc/envoy/envoy.yaml
Im trying to set up a proxy using google envoy with a simple filter : a JWT check from header.
The extension envoy.extensions.filters.http.jwt_authn.v3.JwtHeader seems the way to go.
So here is part my yaml :
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 9911
filter_chains:
- filters:
- name: envoy.filters.http
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtHeader
name: Authorization
value_prefix: "Bearer "
And here what i got :
Didn't find a registered implementation for 'envoy.extensions.filters.http' with type URL: 'envoy.extensions.filters.http.jwt_authn.v3.JwtHeader'
I also tried this :
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 9911
filter_chains:
- filters:
- name: envoy.filters.http
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
jwt_auth:
local_jwks:
inline_string: '{"keys":[{"typ": "JWT", "kty":"oct","alg":"HS256","kid":"df","k":"aGVsbG93b3JsZA=="}]}'
from_headers:
- name: Authorization
value_prefix: "Bearer "
rules:
- match:
prefix: "/"
requires:
provider_name: jwt_auth
With the same result.
What am I doing wrong ? I guess something with "name" and "typed_config" but from the doc i found it should be fine.
Thanks
OK my main mistake was that jwtAuth must be place in a "http_filters" section, it seems the correct configuration is somethings like :
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 9901
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"#type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
route_config:
name: local_route
virtual_hosts:
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
jwt_auth:
local_jwks:
inline_string: '{"keys":[{"alg":"HS256","kty":"oct","k":"BzP7aHv89DvBWeO_YmKfKcTlqCjsifWnjInWSBX0_OA"}]}'
from_headers:
- name: Authorization
value_prefix: "Bearer "
rules:
- match:
prefix: "/"
requires:
provider_name: jwt_auth
- name: envoy.filters.http.router
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
host_rewrite_literal: www.envoyproxy.io
cluster: service_envoyproxy_io
clusters:
- name: service_envoyproxy_io
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: service_envoyproxy_io
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: www.envoyproxy.io
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"#type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: www.envoyproxy.io
That way the proxy is running (if built locally) with "bazel-bin/source/exe/envoy-static -c ../configEnvoy/jwt.yaml -l debug"
The token comes from https://jwt.io/#debugger-io?token=eyJhbGciOiJIUzI1NiIsImt0eSI6Im9jdCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.BzP7aHv89DvBWeO_YmKfKcTlqCjsifWnjInWSBX0_OA
BUT a request like : curl localhost:9901 -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsImt0eSI6Im9jdCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.BzP7aHv89DvBWeO_YmKfKcTlqCjsifWnjInWSBX0_OA' --verbose
Leads to "Jwt verification fails".
After digging a little bit and adding some logs here what i got :
[2023-01-24 15:30:01.654][755170][debug][http] [source/common/http/conn_manager_impl.cc:306] [C0] new stream
[2023-01-24 15:30:01.654][755170][debug][http] [source/common/http/conn_manager_impl.cc:972] [C0][S14930449970890273668] request headers complete (end_stream=true):
':authority', 'localhost:9901'
':path', '/'
':method', 'GET'
'user-agent', 'curl/7.61.1'
'accept', '*/*'
'authorization', 'Bearer eyJhbGciOiJIUzI1NiIsImt0eSI6Im9jdCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.BzP7aHv89DvBWeO_YmKfKcTlqCjsifWnjInWSBX0_OA'
[2023-01-24 15:30:01.654][755170][debug][http] [source/common/http/conn_manager_impl.cc:955] [C0][S14930449970890273668] request end stream
[2023-01-24 15:30:01.654][755170][debug][connection] [./source/common/network/connection_impl.h:92] [C0] current connecting state: false
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:157] Called Filter : setDecoderFilterCallbacks
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:53] Called Filter : decodeHeaders
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/matcher.cc:71] Prefix requirement '/' matched.
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/extractor.cc:254] extract authorizationBearer
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:151] jwt_auth: JWT authentication starts (allow_failed=false), tokens size=1
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:162] jwt_auth: startVerify: tokens size 1
[2023-01-24 15:30:01.654][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:180] jwt_auth: Parse Jwt eyJhbGciOiJIUzI1NiIsImt0eSI6Im9jdCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.BzP7aHv89DvBWeO_YmKfKcTlqCjsifWnjInWSBX0_OA
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:186] jwt_auth: jwt jwt_t eyJhbGciOiJIUzI1NiIsImt0eSI6Im9jdCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.BzP7aHv89DvBWeO_YmKfKcTlqCjsifWnjInWSBX0_OA
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:187] jwt_auth: jwt alg_ HS256
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:188] jwt_auth: jwt audiences_ []
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:189] jwt_auth: jwt header_str_ {"alg":"HS256","kty":"oct"}
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:190] jwt_auth: jwt kid_
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:191] jwt_auth: jwt payload_str_ {"sub":"1234567890","name":"John Doe","iat":1516239022}
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:192] jwt_auth: jwt signature_ 3�h{��;�Y��bb�)���(�������H���
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:193] jwt_auth: jwt iss_
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:306] jwt_auth: jwks alg_ HS256
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:308] jwt_auth: jwks crv_
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:309] jwt_auth: jwks hmac_key_ 3�h{��;�Y��bb�)���(�������H���
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:310] jwt_auth: jwks kid_
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:311] jwt_auth: jkws kty_ oct
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:312] jwt_auth: jwks okp_key_raw_
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/authenticator.cc:394] jwt_auth: JWT token verification completed with: Jwt verification fails
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:109] Jwt authentication completed with: Jwt verification fails
[2023-01-24 15:30:01.655][755170][debug][http] [source/common/http/filter_manager.cc:917] [C0][S14930449970890273668] Preparing local reply with details jwt_authn_access_denied{Jwt_verification_fails}
[2023-01-24 15:30:01.655][755170][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:97] Called Filter : decodeHeaders Stop
[2023-01-24 15:30:01.655][755170][debug][http] [source/common/http/filter_manager.cc:959] [C0][S14930449970890273668] Executing sending local reply.
[2023-01-24 15:30:01.655][755170][debug][http] [source/common/http/conn_manager_impl.cc:1588] [C0][S14930449970890273668] encoding headers via codec (end_stream=false):
':status', '401'
'www-authenticate', 'Bearer realm="http://localhost:9901/", error="invalid_token"'
'content-length', '22'
'content-type', 'text/plain'
'date', 'Tue, 24 Jan 2023 14:30:01 GMT'
'server', 'envoy'
jwt is the sent token and jwks is the local token, i checked authenticator.cc and it goes through google::jwt_verify::verifyJwtWithoutTimeChecking so i really don't get it, why the verification fails, the signature are the same. I'm quite new to jwt, is there a reason (appart from time check) that could make a check with the good signature fail ?
In fact the problem was in the "k" in the yaml configuration, i putted the last part of the token instead of the secret base64 encoded. Since the last part of the token is also encoded the proxy start smoothly anyways.
I guess for advanced user it's obvious but i'm a newbie in JWT.
So here is a simple working yaml config :
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 9901
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.stdout
typed_config:
"#type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
route_config:
name: local_route
virtual_hosts:
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
jwt_auth:
local_jwks:
inline_string: '{"keys":[{"kty":"oct","alg":"HS256","k":"aGVsbG93b3Jk"}]}'
from_headers:
- name: Authorization
value_prefix: "Bearer "
rules:
- match:
prefix: "/"
requires:
provider_name: jwt_auth
- name: envoy.filters.http.router
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match:
prefix: "/"
route:
host_rewrite_literal: www.envoyproxy.io
cluster: service_envoyproxy_io
clusters:
- name: service_envoyproxy_io
type: LOGICAL_DNS
# Comment out the following line to test on v6 networks
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: service_envoyproxy_io
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: www.envoyproxy.io
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"#type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: www.envoyproxy.io
So in the line (i removed all the optional keys): inline_string: '{"keys":[{"kty":"oct","alg":"HS256","k":"aGVsbG93b3Jk"}]}'
aGVsbG93b3Jk is helloword in base64 : https://www.base64encode.org/
Then you can validate https://jwt.io/#debugger-io?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2V4YW1wbGUuY29tIiwic3ViIjoidGVzdEBleGFtcGxlLmNvbSIsImF1ZCI6ImV4YW1wbGVfc2VydmljZSIsImV4cCI6MjAwMTAwMTAwMX0.Bx9ms9QsEoM3jlqD1RFjxUbvpjLebAALzo4q-SL7Rzg
Where you put your secret (here helloworld) in the signature section.
You can test using curl :
curl localhost:9901 -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2V4YW1wbGUuY29tIiwic3ViIjoidGVzdEBleGFtcGxlLmNvbSIsImF1ZCI6ImV4YW1wbGVfc2VydmljZSIsImV4cCI6MjAwMTAwMTAwMX0.Bx9ms9QsEoM3jlqD1RFjxUbvpjLebAALzo4q-SL7Rzg' --verbose
Here is the proxy log :
[2023-01-31 11:04:28.467][252749][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:109] Jwt authentication completed with: OK
[2023-01-31 11:04:28.467][252749][debug][router] [source/common/router/router.cc:470] [C0][S5111927710838764941] cluster 'service_envoyproxy_io' match for URL '/'
[2023-01-31 11:04:28.468][252749][debug][router] [source/common/router/router.cc:678] [C0][S5111927710838764941] router decoding headers:
':authority', 'www.envoyproxy.io'
':path', '/'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.61.1'
'accept', '*/*'
'x-forwarded-proto', 'http'
'x-request-id', '525bfe52-96c7-44f6-af33-4253635e056b'
'x-envoy-expected-rq-timeout-ms', '15000'
I am new to envoyproxy and I am getting 'jwt is missing' error while trying to access the endpoint without providing Authorization header.
How can I access the public api without auth header
i.e
http://hostaddress/v1/fares/locations/location?searchTerm=ABC
Below code shows existing clusters already added that uses Jwt auth.
http_filters:
- name: envoy.filters.http.grpc_http1_bridge
typed_config: {}
- name: envoy.filters.http.cors
typed_config: {}
- name: envoy.filters.http.jwt_authn
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
identity_api:
issuer: 'null'
audiences:
- apigw
forward: true
remote_jwks:
http_uri:
uri: http://abc.identity.api/.well-known/openid-configuration/jwks
cluster: identity_api
timeout: 1s
cache_duration:
seconds: 30
from_headers:
- name: Authorization
value_prefix: "Bearer "
rules:
- match: { prefix: /v1/newhubs/ }
- match: { prefix: / }
requires:
provider_name: identity_api
- name: envoy.filters.http.ext_authz
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
transport_api_version: V3
grpc_service:
envoy_grpc:
cluster_name: permissions_api_grpc
timeout: 3s
with_request_body:
max_request_bytes: 819200
allow_partial_message: true
- name: envoy.filters.http.router
typed_config: {}
Your jwt_authn matches to all paths via the root path: match: { prefix: / }.
Remove the match: { prefix: / } entry to only have your jwt_authn filter match for /v1/newhubs/
I have k8s cluster for using gRPC service with envoy proxy, all gRPC and web request collect Envoy and passed into backend , Envoy SVC started with nlb, and nlb attached with ACM certificate. without nlb certificate request passed into backend correctly and getting response, but need to use nlb-url:443, again if I'm attaching ACM cert into nlb,and its not at all getting response. why?
Or again I need to use another ingress for treat ssl and route?
envoy-svc.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-west-2:12345676789:certificate/ss304s07-3ss2-4s73-8744-bs2sss123460
service.beta.kubernetes.io/aws-load-balancer-type: nlb
creationTimestamp: "2021-06-11T02:50:24Z"
finalizers:
- service.kubernetes.io/load-balancer-cleanup
name: envoy-service
namespace: default
spec:
externalTrafficPolicy: Cluster
ports:
- nodePort: 31156
port: 443
protocol: TCP
targetPort: 80
selector:
name: envoy
sessionAffinity: None
type: LoadBalancer
envoy-conf
admin:
access_log_path: /dev/stdout
address:
socket_address: { address: 0.0.0.0, port_value: 8801 }
static_resources:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/config/listener/v3/listener.proto#config-listener-v3-listener
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 80
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#extensions-filters-network-http-connection-manager-v3-httpconnectionmanager
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
http2_protocol_options: {}
access_log:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/access_loggers/file/v3/file.proto
#
# You can also configure this extension with the qualified
# name envoy.access_loggers.http_grpc
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/access_loggers/grpc/v3/als.proto
- name: envoy.access_loggers.file
typed_config:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/access_loggers/file/v3/file.proto#extensions-access-loggers-file-v3-fileaccesslog
"#type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
# Console output
path: /dev/stdout
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- "*"
routes:
- match:
prefix: /
grpc:
route:
cluster: greeter_service
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
# custom-header-1 is just an example. the grpc-web
# repository was missing grpc-status-details-bin header
# which used in a richer error model.
# https://grpc.io/docs/guides/error/#richer-error-model
allow_headers: accept-language,accept-encoding,user-agent,referer,sec-fetch-mode,origin,access-control-request-headers,access-control-request-method,accept,cache-control,pragma,connection,host,name,x-grpc-web,x-user-agent,grpc-timeout,content-type
expose_headers: grpc-status-details-bin,grpc-status,grpc-message,authorization
max_age: "1728000"
http_filters:
- name: envoy.filters.http.grpc_web
# This line is optional, but adds clarity to the configuration.
typed_config:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/filters/http/grpc_web/v3/grpc_web.proto
"#type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
- name: envoy.filters.http.cors
typed_config:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/filters/http/cors/v3/cors.proto
"#type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
- name: envoy.filters.http.grpc_json_transcoder
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.grpc_json_transcoder.v3.GrpcJsonTranscoder
proto_descriptor: "/etc/envoy-sync/sync.pb"
ignore_unknown_query_parameters: true
services:
- "com.tk.system.sync.Synchronizer"
print_options:
add_whitespace: true
always_print_primitive_fields: true
always_print_enums_as_ints: true
preserve_proto_field_names: true
- name: envoy.filters.http.router
typed_config:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/extensions/filters/http/router/v3/router.proto
"#type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
# https://www.envoyproxy.io/docs/envoy/v1.15.0/api-v3/config/cluster/v3/cluster.proto#config-cluster-v3-cluster
- name: greeter_service
type: LOGICAL_DNS
connect_timeout: 0.25s
lb_policy: round_robin
load_assignment:
cluster_name: greeter_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: micro-deployment
port_value: 8081
http2_protocol_options: {} # Force HTTP/2
I’m trying o communicate from Envoy to Envoy using gRPC for Kubernetes(Amazon EKS).
I have an envoy in my sidecar and I am using grpcurl to validate the request.
The request is delivered to the application container and there are no errors, but the console returns the following results
server closed the stream without sending trailers
I don’t know what the reason for the above problem is, and what could be the reason for this result??
I was able to confirm that the response came back fine when I hit a single service before connecting with envoy.
this my envoy config
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 127.0.0.1
port_value: 10000
static_resources:
listeners:
- name: listener_secure_grpc
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 8443
traffic_direction: INBOUND
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: service_grpc
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: cluster_grpc
max_stream_duration:
grpc_timeout_header_max: 30s
tracing: {}
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck"
pass_through_mode: false
headers:
- name: ":path"
exact_match: "/healthz"
- name: envoy.filters.http.router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext"
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/ssl/grpc/tls.crt
private_key:
filename: /etc/ssl/grpc/tls.key
- name: listener_stats
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10001
traffic_direction: INBOUND
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: AUTO
stat_prefix: ingress_http
route_config:
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: /stats
route:
cluster: cluster_admin
http_filters:
- name: envoy.filters.http.router
- name: listener_healthcheck
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10010
traffic_direction: INBOUND
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: AUTO
stat_prefix: ingress_http
route_config: {}
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck"
pass_through_mode: false
headers:
- name: ":path"
exact_match: "/healthz"
- name: envoy.filters.http.router
clusters:
- name: cluster_grpc
connect_timeout: 1s
type: STATIC
http2_protocol_options: {}
upstream_connection_options:
tcp_keepalive: {}
load_assignment:
cluster_name: cluster_grpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 1443
- name: cluster_admin
connect_timeout: 1s
type: STATIC
load_assignment:
cluster_name: cluster_grpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 10000
P.S 2021.03.19
Here's what else I found out.
When I request from the ingress host, I get the above failure, but when I request from the service, I get a normal response!
My scenario - one example-app service exposed through Nodeport in K8s cluster .
The service is having one envoy side car . OPA is running as separate service on a same node.
My app-deployment specs -
apiVersion: apps/v1
kind: Deployment
metadata:
name: example-app
labels:
app: example-app
spec:
replicas: 1
selector:
matchLabels:
app: example-app
template:
metadata:
labels:
app: example-app
spec:
initContainers:
- name: proxy-init
image: openpolicyagent/proxy_init:v5
# Configure the iptables bootstrap script to redirect traffic to the
# Envoy proxy on port 8000, specify that Envoy will be running as user
# 1111, and that we want to exclude port 8282 from the proxy for the
# OPA health checks. These values must match up with the configuration
# defined below for the "envoy" and "opa" containers.
args: ["-p", "8000", "-u", "1111"]
securityContext:
capabilities:
add:
- NET_ADMIN
runAsNonRoot: false
runAsUser: 0
containers:
- name: app
image: openpolicyagent/demo-test-server:v1
ports:
- containerPort: 8080
- name: envoy
image: envoyproxy/envoy:v1.14.4
securityContext:
runAsUser: 1111
volumeMounts:
- readOnly: true
mountPath: /config
name: proxy-config
args:
- "envoy"
- "--config-path"
- "/config/envoy.yaml"
volumes:
- name: proxy-config
configMap:
name: proxy-config
My Envoy specs(loaded through config maps) -
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 8000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: service
http_filters:
- name: envoy.filters.http.ext_authz
typed_config:
"#type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
grpc_service:
envoy_grpc:
cluster_name: opa
timeout: 0.250s
- name: envoy.filters.http.router
typed_config: {}
clusters:
- name: service
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
load_assignment:
cluster_name: service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 8080
- name: opa
connect_timeout: 0.250s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: opa
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: opa
port_value: 9191
admin:
access_log_path: "/dev/null"
address:
socket_address:
address: 0.0.0.0
port_value: 8001
OPA deployment specs (exposed through nodeport as a service):-
apiVersion: apps/v1
kind: Deployment
metadata:
name: opa
labels:
app: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
- name: opa
image: openpolicyagent/opa:latest-envoy
ports:
- name: http
containerPort: 8181
securityContext:
runAsUser: 1111
args:
- "run"
- "--server"
- "--log-level=info"
- "--log-format=json-pretty"
- "--set=decision_logs.console=true"
- "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
- "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
- "--ignore=.*"
- "/policy/policy.rego"
volumeMounts:
- readOnly: true
mountPath: /policy
name: opa-policy
volumes:
- name: opa-policy
configMap:
name: opa-policy
Sample policy.rego (loaded through configmap)-
package envoy.authz
import input.attributes.request.http as http_request
default allow = false
token = {"valid": valid, "payload": payload} {
[_, encoded] := split(http_request.headers.authorization, " ")
[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}
allow {
is_token_valid
action_allowed
}
is_token_valid {
token.valid
now := time.now_ns() / 1000000000
token.payload.nbf <= now
now < token.payload.exp
}
action_allowed {
http_request.method == "GET"
token.payload.role == "guest"
glob.match("/people*", [], http_request.path)
}
action_allowed {
http_request.method == "GET"
token.payload.role == "admin"
glob.match("/people*", [], http_request.path)
}
action_allowed {
http_request.method == "POST"
token.payload.role == "admin"
glob.match("/people", [], http_request.path)
lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)
}
While calling REST API getting below response -
curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://$SERVICE_URL/people
HTTP/1.1 403 Forbidden
date: Sat, 13 Feb 2021 08:50:29 GMT
server: envoy
content-length: 0
OPA decision logs -
{
"addrs": [
":8181"
],
"diagnostic-addrs": [],
"level": "info",
"msg": "Initializing server.",
"time": "2021-02-13T08:48:27Z"
}
{
"level": "info",
"msg": "Starting decision logger.",
"plugin": "decision_logs",
"time": "2021-02-13T08:48:27Z"
}
{
"addr": ":9191",
"dry-run": false,
"enable-reflection": false,
"level": "info",
"msg": "Starting gRPC server.",
"path": "",
"query": "data.envoy.authz.allow",
"time": "2021-02-13T08:48:27Z"
}
What I am doing wrong here ? there is nothing in decision logs for my rest call to the API . OPA policy should invoke through envoy filter which is not happening.
Please help .