Proxy K8S app delegating authentication of requests from other pods - rest

Background
I have a K8S cluster with a number of different pods that have their own specific service accounts, cluster roles, and cluster role bindings, so that they can execute various read/write requests directly with the K8S REST API. There are some complicated requests that can be issued, and I'd like to make a function to wrap the complex logic. However, the various services in the cluster are written in multiple (i.e. 6+) programming languages, and there does not (yet) seem to be a trivial way to allow all these services to directly re-use this code.
I'm considering creating a "proxy" micro-service, that exposes its own REST API, and issues the necessary requests and handles the "complex logic" on behalf of the client.
Problem
The only problem is that, with the current deployment model, a client could request that the proxy micro-service execute an HTTP request that the client itself isn't authorized to make.
Question
Is there a trivial/straightforward way for one pod, for example, to identify the client pod, and execute some kind of query/result-of-policy operation (i.e. by delegating the authentication to the K8S cluster authentication mechanism itself) to determine if it should honor the request from the client pod?

Kubernetes Authentication model represents a way how the particular user or service account can be entitled in k8s cluster, however Authorization methods determine whether initial request from the cluster visitor, aimed to do some action on cluster resources/objects, has sufficient permissions to make that possible.
Due to the fact that you've used specific service accounts per each Pod entire the cluster and granting them specific RBAC rules, it might be possible to use SelfSubjectAccessReview API in order to inspect requests to k8s REST API and determine whether the client's Pod service account has appropriate permission to perform any action on target's Pod namespace.
That can be achievable using kubectl auth can-i subcommand by submitting essential information for user impersonation.
I assume that you might also be able to query k8s authorization API group within HTTP request schema and then parse structured data from JSON/YAML format, like in the example below:
Regular kubectl auth can-i command to check whether default SA can retrieve data about Pods in default namespace:
kubectl auth can-i get pod --as system:serviceaccount:default:default
Equivalent method via HTTP call to k8s REST API using JSON type of content within Bearer Token authentication:
curl -k \
-X POST \
-d #- \
-H "Authorization: Bearer $MY_TOKEN" \
-H 'Accept: application/json' \
-H "Impersonate-User: system:serviceaccount:default:default" \
-H 'Content-Type: application/json' \
https://<API-Server>/apis/authorization.k8s.io/v1/selfsubjectaccessreviews <<'EOF'
{
"kind": "SelfSubjectAccessReview",
"apiVersion": "authorization.k8s.io/v1",
"spec":{"resourceAttributes":{"namespace":"default","verb":"get","resource":"pods"}}
}
EOF
Output:
.... "status": {
"allowed": true,
"reason": "RBAC: allowed by RoleBinding ....

Related

Accessing Kubernetes APIs from local machine

I wish to access Kubernetes APIs from my local machine. I'm trying to get list of pods using kubernetes Rest APIs.
I've created a kubernetes cluster and some pods on Google Cloud.
On my local Windows machine, I've installed gcloud sdk and kubectl component with it.
I connected to my cluster using:
gcloud container clusters get-credentials my-cluster --region us-central1 --project my-project
I can get the list of pods using kubectl get pods
Although, I want to get pods list using kubernetes Rest APIs.
GET https://kubernetes.default/api/v1/namespaces/default/pods
Authorization: Bearer my_access_token
But I think the request is not going through.
In Postman, I get the error:
Error: tunneling socket could not be established, cause=socket hang up
Or in Python using requests library (from my local machine), I get the error
HTTPSConnectionPool(host='kubernetes.default', port=443): Max retries exceeded with url: /api/v1/namespaces/default/pods (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000277DCD04D90>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
What am I missing here?
The endpoint https://kubernetes.default only works if you want to access Kubernetes REST API from inside the cluster i.e from another pod. For accessing Kubernetes REST API from outside the kubernetes cluster i.e from your local machine you need to use the API server IP or host which is externally accessible i.e the one which is there in kubeconfig file.
For accessing it from outside the kubernetes cruster i.e from your local machine there are three ways referring from the docs here
Run kubectl in proxy mode (recommended). This method is recommended, since it uses the stored apiserver location and verifies the identity of the API server using a self-signed cert. No man-in-the-middle (MITM) attack is possible using this method.
kubectl proxy --port=8080 &
curl http://localhost:8080/api/v1/namespaces/default/pods
It is possible to avoid using kubectl proxy by passing an authentication token directly to the API server, like this:
Check all possible clusters, as your .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="some_server_name"
Point to the API server referring the cluster name
APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(#.name==\"$CLUSTER_NAME\")].cluster.server}")
Gets the token value
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(#.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode)
Explore the API with TOKEN
curl -X GET $APISERVER/api/v1/namespaces/default/pods --header "Authorization: Bearer $TOKEN" --insecure
Using client library
To use Python client, run the following command: pip install kubernetes See Python Client Library page for more installation options.
The Python client can use the same kubeconfig file as the kubectl CLI does to locate and authenticate to the API server. See this example:
from kubernetes import client, config
config.load_kube_config()
v1=client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
You can also do it the way you are doing without using kubeconfig file but it's more work and you need to use the kubernetes API Server IP or hostname from the kubeconfig file.
using below kubectl command start a proxy to the Kubernetes API server:
kubectl proxy --port=8080
Get the API versions:
curl http://localhost:8080/api/
The output should look similar to this:
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.0.2.15:8443"
}
]
}
Your api server address is not correct for external REST access.
Get the address like this.
kubectl config view
Find your cluster name in the list and get the APi.
Here is the cURL (without the real IP or the token) which worked in my local pc.
curl --location --request GET 'https://nnn.nnn.nnnn.nnn/api/v1/namespaces/develop/pods' \
--header 'Authorization: bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
If you run in POSTMAN, you might have to disable certificate verification.

Best practice for managing the cluster from the outside

I have a best-practice question:
I have a Django application running somewhere (non-k8s) where my end-user accounts are tracked. Separately, I have a k8s cluster that has a service for every user. When a new user signs up in Django, a new service should be created in the cluster.
What is the best practice for doing this? Two options I see are:
Have a long-lived service in the cluster, something like user-pod-creator, which exposes an API to the Django side, allowing it to ask for a pod to be created.
Give the Django permissions to use the cluster's API directly; have it create (and delete) pods as it wishes.
Intuitively I prefer the first because of the separation of concerns it creates and because of security reasons. But the second would give a lot of flexibility to the Django app so that it can not only create and delete pods, but it can have more visibility into the cluster if need be with direct API calls, instead of me having to expose new API endpoints in user-pod-creator or some other service.
Option 2 is a valid approach and can be solved with a service account.
Create a ServiceAccount for your Django app:
kubectl create serviceaccount django
This ServiceAccount points to a Secret, and this Secret contains a token.
Find out the Secret associated with the ServiceAccount:
kubectl get serviceaccount django -o yaml
Get the token in the Secret:
kubectl get secret django-token-d2tz4 -o jsonpath='{.data.token}'
Now you can use this token as an HTTP bearer token in the Kubernetes API requests from your Django app outside the cluster.
That is, include the token in the HTTP Authorization header of the Kubernetes API requests like this:
Authorization: Bearer <TOKEN>
In this way, your request passes the authentication stage in the API server. However, the service account has no permissions yet (authorisation).
You can assign the required permissions to your service account with Roles and RoleBindings:
kubectl create role django --verb <...> --resource <...>
kubectl create rolebinding django --role django --serviceaccount django
Regarding security: grant only the minimum permissions needed to the service account (principle of least privileges), and if you think someone stole the service account token, you can just delete the service account with kubectl delete serviceaccount django to invalidate the token.
See also here for an example of the presented approach. Especially:
Service account bearer tokens are perfectly valid to use outside the cluster and can be used to create identities for long standing jobs that wish to talk to the Kubernetes API.

Kubernetes : How to scale a deployment from another service/pod?

I have 2 services. Service A and Service B. They correspond to deployments dA and dB.
I set up my cluster and start both services/deployments. Service A is reachable from the external world. External World --> Service A <--> Service B.
How can I scale dB (change replicaCount and run kubectl apply OR kubectl scale) from within Service A that is responding to a user request.
For example, if a user that is being served by Service A wants some extra resource in my app, I would like to provide that by adding an extra pod to dB. How do I do this programatically?
Every Pod, unless it opts out, has a ServiceAccount token injected into it, which enables it to interact with the kubernetes API according to the Role associated with the ServiceAccount
Thus, one can use any number of kubernetes libraries -- most of which are "in cluster" aware, meaning they don't need any further configuration to know about that injected ServiceAccount token and how to use it -- to issue scale events against any resource the ServiceAccount's Role is authorized to use
You can make it as simple or as complex as you'd like, but the tl;dr is akin to:
curl --cacert /var/run/secrets/kubernetes.io/ca.crt \
--header "Accept: application/json" \
--header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/token)" \
https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}/api/v1/namespaces

Accessing the Kubernetes REST end points using bearer token

Requirement: We need to access the Kubernetes REST end points from our java code. Our basic operations using the REST end points are to Create/Update/Delete/Get the deployments.
We have downloaded the kubectl and configured the kubeconfig file of the cluster in our Linux machine.
We can perform operations in that cluster using the kubectl. We got the bearer token of that cluster running the command 'kubectl get pods -v=8'. We are using this bearer token in our REST end points to perform our required operations.
Questions:
What is the better way to get the bearer token?
Will the bearer token gets change during the lifecycle of the cluster?
follow this simple way
kubectl proxy --port=8080 &
curl http://localhost:8080/api/
from java code use the below approach
# Check all possible clusters, as you .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
# Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="some_server_name"
# Point to the API server refering the cluster name
APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(#.name==\"$CLUSTER_NAME\")].cluster.server}")
# Gets the token value
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(#.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 -d)
# Explore the API with TOKEN
curl -X GET $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
Q: What is the better way to get the bearer token?
A: Since you have configured access to the cluster, you might use
kubectl describe secrets
Q: Will the bearer token gets change during the lifecycle of the cluster?
A: Static tokens do not expire.
Please see Accessing Clusters and Authenticating for more details.
If your token comes from a ServiceAccount, best way should be to add this Service Account to the deployment/statefull or daemonSet where your java runs.This will lead to tokne being written in your pod in "/var/run/secrets/kubernetes.io/serviceaccount/token". Config is explained here
Most of kubernetes client library knows how to discover a service account linked to a running context and "automaticaly" use it. Not an expert be it seems yes for Java client see : github
This token will not be automaticaly refreshed by kubernetes unless someone force a rolling.

Kubernetes get endpoints

I have a set of pods providing nsqlookupd service.
Now I need each nsqd container to have a list of nsqlookupd servers to connect to (while service will point to different every time) simultaneously. Something similar I get with
kubectl describe service nsqlookupd
...
Endpoints: ....
but I want to have it in a variable within my deployment definition or somehow from within nsqd container
Sounds like you would need an extra service running either in your nsqd container or in a separate container in the same pod. The role of that service would be to pole the API regularly in order to fetch the list of endpoints.
Assuming that you enabled Service Accounts (enabled by default), here is a proof of concept on the shell using curl and jq from inside a pod:
# Read token and CA cert from Service Account
CACERT="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
# Replace the namespace ("kube-system") and service name ("kube-dns")
ENDPOINTS=$(curl -s --cacert "$CACERT" -H "Authorization: Bearer $TOKEN" \
https://kubernetes.default.svc/api/v1/namespaces/kube-system/endpoints/kube-dns \
)
# Filter the JSON output
echo "$ENDPOINTS" | jq -r .subsets[].addresses[].ip
# output:
# 10.100.42.3
# 10.100.67.3
Take a look at the source code of Kube2sky for a good implementation of that kind of service in Go.
Could be done with a StatefuSet. Stable names + stable storage