Hi I am trying to add built-in OpenShift(v4.8) prometheus data source to a local grafana server. I have given basic auth with username and password and as of now I have enabled skip tls verify also. Still I'm getting this error
Prometheus URL = https://prometheus-k8s-openshift-monitoring.apps.xxxx.xxxx.xxxx.com
this is the grafana log
logger=tsdb.prometheus t=2022-04-12T17:35:23.47+0530 lvl=eror msg="Instant query failed" query=1+1 err="client_error: client error: 403"
logger=context t=2022-04-12T17:35:23.47+0530 lvl=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=10.100.95.27 time_ms=36 size=65 referer=https://grafana.xxxx.xxxx.com/datasources/edit/6TjZwT87k
You cannot authenticate to the OpenShift prometheus instance using basic authentication. You need to authenticate using a bearer token, e.g. one obtained from oc whoami -t:
curl -H "Authorization: Bearer $(oc whoami -t)" -k https://prometheus-k8s-openshift-monitoring.apps.xxxx.xxxx.xxxx.com/
Or from a ServiceAccount with appropriate privileges:
secret=$(oc -n openshift-monitoring get sa prometheus-k8s -o jsonpath='{.secrets[1].name}')
token=$(oc -n openshift-monitoring get secret $secret -o jsonpath='{.data.token}' | base64 -d)
curl -H "Authorization: Bearer $token" -k https://prometheus-k8s-openshift-monitoring.apps.xxxx.xxxx.xxxx.com/
Related
You can simply interact with K8s using its REST API. For example to get pods:
curl http://IPADDR/api/v1/pods
However I can't find any example of authentication based only on curl or REST. All the examples show the usage of kubectl as proxy or as a way to get credentials.
If I already own the .kubeconfig, and nothing else, is there any way to send the HTTP requests directly (e.g. with a token) without using kubectl?
The kubeconfig file you download when you first install the cluster includes a client certificate and key. For example:
clusters:
- cluster:
certificate-authority-data: ...
server: https://api.cluster1.ocp.virt:6443
name: cluster1
contexts:
- context:
cluster: cluster1
user: admin
name: admin
current-context: admin
preferences: {}
users:
- name: admin
user:
client-certificate-data: ...
client-key-data: ...
If you extract the client-certificate-data and client-key-data to
files, you can use them to authenticate with curl. To extract the
data:
$ yq -r '.users[0].user."client-certificate-data"' kubeconfig | base64 -d > cert
$ yq -r '.users[0].user."client-key-data"' kubeconfig | base64 -d >
key
And then using curl:
$ curl -k --cert cert --key key \
'https://api.cluster1.ocp.virt:6443/api/v1/namespaces/default/pods?limit=500'
{
"kind": "PodList",
"apiVersion": "v1",
"metadata": {
"resourceVersion": "22022"
},
"items": []
Alternately, if your .kubeconfig has tokens in it, like this:
[...]
users:
- name: your_username/api-clustername-domain:6443
user:
token: sha256~...
Then you can use that token as a bearer token:
$ curl -k https://api.mycluster.mydomain:6443/ -H 'Authorization: Bearer sha256~...'
...but note that those tokens typically expire after some time, while the certificates should work indefinitely (unless they are revoked somehow).
I am getting a connection refused response from the server when i try to hit the the endpoint via the proxy i have created on kong.
The curl command i am using to hit the proxy -
curl --location --request GET 'http://localhost:8000/listProducts/'
--header 'Accept: application/json'
--header 'Authorization: Bearer token'
to get the token I use the following curl -
curl --location --request POST 'http://localhost:8180/auth/realms/experimental/protocol/openid-connect/token'
--header 'Content-Type: application/x-www-form-urlencoded'
--data-urlencode 'username=username'
--data-urlencode 'password=password'
--data-urlencode 'grant_type=password'
--data-urlencode 'client_id=myapp'
The Client protocol is open-connect and the access type is public.
The config i have done in the Oidc plugin:
consumer:
response type:
code:
introspection endpoint: http://{MyHostIP}:8180/auth/realms/experimental/protocol/openid-connect/token/introspect
filters:
bearer only: yes
ssl verify: no
session secret:
introspection endpoint auth method:
realm: experimental
redirect after logout uri: /
scope: openid
token endpoint auth method:
client_secret_post:
logout path: /logout
client id: kong
discovery: https://{MyHostIP}:8180/auth/realms/master/.well-known/openid-configuration
client secret: myClientSecret
recovery page path:
redirect uri path:
Thanks in advance
How did u deployed Keycloak? I see 2 points:
Your discovery endpoint is https
Your introspection endpoint is just http
Also, if you are using Docker to deploy Kong + Keycloak, go to your hosts file and add a new line with your local ip with MyHostIP.
sudo nano /etc/hosts
Then add
your.ip.address. keycloak-host
Update the docker-compose file
kong:
build:
context: kong/
dockerfile: Dockerfile
extra_hosts:
- "Keycloak-host:your.ip.address"
Now configure your introspection and discovery url using the keycloak-host
Ex: http://keycloak-host:8180/auth/realms/master/protocol/openid-connect/token/introspect
If you need a functional example about Kong + openID + Keycloak, check this repo.
I used Kubernetes document to create a request for user certificate via API-server.
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: myuser
spec:
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
I generated the certificate, created the kubeconfig file and created the necessary role/rolebindings successfully. However, when I try to access the cluster, I get the below error. I am quite sure that the issue is with the above yaml definition; but could not figure out.
users error: You must be logged in to the server (Unauthorized)
Any idea please?
Seems, the issue is with the "spec" part. It is user authentication not server authentication. Hence, "server auth" should be client auth.
spec:
request: $(cat server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- client auth
I want to automate kubectl and helm commands using Ansible. The target machine is configured properly, so that both works on the cli in a manual shell (e.g. kubectl get nodes or helm list). But when trying to make any API calls like get the server version
- name: List charts
shell: kubectl version -v=8
It breaks with a Forbidden error. The verbose logging doesn't give me much more details:
fatal: [127.0.0.1]: FAILED! => changed=true
cmd: kubectl version -v=10
delta: '0:00:00.072452'
end: '2020-02-27 15:22:36.227928'
msg: non-zero return code
rc: 255
start: '2020-02-27 15:22:36.155476'
stderr: |-
I0227 15:22:36.224517 27321 loader.go:359] Config loaded from file /home/user/.kube/config
I0227 15:22:36.225211 27321 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.3 (linux/amd64) kubernetes/a452946" 'https://k8smaster01:6443/version?timeout=32s'
I0227 15:22:36.225975 27321 round_trippers.go:405] GET https://k8smaster01:6443/version?timeout=32s in 0 milliseconds
I0227 15:22:36.225986 27321 round_trippers.go:411] Response Headers:
I0227 15:22:36.226062 27321 helpers.go:219] Connection error: Get https://k8smaster01:6443/version?timeout=32s: Forbidden
F0227 15:22:36.226080 27321 helpers.go:119] Unable to connect to the server: Forbidden
stderr_lines: <omitted>
stdout: 'Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}'
stdout_lines: <omitted>
However, when sending a manual request to those API url like this
- name: Test master connection
shell: curl -k https://k8smaster01:6443/version?timeout=32s
It works:
stderr_lines: <omitted>
stdout: |-
{
"major": "1",
"minor": "11",
"gitVersion": "v1.11.3",
"gitCommit": "a4529464e4629c21224b3d52edfe0ea91b072862",
"gitTreeState": "clean",
"buildDate": "2018-09-09T17:53:03Z",
"goVersion": "go1.10.3",
"compiler": "gc",
"platform": "linux/amd64"
}
Why API calls with kubectl doesn't work when executed with Ansible?
I'm behind a proxy server, but k8smaster01 is set in no_proxy. Ansible got it, I printed $no_proxy in the task for testing purpose.
For curl I used -k since its a self signed cert from k8s. This sould harm kubectl (which itself works when not running from Ansible). It also doesn't work when calling kubectl --insecure-skip-tls-verify=true get node with Ansible.
I tried to unset the env variables from the proxy (since the proxy is required just for internet access) by setting empty environment variables:
- name: Kubectl test
become: false
shell: kubectl get no -v=10
environment:
http_proxy:
https_proxy:
no_proxy:
HTTP_PROXY:
HTTPS_PROXY:
NO_PROXY:
This was a bad idea, since curl (which seems to be used inside kubectl) parse this to None and fail. Strangely, kubectl failed with an dns error:
skipped caching discovery info due to Get https://k8smaster01:6443/api?timeout=32s: proxyconnect tcp: dial tcp: lookup None on 127.0.0.53:53: server misbehaving
Found out that the main problem was that I set NO_PROXY=$no_proxy in /etc/environment, where no_proxy contains the hostname k8smaster01. Since /etc/environment doesn't resolve bash variables, the uppercase NO_PROXY just contains $no_proxy as string. So it was enough to replace NO_PROXY=$no_proxy by the corresponding value (e.h. NO_PROXY=k8smaster01).
It wasn't an issue before, because most applications seems to follow the Linux specification of using lowercase environment variables for proxy usage.
On a local cluster (server listening on https://0.0.0.0:<any port>), put 0.0.0.0/8 in NO_PROXY, else kubectl (verify with kubectl cluster-info) will try to use your configured proxy.
I wanna have an ingress for all my service in the k8s, and give the ingress a basic auth. But for auth rotation, I want to support a secondary auth for user so the endpoint can be reached when they re-generate the primary key.
I currently can follow this guide to set up an ingress with single basic auth.
Adapting the guide, you can put multiple usernames and passwords in the auth file you're using to generate the basic auth secret. Specifically, if you run the htpasswd command without the -c flag, so e.g. htpasswd <filename> <username> it will add an entry to the file rather than creating a new file from scratch:
$ htpasswd -c auth foo
New password: <bar>
Re-type new password: <bar>
Adding password for user foo
$ cat auth
foo:$apr1$isCec65Z$JNaQ0GJCpPeG8mR1gYsgM1
$ htpasswd auth user2
New password: <pass2>
Re-type new password: <pass2>
Adding password for user user2
$ cat auth
foo:$apr1$isCec65Z$JNaQ0GJCpPeG8mR1gYsgM1
user2:$apr1$.FsOzlqA$eFxym7flDnoDtymRLraA2/
If you've already created the secret in the first place via the given command:
$ kubectl create secret generic basic-auth --from-file=auth
You can then update the secret with this trick:
$ kubectl create secret generic basic-auth --from-file=auth\
--dry-run -o yaml | kubectl apply -f -
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
secret/basic-auth configured
You can confirm setting the secret worked:
$ kubectl get secret basic-auth -ojsonpath={.data.auth} | base64 -D
foo:$apr1$isCec65Z$JNaQ0GJCpPeG8mR1gYsgM1
user2:$apr1$.FsOzlqA$eFxym7flDnoDtymRLraA2/
Finally, you can test basic auth with both usernames and passwords is working:
$ curl http://<minikube_ip>/ -H 'Host: foo.bar.com' \
-s -w"%{http_code}" -o /dev/null
401
$ curl http://<minikube_ip>/ -H 'Host: foo.bar.com' \
-u 'wronguser:wrongpass' \
-s -w"%{http_code}" -o /dev/null
401
$ curl http://<minikube_ip>/ -H 'Host: foo.bar.com' \
-u 'foo:bar' \
-s -w"%{http_code}" -o /dev/null
200
$ curl http://<minikube_ip>/ -H 'Host: foo.bar.com' \
-u 'user2:pass2' \
-s -w"%{http_code}" -o /dev/null
200