I have added the following command-line arguments to kube-apiserver to enable audit logging:
- --audit-log-path=/tmp/k8s-audit.log
- --audit-policy-file=/etc/kubernetes/audit.yaml
- --audit-log-maxage=1
- --audit-log-maxsize=100
- --audit-log-maxbackup=1
The contents of /etc/kubernetes/audit.yaml is:
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "ResponseStarted"
- "ResponseComplete"
rules:
- level: RequestResponse
I have run a command with verbose logging, so that I can see the request body:
$ kubectl --v=10 uncordon cluster-worker2
And the kubectl command logs the request body as follows:
I0328 09:00:07.591869 47228 request.go:942] Request Body: {"spec":{"unschedulable":null}}
But I don't see this request body anywhere in the audit log file on the kubernetes server. What's wrong with my configuration?
The request is actually only logged in the ResponseComplete stage, somewhat unexpectedly. Even though Kubernetes could theoretically log the request as soon as it receives it, it doesn't.
So it's necessary to remove the ResponseComplete line from the omitstages in the policy configuration file (audit.yaml).
Related
What do you use instead og kubectl get ComponentStatus?
kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
Yes, this API is deprecated and as it provided status of etcd, kube-scheduler, and kube-controller-manager components, which we can get through kubectl or using lives endpoint.
so you can try
kubectl get --raw='/readyz?verbose'
#local cluster
curl -k https://localhost:6443/livez?verbose
output
[+]ping ok
[+]log ok
[+]etcd ok
[+]informer-sync ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]shutdown ok
readyz check passed
The current state of this API is problematic, and requires reversing the actual data flow (it requires the API server to call to its clients), and is not functional across deployment topologies.
It should be clearly marked as deprecated.
Mark componentstatus as deprecated
The Kubernetes API server provides 3 API endpoints (healthz, livez and readyz) to indicate the current status of the API server. The healthz endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific livez and readyz endpoints instead.
using-api-health-checks
I tried in incognito as wellbut same issue exists.
Currently I have added in server-deployment.yaml
args: - server - --auth-mode - sso
And in values.yaml
sso:
# #SSO configuration when SSO is specified as a server auth mode.
# #All the values are requied. SSO is activated by adding --auth-mode=sso
# #to the server command line.
#
# #The root URL of the OIDC identity provider.
issuer: http://<keycloak_ip>/auth/realms/demo
# #Name of a secret and a key in it to retrieve the app OIDC client ID from.
clientId:
name: argo
key: client-id
# #Name of a secret and a key in it to retrieve the app OIDC client secret from.
clientSecret:
name: "argo-server-sso"
key: client-secret
# # The OIDC redirect URL. Should be in the form /oauth2/callback.
redirectUrl: http:///argo/oauth2/callback
And in keycloak ui , I have created client and client credentials.
kubectl create secret generic "argo-server-sso" --from-literal=client-secret=9a9c60ba-647d-480c-b6fa-82c19caad26a
kubectl create secret generic "argo" --from-literal=client-id=argo
After hitting the argo server url,manually I need to click on login option but after that keycloak page appears and then again a popup will come "Failed to login:Unauthorized"
Server logs:
kubectl logs argo-server-5c7f8c5cbb-9fcqk
time="2021-01-20T12:06:26.876Z" level=info authModes="[sso]" baseHRef=/ managedNamespace= namespace=default secure=false
time="2021-01-20T12:06:26.877Z" level=warning msg="You are running in insecure mode. Learn how to enable transport layer security: https://argoproj.github.io/argo/tls/"
time="2021-01-20T12:06:26.877Z" level=info msg="config map" name=argo-workflow-controller-configmap
time="2021-01-20T12:06:28.318Z" level=info msg="SSO configuration" clientId="{{argo} client-id }" issuer="http://10.xx.xx.xx:xxxx/auth/realms/demo" redirectUrl="http://xx/argo/oauth2/callback"
time="2021-01-20T12:06:28.318Z" level=info msg="SSO enabled"
time="2021-01-20T12:06:28.322Z" level=info msg="Starting Argo Server" instanceID= version=v2.12.2
time="2021-01-20T12:06:28.322Z" level=info msg="Creating event controller" operationQueueSize=16 workerCount=4
time="2021-01-20T12:06:28.323Z" level=info msg="Argo Server started successfully on http://localhost:2746"
time="2021-01-20T12:07:21.990Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2021-01-20T12:07:21Z" grpc.time_ms=0.379 span.kind=server system=grpc
time="2021-01-20T12:07:22.009Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2021-01-20T12:07:22Z" grpc.time_ms=0.075 span.kind=server system=grpc
I integrated ArgoCD with Keycloak successfully.
You have 1 clear/visible issue: Yaml indentation is wrong
make sure you keep the right indentation as per default values in helm chart :
https://github.com/argoproj/argo-helm/blob/1aea2c41798972ff0077108f926bb9095f3f9deb/charts/argo/values.yaml#L255-L283
Accordingly, your values should be:
( assuming your argo is serving with hostname workflows.company.com )
server:
extraArgs:
- --auth-mode=sso
sso:
issuer: http://<keycloak_ip>/auth/realms/demo
clientId:
name: argo
key: client-id
clientSecret:
name: "argo-server-sso"
key: client-secret
redirectUrl: https://workflows.company.com/argo/oauth2/callback
From keycloak side now, & under your client, make sure you fill-in the Valid Redirect URL as per your ingress hostname :
I am spinning up a container (pod/Job) from a GKE.
I have set up the appropriate Service Account on the cluster's VMs.
Therefore, when I manually perform a curl to a specific CloudRun service endpoint, I can perform the request (and get authorized and have 200 in my response)
However, when I try to automate this by setting an image to run in a Job as follows, I get 401
- name: pre-upgrade-job
image: "google/cloud-sdk"
args:
- curl
- -s
- -X
- GET
- -H
- "Authorization: Bearer $(gcloud auth print-identity-token)"
- https://my-cloud-run-endpoint
Here are the logs on Stackdriver
{
httpRequest: {
latency: "0s"
protocol: "HTTP/1.1"
remoteIp: "gdt3:r787:ff3:13:99:1234:avb:1f6b"
requestMethod: "GET"
requestSize: "313"
requestUrl: "https://my-cloud-run-endpoint"
serverIp: "212.45.313.83"
status: 401
userAgent: "curl/7.59.0"
}
insertId: "29jdnc39dhfbfb"
logName: "projects/my-gcp-project/logs/run.googleapis.com%2Frequests"
receiveTimestamp: "2019-09-26T16:27:30.681513204Z"
resource: {
labels: {
configuration_name: "my-cloud-run-service"
location: "us-east1"
project_id: "my-gcp-project"
revision_name: "my-cloudrun-service-d5dbd806-62e8-4b9c-8ab7-7d6f77fb73fb"
service_name: "my-cloud-run-service"
}
type: "cloud_run_revision"
}
severity: "WARNING"
textPayload: "The request was not authorized to invoke this service. Read more at https://cloud.google.com/run/docs/securing/authenticating"
timestamp: "2019-09-26T16:27:30.673565Z"
}
My question is how can I see if an "Authentication" header does reach the endpoint (the logs do not enlighten me much) and if it does, whether it is appropriately rendered upon image command/args invocation.
In your job you use this container google/cloud-sdk which is a from scratch installation of gcloud tooling. It's generic, without any customization.
When you call this $(gcloud auth print-identity-token) you ask for the identity token of the service account configured in the gcloud tool.
If we put together this 2 paragraphs, you want to generate an identity token from a generic/blank installation of gcloud tool. By the way, you don't have defined service account in your gcloud and your token is empty (like #johnhanley said).
For solving this issue, add an environment variable like this
env:
- GOOGLE_APPLICATION_CREDENTIAL=<path to your credential.json>
I don't know where is your current credential.json of your running environment. Try to perform an echo of this env var to find it and forward it correctly to your gcloud job.
If you are on compute engine or similar system compliant with metadata server, you can get a correct token with this command:
curl -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=<URL of your service>"
UPDATE
Try to run your command outside of the gcloud container. Here the update of your job
- name: pre-upgrade-job
image: "google/cloud-sdk"
entrypoint: "bash"
args:
- -c
- "curl -s -X GET -H \"Authorization: Bearer $(gcloud auth print-identity-token)\" https://my-cloud-run-endpoint"
Not sure that works. Let me know
In your Job, gcloud auth print-identity-token likely does not return any tocken.
The reason is that locally, gcloud uses your identity to mint a token, but in a Job, you are not logged into gcloud.
Problem
Does anyone know how to configure bootstrap.yml to tell Spring Cloud Vault to go to the correct path for k2 v2 and not try other paths first?
Details
I can successfully connect to my Vault, running k2 v2, but Spring Cloud will always try to connect to paths in the vault that don't exist, throwing a 403 on startup.
Status 403 Forbidden [secret/application]: permission denied; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden
The above path, secret/application, doesn't exist because k2 v2 puts data in the path. For example: secret/data/application.
This isn't a show-stopper because Spring Cloud Vault does check other paths, including the correct one that has the data item in the path, but the fact a meaningless 403 is thrown during startup is like a splinter in my mind.
Ultimately, it does try the correct k2 v2 path
2019-03-18 12:22:46.611 INFO 77685 --- [ restartedMain] b.c.PropertySourceBootstrapConfiguration : Located property source: CompositePropertySource {name='vault', propertySources=[LeaseAwareVaultPropertySource {name='secret/data/my-app'}
My configuration
spring.cloud.vault:
kv:
enabled: true
backend: secret
profile-separator: '/'
default-context: my-app
application-name: my-app
host: localhost
port: 8200
scheme: http
authentication: TOKEN
token: my-crazy-long-token-string
Thanks for your help!
Add the following lines in your bootstrap.yml, this disables the generic backend
spring.cloud.vault:
generic:
enabled: false
for more information https://cloud.spring.io/spring-cloud-vault/reference/html/#vault.config.backends.generic
In addition to the accepted answer it's important to turn off (or just remove) fail-fast option:
spring.cloud.vault:
fail-fast: false
spring.cloud.vault.generic.enabled is deprecated in spring-cloud 3.0.0, but the 403 error is still there. To disable the warning (by telling spring to use the exact context), this is what I used:
spring:
config:
import: vault://
application:
name: my-application
cloud:
vault:
host: localhost
scheme: http
authentication: TOKEN
token: my-crazy-long-token-string
kv:
default-context: my-application
Other configs were set to default (such as port = 8200, backend = secret, etc.)
is jsonPath supported in kubernetes http api ?
for ex; how the following translates to in http API ?
kubectl get pods -o=jsonpath='{.items[0]}'
It's not supported by the API, you would need to evaluate that jsonpath against the API response.
You can use verbose flag -v6 and above to see what API calls are actually being made
kubectl get pods -o=jsonpath='{.items[0]}' -v6 2>&1
Output:
I0805 11:16:51.632841 76333 loader.go:375] Config loaded from file: /Users/loganath/firetap/config-ctl1
I0805 11:16:53.666539 76333 round_trippers.go:444] GET https://10.x.x.x:6443/api/v1/namespaces/web/pods?limit=500 200 OK in 2021 milliseconds
I0805 11:16:54.901557 76333 table_printer.go:45] Unable to decode server response into a Table. Falling back to hardcoded types: attempt to decode non-Table object