keycloak and cockroachdb cloud - kubernetes

I tried to use keycloak db against crdb cloud. I have used https://github.com/codecentric/helm-charts/tree/master/charts/keycloak charts for deployment k8s. I create a db for the keycloak and give the above configuration to success connection. I use my values.yaml and added additional env var:
extraEnv: |
- name: DB_VENDOR
value: postgres
- name: DB_ADDR
value: xxxx.xxx.cockroachlabs.cloud
- name: DB_PORT
value: "xxx"
- name: DB_DATABASE
value: keycloak
- name: DB_USER_FILE
value: /secrets/db-creds/user
- name: DB_PASSWORD_FILE
value: /secrets/db-creds/password
- name: JDBC_PARAMS
value: sslmode=verify-ca&sslrootcert=/secrets/crdb-creds/xxx.crt
- name: JDBC_PARAMS_FILE
value: /secrets/crdb-creds/xxx.crt
and also
# Add additional volumes, e. g. for custom themes
extraVolumes: |
- name: crdb-creds
secret:
secretName: keycloak-crdb-creds
- name: db-creds
secret:
secretName: keycloak-db-creds
and mounting
# Add additional volumes mounts, e. g. for custom themes
extraVolumeMounts: |
- name: crdb-creds
mountPath: /secrets/crdb-creds
readOnly: true
- name: db-creds
mountPath: /secrets/db-creds
readOnly: true
So in theory there is no restriction for using cockroach the with postgres db vendor in keycloak(!). And I am going to give a try for this and actually it wasn't give an error but it restaring after while and keeping restarting same period. So it gives an :
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#3a612dd6
or
10:55:31,907 FATAL [org.keycloak.services] (ServerService Thread Pool -- 64) Error during startup: java.lang.IllegalStateException: Failed to retrieve lock
So my question is what is the variable for giving .crt path and is there any additional progress need to run this correctly

Related

Debezium with GCP Cloudsql postgresql

Google finally added the support for Cloudsql postgresql logical replication/decoding support.
What I am trying to do is to use Debezium to capture the changes on the database tables, and then publish the changes in json to Gcp pubsub.
I am using Debezium server with Gcp Pubsub Sink just to verify the connection is working, but keep getting following error message org.postgresql.util.PSQLException: ERROR: could not access file "decoderbufs": No such file or directory
So seems like it defaults the plugin to decoderbufs, however I did set the database.source.plugin.name in the environment variable. According to this doc https://hub.docker.com/r/debezium/server, I am using DATABASE_SOURCE_PLUGIN_NAME. Here is the snippet of my k8s manifest
serviceAccountName: cloudsql-client
containers:
- name: debezium-standalone-server-dv
image: debezium/server:1.6
env:
- name: DEBEZIUM_SOURCE_CONNECTOR_CLASS
value: io.debezium.connector.postgresql.PostgresConnector
- name: DEBEZIUM_SOURCE_DATABASE_HOSTNAME
value: localhost
- name: DEBEZIUM_SOURCE_DATABASE_PORT
value: "5432"
- name: DEBEZIUM_SOURCE_DATABASE_USER
value: cdc
- name: DEBEZIUM_SOURCE_DATABASE_PASSWORD
value: cdc
- name: DEBEZIUM_SOURCE_DATABASE_DBNAME
value: postgres
- name: DEBEZIUM_SINK_TYPE
value: pubsub
- name: DEBEZIUM_SINK_PUBSUB_PROJECT_ID
value: myproject
- name: DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME
value: data/offsets.dat
- name: DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS
value: "0"
- name: DEBEZIUM_SOURCE_DATABASE_SERVER_NAME
value: CDC_POC
- name: DATABASE_SOURCE_PLUGIN_NAME
value: wal2json
From Cloudsql, i have already turn on the logical decoding and add the REPLICATION attribute to the user, and i am able to "manually" create it by running SELECT * FROM pg_create_logical_replication_slot('test_slot', 'wal2json'); I can also see the changes by querying the replication slot SELECT * FROM pg_logical_slot_get_changes('test_slot', NULL, NULL);
so what am i missing here?
I am guessing it is a typo on the docker configuration documentation.
After i changed the configuration from DATABASE_SOURCE_PLUGIN_NAME to DEBEZIUM_SOURCE_PLUGIN_NAME it chooses the plugin correctly

Pathing Issue when Executing Airflow run & backfill commands on Kubernetes

Verions
Airflow: 1.10.7
Kubernetes: 1.14.9
Setup
Airflow is configured to use Kubernetes Executors; Normal operations work just fine;
Dags and logs are accessed via EFS volumes defined with PersistentVolume & PersistentVolumeClaim specs ;
I have the following k8s spec, which I want to run backfill jobs with;
apiVersion: v1
kind: Pod
metadata:
name: backfill-test
namespace: airflow
spec:
serviceAccountName: airflow-service-account
volumes:
- name: airflow-dags
persistentVolumeClaim:
claimName: airflow-dags
- name: airflow-logs
persistentVolumeClaim:
claimName: airflow-logs
containers:
- name: somename
image: myimage
volumeMounts:
- name: airflow-dags
mountPath: /usr/local/airflow/dags
readOnly: true
- name: airflow-logs
mountPath: /usr/local/airflow/logs
readOnly: false
env:
- name: AIRFLOW__CORE__EXECUTOR
value: KubernetesExecutor
- name: AIRFLOW__KUBERNETES__NAMESPACE
value: airflow
- name: AIRFLOW__CORE__DAGS_FOLDER
value: dags
- name: AIRFLOW__CORE__BASE_LOG_FOLDER
value: logs
# - name: AIRFLOW__KUBERNETES__DAGS_VOLUME_MOUNT_POINT
# value: /usr/local/airflow/dags
- name: AIRFLOW__KUBERNETES__DAGS_VOLUME_SUBPATH
value: dags
- name: AIRFLOW__KUBERNETES__LOGS_VOLUME_SUBPATH
value: logs
- name: AIRFLOW__KUBERNETES__DAGS_VOLUME_CLAIM
value: airflow-dags
- name: AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM
value: airflow-logs
- name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY
value: someimage_uri
- name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_TAG
value: latest
- name: AIRFLOW__KUBERNETES__WORKER_SERVICE_ACCOUNT_NAME
value: airflow-service-account
- name: AIRFLOW_HOME
value: usr/local/airflow
# command: ["sleep", "1h"]
command: ["airflow", "backfill",
"my_dag",
# # "--subdir", ".",
# "--local",
"--task_regex", "my_task_task",
"--start_date", "2020-07-01T00:00:00",
"--end_date", "2020-08-01T00:00:00"]
restartPolicy: Never
Problem
The issue with this seems to be some pathing problem when the task is added to the queue
When running the initial command, the cli finds the dag and associated task;
airflow#backfill-test:~$ airflow run my_dag my_task 2020-07-01T01:15:00+00:00 --local --raw --force
[2020-08-27 23:14:42,038] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2020-08-27 23:14:42,040] {dagbag.py:403} INFO - Filling up the DagBag from /usr/local/airflow/dags
Running %s on host %s <TaskInstance: my_dag.my_task 2020-07-01T01:15:00+00:00 [failed]> backfill-test
However, the task gets added to the queue with some weird path. Log of the actual task execution attempt below.
[2020-08-27 23:14:43,019] {taskinstance.py:867} INFO - Starting attempt 3 of 2
[2020-08-27 23:14:43,019] {taskinstance.py:868} INFO -
--------------------------------------------------------------------------------
[2020-08-27 23:14:43,043] {taskinstance.py:887} INFO - Executing <Task(PostgresOperator): my_task> on 2020-07-01T01:15:00+00:00
[2020-08-27 23:14:43,046] {standard_task_runner.py:52} INFO - Started process 191 to run task
[2020-08-27 23:14:43,085] {logging_mixin.py:112} INFO - [2020-08-27 23:14:43,085] {dagbag.py:403} INFO - Filling up the DagBag from /usr/local/airflow/dags/usr/local/airflow/my_dag.py
[2020-08-27 23:14:53,006] {logging_mixin.py:112} INFO - [2020-08-27 23:14:53,006] {local_task_job.py:103} INFO - Task exited with return code 1
Adding --subdir to the initial command doesn't actually get propagated to the task queue, and results in the same log output.

Google Cloud, Kubernetes and Cloud SQL proxy: default Compute Engine service account issue

I have Google Cloud projects A, B, C, D. They all use similar setup for Kubernetes cluster and deployment. Projects A,B and C have been build months ago. They all use Google Cloud SQL proxy to connect to Google Cloud SQL service. Now when recently I started setting up the Kubernetes for project D, I get following error visible in the Stackdriver logging:
the default Compute Engine service account is not configured with sufficient permissions to access the Cloud SQL API from this VM. Please create a new VM with Cloud SQL access (scope) enabled under "Identity and API access". Alternatively, create a new "service account key" and specify it using the -credential_file parameter
I have compared the difference between the Kubernetes cluster between A,B,C and D but they appear to be same.
Here is the deployment I am using
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: my-site
labels:
system: projectA
spec:
selector:
matchLabels:
system: projectA
template:
metadata:
labels:
system: projectA
spec:
containers:
- name: web
image: gcr.io/customerA/projectA:alpha1
ports:
- containerPort: 80
env:
- name: DB_HOST
value: 127.0.0.1:3306
# These secrets are required to start the pod.
# [START cloudsql_secrets]
- name: DB_USER
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: username
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: password
# [END cloudsql_secrets]
# Change <INSTANCE_CONNECTION_NAME> here to include your GCP
# project, the region of your Cloud SQL instance and the name
# of your Cloud SQL instance. The format is
# $PROJECT:$REGION:$INSTANCE
# [START proxy_container]
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command:
- sh
- -c
- /cloud_sql_proxy -instances=my-gcloud-project:europe-west1:databaseName=tcp:3306
- -credential_file=/secrets/cloudsql/credentials.json
# [START cloudsql_security_context]
securityContext:
runAsUser: 2 # non-root user
allowPrivilegeEscalation: false
# [END cloudsql_security_context]
volumeMounts:
- name: cloudsql-instance-credentials
mountPath: /secrets/cloudsql
readOnly: true
# [END proxy_container]
# [START volumes]
volumes:
- name: cloudsql-instance-credentials
secret:
secretName: cloudsql-instance-credentials
# [END volumes]
So it would appear that the default service account doesn't have enough permissions? Google Cloud doesn't allow enabling the Cloud SQL API when creating the cluster via Google Cloud console.
From what I have googled this issue some say that the problem was with the gcr.io/cloudsql-docker/gce-proxy image but I have tried newer versions but the same error still occurs.
I found solution to this problem and it was setting the service-account argument when creating the cluster. Note that I haven't tested what are the minimum required permissions for the new service account.
Here are the steps:
Create new service account, doesn't require API key. Name I used was "super-service"
Assign roles Cloud SQL admin, Compute Admin, Kubernetes Engine Admin, Editor to the new service account
Use gcloudto create the cluster like this using the new service account
gcloud container clusters create my-cluster \
--zone=europe-west1-c \
--labels=system=projectA \
--num-nodes=3 \
--enable-master-authorized-networks \
--enable-network-policy \
--enable-ip-alias \
--service-account=super-service#project-D.iam.gserviceaccount.com \
--master-authorized-networks <list-of-my-ips>
Then the cluster and the deployment at least was deployed without errors.

How to pass environmental variables in envconsul config file?

I read in the envconsul documentation this:
For additional security, tokens may also be read from the environment
using the CONSUL_TOKEN or VAULT_TOKEN environment variables
respectively. It is highly recommended that you do not put your tokens
in plain-text in a configuration file.
So, I have this envconsul.hcl file:
# the settings to connect to vault server
# "http://10.0.2.2:8200" is the Vault's address on the host machine when using Minikube
vault {
address = "${env(VAULT_ADDR)}"
renew_token = false
retry {
backoff = "1s"
}
token = "${env(VAULT_TOKEN)}"
}
# the settings to find the endpoint of the secrets engine
secret {
no_prefix = true
path = "secret/app/config"
}
However, I get this error:
[WARN] (view) vault.read(secret/app/config): vault.read(secret/app/config): Get $%7Benv%28VAULT_ADDR%29%7D/v1/secret/app/config: unsupported protocol scheme "" (retry attempt 1 after "1s")
As I understand it, it cannot do the variable substitution.
I tried to set "http://10.0.2.2:8200" and it works.
The same happens with the VAULT_TOKEN var.
If I hardcode the VAULT_ADDR, then I get this error:
[WARN] (view) vault.read(secret/app/config): vault.read(secret/app/config): Error making API request.
URL: GET http://10.0.2.2:8200/v1/secret/app/config
Code: 403. Errors:
* permission denied (retry attempt 2 after "2s")
Is there a way for this file to understand the environmental variables?
EDIT 1
This is my pod.yml file
---
apiVersion: v1
kind: Pod
metadata:
name: sample
spec:
serviceAccountName: vault-auth
restartPolicy: Never
# Add the ConfigMap as a volume to the Pod
volumes:
- name: vault-token
emptyDir:
medium: Memory
# Populate the volume with config map data
- name: config
configMap:
# `name` here must match the name
# specified in the ConfigMap's YAML
# -> kubectl create configmap vault-cm --from-file=./vault-configs/
name: vault-cm
items:
- key : vault-agent-config.hcl
path: vault-agent-config.hcl
- key : envconsul.hcl
path: envconsul.hcl
initContainers:
# Vault container
- name: vault-agent-auth
image: vault
volumeMounts:
- name: vault-token
mountPath: /home/vault
- name: config
mountPath: /etc/vault
# This assumes Vault running on local host and K8s running in Minikube using VirtualBox
env:
- name: VAULT_ADDR
value: http://10.0.2.2:8200
# Run the Vault agent
args:
[
"agent",
"-config=/etc/vault/vault-agent-config.hcl",
"-log-level=debug",
]
containers:
- name: python
image: myappimg
imagePullPolicy: Never
ports:
- containerPort: 5000
volumeMounts:
- name: vault-token
mountPath: /home/vault
- name: config
mountPath: /etc/envconsul
env:
- name: HOME
value: /home/vault
- name: VAULT_ADDR
value: http://10.0.2.2:8200
I. Within container specification set environmental variables (values in double quotes):
env:
- name: VAULT_TOKEN
value: "abcd1234"
- name: VAULT_ADDR
value: "http://10.0.2.2:8200"
Then refer to the values in envconsul.hcl
vault {
address = ${VAULT_ADDR}
renew_token = false
retry {
backoff = "1s"
}
token = ${VAULT_TOKEN}
}
II. Another option is to unseal the vault cluster (with the unseal key which was printed while initializing the vault cluster)
$ vault operator unseal
and then authenticate to the vault cluster using a root token.
$ vault login <your-generated-root-token>
More details
I tried many suggestions and nothing worked until I passed -vault-token argument to envconsul command like this:
envconsul -vault-token=$VAULT_TOKEN -config=/app/config.hcl -secret="/secret/debug/service" env
and in config.hcl it should be like this:
vault {
address = "http://kvstorage.try.direct:8200"
token = "${env(VAULT_TOKEN)}"
}

How to use environment variable in kubernetes container command?

I am trying to deploy cloudsql proxy as sidecar contaier like this:
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy",
"-instances=${CLOUDSQL_INSTANCE}=tcp:5432",
"-credential_file=/secrets/cloudsql/google_application_credentials.json"]
env:
- name: CLOUDSQL_INSTANCE
valueFrom:
secretKeyRef:
name: persistence-cloudsql-instance-creds
key: instance_name
volumeMounts:
- name: my-secrets-volume
mountPath: /secrets/cloudsql
readOnly: true
But when I deploy this, I get following error in logs:
2019/06/20 13:42:38 couldn't connect to "${CLOUDSQL_INSTANCE}": googleapi: Error 400: Missing parameter: project., required
How could I use environment variable in command that runs inside kubernetes container?
If you want to reference environment variables in the command you need to put them in parentheses, something like: $(CLOUDSQL_INSTANCE).