Google finally added the support for Cloudsql postgresql logical replication/decoding support.
What I am trying to do is to use Debezium to capture the changes on the database tables, and then publish the changes in json to Gcp pubsub.
I am using Debezium server with Gcp Pubsub Sink just to verify the connection is working, but keep getting following error message org.postgresql.util.PSQLException: ERROR: could not access file "decoderbufs": No such file or directory
So seems like it defaults the plugin to decoderbufs, however I did set the database.source.plugin.name in the environment variable. According to this doc https://hub.docker.com/r/debezium/server, I am using DATABASE_SOURCE_PLUGIN_NAME. Here is the snippet of my k8s manifest
serviceAccountName: cloudsql-client
containers:
- name: debezium-standalone-server-dv
image: debezium/server:1.6
env:
- name: DEBEZIUM_SOURCE_CONNECTOR_CLASS
value: io.debezium.connector.postgresql.PostgresConnector
- name: DEBEZIUM_SOURCE_DATABASE_HOSTNAME
value: localhost
- name: DEBEZIUM_SOURCE_DATABASE_PORT
value: "5432"
- name: DEBEZIUM_SOURCE_DATABASE_USER
value: cdc
- name: DEBEZIUM_SOURCE_DATABASE_PASSWORD
value: cdc
- name: DEBEZIUM_SOURCE_DATABASE_DBNAME
value: postgres
- name: DEBEZIUM_SINK_TYPE
value: pubsub
- name: DEBEZIUM_SINK_PUBSUB_PROJECT_ID
value: myproject
- name: DEBEZIUM_SOURCE_OFFSET_STORAGE_FILE_FILENAME
value: data/offsets.dat
- name: DEBEZIUM_SOURCE_OFFSET_FLUSH_INTERVAL_MS
value: "0"
- name: DEBEZIUM_SOURCE_DATABASE_SERVER_NAME
value: CDC_POC
- name: DATABASE_SOURCE_PLUGIN_NAME
value: wal2json
From Cloudsql, i have already turn on the logical decoding and add the REPLICATION attribute to the user, and i am able to "manually" create it by running SELECT * FROM pg_create_logical_replication_slot('test_slot', 'wal2json'); I can also see the changes by querying the replication slot SELECT * FROM pg_logical_slot_get_changes('test_slot', NULL, NULL);
so what am i missing here?
I am guessing it is a typo on the docker configuration documentation.
After i changed the configuration from DATABASE_SOURCE_PLUGIN_NAME to DEBEZIUM_SOURCE_PLUGIN_NAME it chooses the plugin correctly
Related
I follow this but it is not working.
I created custom secret:
apiVersion: v1
kind: Secret
metadata:
name: keycloak-db-secret
data:
POSTGRES_DATABASE: ...
POSTGRES_EXTERNAL_ADDRESS: ...
POSTGRES_EXTERNAL_PORT: ...
POSTGRES_HOST: ...
POSTGRES_USERNAME: ...
POSTGRES_PASSWORD: ...
and keycloak with external db:
apiVersion: keycloak.org/v1alpha1
kind: Keycloak
metadata:
labels:
app: keycloak
name: keycloak
spec:
externalDatabase:
enabled: true
instances: 1
but when I check log, keycloak can not connect to db. It is still using default vaule: keycloak-postgresql.keycloak not value defined in my custom secret ? Why it is not using my value from secrets ?
UPDATE
when I check keycloak pod which was created by operator I can see:
env:
- name: DB_VENDOR
value: POSTGRES
- name: DB_SCHEMA
value: public
- name: DB_ADDR
value: keycloak-postgresql.keycloak
- name: DB_PORT
value: '5432'
- name: DB_DATABASE
value: keycloak
- name: DB_USER
valueFrom:
secretKeyRef:
name: keycloak-db-secret
key: POSTGRES_USERNAME
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-db-secret
key: POSTGRES_PASSWORD
so now I know why I can not connect to db. It use different DB_ADDR. How I can use address: my-app.postgres (db in another namespace).
I dont know why POSTGRES_HOST in secret not working and pod still using default service name
To connect with service in another namespace you can use.
<servicename>.<namespace>.svc.cluster.local
suppose your Postgres deployment and service running in test namespace it will go like
postgres.test.svc.cluster.local
this is what i am using : https://github.com/harsh4870/Keycloack-postgres-kubernetes-deployment/blob/main/keycload-deployment.yaml
i have also attached the Postgres file you can use it however in my case i have setup both in the same namespace keycloak and Postgres so working like charm.
I'm using Azure PostgreSQL for that, and it works correctly. In pod configuration, it also uses keycloak-postgresql.keycloak as DB_ADDR, but this is pointing to my internal service created by operator based on keycloak-db-secret.
keycloak-postgresql.keycloak this is the another service created by Keycloak Operator, which is used to connect to Postgresql's service.
You can check its endpoint.
$ kubectl get endpoints keycloak-postgresql -n keycloak
NAME ENDPOINTS AGE
keycloak-postgresql {postgresql's service ip}:5432 4m31s
However, the reason why it fails is due to the selector of this service:
selector:
app: keycloak
component: database
So if your DB Pod has the different Label, the selector will not work.
I reported this issue to the community. If they reply me, I will try to fix this bug by submitting a patch.
I was having this same issue, and then after looking at #JiyeYu 's answer, I have searched the project's issue backlog, and I've found some related issues that are still open (at the moment of this reply).
Particularly this one: https://issues.redhat.com/browse/KEYCLOAK-18602
After reading this, and its comments, I did the following:
Don't use IPs on POSTGRES_EXTERNAL_ADDRESS. If your PostGres is hosted within K8s via a StatefulSet, use the full <servicename>.<namespace>.svc.cluster.local (like #Harsh Manvar 's answer)
Remove the POSTGRES_HOST setting from the secret (don't just set it to the default, delete it). Apparently, it is not only being ignored, but also breaking the keycloak pod initialization process somehow.
After I applied these changes the issue was solved for me.
I also had similar problem, it turned out since I was using SSLMODE: "verify-full", keycloak expected correct hostname of my external db.
Since somehow Keycloak translates internally the real external db address into "keycloak-postgresql.keycloak", it expected something like "keycloak-postgresql.my-keycloak-namespace"
The log went something like this:
SEVERE [org.postgresql.ssl.PGjdbcHostnameVerifier] (ServerService Thread Pool -- 57) Server name validation failed: certificate for host keycloak-postgresql.my-keycloak-namespace dNSName entries subjectAltName, but none of them match. Assuming server name validation failed
After I added the host keycloak-postgresql.my-keycloak-namespace on the db certificate, it worked like advertised.
I tried to use keycloak db against crdb cloud. I have used https://github.com/codecentric/helm-charts/tree/master/charts/keycloak charts for deployment k8s. I create a db for the keycloak and give the above configuration to success connection. I use my values.yaml and added additional env var:
extraEnv: |
- name: DB_VENDOR
value: postgres
- name: DB_ADDR
value: xxxx.xxx.cockroachlabs.cloud
- name: DB_PORT
value: "xxx"
- name: DB_DATABASE
value: keycloak
- name: DB_USER_FILE
value: /secrets/db-creds/user
- name: DB_PASSWORD_FILE
value: /secrets/db-creds/password
- name: JDBC_PARAMS
value: sslmode=verify-ca&sslrootcert=/secrets/crdb-creds/xxx.crt
- name: JDBC_PARAMS_FILE
value: /secrets/crdb-creds/xxx.crt
and also
# Add additional volumes, e. g. for custom themes
extraVolumes: |
- name: crdb-creds
secret:
secretName: keycloak-crdb-creds
- name: db-creds
secret:
secretName: keycloak-db-creds
and mounting
# Add additional volumes mounts, e. g. for custom themes
extraVolumeMounts: |
- name: crdb-creds
mountPath: /secrets/crdb-creds
readOnly: true
- name: db-creds
mountPath: /secrets/db-creds
readOnly: true
So in theory there is no restriction for using cockroach the with postgres db vendor in keycloak(!). And I am going to give a try for this and actually it wasn't give an error but it restaring after while and keeping restarting same period. So it gives an :
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: java.sql.SQLException: IJ031040: Connection is not associated with a managed connection: org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8#3a612dd6
or
10:55:31,907 FATAL [org.keycloak.services] (ServerService Thread Pool -- 64) Error during startup: java.lang.IllegalStateException: Failed to retrieve lock
So my question is what is the variable for giving .crt path and is there any additional progress need to run this correctly
As the title says, I'm setting a POSTGRES_PASSWORD and after spinning up the cluster with Skaffold (--port-forward on so I can access the DB with pgAdmin), I can access the database
with or without the correct password. POSTGRES_DB and POSTGRES_USER work as expected.
I am seeing in the documentation on Docker Hub for Postgres:
Note 1: The PostgreSQL image sets up trust authentication locally so you may notice a password is not required when connecting from localhost (inside the same container). However, a password will be required if connecting from a different host/container.
I think the --port-forward could possibly be the culprit since it is registering as localhost.
Anyway to prevent this behavior?
I guess the concern is someone having access to my laptop and easily being able to connect to the DB.
This is my postgres.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres-deployment
spec:
replicas: 1
selector:
matchLabels:
component: postgres
template:
metadata:
labels:
component: postgres
spec:
containers:
- name: postgres
image:testproject/postgres
ports:
- containerPort: 5432
env:
- name: POSTGRES_DB
value: dev
- name: POSTGRES_USER
value: dev
- name: POSTGRES_PASSWORD
value: qwerty
volumeMounts:
- name: postgres-storage
mountPath: /var/lib/postgresql/data
subPath: postgres
volumes:
- name: postgres-storage
persistentVolumeClaim:
claimName: postgres-storage
---
apiVersion: v1
kind: Service
metadata:
name: postgres-cluster-ip-service
spec:
type: ClusterIP
selector:
component: postgres
ports:
- port: 5432
targetPort: 5432
And the skaffold.yaml:
apiVersion: skaffold/v1beta15
kind: Config
build:
local:
push: false
artifacts:
- image: testproject/postgres
docker:
dockerfile: ./db/Dockerfile.dev
sync:
manual:
- src: "***/*.sql"
dest: .
- image: testproject/server
docker:
dockerfile: ./server/Dockerfile.dev
sync:
manual:
- src: "***/*.py"
dest: .
deploy:
kubectl:
manifests:
- k8s/ingress.yaml
- k8s/postgres.yaml
- k8s/server.yaml
The Dockerfile.dev too:
FROM postgres:11-alpine
EXPOSE 5432
COPY ./db/*.sql /docker-entrypoint-initdb.d/
Ok, reread the postgres Docker docs and came across this:
POSTGRES_INITDB_ARGS
This optional environment variable can be used to send arguments to postgres initdb. The value is a space separated string of arguments as postgres initdb would expect them. This is useful for adding functionality like data page checksums: -e POSTGRES_INITDB_ARGS="--data-checksums".
That brought me to the initdb docs:
--auth=authmethod
This option specifies the authentication method for local users used in pg_hba.conf (host and local lines). Do not use trust unless you trust all local users on your system. trust is the default for ease of installation.
That brought me to the Authentication Methods docs:
19.3.2. Password Authentication
The password-based authentication methods are md5 and password. These methods operate similarly except for the way that the password is sent across the connection, namely MD5-hashed and clear-text respectively.
If you are at all concerned about password "sniffing" attacks then md5 is preferred. Plain password should always be avoided if possible. However, md5 cannot be used with the db_user_namespace feature. If the connection is protected by SSL encryption then password can be used safely (though SSL certificate authentication might be a better choice if one is depending on using SSL).
PostgreSQL database passwords are separate from operating system user passwords. The password for each database user is stored in the pg_authid system catalog. Passwords can be managed with the SQL commands CREATE USER and ALTER ROLE, e.g., CREATE USER foo WITH PASSWORD 'secret'. If no password has been set up for a user, the stored password is null and password authentication will always fail for that user.
Long story short, I just did this and it takes only the actual password now:
env:
...
- name: POSTGRES_INITDB_ARGS
value: "-A md5"
I've installed the DataDog agent on my Kubernetes cluster using the Helm chart (https://github.com/helm/charts/tree/master/stable/datadog).
This works very well except for one thing. I have a number of Redis containers that have passwords set. This seems to be causing issues for the DataDog agent because it can't connect to Redis without a password.
I would like to either disable monitoring Redis completely or somehow bypass the Redis authentication. If I leave it as is I get a lot of error messages in the DataDog container logs and the redisdb integration shows up in yellow in the DataDog dashboard.
What are my options here?
I am not a fan of helm, but you can accomplish this in 2 ways:
via env vars: make use of DD_AC_EXCLUDE variable to exclude the Redis containers: eg DD_AC_EXCLUDE=name:prefix-redis
via a config map: mount an empty config map in /etc/datadog-agent/conf.d/redisdb.d/, below is an example where I renamed the auto_conf.yaml to auto_conf.yaml.example.
apiVersion: v1
data:
auto_conf.yaml.example: |
ad_identifiers:
- redis init_config: instances:
## #param host - string - required
## Enter the host to connect to.
#
- host: "%%host%%" ## #param port - integer - required
## Enter the port of the host to connect to.
#
port: "6379"
conf.yaml.example: |
init_config: instances: ## #param host - string - required
## Enter the host to connect to.
# [removed content]
kind: ConfigMap
metadata:
creationTimestamp: null
name: redisdb-d
alter the daemonset/deployment object:
[....]
volumeMounts:
- name: redisdb-d
mountPath: /etc/datadog-agent/conf.d/redisdb.d
[...]
volumes:
- name: redisdb-d
configMap:
name: redisdb-d
[...]
Trying to connect to a 2nd gen cloud sql database from a GCP Container.
I created the cloud proxy, but am a bit confused on what my app needs to connect via the proxy. My app is looking to connect on 127.0.0.1:3306 already within the application with all the needed mysql connection information which works fine outside of GCP. My app is currently logging connection errors against 127.0.0.1:3306 when deployed on GCP container.
Error: connect ECONNREFUSED 127.0.0.1:3306 at Object.exports._errnoException
Any additional sample files available for a simple node app to better understand the needed application config?
The sample below seems to address what wordpress needs, but what do I need for simple node app?
https://github.com/GoogleCloudPlatform/container-engine-samples/blob/master/cloudsql/cloudsql_deployment.yaml
Related Link:
https://cloud.google.com/sql/docs/mysql/connect-container-engine
Provide 127.0.0.1:3306 as the host address your application uses to access the database.
I have this hard coded in my app.
Because the proxy runs in a second container in the same pod, it appears to your application as localhost, so you use 127.0.0.1:3306 to connect to it.
Right, I have this hard coded in my app
Provide the cloudsql-db-credentials secret to enable the application to log in to the database.
Ok, if I have to add this, what
For example, assuming the application expected DB_USER and DB_PASSWORD:
name: DB_USER
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: username
If your proxy user requires a password, you would also add:
So what variable name would I be using here? Is this asking for the mysql user name?
name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: password
What variable is needed here? Is this asking for the mysql pw for the id above?
In the wordpress sample from the link above, I'm trying to figure out what variables are needed for a simple node app.
containers:
- image: wordpress:4.4.2-apache
name: web
env:
- name: WORDPRESS_DB_HOST
# Connect to the SQL proxy over the local network on a fixed port.
# Change the [PORT] to the port number used by your database
# (e.g. 3306).
value: 127.0.0.1:[PORT]
# These secrets are required to start the pod.
# [START cloudsql_secrets]
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: password
- name: WORDPRESS_DB_USER
valueFrom:
secretKeyRef:
name: cloudsql-db-credentials
key: username
# [END cloudsql_secrets]
ports:
- containerPort: 80
name: wordpress
Thanks!