Postgres DB URL string from a K8s deployment yaml file - postgresql

I am trying to contact from a customized helm chart to a fully managed Postgres service on azure, and then I have to put the url connection string according to the app I want to deploy.
I want to ask which value should be the DATABASE_URL at the helm chart deployment?
My situation is the following:
I want to use an external Azure managed PostgreSQL and no the PostgreSQL container that comes with the helm chart.
So in consequence, I modified the DATABASE_URL value, given here to connect to the container inside K8s, I've modified in this way:
name: DATABASE_URL
# value: "postgres://{{ .Values.postgresql.postgresqlUsername }}:$(POSTGRESQL_PASSWORD)#{{ .Release.Name }}-postgresql"
value: "postgres://nmbrs#postgresql-nmb-psfc-stag:$(POSTGRESQL_PASSWORD)#postgresql-nmb-psfc-stag.postgres.database.azure.com/postfacto-staging-db"
but I am getting this error
/usr/local/lib/ruby/2.7.0/uri/generic.rb:208:in `initialize': the scheme postgres does not accept registry part: nmbrs#postgresql-nmb-psfc-stag:mypassword*#postgresql-nmb-psfc-stag.postgres.database.azure.com (or bad hostname?) (URI::InvalidURIError)
Which should be the real DATABASE_URL value if I want to contact to a fully Postgres managed service?
Which is the equivalent value to this?:
value: "postgres://{{ .Values.postgresql.postgresqlUsername }}:$(POSTGRESQL_PASSWORD)#{{ .Release.Name }}-postgresql"
I mean is
postgres//<username>:<my-pg-password>#<WHICH VALUE SHOULD BE HERE?>
What is the value of {{ .Release.Name }}-postgresql" ?
Just for the record, my customize postfacto/deployment/helm/templates/deployment.yaml is this
UPDATE
I changed the value for this
- name: DATABASE_URL
# value: "postgres://{{ .Values.postgresql.postgresqlUsername }}:$(POSTGRESQL_PASSWORD)#{{ .Release.Name }}-postgresql"
# value: "postgres://nmbrs#postgresql-nmb-psfc-stag:$(POSTGRESQL_PASSWORD)#postgresql-nmb-psfc-stag.postgres.database.azure.com:5432/postfacto-staging-db"
value: "postgres://postgresql-nmb-psfc-stag.postgres.database.azure.com/postfacto-staging-db"
And I got a different error:
Caused by:
PG::ConnectionBad: FATAL: Invalid Username specified. Please check the Username and retry connection. The Username should be in <username#hostname> format.
FATAL: Invalid Username specified. Please check the Username and retry connection. The Username should be in <username#hostname> format.
But is not clear how should be the syntax format since this article says:
Next, encode the database credentials. Use the format DB_ADAPTER://USER:PASSWORD#HOSTNAME/DB_NAME. If you are using mysql with a user ‘deploy’ and a password ‘secret’ on 127.0.0.1 and have a database railsapp, run
The format DB_ADAPTER://USER:PASSWORD#HOSTNAME/DB_NAME, it is the same I was using at the beginning

I think the problem with your connection string is, its username has a special character #, which might be breaking the connection string format and causing the validation error.
Your value
- name: DATABASE_URL
value: "postgres://nmbrs#postgresql-nmb-psfc-stag:$(POSTGRESQL_PASSWORD)#postgresql-nmb-psfc-stag.postgres.database.azure.com/postfacto-staging-db"
You can try to do an URL encoding for the username part like
- name: DATABASE_URL
value: "postgres://nmbrs%40postgresql-nmb-psfc-stag:$(POSTGRESQL_PASSWORD)#postgresql-nmb-psfc-stag.postgres.database.azure.com/postfacto-staging-db"

From what I see this helm chart is poorly written, and badly assumes things i.e. the DATABASE_URL has is build to only properly format a kubernetes posgress helm release and nothing else.
What I would suggest:
Instead of installing this chart on k8s by helm, use helm template functionality to locally render the template.
Edit the exported plain manifest, to match your needs
Go to your Azure DB, and get it ConnectionString, depending on how you mange your secrets in k8s, pass it to the DATABASE_URL environment
Manually apply the manifests

Related

Helm - How to change the storage class used by PostgreSQL when deploying Keycloak

I am using the Keycloak helm chart from codecentric. The chart depends on the PostgreSQL chart form bitnami, and by default the PostgreSQL chart uses the default storage class.
Looking values.yaml, I can see there is a section for postgresql, but there is no mention of the storage class. Is it possible to use Keycloak values.yaml file to override that setting?
The other option, that I would rather avoid, is to set enabled: false and deploy the PostgreSQL chart separately - hopefully that is not necessary.
postgresql:
# If `true`, the Postgresql dependency is enabled
enabled: true
# PostgreSQL User to create
postgresqlUsername: keycloak
# PostgreSQL Password for the new user
postgresqlPassword: keycloak
# PostgreSQL Database to create
postgresqlDatabase: keycloak
# PostgreSQL network policy configuration
networkPolicy:
enabled: false
Looking at the postgres values file I can see that you can change the value under global.storageClass.
Since postgres is a subchart, you can either add the above to your custome values file, or pass the storage class in your helm install command, like helm install --set postgresql.global.storageClass="yourStorageClass"`
You can also refer to this.

How to change database to Postges in JupyterHub?

I am trying to run jupyterhub with my config and I would like to change database form SQLite, that is created by default to PostgreSQL that alredy exists and has some tables (jupyterhub and other app would work concurrently and share database). On website only thing I see is:
We recommend using PostgreSQL for production if you are unsure ...
But no word how to change this database. Have you done this before and can describe it? Like do I ahve to create some tables on my own or do I just pass a link and jupyterhub willl do the rest?
You need create an empty database in postgres and then set the db_url property in the jupyterhub config. For example, for a postgres database on the local machine:
Connect to your postgres instance with a user that has the 'Create DB' attribute and run:
CREATE DATABASE jupyterhub1;
In your jupyterhub_config.py file set this property:
c.JupyterHub.db_url = 'postgresql://username:password#localhost:5432/jupyterhub1'
When you start jupyterhub it will create the required tables automatically. Also note that you don't have to hard code the credentials into the db_url property, you could access them from environment variables using os.environ["VAR_NAME"]
Thanks
Well, I will share what have worked for me. I hope this instruction help other people that face the same difficulty also.
Make a full backup of your database, just in case things go bad. Source
If you read the source above you got that you need to add some fields to your config.yaml.
you'll have to add this part to your config.yaml:
hub:
db:
upgrade: true
Keep following the source mentioned above and you will understand, because there are a little few details to do.
It is not that all, yet! You need to create your database(using Mysql or Postgres). After you have done your database, now you should have username, password, database_name, host and port.
Now, have a look on this info. This another page highlights one interesting point. JupyterHub doc says that using Postgres is easier than Mysql. You 'll have to add hub.db.type and hub.db.url. Have a look on doc to understand the string connection.
postgresql+psycopg2://<db-username>:<db-password>#<db-hostname>:<db-port>/<db-name>
Pay attention that in this example I just put the username and put nothing as password within string. Once it is possible to add hub.db.password I used this option. I also declared the size of database as 20Gi, in my example case.
hub:
db:
upgrade: true
type: postgres
url: postgresql+psycopg2://postgres:#db_jupyterhub_xxxxx.amazonaws.com:5432/db_jupyterhub
password: ~
pvc:
accessModes:
- ReadWriteMany
storage: 20Gi
At this point you realized that I did not put the password. In this case when I will deploy JupyterHub(or make a helm upgrade ...) I need to pass the --set parameter. So for this example I used this way to upgrade the helm chart that already existed.
helm upgrade -f config.yaml jupyterhub . \
--set hub.db.password=TYPE-PASSWORD-OF-DATABASE
With these steps the deploy should work. I used in this way and it worked fine. I have also another point to highlight, cookieSecret. In the docs they mentioned the needs of recreate it in case of pods restarting. Please have a look in this topic about deafult behavior of jupyterhub_cookie_secret and in this link explaining about cookie generation and uses.
I hope these steps help some of you.

How to configure conjur DATABASE_URL with postgres ssl_mode=verify-full

I would like to configure Conjur with ssl_mode=verify-full to connect to my postgres database.
I use the Docker image cyberark/conjur:1.8.1#sha256:01d601d763edf1d98ca81dda36d4744e78244a4836cfa804570a47da5fd50405
Adding it as a string parameter (like that DATABASE_URL=postgres://conjur:$CONJURDBPASSWORD#postgres-conjur:5432/conjurdb?sslmode=verify-full) does not seem to work.
The db library used by Conjur is Sequel and it supports it https://sequel.jeremyevans.net/rdoc/files/doc/opening_databases_rdoc.html#label-postgres
How can I achieve that without altering the Conjur code ? Ideally, via ENV or mounting a config file.
A project like Gemstash uses the same library and gives a way to achieve that easily. with a config.yml file containing (for instance):
:db_adapter: postgres
:db_url: postgres://{{ .Env.DB_HOST }}/gemstashdb?user=gemstash&password={{ .Env.DB_PASSWD }}
:db_connection_options:
:connect_timeout: 10
:read_timeout: 5
:timeout: 30
:sslmode: 'verify-full'
:sslrootcert: '{{ .Env.HOME }}/.ssl/root.crt'
I didn't find anything similar in Conjur.

How do I store a SQL connection password in Airflow cfg?

In the .cfg file, I connected sql alchemy to Postgres with user: airflow_admin and password: pass:
sql_alchemy_conn = postgresql+psycopg2://airflow_admin:pass#localhost:5432/airflow_backend
How do I anonymize this so that the password doesn't show? Do I create a .env file and store the password as a variable and then reference that variable in .cfg conn string?
I read the following but an example would be helpful: https://airflow.readthedocs.io/en/stable/howto/set-config.html
There are several way to do it:
1. change the configuration file permision to read only by airflow account
2. used airflow by docker mode, and encrpyt the configurationfile by docker secret create and map it.
THe mode 1 is easy and convinent. The mode 2 is flexible and can be used in production environment.
Good luck.
If you think the answer is good, pls up vote

Docker YAML enviroment for spring.datasource.password have symbols. How do i work this around?

Okay so I have a yaml file for deploying a spring boot image. So i want to set my spring datasource password in my environment yaml. But when I use docker-compose up , it shows error
ERROR: Invalid interpolation format for "environment" option in service "apps": "spring.datasource.password= "my_password!##$"
I believe that this error occured because my datasource password has symbols in it. How do I resolve it?
Here is my .yml file:
version: '3.0'
services:
apps:
image: student
ports:
- 8085:8080
environment:
- spring.datasource.url=jdbc:postgresql://192.168.100.3/my_database
- spring.datasource.username= my_user
- spring.datasource.password= my_password!##$
The $ at the end signals to Compose that it needs to do variable substitution; without an actual environment variable name after it, you get the "invalid interpolation" error. To get around this:
You can use a $$ (double-dollar sign) when your configuration needs a literal dollar sign.
So set:
- spring.datasource.password=my_password!##$$
(Double-check: should the username and password begin with spaces? You might need to remove the space after the equals sign, the environment: documentation does not show spaces here and does not suggest it removes whitespace from values.)