How to use (env) variables instead of plain secrets in data.metadataConnection in values.yaml
VAR_USER = "username"
VAR_PASS = "password"
data:
metadataConnection:
user: VAR_USER
pass: VAR_PASS
protocol: postgresql
host: ~
port: 5432
db: VAR_DB
sslmode: disable
instead of
data:
metadataConnection:
user: postgres
pass: postgres
protocol: postgresql
host: ~
port: 5432
db: postgres
sslmode: disable
Related
I have made this task definition in ansible to create two container in an ECS cluster:
- name: Create task definition
community.aws.ecs_taskdefinition:
containers:
- name: mongo-db
essential: true
image: "mongo:5.0.9"
environment:
- name: MONGO_INITDB_ROOT_USERNAME
value: "{{ lookup('env', 'DB_USER') }}"
- name: MONGO_INITDB_ROOT_PASSWORD
value: "{{ lookup('env', 'DB_PASSWORD') }}"
mountPoints:
- containerPath: "/data"
sourceVolume: mongodb-data-local
readOnly: true
- containerPath: "/dump"
sourceVolume: mongodb-dump
- containerPath: "/docker-entrypoint-initdb.d/init-mongo.sh"
sourceVolume: db-init
readOnly: true
portMappings:
- containerPort: 27017
hostPort: 27017
- name: roadmap-mongo-db
essential: false
image: "mongo:5.0.9"
command: ["mongod", "--port", "27018"]
environment:
- name: MONGO_INITDB_ROOT_USERNAME
value: "{{ lookup('env', 'DB_USER') }}"
- name: MONGO_INITDB_ROOT_PASSWORD
value: "{{ lookup('env', 'DB_PASSWORD') }}"
mountPoints:
- containerPath: "/data"
sourceVolume: roadmap-mongodb-data-local
readOnly: true
- containerPath: "/dump"
sourceVolume: roadmap-mongodb-dump
- containerPath: "/docker-entrypoint-initdb.d/init-mongo.sh"
sourceVolume: roadmap-db-init
portMappings:
portMappings:
- containerPort: 27018
hostPort: 27018
volumes:
- name: mongodb-data-local
- name: roadmap-mongodb-data-local
- name: mongodb-dump
host:
sourcePath: "/home/ec2-user/dump"
- name: roadmap-mongodb-dump
host:
sourcePath: "/home/ec2-user/roadmap-dump"
- name: db-init
host:
sourcePath: "/home/ec2-user/init-mongo.sh"
- name: roadmap-db-init
host:
sourcePath: "/home/ec2-user/roadmapdb/init-mongo.sh"
cpu: 128
memory: 340
family: showcase-task
force_create: True
state: present
basically I am creating two mongodb containers. The two containers start just fine, the first container (mongo-db) runs properly, I can exec into it and run mongo -u root to connect to it and also the services that interact with it can communicate just fine, so no problem with the first container.
Instead the second container is giving me problems, even if it runs ok and I can exec into it, I cannot do mongo -u root
mongo -u root gives this:
MongoDB shell version v5.0.9
Enter password:
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect#src/mongo/shell/mongo.js:372:17
#(connect):2:6
exception: connect failed
exiting with code 1
It seems it still wants to connect to 27017 instead of 27018. What am I doing wrong?
However the logs seems to be fine as it says:
{"t":{"$date":"2022-11-03T12:46:27.654+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27018.sock"}}
{"t":{"$date":"2022-11-03T12:46:27.753+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
{"t":{"$date":"2022-11-03T12:46:27.756+00:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27018,"ssl":"off"}}
I'm running Postgres DB with pg-admin and GO on the docker-compose.
Problem: I can connect from pg-admin to Postgres. But cannot establish a connection from Go.
I tried different combinations of authentication string but it does not work. String format same as here https://github.com/karlkeefer/pngr - but different container name - database
(ERROR) Connection URl:
backend_1 | 2021/08/08 14:24:40 DB connection: database://main:fugZwypczB94m0LP7CcH#postgres:5432/temp_db?sslmode=disable
backend_1 | 2021/08/08 14:24:40 Unalble to open DB connection: dial tcp 127.0.0.1:5432: connect: connection refused
(URI generation same as here https://github.com/karlkeefer/pngr)
Docker:
version: '3.8'
services:
backend:
restart: always
build:
context: backend
target: dev
volumes:
- ./backend:/root
ports:
- "5000:5000"
env_file: .env
depends_on:
- database
database:
build: database
restart: always
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
PGDATA: /var/lib/postgresql/data
volumes:
- ./database/data:/var/lib/postgresql/data
- ./logs/databse:/var/log/postgresql
- ./database/migrations:/docker-entrypoint-initdb.d/migrations
ports:
- "5432:5432"
database-admin:
image: dpage/pgadmin4:5.5
restart: always
environment:
PGADMIN_DEFAULT_EMAIL: ${PG_ADMIN_EMAIL}
PGADMIN_DEFAULT_PASSWORD: ${PG_ADMIN_PASSWORD}
PGADMIN_LISTEN_PORT: 80
ports:
- "8080:80"
volumes:
- ./database/admin:/var/lib/pgadmin
links:
- "database:pgsql-server"
depends_on:
- database
volumes:
database:
database-admin:
Environment:
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=temp_db
POSTGRES_USER=main
POSTGRES_PASSWORD=fugZwypczB94m0LP7CcH
PG_ADMIN_EMAIL=admin#temp.com
PG_ADMIN_PASSWORD=ayzi2ta8f1TnX3vKQSN1
PG_ADMIN_PORT=80
GO Code:
db, err = sqlx.Open("postgres", str)
str
func buildConnectionString() string {
user := os.Getenv("POSTGRES_USER")
pass := os.Getenv("POSTGRES_PASSWORD")
if user == "" || pass == "" {
log.Fatalln("You must include POSTGRES_USER and POSTGRES_PASSWORD environment variables")
}
host := os.Getenv("POSTGRES_HOST")
port := os.Getenv("POSTGRES_PORT")
dbname := os.Getenv("POSTGRES_DB")
if host == "" || port == "" || dbname == "" {
log.Fatalln("You must include POSTGRES_HOST, POSTGRES_PORT, and POSTGRES_DB environment variables")
}
str := fmt.Sprintf("database://%s:%s#%s:%s/%s?sslmode=disable", user, pass, host, port, dbname)
log.Println("DB connection: " + str)
return str
}
Thanks in advance!
You reference the database hostname as postgres (POSTGRES_HOST=postgres) which is fine, but the container/service name is database.
Either change the name in your compose.yaml from database to postgres or add an explicit hostname field:
database:
build: database
restart: always
hostname: postgres # <- add this
You may also want to add a dedicated network for multiple container services to talk to one another (or prevent others from). To do this, add this to each service your want to use a specific network e.g.
database:
# ...
networks:
- mynet
backend:
# ...
networks:
- mynet
and define the network at the end of your compose.yaml
networks:
mynet:
name: my-shared-db-network
I've set up a GitHub workflow to start a Postgres instance inside a docker container. I then execute my web API to simply prove it connects. This satisfies the workflow.
My workflow
name: Build
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:10.8
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: test
POSTGRES_PORT: 5432
ports:
- 5432/tcp
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
steps:
- uses: actions/checkout#v2
- name: Set up Go
uses: actions/setup-go#v2
with:
go-version: 1.15
- name: Run
env:
# These are the expected envs in my code
DB_HOST: localhost
DB_USER: test
DB_PASSWORD: test
DB_NAME: test
DB_PORT: 5432
DB_DIALECT: postgres
PORT: 8080
run: make run
For brevity I've trimmed irrelavent bits of code out, this is my database package:
// Config Model
type Config struct {
Host string
Name string
User string
Password string
Port string
Dialect string
}
// Open a new connection to a database
func Open(c Config) (*gorm.DB, error) {
dsn := fmt.Sprintf("host=%s port=%s dbname=%s password=%s user=%s sslmode=disable",
c.Host,
c.Port,
c.Name,
c.Password,
c.User,
)
db, err := gorm.Open(postgres.New(postgres.Config{
DriverName: c.Dialect,
DSN: dsn,
}), &gorm.Config{})
if err != nil {
return nil, fmt.Errorf("sql.Open: %v", err)
}
return db, nil
}
And this is called from my main package
func main() {
// Create Database Configuration
dbConfig := database.Config{
Host: os.Getenv("DB_HOST"),
Name: os.Getenv("DB_NAME"),
User: os.Getenv("DB_USER"),
Port: os.Getenv("DB_PORT"),
Password: os.Getenv("DB_PASSWORD"),
Dialect: os.Getenv("DB_DIALECT"),
}
// Connect to database
db, err := database.Open(dbConfig)
if err != nil {
log.Fatalf("failed to connect to database: %s", err)
}
}
The code fails to connect during the workflow with the given error:
go run -ldflags "-X main.Version=afc0042" cmd/api/main.go
2021/02/10 09:55:12 Running Version: afc0042
2021/02/10 09:55:12 failed to connect to database: sql.Open: dial tcp [::1]:5432: connect: connection refused
2021/02/10 09:55:12 /home/runner/work/database/database.go:32
[error] failed to initialize database, got error dial tcp [::1]:5432: connect: connection refused
you can use localhost when only you are trying to connect your server locally but as docker is not local in your machine so you can't use localhost except you are inside docker container.
try with port forward.
ports:
- 5432:5432 //update this one
With Ansible, I want to find which port is available in a range on a K8s cluster and use this port to expose a service temporary.
I'm able to find and extract the port but when I'm declaring the Nodeport using that port the tasks fail.
It seems that ansible is not converting my "port" variable to an int with the instruction {{ port|int }}.
- block:
- name: List all ports in range 32200 to 32220
wait_for:
port: "{{ item|int }}"
timeout: 1
state: stopped
msg: "Port {{ item }} is already in use"
register: available_ports
with_sequence: start=32200 end=32220
ignore_errors: yes
- name: extract first unused port from list
set_fact:
port: "{{ available_ports.results | json_query(\"[? state=='stopped'].port\") | first }}"
- debug:
var: port
- name: Expose service as a nodeport service
k8s:
state: present
definition:
apiVersion: v1
kind: Service
metadata:
name: "{{ namespace }}-service-nodeport"
namespace: "{{ namespace }}"
spec:
type: NodePort
selector:
component: my-app
ports:
- protocol: TCP
targetPort: 5432
nodePort: "{{ port|int }}"
port: 5432
This outputs the following:
TASK [../roles/my-role : debug] ***************************************************************************************************************************************************************************************************
ok: [127.0.0.1] => {
"port": "32380"
}
TASK [../roles/my-role : Expose service as a nodeport service] *******************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "error": 400, "msg": "Failed to create object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Service in version \\\\\"v1\\\\\" cannot be handled as a Service: v1.Service.Spec: v1.ServiceSpec.Ports: []v1.ServicePort: v1.ServicePort.NodePort: readUint32: unexpected character: \\\\ufffd, error found in #10 byte of ...|dePort\\\\\": \\\\\"32380\\\\\", \\\\\"p|..., bigger context ...|rotocol\\\\\": \\\\\"TCP\\\\\", \\\\\"targetPort\\\\\": 5432, \\\\\"nodePort\\\\\": \\\\\"32380\\\\\", \\\\\"port\\\\\": 5432}]}}|...\",\"reason\":\"BadRequest\",\"code\":400}\\n'", "reason": "Bad Request", "status": 400}
If I set the nodeport to a fix value such as 32800, it works.
Trying to open Database but it is saying password authentication failed for user "postgres"
I am not able to find the root cause of this problem.First time,i am using Docker. Pleas help
func openDB() (*sqlx.DB, error) {
q := url.Values{}
q.Set("sslmode", "disable")
q.Set("timezone", "utc")
u := url.URL{
Scheme: "postgres",
User: url.UserPassword("postgres", "postgres"),
Host: "localhost",
Path: "postgres",
RawQuery: q.Encode(),
}
fmt.Println(u.String())
// fmt.Println(u.String()) is
// postgre://postgres:postgres#localhost/postgres?sslmode=disable&timezone=utc
return sqlx.Open("postgres", u.String())
}
docker-compose.yaml looks like this.
version: '3'
networks:
shared-network:
driver: bridge
services:
db:
container_name: sales_db
networks:
- shared-network
image: postgres:11.1-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- 5432:5432
There was no problem in code. I changed the port eveything is working now.
func openDB() (*sqlx.DB, error) {
q := url.Values{}
q.Set("sslmode", "disable")
q.Set("timezone", "utc")
u := url.URL{
Scheme: "postgres",
User: url.UserPassword("postgres", "postgres"),
Host: "localhost:5433", // change here
Path: "postgres",
RawQuery: q.Encode(),
}
fmt.Println(u.String())
// fmt.Println(u.String()) is
// postgre://postgres:postgres#localhost/postgres?sslmode=disable&timezone=utc
return sqlx.Open("postgres", u.String())
}
and
version: '3'
networks:
shared-network:
driver: bridge
services:
db:
container_name: sales_db
networks:
- shared-network
image: postgres:11.1-alpine
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
ports:
- 5433:5432 //change here