PostgreSQL Exporter for docker - Prometheus - postgresql

I have been reading this page for years, and now I need some help.
I m starting to configure Prometheus to collect metrics from Docker Swarm and Docker containers, It works really good with cAdvisor and Node Exporter, but now i m having issues to collect metrics from PostgreSQL docker container. I m using this exporter --> https://github.com/wrouesnel/postgres_exporter
This is the service in docker-compose.yml:
postgresql-exporter:
image: wrouesnel/postgres_exporter
ports:
- 9187:9187
networks:
- backend
environment:
- DATA_SOURCE_NAME=postgresql://example:<password>#localhost:5432/example?sslmode=disable
And this is in prometheus.yml
- job_name: 'postgresql-exporter'
static_configs:
- targets: ['postgresql-exporter:9187']
We have two stacks, one with the db and the other one with the monitoring stack.
Logs in the postgresql service:
monitoring_postgresql-exporter.1.krslcea4hz20#master1.xxx.com | time="2019-11-02T16:12:20Z" level=error msg="Error opening connection to database (postgresql://example:PASSWORD_REMOVED#localhost:5432/example?sslmode=disable): dial tcp 127.0.0.1:5432: connect: connection refused" source="postgres_exporter.go:1403"
monitoring_postgresql-exporter.1.krslcea4hz20#master1.xxx.com | time="2019-11-02T16:12:29Z" level=info msg="Established new database connection to \"localhost:5432\"." source="postgres_exporter.go:814"
monitoring_postgresql-exporter.1.krslcea4hz20#master1.xxx.com | time="2019-11-02T16:12:30Z" level=info msg="Established new database connection to \"localhost:5432\"." source="postgres_exporter.go:814"
monitoring_postgresql-exporter.1.krslcea4hz20#master1.xxx.com | time="2019-11-02T16:12:32Z" level=info msg="Established new database connection to \"localhost:5432\"." source="postgres_exporter.go:814"
monitoring_postgresql-exporter.1.krslcea4hz20#master1.xxx.com | time="2019-11-02T16:12:35Z" level=error msg=**"Error opening connection to database (postgresql://example:PASSWORD_REMOVED#localhost:5432/example?sslmode=disable): dial tcp 127.0.0.1:5432: connect: connection refused" source="postgres_exporter.go:1403"**
But when I look in prometheus targets section, the postresql-exporter endpoint says is "UP"
And when I check the pg_up metric it says 0, no conection.
Any idea of how can I solve this?
Any help will be appreciated, thanks!
EDIT: Here is the config of postgreSQL docker service db:
pg:
image: registry.xxx.com:443/pg:201908221000
environment:
- POSTGRES_DB=example
- POSTGRES_USER=example
- POSTGRES_PASSWORD=example
volumes:
- ./postgres/db_data:/var/lib/postgresql/data
networks:
- allnet
deploy:
mode: replicated
replicas: 1
placement:
constraints: [node.role == manager]
restart_policy:
condition: on-failure
max_attempts: 3
window: 120s
Thanks

Related

Mongodb connection refused from other application in docker-compose

I have below mongodb configuration in docker-compose.yml file -
version: '3.7'
networks:
app-tier:
driver: bridge
mongodb:
image: 'bitnami/mongodb:latest'
container_name: "mongodb"
environment:
MONGODB_INITIAL_PRIMARY_HOST: mongodb
MONGODB_ADVERTISED_HOSTNAME: mongodb
MONGODB_REPLICA_SET_MODE: primary
MONGODB_INITDB_DATABASE: testdb
MONGODB_REPLICA_SET_NAME: rs0
ALLOW_EMPTY_PASSWORD: 'yes'
ports:
- "27017:27017"
volumes:
- ./scripts/mongorestore.sh:/docker-entrypoint-initdb.d/mongorestore.sh
- ./data/mongodb:/data/mongodb
networks:
- app-tier
infrastructure:
build:
context: .
dockerfile: Dockerfile
target: base
container_name: infra
environment:
- SPRING_PROFILES_ACTIVE=dev
- KAFKA_BROKERS=kafka:9092
- REDIS_ENDPOINT=redis
- APP_NAME=infrastructure
volumes:
- ~/.m2:/root/.m2
depends_on:
- "kafka"
- "redis"
- "mongodb"
networks:
- app-tier
Whenever I run docker-compose my app infrastructure giving below error -
error connecting to host: could not connect to server: server selection error: server selection timeout, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp 127.0.0.1:27017: connect: connection refused }, ] }
Inside application I am not even trying to connect mongodb, I am just trying to set up my application first using docker-compose
Am I missing anything here?
Something in infrastructure image is trying to connect to mongodb. localhost is likely a default host, if you didn't set it explicitly. You need to find out who is that and set host name to mongodb

launching containers which use fluentd logging will return port error

When I launch my containers, i always notice that when i launch EFK, if i launch my other containers which use F logging, that if launched too quickly will fail because f ports are not yet active.
I was think that there was some sort of FluentD healtthcheck I can use which will wait containers until fluentd is running before it continues?
If i wait like 1 minute and start the rest of my containers it will work fine.
The error i will get is:
Error response from daemon: failed to initialize logging driver: dial tcp [::1]:24224: connect: connection refused
The general format of the 2 different stacks is as follows:
services:
database:
build: db/.
depends_on:
- fluentd
ports:
- "5432:5432"
logging:
driver: "fluentd"
options:
fluentd-address: localhost:24224
tag:home-database
and my fluentd will look like:
services:
fluentd:
build: ./fluentd
volumes:
- ./fluentd/log:/fluentd/log
- "./fluentd/conf:/fluentd/etc"
ports:
- "24224:24224"
- "24224:24224/udp"
Is there a good healthcheck i should do? a curl request? Would a healthcheck solve the problem of delaying other containers until its status is green?

How to use fluent-bit with Docker-compose

I want to use the fluent-bit docker image to help me persist the ephemeral docker container logs to a location on my host (and later use it to ship logs elsewhere).
I am facing issues such as:
Cannot start service clamav: failed to initialize logging driver: dial tcp 127.0.0.1:24224: connect: connection refused
I have read a number of post including
configuring fluentbit with docker but I'm still at a lost.
My docker-compose is made up of nginx, our app, keycloak, elasticsearch and clamav. I have added fluent-bit, made it first to starts via depends on. I changed the other services to use the fluentd logging driver.
Part of config:
clamav:
container_name: clamav-app
image: tiredofit/clamav:latest
restart: always
volumes:
- ./clamav/data:/data
- ./clamav/logs:/logs
environment:
- ZABBIX_HOSTNAME=clamav-app
- DEFINITIONS_UPDATE_FREQUENCY=60
networks:
- iris-network
expose:
- "3310"
depends_on:
- fluentbit
logging:
driver: fluentd
fluentbit:
container_name: iris-fluent
image: fluent/fluent-bit:latest
restart: always
networks:
- iris-network
volumes:
- ./fluent-bit/etc:/fluent-bit/etc
ports:
- "24224:24224"
- "24224:24224/udp"
I have tried to proxy_pass 24224 to fluentbit in nginx and start nginx first, and that avoided the error on clamav and es, but same error with keycloak.
So how can I configure the service to use the host or is it that localhost is not the "external" host?

Speedtest is found down in Prometheus

I used this docker-compose (kinda basic), however after configuring it and building it I got after entering http://[server-ip]:9090/targets information that:
speedtest (0/1 up)
Error: Get "http://speedtest:9798/metrics": dial tcp: lookup speedtest on 127.0.0.11:53: no such host
And I understand that it can't find that host, it's just that the configuration itself wasn't touched and it actually looks legit to me:
docker-compose
service:
speedtest:
tty: true
stdin_open: true
expose:
- 9798
ports:
- 9798:9798
image: miguelndecarvalho/speedtest-exporter
restart: always
networks:
- back-tier
prometheus.yml
- job_name: 'speedtest'
metrics_path: /metrics
scrape_interval: 5m
scrape_timeout: 60s # running speedtest needs time to complete
static_configs:
- targets: ['speedtest:9798']
Can someone spot the issue? How the speedtest is not found on local DNS server? Everything is exposed and still not finding the right stuff.
#Edit I have DNS server configured by dnsmasq
If Prometheus is bound to the host's network and you're trying to access speedtest on the host's network too, then you should reference speedtest as localhost not speedtest:
static_configs:
- targets: ['localhost:9798']
NOTE Docker (Compose) only provides DNS resolution for e.g. services (i.e. speedtest) within the process. If you were to run Prometheus within the Docker Compose services too, then you'd be able to use Docker (Compose) DNS resolution to resolve speedtest to the container on port 9798.

Drone.io | Create docker-compose settings

I can't connect to Drone.io with my GitHub.
And have several problems with the app:
1) drone-agent can't connect to server
dodge#comp:$drone agent
28070:M 15 Nov 22:04:01.906 * connecting to server http://<my_ip>
28070:M 15 Nov 22:04:01.906 # connection failed, retry in 15s. websocket.Dial http://<my_ip>: bad scheme
2) I can't add Postgresql to docker-compose.
When I add this text from your site
DRONE_DATABASE_DRIVER: postgres
DRONE_DATABASE_DATASOURCE: postgres://root:password#1.2.3.4:5432/postgres?sslmode=disable
I have this error
INFO: 2017/11/15 19:42:33 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 172.18.0.2:9000: getsockopt: connection refused"; Reconnecting to {drone-server:9000 <nil>}
3) When I use only a server and an agent in docker-compose I have this error
dodge#comp:$drone server
ERRO[0000] sql: unknown driver "sqlite3" (forgotten import?)
FATA[0000] database connection failed
docker-compose.yml
version: '2'
services:
drone-server:
image: drone/drone:0.8
ports:
- 80:8000
- 9000
volumes:
- /var/lib/drone:/var/lib/drone/
- ./drone:/var/lib/drone/
restart: always
environment:
- DRONE_DEBUG=true
- DRONE_OPEN=true
- DRONE_HOST=http://172.18.0.2
- DRONE_GITHUB=true
- DRONE_GITHUB_CLIENT=secretid
- DRONE_GITHUB_SECRET=secretpass
- DRONE_SECRET=password
drone-agent:
image: drone/agent:0.8
command: agent
restart: always
depends_on: [ drone-server ]
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DRONE_SERVER=drone-server:9000
- DRONE_SECRET=password
4) I cannot start tests in my project. Maybe I missed something during the setup.
$ drone server
$ drone agent
I see the above commands in your examples. These commands are only available in drone 0.7 and below. Drone 0.8 uses drone-server and drone-agent binaries. There seems to be some version disconnect here.
connection failed, retry in 15s. websocket.Dial
drone 0.7 and below used websockets. I see in the docker-compose example you are using drone 0.8 which uses http2 and grpc. There seems to be a disconnect in your configuration vs the version of drone you are using.
sql: unknown driver "sqlite3"
this happens when you compile drone with CGO disabled, or use a version of drone that has been compiled with CGO disabled. If CGO is disabled the sqlite3 driver is not compiled into the binary. Are you trying to build drone from source?
grpc: addrConn.resetTransport failed to create client transport
This error comes from the agent, and is therefore unrelated to a postgres configuration. You should not be providing your agent with a postgres configuration, only the server.
version: '2'
services:
drone-server:
image: drone/drone:latest
ports:
- 80:8000
- 9000:9000
volumes:
- /var/lib/drone:/var/lib/drone/
- ./drone:/var/lib/drone/
restart: always
environment:
- DRONE_DEBUG=true
- DRONE_HOST=http://<container_ip_server>
- DRONE_OPEN=true
- DRONE_GITHUB=true
- DRONE_GITHUB_CLIENT=<client_git>
- DRONE_GITHUB_SECRET=<secret_git>
- DRONE_SECRET=<secret_drone>
- DRONE_GITHUB_MERGE_REF=true
drone-agent:
image: drone/agent:latest
command: agent
restart: always
depends_on: [ drone-server ]
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DRONE_SERVER=drone-server:9000
- DRONE_SECRET=<drone_secret>
This workes fine.