I’ve been working on creating a new syslog setup and have run into an issue, that i cannot find a solution for, so i thought maybe someone here could help me out.
I have a setup with 2 syslog servers and 2 haproxy nodes(in HA with keepalived). i have 2 endpoints on configured on the haproxy nodes: “endpoint_X” and “endpoint_Y” for different types of logs. I would like to control the flow of syslog messages, so that when syslog is send to “endpoint_X”:514 its send to syslog01 and when “endpoint_Y”:514 its send to syslog02. this is normally done with the use of ACL’s for normal frontends. But for syslog I use HAproxy’s “log-forward” function, where ACL’s is not supported for.
Below is an example of my config:
ring syslog01
description " "
format rfc3164
maxlen 1200
size 357913941
server syslog01 XXXXX_01:514 source YYYYY check
timeout client 90s
timeout connect 10s
timeout server 90s
timeout check 10s
ring syslog02
description " "
format rfc3164
maxlen 1200
size 357913941
server syslog02 XXXXX_02:514 source YYYYYY check
timeout client 90s
timeout connect 10s
timeout server 90s
timeout check 10s
log-forward syslog
bind 0.0.0.0:514
bind [::]:514
dgram-bind 0.0.0.0:514
dgram-bind [::]:514
log ring#syslog01 local0
log ring#syslog02 local0
does anyone have an idea if there is something i can do to get around this issue , so i can control the data flow in log-forward, other than using differen ports? I use haproxy version 2.6
i have tried some like the following, but as stated ACL does not work with log-forward:
acl acl_endpoint_X hdr(host) -i endpoint_X
acl acl_endpoint_X hdr(host) -i endpoint_Y
log ring#syslog01 local0 if endpoint_X hdr(host)
log ring#syslog02 local0 if endpoint_Y hdr(host)
Motivations
I am a running into an issue when trying to proxy PostgreSQL with Traefik over SSL using Let's Encrypt.
I did some research but it is not well documented and I would like to confirm my observations and leave a record to everyone who faces this situation.
Configuration
I use latest versions of PostgreSQL v12 and Traefik v2. I want to build a pure TCP flow from tcp://example.com:5432 -> tcp://postgresql:5432 over TLS using Let's Encrypt.
Traefik service is configured as follow:
version: "3.6"
services:
traefik:
image: traefik:latest
restart: unless-stopped
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./configuration/traefik.toml:/etc/traefik/traefik.toml:ro"
- "./configuration/dynamic_conf.toml:/etc/traefik/dynamic_conf.toml"
- "./letsencrypt/acme.json:/acme.json"
networks:
- backend
ports:
- "80:80"
- "443:443"
- "5432:5432"
networks:
backend:
external: true
With the static setup:
[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.web.http]
[entryPoints.web.http.redirections.entryPoint]
to = "websecure"
scheme = "https"
[entryPoints.websecure]
address = ":443"
[entryPoints.websecure.http]
[entryPoints.websecure.http.tls]
certresolver = "lets"
[entryPoints.postgres]
address = ":5432"
PostgreSQL service is configured as follow:
version: "3.6"
services:
postgresql:
image: postgres:latest
environment:
- POSTGRES_PASSWORD=secret
volumes:
- ./configuration/trial_config.conf:/etc/postgresql/postgresql.conf:ro
- ./configuration/trial_hba.conf:/etc/postgresql/pg_hba.conf:ro
- ./configuration/initdb:/docker-entrypoint-initdb.d
- postgresql-data:/var/lib/postgresql/data
networks:
- backend
#ports:
# - 5432:5432
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.tcp.routers.postgres.entrypoints=postgres"
- "traefik.tcp.routers.postgres.rule=HostSNI(`example.com`)"
- "traefic.tcp.routers.postgres.tls=true"
- "traefik.tcp.routers.postgres.tls.certresolver=lets"
- "traefik.tcp.services.postgres.loadBalancer.server.port=5432"
networks:
backend:
external: true
volumes:
postgresql-data:
It seems my Traefik configuration is correct. Everything is OK in the logs and all sections in dashboard are flagged as Success (no Warnings, no Errors). So I am confident with the Traefik configuration above. The complete flow is about:
EntryPoint(':5432') -> HostSNI(`example.com`) -> TcpRouter(`postgres`) -> Service(`postgres#docker`)
But, it may have a limitation at PostgreSQL side.
Debug
The problem is that I cannot connect the PostgreSQL database. I always get a Timeout error.
I have checked PostgreSQL is listening properly (main cause of Timeout error):
# - Connection Settings -
listen_addresses = '*'
port = 5432
And I checked that I can connect PostgreSQL on the host (outside the container):
psql --host 172.19.0.4 -U postgres
Password for user postgres:
psql (12.2 (Ubuntu 12.2-4), server 12.3 (Debian 12.3-1.pgdg100+1))
Type "help" for help.
postgres=#
Thus I know PostgreSQL is listening outside its container, so Traefik should be able to bind the flow.
I also have checked external traefik can reach the server:
sudo tcpdump -i ens3 port 5432
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes
09:02:37.878614 IP x.y-z-w.isp.com.61229 > example.com.postgresql: Flags [S], seq 1027429527, win 64240, options [mss 1452,nop,wscale 8,nop,nop,sackOK], length 0
09:02:37.879858 IP example.com.postgresql > x.y-z-w.isp.com.61229: Flags [S.], seq 3545496818, ack 1027429528, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
09:02:37.922591 IP x.y-z-w.isp.com.61229 > example.com.postgresql: Flags [.], ack 1, win 516, length 0
09:02:37.922718 IP x.y-z-w.isp.com.61229 > example.com.postgresql: Flags [P.], seq 1:9, ack 1, win 516, length 8
09:02:37.922750 IP example.com.postgresql > x.y-z-w.isp.com.61229: Flags [.], ack 9, win 502, length 0
09:02:47.908808 IP x.y-z-w.isp.com.61229 > example.com.postgresql: Flags [F.], seq 9, ack 1, win 516, length 0
09:02:47.909578 IP example.com.postgresql > x.y-z-w.isp.com.61229: Flags [P.], seq 1:104, ack 10, win 502, length 103
09:02:47.909754 IP example.com.postgresql > x.y-z-w.isp.com.61229: Flags [F.], seq 104, ack 10, win 502, length 0
09:02:47.961826 IP x.y-z-w.isp.com.61229 > example.com.postgresql: Flags [R.], seq 10, ack 104, win 0, length 0
So, I am wondering why the connection cannot succeed. Something must be wrong between Traefik and PostgreSQL.
SNI incompatibility?
Even when I remove the TLS configuration, the problem is still there, so I don't expect the TLS to be the origin of this problem.
Then I searched and I found few posts relating similar issue:
Introducing SNI in TLS handshake for SSL connections
Traefik 2.0 TCP routing for multiple DBs;
As far as I understand it, the SSL protocol of PostgreSQL is a custom one and does not support SNI for now and might never support it. If it is correct, it will confirm that Traefik cannot proxy PostgreSQL for now and this is a limitation.
By writing this post I would like to confirm my observations and at the same time leave a visible record on Stack Overflow to anyone who faces the same problem and seek for help. My question is then: Is it possible to use Traefik to proxy PostgreSQL?
Update
Intersting observation, if using HostSNI('*') and Let's Encrypt:
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.tcp.routers.postgres.entrypoints=postgres"
- "traefik.tcp.routers.postgres.rule=HostSNI(`*`)"
- "traefik.tcp.routers.postgres.tls=true"
- "traefik.tcp.routers.postgres.tls.certresolver=lets"
- "traefik.tcp.services.postgres.loadBalancer.server.port=5432"
Everything is flagged as success in Dashboard but of course Let's Encrypt cannot perform the DNS Challenge for wildcard *, it complaints in logs:
time="2020-08-12T10:25:22Z" level=error msg="Unable to obtain ACME certificate for domains \"*\": unable to generate a wildcard certificate in ACME provider for domain \"*\" : ACME needs a DNSChallenge" providerName=lets.acme routerName=postgres#docker rule="HostSNI(`*`)"
When I try the following configuration:
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.tcp.routers.postgres.entrypoints=postgres"
- "traefik.tcp.routers.postgres.rule=HostSNI(`*`)"
- "traefik.tcp.routers.postgres.tls=true"
- "traefik.tcp.routers.postgres.tls.domains[0].main=example.com"
- "traefik.tcp.routers.postgres.tls.certresolver=lets"
- "traefik.tcp.services.postgres.loadBalancer.server.port=5432"
The error vanishes from logs and in both setups the dashboard seems ok but traffic is not routed to PostgreSQL (time out). Anyway, removing SSL from the configuration makes the flow complete (and unsecure):
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.tcp.routers.postgres.entrypoints=postgres"
- "traefik.tcp.routers.postgres.rule=HostSNI(`*`)"
- "traefik.tcp.services.postgres.loadBalancer.server.port=5432"
Then it is possible to connect PostgreSQL database:
time="2020-08-12T10:30:52Z" level=debug msg="Handling connection from x.y.z.w:58389"
I'm using Traefik to proxy PostgreSQL, so answer is yes. But I'm not using TLS, because my setup is a bit different. First of all, if PostgreSQL doesn't support SNI, then I would suggest to try to modify labels, especially HostSNI rule to this:
"traefik.tcp.routers.postgres.rule=HostSNI(`*`)"
That says: ignore SNI and just take any name from specified entrypoint as valid.
SNI routing for postgres with STARTTLS has been added to Traefik in this PR. Now Treafik will listen to the initial bytes sent by postgres and if its going to initiate a TLS handshake (Note that postgres TLS requests are created as non-TLS first and then upgraded to TLS requests), Treafik will handle the handshake and then is able to receive the TLS headers from postgres, which contains the SNI information that it needs to route the request properly. This means that you can use HostSNI("example.com") along with tls to expose postgres databases under different subdomains.
As of writing this answer, I was able to get this working with the v3.0.0-beta2 image (Reference)
I am trying to perform a load balancing for my Node.js application which uses websockets. I need haproxy to stop load balance new connections on a server, which has reached its maximum number of connections, keeping the existing ones intact in the same time.
I do this by performing an agent-check for each of my servers. If a server cannot accept new connections it responds with "drain" to an agent-check. If a server is able to respond to a new connection, it responds with "ready" to an agent-check.
Here is my haproxy.cfg configuration file:
global
daemon
maxconn 240000
log /dev/log local0 debug
log /dev/log local1 notice
tune.ssl.default-dh-param 2048
defaults
mode http
log global
option httplog
option dontlognull
option dontlog-normal
option http-server-close
option redispatch
timeout connect 20000ms
timeout http-request 1m
timeout client 2100000ms
timeout server 2100000ms
timeout queue 30s
timeout check 5s
timeout http-keep-alive 180s
timeout tunnel 3600s
timeout tarpit 60s
frontend stats
bind *:8084
stats enable
stats uri /stats
stats refresh 10s
stats admin if TRUE
frontend test
mode http
bind *:5000
default_backend ws
backend ws
mode http
fullconn 100000
balance roundrobin
cookie SERVERID insert indirect nocache
server 1 backend1:9999 check agent-check agent-port 8080 cookie 1 inter 500 fall 1 rise 2
And here is how I respond to haproxy agent-check in my Node.js app:
const healthCheckServer = net.createServer((c) => {
let data = '';
if (currentConn < MAX_CONN) {
data += 'ready';
} else {
data += 'drain';
}
c.write(data + '\r\n');
c.destroy();
});
healthCheckServer.listen(8080, '0.0.0.0');
When number of connections to my application is reaching its maximum, haproxy correctly changes server status to DRAIN (agent) (I can observe this in haproxy web dashboard). The problem is, that new connections are still accepted by the application.
I am new to haproxy, so can someone point me to where I am wrong?
Found out that when server is drained by agent (status is set to DRAIN (agent)) and if server is the only one existing in the backend it will still accept new connections.
When there are multiple servers present and each server is drained the behavior is just like expected: haproxy returns HTTP 503.
UPDATE:
Turned out that I was looking in the wrong direction all the time.
First I had to mark my backend which processes WebSocket connections as a non-http (remove mode http line). My guess is haproxy incorrectly counts current sessions for http backend when using WebSockets. Removing mode http solved my problem.
Second, returning maxconn:<conn> in agent-check looks like much simple and more idiomatic way to limit the number of concurrent connections.
Sources:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-agent-check
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/
I hope this will help someone.
I'm fighting with kibana since few days and I don't overcome to start it on my FreeBSD server.
This is my environment:
FreeBSD 11.1-STABLE
ElasticSearch 5.3.0
Kibana 5.3.0
Logstash 5..
ElasticSearch and Logstash work fine. But I don't overcome to start kibana service.
This is files according to kibana:
kibana.yml file:
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are
both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
server.basePath: "/qual/kibana"
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://localhost:9200"
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "discover"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 0
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000
# Specifies the path where Kibana creates the process ID file.
pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
logging.dest: /var/log/kibana.log
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# The default locale. This locale can be used in certain circumstances to substitute any missing
# translations.
#i18n.defaultLocale: "en"
/usr/local/etc/rc.d/kibana:
#!/bin/sh
#
# $FreeBSD: head/textproc/kibana5/files/kibana.in 462830 2018-02-24 14:17:41Z feld $
#
# PROVIDE: kibana
# REQUIRE: DAEMON
# KEYWORD: shutdown
. /etc/rc.subr
name=kibana
rcvar=kibana_enable
load_rc_config $name
: ${kibana_enable:="NO"}
: ${kibana_config:="/usr/local/etc/kibana.yml"}
: ${kibana_user:="www"}
: ${kibana_group:="www"}
: ${kibana_log:="/var/log/kibana.log"}
required_files="${kibana_config}"
pidfile="/var/run/${name}/${name}.pid"
start_precmd="kibana_precmd"
procname="/usr/local/bin/node"
command="/usr/sbin/daemon"
command_args="-f -p ${pidfile} env BABEL_DISABLE_CACHE=1 ${procname} /usr/local/www/kibana5/src/cli serve --config ${kibana_config} --log-file ${kibana_log}"
kibana_precmd()
{
if [ ! -d $(dirname ${pidfile}) ]; then
install -d -o ${kibana_user} -g ${kibana_group} $(dirname ${pidfile})
fi
if [ ! -f ${kibana_log} ]; then
install -o ${kibana_user} -g ${kibana_group} -m 640 /dev/null ${kibana_log}
fi
if [ ! -d /usr/local/www/kibana5/optimize ]; then
install -d -o ${kibana_user} -g ${kibana_group} /usr/local/www/kibana5/optimize
fi
}
run_rc_command "$1"
/etc/rc.conf:
kibana_enable="YES"
But when I execute: service kibana start
I get:
root#server:/var/log # service kibana start
Starting kibana.
root#server:/var/log # service kibana status
kibana is not running.
I don't know why ?
Start the service in debug mode
sh -x /usr/local/etc/rc.d/kibana start
find which command is used to start the kibana service. For kibana, the command should be something like /usr/local/bin/node /usr/local/www/kibana6/src/cli serve --config /usr/local/etc/kibana/kibana.yml
Start the process in foreground
It is possible that node is not properly installed or some permission issue.
I am configuring an Eureka client app, but always the registered port result in 80.
The server config is obtained from Eureka with auto-discovery enabled, when auto-discovery is disabled the port is registered correctly.
The port of the app is assigned only from the command line (--server.port=8080) and deleted in all other properties files (app.yaml, boostrap.yaml and in the config server git repo)
I have notice that in this code:
EurekaClientConfiguration.java
if (port != 0 && instanceConfig.getNonSecurePort() == 0) {
instanceConfig.setNonSecurePort(port);
}
The instanceConfig.getNonSecurePort() never is 0, due to this, the nonSecurePort property is never changed.
I have to register the port property in some other place?
Edited to add some detail:
I mean that my bootstrap.yml has the following lines:
cloud:config:discovery:enabled: true
The yml config is in a github repository and doesn’t have the port assigned
The app is running in the port 8082 with the app param --server.port=8082 But when it is registered in Eureka the port is always 80 instead of 8082
<port enabled="true">8082</port>
<securePort enabled="false">443</securePort>
This cause every based ribbon invocation doesn’t get the correct URL.
I notice that the port is setting correctly in this handler event after the init
EurekaClientConfiguration.java:
public void onApplicationEvent(EmbeddedServletContainerInitializedEvent event) {
...
EurekaClientConfiguration.this.port = event.getEmbeddedServletContainer().getPort();
But the flag "running" is active now and doesn't have any effect
Thanks a lot for your help