I am having a strange issue with the Liveness Probe constantly failing but connecting into the Pod and checking the endpoint with cURL looks good.
Here is the output of the CURL command.
curl -v localhost:7000/health
* Expire in 0 ms for 6 (transfer 0x5595637270f0)
...
* Expire in 0 ms for 1 (transfer 0x5595637270f0)
* Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 150000 ms for 3 (transfer 0x5595637270f0)
* Expire in 200 ms for 4 (transfer 0x5595637270f0)
* Connected to localhost (127.0.0.1) port 7000 (#0)
> GET /health HTTP/1.1
> Host: localhost:7000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: gunicorn
< Date: Mon, 13 Feb 2023 19:41:44 GMT
< Connection: close
< Content-Type: text/html; charset=utf-8
< Content-Length: 24
<
* Closing connection 0
Now here is the section of the YAML that has the probe for the Pod:
containers:
- name: flask-container
image: path
imagePullPolicy: Always
volumeMounts:
- name: cert-and-key
mountPath: /etc/certs
readOnly: true
ports:
- containerPort: 7000
livenessProbe:
httpGet:
path: /health
port: 7000
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 20
imagePullSecrets:
- name: pullsecret
For some reason the Liveness probe keeps failing after creating the Pod:
Liveness probe failed: Get "http://10.224.0.130:7000/health": dial tcp 10.224.0.130:7000: connect: connection refused
Thanks in advance for any pointers!
Fixed. The issue was with the Docker image exposing the container on localhost or 127.0.0.1 instead of correctly exposing on 0.0.0.0.
Related
I getting problem with the connecting nginx server, postgresql and Matrix Synapse
Postgresql
it is running see the systemctl status below .
-Synapse1 is the database and roshyara is user which I have already added in the postgresql .
hb_pga_conf files are as following
1 # TYPE DATABASE USER ADDRESS METHOD
2 local all all md5
3
4 # The same using local loopback TCP/IP connections.
5 #
6 # TYPE DATABASE USER ADDRESS METHOD
7 host all all 127.0.0.1/32 md5
8 host all all 0.0.0.0/0 md5
9 host all all ::1/128 md5
10 # IPv4 local connections:
11 host all all 127.0.0.1/32 md5
12 host all all 172.19.0.0/16 md5
Synapse homeserver.yaml file is as follwoing
1 # Configuration file for Synapse.
2 #
3 # This is a YAML file: see [1] for a quick introduction. Note in particular
4 # that *indentation is important*: all the elements of a list or dictionary
5 # should have the same indentation.
6 #
7 # [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
8 #
9 # For more information on how to configure Synapse, including a complete accounting of
10 # each option, go to docs/usage/configuration/config_documentation.md or
11 # https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html
12
13 #server_name: "192.168.11.88"
14 server_name: 192.168.11.88
15 #
16 pid_file: /root/synapse1/homeserver.pid
17 #web_client: True
18 #soft_file_limit: 0
19 #
20 #type: http
21 #tls: true
22 #x_forwarded: true
23
24 #user_directory:
25 enabled: true
26
27 database:
28 name: psycopg2
29 args:
30 user: roshyara
31 password: 12345678
32 database: synapse1
33 host: 127.0.0.1
34 port: 5432
35 cp_min: 5
36 cp_max: 10
37 #database: /root/synapse1/homeserver.db
38 # seconds of inactivity after which TCP should send a keepalive message to the server
39 keepalives_idle: 10
40
41 # the number of seconds after which a TCP keepalive message that is not
42 # acknowledged by the server should be retransmitted
43 #keepalives_interval: 10
44
45 # the number of TCP keepalives that can be lost before the client's connection
46 # to the server is considered dead
47 # keepalives_count: 3
48
50 log_config: "/root/synapse1/192.168.11.88.log.config"
51 media_store_path: /root/synapse/media_store
52 #registration_shared_secret: ";6NfAHoYP#xt3vQpi-o^4-8rJDeBnujn*rLdk-R7h6:,&~rjm."
53 report_stats: true
54 macaroon_secret_key: "D=:YD_lc_^;QhiKhj.iGV&#AEW3rmcna6rAq9O~.2=b6^lwyr6"
55 form_secret: "r,:c#PA6PEwk3B9e7d=AKjUD--Iw#X+zB4R_C^4aB.zWGZt+K1"
56 signing_key_path: "/root/synapse/matrix.ginmbh.de.signing.key"
57 trusted_key_servers:
58 - server_name: "matrix.org"
59
-synapse is also running
Nginx sever is also runnung
nginx setting is as follwoing
/etc/nginx/nginx.conf
1 #user
2 user nginx;
3 worker_processes auto;
4 # include config file
5
6 #include /etc/nginx/conf.d/*.conf;
7 #
8 #load_module modules/ngx_postgres_module.so;
9
10 #
11 error_log /var/log/nginx/error.log notice;
12 pid /var/run/nginx.pid;
13
14
15 events {
16 worker_connections 1024;
17 }
18
19
20 http {
21 include /etc/nginx/mime.types;
22 default_type application/octet-stream;
23
24 log_format main '$remote_addr - $remote_user [$time_local] "$request" '
25 '$status $body_bytes_sent "$http_referer" '
26 '"$http_user_agent" "$http_x_forwarded_for"';
27
28 access_log /var/log/nginx/access.log main;
29
30 sendfile on;
31 #tcp_nopush on;
32
33 keepalive_timeout 65;
34
35 include /etc/nginx/conf.d/*.conf;
36 }
/etc/nginx/conf.d/matrix.conf file
1 #
2 server {
3 listen 443 ssl http2;
4 listen [::]:443 ssl http2;
5
6 # For the federation port
7 listen 8448 ssl http2 default_server;
8 listen [::]:8448 ssl http2 default_server;
9
10 server_name 192.168.11.88;
11 #ssl on;
12 ssl_certificate /etc/letsencrypt/live/matrix.ginmbh.de/fullchain.pem;
13 ssl_certificate_key /etc/letsencrypt/live/matrix.ginmbh.de/privkey.pem;
14
15 #location ~ ^(/_matrix|/_synapse/static) {
16 location / {
17 # note: do not add a path (even a single /) after the port in `proxy_pass`,
18 # otherwise nginx will canonicalise the URI and cause signature verification
19 # errors.
20 proxy_pass http://localhost:8008;
21 proxy_set_header X-Forwarded-For $remote_addr;
22 proxy_set_header X-Forwarded-Proto $scheme;
23 proxy_set_header Host $host;
24
25 # Nginx by default only allows file uploads up to 1M in size
26 # Increase client_max_body_size to match max_upload_size defined in homeserver.yaml
27 client_max_body_size 50M;
28
29 # Synapse responses may be chunked, which is an HTTP/1.1 feature.
30 proxy_http_version 1.1;
31 }
32 }
-tcp connection
(env) [root#matrix-clon synapse1]# netstat -tunpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 822/sshd
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 2459/postmaster
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 1105/nginx: master
tcp 0 0 0.0.0.0:8448 0.0.0.0:* LISTEN 1105/nginx: master
tcp6 0 0 :::22 :::* LISTEN 822/sshd
tcp6 0 0 :::443 :::* LISTEN 1105/nginx: master
tcp6 0 0 :::8448 :::* LISTEN 1105/nginx: master
tcp6 0 0 :::9090 :::* LISTEN 1/systemd
(env) [root#matrix-clon synapse1]#
(env) [root#matrix-clon synapse1]# ps aux |grep nginx
root 1105 0.0 0.0 44768 920 ? Ss 11:52 0:00 nginx: master process /usr/sbin/nginx
nginx 1106 0.0 0.1 77860 7688 ? S 11:52 0:02 nginx: worker process
nginx 1107 0.0 0.1 77468 5212 ? S 11:52 0:00 nginx: worker process
root 1202 0.0 0.0 7352 908 pts/1 S+ 11:52 0:00 tail -f /var/log/nginx/error.log
root 2615 0.0 0.0 12136 1152 pts/0 S+ 12:35 0:00 grep --color=auto nginx
port is also open
(env) [root#matrix-clon synapse1]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: eth0
sources:
services: cockpit dhcpv6-client http https ssh
ports: 8448/tcp 5432/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
(env) [root#matrix-clon synapse1]#
However, nginx is showing the follwoing error . What can I do now and how can I test which connection is creating problem?
2023/02/12 12:08:38 [error] 1106#0: *249 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: 192.168.11.88, request: "GET /_synapse/admin/v1/register HTTP/1.1", upstream: "http://[::1]:8008/_synapse/admin/v1/register", host: "localhost:8448"
2023/02/12 12:08:38 [warn] 1106#0: *249 upstream server temporarily disabled while connecting to upstream, client: ::1, server: 192.168.11.88, request: "GET /_synapse/admin/v1/register HTTP/1.1", upstream: "http://[::1]:8008/_synapse/admin/v1/register", host: "localhost:8448"
2023/02/12 12:08:38 [error] 1106#0: *249 connect() failed (111: Connection refused) while connecting to upstream, client: ::1, server: 192.168.11.88, request: "GET /_synapse/admin/v1/register HTTP/1.1", upstream: "http://127.0.0.1:8008/_synapse/admin/v1/register", host: "localhost:8448"
2023/02/12 12:08:38 [warn] 1106#0: *249 upstream server temporarily disabled while connecting to upstream, client: ::1, server: 192.168.11.88, request: "GET /_synapse/admin/v1/register HTTP/1.1", upstream: "http://127.0.0.1:8008/_synapse/admin/v1/register", host: "localhost:8448"
2023/02/12 12:11:52 [error] 1106#0: *294 connect() failed (111: Connection refused) while connecting to upstream, client: 10.176.8.89, server: 192.168.11.88, request: "GET /_matrix/static/ HTTP/2.0", upstream: "http://127.0.0.1:8008/_matrix/static/", host: "192.168.11.88"
2023/02/12 12:11:52 [warn] 1106#0: *294 upstream server temporarily disabled while connecting to upstream, client: 10.176.8.89, server: 192.168.11.88, request: "GET /_matrix/static/ HTTP/2.0", upstream: "http://127.0.0.1:8008/_matrix/static/", host: "192.168.11.88"
2023/02/12 12:11:52 [error] 1106#0: *294 connect() failed (111: Connection refused) while connecting to upstream, client: 10.176.8.89, server: 192.168.11.88, request: "GET /_matrix/static/ HTTP/2.0", upstream: "http://[::1]:8008/_matrix/static/", host: "192.168.11.88"
2023/02/12 12:11:52 [warn] 1106#0: *294 upstream server temporarily disabled while connecting to upstream, client: 10.176.8.89, server: 192.168.11.88, request: "GET /_matrix/static/ HTTP/2.0", upstream: "http://[::1]:8008/_matrix/static/", host: "192.168.11.88"
installed nginx
installed postgresql
installed matrix synapse
created homeserver.yaml
now the nginx server is showing upstream server is not available
I have a sample webapp and redis that I am running in Kubernetes.
I am using probes for the basic checks like below
Now I want to make sure that redis is up and running before the application.
below code snippet is from webapp.
when I run a command nc -zv <redis service name> 6379 it works well, but when I use it as command in startupProbe it gives me errors. I think the way I am passing command is not right, can someone help me understand what is wrong
error I get
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "nc -zv redis 6379": executable file not found in $PATH: unknown
readinessProbe:
httpGet:
path: /
port: 5000
initialDelaySeconds: 20
periodSeconds: 5
livenessProbe:
httpGet:
path: /
port: 5000
initialDelaySeconds: 30
periodSeconds: 5
startupProbe:
exec:
command:
- nc -zv redis 6379
failureThreshold: 20
periodSeconds: 5
The command has to be entered in proper format as it is an array. The below code is in expected format.
startupProbe:
exec:
command:
- nc
- -zv
- redis
- "6379"
failureThreshold: 30
periodSeconds: 5
I have the following pod setup:
apiVersion: v1
kind: Pod
metadata:
name: proxy-test
namespace: test
spec:
containers:
- name: container-a
image: <Image>
imagePullPolicy: Always
ports:
- name: http-port
containerPort: 8083
- name: container-proxy
image: <Image>
ports:
- name: server
containerPort: 7487
protocol: TCP
- name: container-b
image: <Image>
I exec into container-b and execute following curl request:
curl --proxy localhost:7487 -X POST http://localhost:8083/
Due to some reason, http://localhost:8083/ is directly getting called and proxy is ignored. Can someone explain why this can happen ?
Environment
I replicated the scenario on kubeadm and GCP GKE kubernetes clusters to see if there is any difference - no, they behave the same, so I assume AWS EKS should behave the same too.
I created a pod with 3 containers within:
apiVersion: v1
kind: Pod
metadata:
name: proxy-pod
spec:
containers:
- image: ubuntu # client where connection will go from
name: ubuntu
command: ['bash', '-c', 'while true ; do sleep 60; done']
- name: proxy-container # proxy - that's obvious
image: ubuntu
command: ['bash', '-c', 'while true ; do sleep 60; done']
- name: server # regular nginx server which listens to port 80
image: nginx
For this test stand I installed squid proxy on proxy-container (what is squid and how to install it). By default it listens to port 3128.
As well as curl was installed on ubuntu - client container. (net-tools package as a bonus, it has netstat).
Tests
Note!
I used 127.0.0.1 instead of localhost because squid has some resolving questions, didn't find an easy/fast solution.
curl is used with -v flag for verbosity.
We have proxy on 3128 and nginx on 80 within the pod:
# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:3128 0.0.0.0:* LISTEN -
tcp6 0 0 :::80 :::* LISTEN -
curl directly:
# curl 127.0.0.1 -vI
* Trying 127.0.0.1:80... # connection goes directly to port 80 which is expected
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*
curl via proxy:
# curl --proxy 127.0.0.1:3128 127.0.0.1:80 -vI
* Trying 127.0.0.1:3128... # connecting to proxy!
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0) # connected to proxy
> HEAD http://127.0.0.1:80/ HTTP/1.1 # going further to nginx on `80`
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*
squid logs:
# cat /var/log/squid/access.log
1635161756.048 1 127.0.0.1 TCP_MISS/200 958 GET http://127.0.0.1/ - HIER_DIRECT/127.0.0.1 text/html
1635163617.361 0 127.0.0.1 TCP_MEM_HIT/200 352 HEAD http://127.0.0.1/ - HIER_NONE/- text/html
NO_PROXY
NO_PROXY environment variable might be set up, however by default it's empty.
I added it manually:
# export NO_PROXY=127.0.0.1
# printenv | grep -i proxy
NO_PROXY=127.0.0.1
Now curl request via proxy will look like:
# curl --proxy 127.0.0.1:3128 127.0.0.1 -vI
* Uses proxy env variable NO_PROXY == '127.0.0.1' # curl detects NO_PROXY envvar
* Trying 127.0.0.1:80... # and ignores the proxy, connection goes directly
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> HEAD / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*
It's possible to override NO_PROXY envvar while executing curl command with --noproxy flag.
--noproxy no-proxy-list
Comma-separated list of hosts which do not use a proxy, if one is specified. The only wildcard is a single *
character, which matches all hosts, and effectively disables the
proxy. Each name in this list is matched as either a domain which
contains the hostname, or the hostname itself. For example, local.com
would match local.com, local.com:80, and www.local.com, but not
www.notlocal.com. (Added in 7.19.4).
Example:
# curl --proxy 127.0.0.1:3128 --noproxy "" 127.0.0.1 -vI
* Trying 127.0.0.1:3128... # connecting to proxy as it was supposed to
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 3128 (#0) # connection to proxy is established
> HEAD http://127.0.0.1/ HTTP/1.1 # connection to nginx on port 80
> Host: 127.0.0.1
> User-Agent: curl/7.68.0
> Accept: */*
This proves that proxy works! with localhost.
Another option is something incorrectly configured in proxy which is used in the question. You can get this pod and install squid and curl into both containers and try yourself.
Stages:
connect a container's shell
curl www.xxx.com (//this will always waiting )
...
Then I use tcpdump in host machine and filter by ip
tcpdump -i eth0 host ip
3 11:05:05 2019/12/2 133.5701630 10.171.162.231 111.111.222.333 TCP TCP: [Bad CheckSum]Flags=......S., SrcPort=48836, DstPort=HTTP(80), PayloadLen=0, Seq=126843476, Ack=0, Win=29200 ( Negotiating scale factor 0x7 ) = 29200
4 11:05:05 2019/12/2 133.5704230 111.111.222.333 10.171.162.231 TCP TCP:Flags=...A..S., SrcPort=HTTP(80), DstPort=48836, PayloadLen=0, Seq=3228156738, Ack=126843477, Win=2896 ( Negotiated scale factor 0x9 ) = 1482752
5 11:05:05 2019/12/2 133.5704630 10.171.162.231 111.111.222.333 TCP TCP: [Bad CheckSum]Flags=...A...., SrcPort=48836, DstPort=HTTP(80), PayloadLen=0, Seq=126843477, Ack=3228156739, Win=229 (scale factor 0x7) = 29312
6 11:05:05 2019/12/2 133.5705430 10.171.162.231 111.111.222.333 HTTP HTTP:Request, GET /api/test, Query:debug
7 11:05:05 2019/12/2 133.5707110 111.111.222.333 10.171.162.231 TCP TCP:Flags=...A...., SrcPort=HTTP(80), DstPort=48836, PayloadLen=0, Seq=3228156739, Ack=126843596, Win=6 (scale factor 0x9) = 3072
The tcp flag is
src -> dst syn
dst -> src syn/ack
src -> dst ack
src -> dst ack/push
dst -> src ack
The curl command will waiting a long time and then throw a timeout error. in normal request there has a dst -> src ack/push packet, but I never received.
I don't know why and how to resolve it.
--- my deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-app-dep
labels:
app: test-app
version: stable
spec:
replicas: 2
selector:
matchLabels:
app: test-app
version: stable
template:
metadata:
labels:
app: test-app
version: stable
spec:
containers:
- image: test-app
name: test-app
livenessProbe:
httpGet:
path: /health/status
port: 80
initialDelaySeconds: 3
periodSeconds: 10
ports:
- containerPort: 80
I'm having network issues running services in docker-compose. Essentially I'm just trying to make a get request through Kong to a simple Flask API I have setup. The docker-compose.yml is below
version: "3.0"
services:
postgres:
image: postgres:9.4
container_name: kong-database
ports:
- "5432:5432"
environment:
- POSTGRES_USER=kong
- POSTGRES_DB=kong
web:
image: kong:latest
container_name: kong
environment:
- DATABASE=postgres
- KONG_PG_HOST=postgres
restart: always
ports:
- "8000:8000"
- "443:8443"
- "8001:8001"
- "7946:7946"
- "7946:7946/udp"
links:
- postgres
ui:
image: pgbi/kong-dashboard
container_name: kong-dashboard
ports:
- "8080:8080"
employeedb:
build: test-api/
restart: always
ports:
- "5002:5002"
I add the API to kong with the command curl -i -X POST --url http://localhost:8001/apis/ --data name=employeedb --data upstream_url=http://localhost:5002 --data hosts=employeedb --data uris=/employees. I've tried this with many combinations of inputs, including different names, passing in the Docker network IP and the name of the test-api as hostname for the upstreamurl. After adding the API to Kong I get
HTTP/1.1 502 Bad Gateway
Date: Tue, 11 Jul 2017 14:17:17 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: kong/0.10.3
Additionally I've gotten into the docker containers running docker exec it <container-id> /bin/bash and attempted to make curl requests to the expected flask endpoint. While on the container running the API I was able to make a sucessful call to both localhost:5002/employees as well as to employeedb:5002/employees. However when making it from the container running Kong I see
curl -iv -X GET --url 'http://employeedb:5002/employees'
* About to connect() to employeedb port 5002 (#0)
* Trying X.X.X.X...
* Connection refused
* Failed connect to employeedb:5002; Connection refused
* Closing connection 0
Am I missing some sort of configuration that exposes the containers to one another?
You need to make the employeedb container visible to kong by defining a link like you did with the PostgreSQL database. Just add it as an additional entry directly below - postgres and it should be reachable by Kong:
....
links:
- postgres
- employeedb
....