LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:443 - kubernetes

I was following this doc (https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/) to set up my Kubernetes on my Docker-Desktop and used istio ingress gateway. I deployed an echo test app, added virtual service that points to the test app endpoint at port 8081. Then I set the istio gateway to open port 443 with the following:
servers:
- hosts:
- some.random.host
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: test-app-tls
where I also created a tls type secret with name test-app-tls using the certs and private key I generated.
(Just in case I forgot to mention something here, I tried with port 80 and http and everything works. Here is an example)
curl -i -H 'Host: some.random.host' 'localhost:80/host'
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 02 Aug 2022 21:10:31 GMT
content-length: 148
x-envoy-upstream-service-time: 1
server: istio-envoy
{"name":"some-expected-response","address":"some-other-expected-response"}
Then I tried to curl my localhost to hit the test app in the cluster with the following command
curl -k -i -v -H 'Host: some.random.host' 'https://localhost:443/host'
it gave me this error
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:443
I also tried with https://127.0.0.1:443/host and still doesn't work.
I'm fairly new to setting up TLS for Kubernetes. Could anyone please help me with this?
Thank you very much!!

Related

OCP 4.7.1 - Curl oauth-openshift.apps resutls in SSL_ERROR_SYSCALL

I am having an issue with the authentication operator not becoming stable (bouncing Between Avaialbe = True, and Degraded = True). The operator is trying to check the health using the endpoing https://oauth-openshift.apps.oc.sow.expert/healthz. and it sees it as not available (at least sometimes).
Cluster version :
[root#bastion ~]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.1 True False 44h Error while reconciling 4.7.1: the cluster operator ingress is degraded
Cluster operator describe:
[root#bastion ~]# oc describe clusteroperator authentication
Name: authentication
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
include.release.openshift.io/self-managed-high-availability: true
include.release.openshift.io/single-node-developer: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2021-03-15T19:54:21Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:include.release.openshift.io/self-managed-high-availability:
f:include.release.openshift.io/single-node-developer:
f:spec:
f:status:
.:
f:extension:
Manager: cluster-version-operator
Operation: Update
Time: 2021-03-15T19:54:21Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:relatedObjects:
f:versions:
Manager: authentication-operator
Operation: Update
Time: 2021-03-15T20:03:18Z
Resource Version: 1207037
Self Link: /apis/config.openshift.io/v1/clusteroperators/authentication
UID: b7ca7d49-f6e5-446e-ac13-c5cc6d06fac1
Spec:
Status:
Conditions:
Last Transition Time: 2021-03-17T11:42:49Z
Message: OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.oc.sow.expert/healthz": EOF
Reason: AsExpected
Status: False
Type: Degraded
Last Transition Time: 2021-03-17T11:42:53Z
Message: All is well
Reason: AsExpected
Status: False
Type: Progressing
Last Transition Time: 2021-03-17T11:43:21Z
Message: OAuthRouteCheckEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.oc.sow.expert/healthz": EOF
Reason: OAuthRouteCheckEndpointAccessibleController_EndpointUnavailable
Status: False
Type: Available
Last Transition Time: 2021-03-15T20:01:24Z
Message: All is well
Reason: AsExpected
Status: True
Type: Upgradeable
Extension: <nil>
Related Objects:
Group: operator.openshift.io
Name: cluster
Resource: authentications
Group: config.openshift.io
Name: cluster
Resource: authentications
Group: config.openshift.io
Name: cluster
Resource: infrastructures
Group: config.openshift.io
Name: cluster
Resource: oauths
Group: route.openshift.io
Name: oauth-openshift
Namespace: openshift-authentication
Resource: routes
Group:
Name: oauth-openshift
Namespace: openshift-authentication
Resource: services
Group:
Name: openshift-config
Resource: namespaces
Group:
Name: openshift-config-managed
Resource: namespaces
Group:
Name: openshift-authentication
Resource: namespaces
Group:
Name: openshift-authentication-operator
Resource: namespaces
Group:
Name: openshift-ingress
Resource: namespaces
Group:
Name: openshift-oauth-apiserver
Resource: namespaces
Versions:
Name: oauth-apiserver
Version: 4.7.1
Name: operator
Version: 4.7.1
Name: oauth-openshift
Version: 4.7.1_openshift
Events: <none>
When I curl multiple times to the same endpoint from bastion server, it results in two different responses once with the error "OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to oauth-openshift.apps.oc.sow.expert:443" and the other seems to be successful as follows:
[root#bastion ~]# curl -vk https://oauth-openshift.apps.oc.sow.expert/healthz
* Trying 192.168.124.173...
* TCP_NODELAY set
* Connected to oauth-openshift.apps.oc.sow.expert (192.168.124.173) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to oauth-openshift.apps.oc.sow.expert:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to oauth-openshift.apps.oc.sow.expert:443
[root#bastion ~]# curl -vk https://oauth-openshift.apps.oc.sow.expert/healthz
* Trying 192.168.124.173...
* TCP_NODELAY set
* Connected to oauth-openshift.apps.oc.sow.expert (192.168.124.173) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=*.apps.oc.sow.expert
* start date: Mar 15 20:05:53 2021 GMT
* expire date: Mar 15 20:05:54 2023 GMT
* issuer: CN=ingress-operator#1615838672
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /healthz HTTP/1.1
> Host: oauth-openshift.apps.oc.sow.expert
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Content-Type: text/plain; charset=utf-8
< Expires: 0
< Pragma: no-cache
< Referrer-Policy: strict-origin-when-cross-origin
< X-Content-Type-Options: nosniff
< X-Dns-Prefetch-Control: off
< X-Frame-Options: DENY
< X-Xss-Protection: 1; mode=block
< Date: Wed, 17 Mar 2021 11:49:50 GMT
< Content-Length: 2
<
* Connection #0 to host oauth-openshift.apps.oc.sow.expert left intact
ok
In the Bastion server, I am hosting the HAProxy load balancer and the squid proxy to allow internal instalnces to access the internet.
HAProxy configurations is as follows:
[root#bastion ~]# cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# https://www.haproxy.org/download/1.8/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# utilize system-wide crypto-policies
#ssl-default-bind-ciphers PROFILE=SYSTEM
#ssl-default-server-ciphers PROFILE=SYSTEM
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
log global
option tcplog
option dontlognull
option http-server-close
#option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
# Control Plane config - external
frontend api
bind 192.168.124.174:6443
mode tcp
default_backend api-be
# Control Plane config - internal
frontend api-int
bind 10.164.76.113:6443
mode tcp
default_backend api-be
backend api-be
mode tcp
balance roundrobin
# server bootstrap 10.94.124.2:6443 check
server master01 10.94.124.3:6443 check
server master02 10.94.124.4:6443 check
server master03 10.94.124.5:6443 check
frontend machine-config
bind 10.164.76.113:22623
mode tcp
default_backend machine-config-be
backend machine-config-be
mode tcp
balance roundrobin
# server bootstrap 10.94.124.2:22623 check
server master01 10.94.124.3:22623 check
server master02 10.94.124.4:22623 check
server master03 10.94.124.5:22623 check
# apps config
frontend https
mode tcp
bind 10.164.76.113:443
default_backend https
frontend http
mode tcp
bind 10.164.76.113:80
default_backend http
frontend https-ext
mode tcp
bind 192.168.124.173:443
default_backend https
frontend http-ext
mode tcp
bind 192.168.124.173:80
default_backend http
backend https
mode tcp
balance roundrobin
server storage01 10.94.124.6:443 check
server storage02 10.94.124.7:443 check
server storage03 10.94.124.8:443 check
server worker01 10.94.124.15:443 check
server worker02 10.94.124.16:443 check
server worker03 10.94.124.17:443 check
server worker04 10.94.124.18:443 check
server worker05 10.94.124.19:443 check
server worker06 10.94.124.20:443 check
backend http
mode tcp
balance roundrobin
server storage01 10.94.124.6:80 check
server storage02 10.94.124.7:80 check
server storage03 10.94.124.8:80 check
server worker01 10.94.124.15:80 check
server worker02 10.94.124.16:80 check
server worker03 10.94.124.17:80 check
server worker04 10.94.124.18:80 check
server worker05 10.94.124.19:80 check
server worker06 10.94.124.20:80 check
And Here is the squid proxy configurations:
[root#bastion ~]# cat /etc/squid/squid.conf
#
# Recommended minimum configuration:
#
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN)
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
#http_access deny !Safe_ports
# Deny CONNECT to other than secure SSL ports
#http_access deny CONNECT !SSL_ports
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
# And finally deny all other access to this proxy
http_access deny all
# Squid normally listens to port 3128
http_port 3128
http_port 10.164.76.113:3128
# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256
# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid
#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
Can someone please help me resolve the connection problem when hitting the application endpoint?
EDITED:
I get the following error in the console pod logs:
[root#bastion cp]# oc logs -n openshift-console console-6697f85d68-p8jxf
W0404 14:59:30.706793 1 main.go:211] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
I0404 14:59:30.706887 1 main.go:288] cookies are secure!
E0404 14:59:31.221158 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 14:59:41.690905 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 14:59:52.155373 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 15:00:02.618751 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 15:00:13.071041 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 15:00:23.531058 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 15:00:33.999953 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 15:00:44.455873 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
E0404 15:00:54.935240 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.oc.sow.expert/oauth/token failed: Head "https://oauth-openshift.apps.oc.sow.expert": EOF
I0404 15:01:05.666751 1 main.go:670] Binding to [::]:8443...
I0404 15:01:05.666776 1 main.go:672] using TLS
I just resolved this issue. To check you have the same issue:
oc logs -n openshift-console console-xxxxxxx-yyyyy
Check if you have messages like these:
error contacting auth provider (retrying in 10s): request to OAuth
issuer endpoint
https://oauth-openshift.apps.oc4.tt.testing/oauth/token failed: Head
"https://oauth-openshift.apps.oc4.tt.testing": dial tcp: lookup
oauth-openshift.apps.oc4.tt.testing on 172.30.0.10:53: no such host
In my case I'm deploying through libvirt. Libvirt does part of the DNS resolving.
I had already added this entry to the libvirt network however I had to delete and add it again.
WORKER_IP=192.168.126.51
virsh net-update oc4-xxxx delete dns-host "<host ip='$WORKER_IP'><hostname>oauth-openshift.apps.oc4.tt.testing</hostname></host>"
virsh net-update oc4-xxxx add dns-host "<host ip='$WORKER_IP'><hostname>oauth-openshift.apps.oc4.tt.testing</hostname></host>"

Istio mtls misconfiguration causes inconsistent behavior

I have deployed 2 istio enabled services on a GKE cluster.
istio version is 1.1.5 and GKE is on v1.15.9-gke.24
istio has been installed with global.mtls.enabled=true
serviceA communicates properly
serviceB apparently has TLS related issues.
I spin up a non-istio enabled deployment just for testing and exec into this test pod to curl these 2 service endpoints.
/ # curl -v serviceA
* Rebuilt URL to: serviceA/
* Trying 10.8.61.75...
* TCP_NODELAY set
* Connected to serviceA (10.8.61.75) port 80 (#0)
> GET / HTTP/1.1
> Host: serviceA
> User-Agent: curl/7.57.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 130
< server: istio-envoy
< date: Sat, 25 Apr 2020 09:45:32 GMT
< x-envoy-upstream-service-time: 2
< x-envoy-decorator-operation: serviceA.mynamespace.svc.cluster.local:80/*
<
{"application":"Flask-Docker Container"}
* Connection #0 to host serviceA left intact
/ # curl -v serviceB
* Rebuilt URL to: serviceB/
* Trying 10.8.58.228...
* TCP_NODELAY set
* Connected to serviceB (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> Host: serviceB
> User-Agent: curl/7.57.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Execing into the envoy proxy of the problematic service and turning trace level logging on, I see this error
serviceB-758bc87dcf-jzjgj istio-proxy [2020-04-24 13:15:21.180][29][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C1484] handshake error: 1
serviceB-758bc87dcf-jzjgj istio-proxy [2020-04-24 13:15:21.180][29][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:201] [C1484] TLS error: 268435612:SSL routines:OPENSSL_internal:HTTP_REQUEST
The envoy sidecars of both containers, display similar information when debugging their certificates.
I verify this by execing into both istio containers, cd-ing to /etc/certs/..data and running
openssl x509 -in root-cert.pem -noout -text
The two root-cert.pem are identical!
Since those 2 istio proxies have exactly the same tls configuration in terms of certs, why this cryptic SSL error on serviceB?
FWIW serviceB communicates with a non-istio enabled postgres service.
Could that be causing the issue?
curling the container of serviceB from within itself however, returns a healthy response.

How can I deploy Istio Jaeger UI tracing in production without using kubectl port forwarding

I am trying to deploy Istio Jaeger UI for distributed tracing. Currently I am using kubectl port forwarding using the command kubectl port-forward -n monitoring prometheus-prometheus-operator-prometheus-0 9090. But it runs on http://localhost:port So how can I do it in production? Is there any other way to deploy in production. And also how can I make it run on https?
According to the documentation Remotely Accessing Telemetry Addons. There are different ways how to acces telemetry.
The Recommended way is to create Secure acces using https instead of http.
Note for both methods:
This option covers securing the transport layer only. You should also configure the telemetry addons to require authentication when exposing them externally.
Please note that jaeger itself doesn't support authentication methods github and workaround using Apache httpd server here.
With your recruitments you can use Gateways (SDS) with self-signed certificates:
a.) Make sure your that during istio instalation youe have enabled SDS at ingress gateway --set gateways.istio-ingressgateway.sds.enabled=true and --set tracing.enabled=true for tacing purposes.
b.) Create self signed certificates for testing purposes you can use this example and repository.
c.) Please follow Generate client and server certificates and keys and Configure a TLS ingress gateway using SDS.
Create Virtualservice and Gateway:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: mygateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: "httpbin-credential" # must be the same as secret crated in the step 2.
hosts:
- "httpbin.example.com" ## You can apply "*" for all hosts
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: tracing
spec:
hosts:
- "httpbin.example.com" ## You can apply "*" for all hosts
gateways:
- mygateway
http:
- match:
- port: 443
route:
- destination:
port:
number: 80
host: tracing.istio-system.svc.cluster.local
curl -kvI https ://xx.xx.xx.xx/
* Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
> HEAD / HTTP/1.1
> Host: xx.xx.xx.xx
> User-Agent: curl/7.52.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
HTTP/2 200
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< date: Thu, 07 Nov 2019 10:01:33 GMT
date: Thu, 07 Nov 2019 10:01:33 GMT
< x-envoy-upstream-service-time: 1
x-envoy-upstream-service-time: 1
< server: istio-envoy
server: istio-envoy
Hope this help

Istio - Connection timeout when calling service-two from service-one (examples)

I'm following the tutorials to evaluate Istio as the service mesh for my K8s cluster, but for some reason I cannot make the simple example that uses a couple of services to work properly:
https://istio.io/docs/tasks/integrating-services-into-istio.html
If I try to call service-two from service-one, I get this error:
# kubectl exec -ti ${CLIENT} -- curl -v service-two:80
Defaulting container name to app.
Use 'kubectl describe pod/service-one-714088666-73fkp' to see all of the containers in this pod.
* Rebuilt URL to: service-two:80/
* Trying 10.102.51.89...
* connect to 10.102.51.89 port 80 failed: Connection refused
* Failed to connect to service-two port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to service-two port 80: Connection refused
However, if I try to connect to service-two from another service in my cluster, even in a different namespace, then it works:
# kubectl exec -ti redis-4054078334-mj287 -n redis -- curl -v service-two.default:80
* Rebuilt URL to: service-two.default:80/
* Hostname was NOT found in DNS cache
* Trying 10.102.51.89...
* Connected to service-two.default (10.102.51.89) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: service-two.default
> Accept: */*
>
< HTTP/1.1 200 OK
* Server envoy is not blacklisted
< server: envoy
< date: Sat, 19 Aug 2017 14:43:01 GMT
< content-type: text/plain
< x-envoy-upstream-service-time: 2
< transfer-encoding: chunked
<
CLIENT VALUES:
client_address=127.0.0.1
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://service-two.default:8080/
SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001
HEADERS RECEIVED:
accept=*/*
content-length=0
host=service-two.default
user-agent=curl/7.38.0
x-b3-sampled=1
x-b3-spanid=00000caf6e052e86
x-b3-traceid=00000caf6e052e86
x-envoy-expected-rq-timeout-ms=15000
x-forwarded-proto=http
x-ot-span-context=00000caf6e052e86;00000caf6e052e86;0000000000000000;cs
x-request-id=1290973c-7bca-95d2-8fa8-80917bb404ad
BODY:
* Connection #0 to host service-two.default left intact
-no body in request-
Any reason or explanation why I get this unexpected behaviour?
Thanks.
I figured out what happened: on service-one the init containers had not been properly completed, so it was not resolving correctly.

Kubernetes unable to access the kube-apiserver from inside pod on node

I have configured a vagrant backed kubernetes cluster but I am unable to access the kube-apiserver running on master from within pods running on nodes. I am trying to look up a service from within a pod via the api but it looks like the api keeps dropping the connection.
Using curl from within the pod I get the following output
root#itest-pod-2:/# curl -v \
--insecure -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://$KUBERNETES_SERVICE_HOST:443/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
* Trying 10.245.0.1...
* Connected to 10.245.0.1 (10.245.0.1) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 10.245.0.1:443
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to 10.245.0.1:443
root#itest-pod-2:/#
However if I configure a single machine environment by simply installing all the node components onto the master I am able to contact the api from within a pod.
root#itest-pod-3:/# curl -v --insecure \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://$KUBERNETES_SERVICE_HOST:443/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
* Trying 10.245.0.1...
* Connected to 10.245.0.1 (10.245.0.1) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA
* Server certificate:
* subject: CN=10.0.2.15#1452869292
* start date: 2016-01-15 14:48:12 GMT
* expire date: 2017-01-14 14:48:12 GMT
* issuer: CN=10.0.2.15#1452869292
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /api/v1/namespaces/default/services?labelSelector=name%3Dtest-server HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 10.245.0.1
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tdDY3cXUiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImIxNGI4YWE3LWJiOTgtMTFlNS1iNjhjLTA4MDAyN2FkY2NhZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.HhPnit7Sfv-yUkMW6Cy9ZVbuiV2wt5PLaPSP-uZtaByOPagkb8d-8zBQE8Lx53lqxMFwBmjjxSWl-vKtSGa0ka6rEkq_xWtFJb8uDDlxz_R63R6IJBWB8YhwB7SzPVWgtHohj55D3pL8-r8NOQSQVXFAHaiGTlzmtwiE3CmJv3yBzBLALG0yvtW2YgwrO9jlxCGdFIOKae-5eduiOyZHUimxAgfBkbwSNfSzXYZTJNryfPiDBKZybh9c3Wd-pNsSZyw9gbBhbGFM7EiK9pWsdViQ__fZA2JbxX78YbajWE6CeL4FWLKFu4MuIlnmhLOvOXia_9WXz1B8XJ-MlzclZQ
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 15 Jan 2016 16:37:40 GMT
< Content-Length: 171
<
{
"kind": "ServiceList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/default/services",
"resourceVersion": "1518"
},
"items": []
}
* Connection #0 to host 10.245.0.1 left intact
What's confusing me is that the configuration is the same in both cases except that the node components have been installed into the master, which makes me think it is not a misconfiguration of ssl/https so much as it is something to do with the kubernetes network configuration.
I've looked into the logs of the apiserver but I can't see anything related to these dropped connections.
Any help would be greatly appreciated.
I assume you pass the kubelet the worker/minion certificate/key pair.
Does the api-server certificate include the distinguished name / alternate name pointing to the master (the subjectAltName on your openssl.conf should have the IP of the master when generating the API server cert)
That is I think the most common reason for this problem.
Create your CA key, and cert
then:
cat <<EOT >openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = #alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
IP.1 = ${K8S_SERVICE_IP}
IP.2 = ${MASTER_HOST}
EOT
openssl genrsa -out apiserver-key.pem 2048
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config ./openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile ./openssl.cnf
The problem was that we had not set the bind address for the
apiserver (we had set insecure bind address but not --bind-address)
we thought this would not be a problem since by default the apiserver binds on all interfaces.
When bound on all interfaces calls to /api/v1/endpoints return the eth0 IP for the apiserver secure port. In most cases this would probably be fine
but since we were running kubernetes on a virtualbox vm eth0 is the
NAT interface created by by virtualbox that can onl be reached through
host ports on which VBoxHeadless is listening.
When outgoing traffic leaves a pod it hits a set of iptables rules
matching cluster service ips and redirecting to a port on the proxy
the proxy then forwards the request to the actual machine in the
cluster.
In this case kube-proxy did not have available the
externally reachable ip for the apiservice instead it had unreachable the eth0 address used by virtualbox.
Oddly though it seems as if the proxy then attempts to contact the
api on its insecure port (it knows the intended destination for the
request due to the iptables rules which it creates). Since our request
in this case is https the apiserver drops it after the first client
hello.
Normally in curl this looks like this
root#app-master-0:/home/vagrant# curl -v --insecure \
https://10.235.1.2:8080/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
* Trying 10.235.1.2...
* Connected to 10.235.1.2 (10.235.1.2) port 8080 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol
But when being proxied through the kube proxy it looks like this
root#itest-pod-2:/# curl -v --insecure \
https://$KUBERNETES_SERVICE_HOST:443/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
* Trying 10.245.0.1...
* Connected to 10.245.0.1 (10.245.0.1) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 10.245.0.1:443
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to 10.245.0.1:443
by adding --bind-address=xxxx with the externally reachable eth1 ip to the apiserver's args we were able to fix this.