Route TCP traffic trough Istio egressgateway - kubernetes

I have a deployment running frontier-squid proxy for caching, and I need to route all outbound TCP traffic through an istio-egressgateway in order to exit the mesh always from the same host/ip.
I tried to follow the egress-mongo example but unsuccessfully.
For reference the inbound part is working fine and traffic is also exiting the mesh, from the host where the pod is running (unwanted behavior).
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cvmfs
namespace: lcg
annotations:
co.elastic.logs/enabled: "true"
co.elastic.logs/module: "squid"
labels:
app: cvmfs
spec:
replicas: 1
selector:
matchLabels:
app: cvmfs
template:
metadata:
labels:
app: cvmfs
spec:
containers:
- name: cvmfs
image: opensciencegrid/frontier-squid:stable
env:
## SQUID_IPRANGE: Don't add external ip here. Proxied traffic dosen't have external ip. See below AuthorizationPolicy ingress-policy-cvmfs
- name: "SQUID_IPRANGE"
value: "10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 fc00::/7 fe80::/10 127.0.0.0/24"
- name: "SQUID_CACHE_DISK"
value: "50000"
- name: "SQUID_CACHE_MEM"
value: "16 GB"
ports:
- containerPort: 3128
resources:
requests:
cpu: 4
memory: 20G
limits:
cpu: 8
memory: 32G
---
apiVersion: v1
kind: Service
metadata:
labels:
app: cvmfs
name: cvmfs
spec:
ports:
- port: 3128
selector:
app: cvmfs
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: lcg-cvmfs-gw
namespace: lcg
spec:
selector:
app: clustername-vlan320-gateway-cvmfs1
istio: ingressgateway
servers:
- port:
number: 3128
name: tcp-squid
protocol: TCP
hosts:
- cvmfs1.sub.domain.com
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: lcg-cvmfs-vs
namespace: lcg
spec:
hosts:
- cvmfs1.sub.domain.com
gateways:
- lcg-cvmfs-gw
tcp:
- match:
- port: 3128
route:
- destination:
host: cvmfs
port:
number: 3128
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ingress-policy-cvmfs
namespace: istio-system
spec:
selector:
matchLabels:
app: clustername-vlan320-gateway-cvmfs1
istio: ingressgateway
action: ALLOW
rules:
- from:
- source:
remoteIpBlocks:
- aa.aa.aa.aa/19
- bb.bb.bb.bb/25
Here my attempt to setup all steps to exit the mesh from the istio-egressgateway:
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: external-stratum-1
namespace: lcg
spec:
hosts:
- external.tcp.svc # Not used in TCP mode
addresses:
- xx.xx.xx.xx/32
- xx.xx.xx.xx/32
ports:
- number: 8000
name: tcp-stratum1
protocol: TCP
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: xx.xx.xx.xx/32
- address: xx.xx.xx.xx/32
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
namespace: lcg
spec:
selector:
istio: egressgateway
servers:
- port:
number: 55555
name: tcp-exit1
protocol: TCP
hosts:
- external.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-cvmfs-stratum1
namespace: lcg
spec:
host: istio-egressgateway.istio-system.svc.clustername.domain.com
subsets:
- name: external-stratum-1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: external-stratum-1
spec:
host: external.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-stratum1-through-egress-gateway
namespace: lcg
spec:
hosts:
- external.tcp.svc
gateways:
- mesh
- istio-egressgateway
tcp:
- match:
- gateways:
- mesh
destinationSubnets:
- xx.xx.xx.xx/32
- xx.xx.xx.xx/32
port: 8000
route:
- destination:
host: istio-egressgateway.istio-system.svc.clustername.domain.com
subset: external-stratum-1
port:
number: 55555
- match:
- gateways:
- istio-egressgateway
port: 55555
route:
- destination:
host: external.tcp.svc
port:
number: 8000
weight: 100
I'm running vanilla kubernetes version 1.21 and instio version 1.10
Can someone help me setting up this egressgateway or maybe suggest a better istio example?

Related

Facing 502 error while implementing Istio-ingress with envoy proxy

Below is the configuration which I am using in my environment.
And I am able to launch the site but some how inbound is getting blocked by Istio/envoy so i am not able to navigate my sites other pages which is called bu ajax getting below attached error
apiVersion: v1
kind: Service
metadata:
name: svc-controlcenter
namespace: ns-test
labels:
app: controlcenter
app.kubernetes.io/managed-by: Helm
env: dev
annotations:
meta.helm.sh/release-name: controlcenter
meta.helm.sh/release-namespace: ns-test
spec:
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8001
selector:
app: controlcenter
env: dev
Istio-Gateway
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: cc-gw-apps
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- "*.<URL>"
port:
name: https
number: 443
protocol: HTTPS
tls:
credentialName: secret
mode: SIMPLE
Virtual Services
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vs-cc
namespace: ns-test
spec:
hosts:
- "<FQDN>"
gateways:
- istio-system/cc-gw-apps
http:
- match:
- uri:
prefix: "/"
route:
- destination:
host: <same>.ns-test.svc.cluster.local
port:
number: 80

Kiali: Avoid KIA1106 - More than one Virtual Service for same host

I'm using Kiali on Istio/Kubernetes to monitor my mesh.
I need to route 2 different pods based in URL contain, and for this, I'm following the tutorial in Split large virtual services and destination rules into multiple resources. So, I created 2 VirtualServices for the same host and gateway:
Service 1:
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vs-product-composite
spec:
hosts:
- "kubernetes.b-thinking.com"
gateways:
- gw-ingress
http:
- match:
- uri:
prefix: /product-composite
route:
- destination:
port:
number: 80
host: product-composite
Service 2:
apiVersion: apps/v1
kind: Deployment
metadata:
name: uaa
spec:
replicas: 1
selector:
matchLabels:
app: uaa
template:
metadata:
labels:
app: uaa
spec:
containers:
- name: uaa
image: bthinking/uaa
imagePullPolicy: Never
env:
- name: LOGGING_LEVEL_ROOT
value: DEBUG
ports:
- containerPort: 8090
resources:
limits:
memory: 350Mi
---
apiVersion: v1
kind: Service
metadata:
name: uaa
spec:
type: NodePort
selector:
app: uaa
ports:
- port: 8090
nodePort: 31090
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: vs-auth
spec:
hosts:
- "kubernetes.b-thinking.com"
gateways:
- gw-ingress
http:
- match:
- uri:
prefix: /oauth
rewrite:
uri: "/uaa/oauth"
route:
- destination:
port:
number: 8090
host: uaa
But, I'm getting the warning KIA1106 - More than one Virtual Service for same host
Looking in docs, it explain the case (it's my case), but it redirect to the same guide I have followed.

Egress gateway does't work correctly when adding multiple external services

I was trying to define multiple external services(redis: AWS ElastiCache) to route through egress gateway. Two gateways were configured with reference to https://github.com/istio/istio/issues/16806#issuecomment-538718737. I applied the following config, but it didn't work correctly and I could find error logs. It seemed that connecting to both of the destinations was forward to only one of them(random?)
Is this a bug or is there any solution?
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway-redis-1
namespace: test
spec:
selector:
istio: egressgateway
servers:
- port:
name: redis
number: 6379
protocol: TCP
hosts:
- "aaa.aaa.ng.0001.apne1.cache.amazonaws.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: egress-redis-1
namespace: test
spec:
hosts:
- aaa.aaa.ng.0001.apne1.cache.amazonaws.com
ports:
- name: egress-redis-1
number: 6379
protocol: TCP
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: egress-redis-1
namespace: test
spec:
hosts:
- aaa.aaa.ng.0001.apne1.cache.amazonaws.com
gateways:
- istio-egressgateway-redis-1
- mesh
tcp:
- match:
- gateways:
- mesh
port: 6379
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: egress-redis-1
port:
number: 6379
- match:
- gateways:
- istio-egressgateway-redis-1
port: 6379
route:
- destination:
host: aaa.aaa.ng.0001.apne1.cache.amazonaws.com
port:
number: 6379
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egress-redis-1
namespace: test
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: egress-redis-1
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway-redis-2
namespace: test
spec:
selector:
istio: egressgateway
servers:
- port:
name: redis
number: 6379
protocol: TCP
hosts:
- "bbb.bbb.clustercfg.apne1.cache.amazonaws.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: egress-redis-2
namespace: test
spec:
hosts:
- bbb.bbb.clustercfg.apne1.cache.amazonaws.com
ports:
- name: egress-redis-2
number: 6379
protocol: TCP
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: egress-redis-2
namespace: test
spec:
hosts:
- bbb.bbb.clustercfg.apne1.cache.amazonaws.com
gateways:
- istio-egressgateway-redis-2
- mesh
tcp:
- match:
- gateways:
- mesh
port: 6379
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: egress-redis-2
port:
number: 6379
- match:
- gateways:
- istio-egressgateway-redis-2
port: 6379
route:
- destination:
host: bbb.bbb.clustercfg.apne1.cache.amazonaws.com
port:
number: 6379
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egress-redis-2
namespace: test
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: egress-redis-2
Istio error logs:
2020-07-14T08:24:42.803875Z#011info#011ads#011Push Status: {
"pilot_conflict_outbound_listener_tcp_over_current_tcp": {
"0.0.0.0:6379": {
"proxy": "member-11111-22222.test",
"message": "Listener=0.0.0.0:6379 AcceptedTCP=aaa.aaa.clustercfg.apne1.cache.amazonaws.com RejectedTCP=bbb.bbb.ng.0001.apne1.cache.amazonaws.com TCPServices=1"
}
}
}
Version
Kubernetes: 1.18.5
Istio: 1.6.4
outboundTrafficPolicy
REGISTRY_ONLY
After updated port names of Gateway to unique, it still doesn't work.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway-redis-1
namespace: qa2
spec:
selector:
istio: egressgateway
servers:
- port:
name: egress-redis-1
number: 6379
protocol: TCP
hosts:
- "aaa.aaa.ng.0001.apne1.cache.amazonaws.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway-redis-2
namespace: qa2
spec:
selector:
istio: egressgateway
servers:
- port:
name: egress-redis-2
number: 6379
protocol: TCP
hosts:
- "bbb.bbb.clustercfg.apne1.cache.amazonaws.com"

How to deny default but allow HTTP and TCP traffic in istio kubernetes cluster?

I have a cluster with istio injection enabled and cockroach db stateful set defined:
apiVersion: v1
kind: ServiceAccount
metadata:
name: cockroachdb-serviceaccount
---
apiVersion: v1
kind: Service
metadata:
# This service is meant to be used by clients of the database. It exposes a ClusterIP that will
# automatically load balance connections to the different database pods.
name: cockroachdb-public
labels:
app: cockroachdb
spec:
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
# traffic and the cli.
- port: 26257
targetPort: 26257
name: tcp
# The secondary port serves the UI as well as health and debug endpoints.
- port: 8080
targetPort: 8080
name: http
selector:
app: cockroachdb
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cockroachdb-statefulset
labels:
version: v20.1.2
spec:
serviceName: cockroachdb
replicas: 3
selector:
matchLabels:
app: cockroachdb
template:
metadata:
labels:
app: cockroachdb
version: v20.1.2
spec:
serviceAccountName: cockroachdb-serviceaccount
containers:
- name: cockroachdb
image: cockroachdb/cockroach:v20.1.2
ports:
- containerPort: 26257
name: tcp
- containerPort: 8080
name: http
volumeMounts:
- name: datadir
mountPath: /cockroach/cockroach-data
env:
- name: COCKROACH_CHANNEL
value: kubernetes-insecure
command:
- "/bin/bash"
- "-ecx"
# The use of qualified `hostname -f` is crucial:
# Other nodes aren't able to look up the unqualified hostname.
- "exec /cockroach/cockroach start --logtostderr --insecure --advertise-host $(hostname -f) --http-addr 0.0.0.0 --join cockroachdb-statefulset-0.cockroachdb,cockroachdb-statefulset-1.cockroachdb,cockroachdb-statefulset-2.cockroachdb --cache 25% --max-sql-memory 25%"
# No pre-stop hook is required, a SIGTERM plus some time is all that's
# needed for graceful shutdown of a node.
terminationGracePeriodSeconds: 5
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 4Gi
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: cockroachdb-public
spec:
host: cockroachdb-public
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: cockroachdb-public
spec:
hosts:
- cockroachdb-public
http:
- match:
- port: 8080
route:
- destination:
host: cockroachdb-public
port:
number: 8080
tcp:
- match:
- port: 26257
route:
- destination:
host: cockroachdb-public
port:
number: 26257
and a service that accesses it:
apiVersion: v1
kind: ServiceAccount
metadata:
name: downstream-serviceaccount
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: downstream-deployment-v1
labels:
app: downstream
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: downstream
version: v1
template:
metadata:
labels:
app: downstream
version: v1
spec:
serviceAccountName: downstream-serviceaccount
containers:
- name: downstream
image: downstream:0.1
ports:
- containerPort: 80
env:
- name: DATABASE_URL
value: postgres://roach#cockroachdb-public:26257/roach?sslmode=disable
---
apiVersion: v1
kind: Service
metadata:
name: downstream-service
labels:
app: downstream
spec:
type: ClusterIP
selector:
app: downstream
ports:
- port: 80
targetPort: 80
name: http
protocol: TCP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: downstream-service
spec:
host: downstream-service
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: downstream-service
spec:
hosts:
- downstream-service
http:
- name: "downstream-service-routes"
match:
- port: 80
route:
- destination:
host: downstream-service
port:
number: 80
Now I'd like to restrict access to cockroach db only to downstream-service and to cockroachdb itself (since nodes need intercommunication between each other).
I'm trying to restrict the traffic with something like this:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: default-deny-all
namespace: default
spec:
{}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cockroachdb-authorizationpolicy-allow-from-downstream
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/downstream-serviceaccount"]
- to:
- operation:
ports: ["26257"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cockroachdb-authorizationpolicy-allow-from-cockroachdb
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/cockroachdb-serviceaccount"]
- to:
- operation:
ports: ["26257"]
but doesn't seem to do anything. I can still e.g. access cockroachdb-public:8080 cluster HTTP UI from downstream-service.
Now when I add the following:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: default-deny-all-to-cockroachdb
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: DENY
rules:
- to:
- operation:
ports: ["26257"]
then all the traffic is blocked (including the traffic between cockroachdb nodes).
What am I doing wrong here?
You are having the same problem that a guy couple of days ago. In your authorization policy you have two policies:
service account downstream-serviceaccount (and cockroachdb-serviceaccount for the other authorization policy) from default namespace can access the service with labels app: cockroachdb on any port on default namespace.
Any service account, from any namespace can access the service with labels app: cockroachdb, on port 26257.
In order to make it an AND, you would do this:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cockroachdb-authorizationpolicy-allow-from-cockroachdb
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/cockroachdb-serviceaccount"]
to: <- remove the dash from here
- operation:
ports: ["26257"]
Same with the other AuthorizationPolicy object. Also note that you don't need to explicitly create a DENY policy. When you create an ALLOW one, it automatically denies everything else.

How to write a custom ingressgateway in istio?

I'm new to istio, I have a simple test yaml file which is a little long. What I want to do is to write a custom ingressgateway service for my gateway. And after testing, the incorrect part is the definition of ingressgateway which is at the top. The entire yaml is below:
apiVersion: v1
kind: Service
metadata:
name: batman-ingressgateway
labels:
app: batman-ingressgateway
spec:
type: LoadBalancer
selector:
app: batman-ingressgateway
ports:
- port: 80
targetPort: 80
nodePort: 31389
name: http
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: batman-gateway
spec:
selector:
app: batman-ingressgateway
#istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: batman
spec:
hosts:
- "*"
gateways:
- batman-gateway
http:
- match:
route:
- destination:
host: batman
port:
number: 8000
subset: v1
weight: 80
- destination:
host: batman
port:
number: 8000
subset: v2
weight: 20
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: batman-destination
spec:
host: batman
subsets:
- name: v1
labels:
version: v1
run: batman
- name: v2
labels:
version: v2
run: batman
I want to access my app from browser with the address like: http://my_host_ip:31389/article. The problem now is the ingressgateway doesn't route traffic to my gateway. Is there any one can help me?
Thanks.
Documentation on istio gateway routing is here https://istio.io/docs/tasks/traffic-management/ingress/ingress-control/.
If you look at gateway spec they have
selector:
istio: ingressgateway # use Istio default gateway implementation
While you have
selector:
app: batman-ingressgateway
#istio: ingressgateway
For VirtualService definition you can look here https://istio.io/docs/reference/config/networking/v1alpha3/virtual-service/
You can try with routing requests to /article to your service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: article-route
spec:
hosts:
- *
http:
- match:
- uri:
prefix: "/article"
route:
- destination:
host: <name of your service>