Can we trace external API calls with Istio behind proxy via Kiali? - kubernetes

Question

We have a Nodejs based microservices running in our on-prem kubernetes v1.19 with Istio v1.8.0. What I would like to achieve is trace or display the external API calls in Kiali where we have Jaeger clients for each microservices and able to trace internal traffics.
But so far I could not able to trace any external API calls hits from any microservices.The only thing that I can see the traffic for proxy in Kiali's graph overview.
We have a cooperate proxy, and each container have env proxies set for both http_proxy, https_proxy.Any external service accessible via a cooperate proxy thus traffics should go through the our cooperate proxy first. We have a secured gateway with TLS and we do not have egressgateway where only have istio-ingressgateway.
So is there anyway to trace external traffics likewise the internal traffics inside cluster?If yes what might be the missing thing?
$ kubectl get pods -n dev
NAME READY STATUS RESTARTS AGE
api-dev-74896ff4f9-slxt5 3/3 Running 0 7h1m
auth-dev-98f77d487-qt5zd 3/3 Running 0 3d5h
backend-dev-bb7765464-b7bpr 2/2 Running 0 7d3h
mp-dev-86d6b8b978-slqp7 3/3 Running 0 5d9h
ui-dev-d5667946b-sdvlc 2/2 Running 0 5d4h
Here are the ServiceEntries and VirtualServices that I created where I would like to use the retry feature as well the calls for proxy and externalAPI
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: company-proxy
namespace: dev
spec:
hosts:
- foo-proxy.net
ports:
- number: PORT
name: tcp
protocol: TCP
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: proxy
namespace: dev
spec:
hosts:
- "foo-proxy.net"
http:
- name: "company-proxy"
match:
- uri:
prefix: "/"
route:
- destination:
host: "foo-proxy.com"
timeout: 90s
retries:
retryOn: "5xx"
attempts: 3
perTryTimeout: 30s
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: foo-example.com
namespace: dev
spec:
hosts:
- "foo-example.com"
ports:
- number: 80
name: http
protocol: HTTP
- number: 443
name: https
protocol: HTTPS
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: foo-example.com
namespace: dev
spec:
hosts:
- "foo-example.com"
http:
- name: "developer-api"
match:
- uri:
prefix: "/"
route:
- destination:
host: "foo-example.com"
timeout: 90s
retries:
retryOn: "5xx"
attempts: 3
perTryTimeout: 30s

Answer 1

I am not sure why Istio doesn't automatically trace your calls to external APIs. Perhaps it requires an egress gateway to be used, I'm not sure. Note also that Istio creates traces for http(s) traffic, not TCP.
However, this is something you can still do programmatically. You can use any of the Jaeger client libraries to augment"the traces already created by Envoy by appending your own spans.
To do so, you need first to extract the trace context from the HTTP headers of the incoming request (assuming that your external API calls are consecutive to an incoming request), and then create a new span as child of that previous span context. A good idea would be to use OpenTracing semantic conventions when you tag your new span. Tools like Kiali will be able to leverage some information if it follows this convention.
I've found this blog post that explains how to do it with the nodejs jaeger client: https://rhonabwy.com/2019/01/06/adding-tracing-with-jaeger-to-an-express-application/