server closed the stream without sending trailers - kubernetes

Question

I’m trying o communicate from Envoy to Envoy using gRPC for Kubernetes(Amazon EKS).
I have an envoy in my sidecar and I am using grpcurl to validate the request.
The request is delivered to the application container and there are no errors, but the console returns the following results
server closed the stream without sending trailers
I don’t know what the reason for the above problem is, and what could be the reason for this result??
I was able to confirm that the response came back fine when I hit a single service before connecting with envoy.
this my envoy config
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 127.0.0.1
port_value: 10000
static_resources:
listeners:
- name: listener_secure_grpc
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 8443
traffic_direction: INBOUND
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: service_grpc
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: cluster_grpc
max_stream_duration:
grpc_timeout_header_max: 30s
tracing: {}
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck"
pass_through_mode: false
headers:
- name: ":path"
exact_match: "/healthz"
- name: envoy.filters.http.router
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"#type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext"
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/ssl/grpc/tls.crt
private_key:
filename: /etc/ssl/grpc/tls.key
- name: listener_stats
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10001
traffic_direction: INBOUND
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: AUTO
stat_prefix: ingress_http
route_config:
virtual_hosts:
- name: backend
domains:
- "*"
routes:
- match:
prefix: /stats
route:
cluster: cluster_admin
http_filters:
- name: envoy.filters.http.router
- name: listener_healthcheck
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 10010
traffic_direction: INBOUND
filter_chains:
- filters:
- name: envoy.http_connection_manager
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
codec_type: AUTO
stat_prefix: ingress_http
route_config: {}
http_filters:
- name: envoy.filters.http.health_check
typed_config:
"#type": "type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck"
pass_through_mode: false
headers:
- name: ":path"
exact_match: "/healthz"
- name: envoy.filters.http.router
clusters:
- name: cluster_grpc
connect_timeout: 1s
type: STATIC
http2_protocol_options: {}
upstream_connection_options:
tcp_keepalive: {}
load_assignment:
cluster_name: cluster_grpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 1443
- name: cluster_admin
connect_timeout: 1s
type: STATIC
load_assignment:
cluster_name: cluster_grpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 10000
P.S 2021.03.19
Here's what else I found out.
When I request from the ingress host, I get the above failure, but when I request from the service, I get a normal response!