Open-telemetry auto instrumentation does not work without sidecar - kubernetes

Question

I work at a startup, and we recently migrated our workloads to use Kubernetes, specifically we are running inside a cluster in EKS (AWS).
I'm currently trying to implement a observability stack on our cluster. I'm running Signoz on a separate EC2 instance (for tests, and because our cluster uses small machines that are not supported by their helm chart).
In the cluster, I'm running the Open Telemetry Operator, and have managed to deploy a Collector in deployment mode, and have validated that it is able to connect to the signoz instance. However, when I try to auto instrument my applications, I'm not able to do it without using sidecars.
My manifest file for the elements above is below.
apiVersion: v1
kind: Namespace
metadata:
name: opentelemetry
labels:
name: opentelemetry
---
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-collector
namespace: opentelemetry
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
otlp:
endpoint: obs.stg.company.domain:4317
tls:
insecure: true
logging:
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp, logging]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp, logging]
logs:
receivers: [otlp]
processors: []
exporters: [otlp, logging]
---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
namespace: opentelemetry
spec:
exporter:
endpoint: http://otel-collector-collector.opentelemetry.svc.cluster.local:4317
propagators:
- tracecontext
- baggage
- b3
sampler:
type: parentbased_traceidratio
argument: "0.25"
dotnet:
nodejs:
When I apply the annotation instrumentation.opentelemetry.io/inject-dotnet=opentelemetry/auto-instrumentation to the deployment of the application, or even to the namespace, and delete the pod (so it is recreated), I can see that the init container for dotnet auto instrumentation runs without a problem, but I get no traces, metrics or logs, either on the Collector or in Signoz.
If I create another collector in sidecar mode, like the one below, point the instrumentation to this collector, and also apply annotation sidecar.opentelemetry.io/inject=sidecar to the namespace, everything works fine.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: sidecar
namespace: application
spec:
mode: sidecar
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
logging:
otlp:
endpoint: "http://otel-collector-collector.opentelemetry.svc.cluster.local:4317"
tls:
insecure: true
service:
telemetry:
logs:
level: "debug"
pipelines:
traces:
receivers: [otlp]
processors: []
exporters: [logging, otlp]
The reason I am trying to do it without sidecars is that, as I said before, we have quite a small cluster, and would like to keep overhead to a minimum.
So, I would like first to understand if I should even be worried about sidecars, if their overhead is measurably different than not using them.
And second, I would like to understand what went wrong with my config, since I believe I followed all the instructions in Signoz's documentation.
Thank you for any help you guys can provide.