How to integrate opensearch with logstash on the Kubernetes Cluster? - kubernetes

Question

I have a kubernetes cluster on which I have deployed a opensearch cluster and opensearch dashboard using Helm, I am also able to deploy logstash using helm successfully but I am confused on how to integrate those, I want to feed data to my Opensearch using logstash as my OBJECTIVE as I am not able to find much documentation on it as well. Any help is appreciated....Thanks in advance!
Deployed opensearch using Helm and logstash as well but unable to integrate them
Update here!!!
Have made a few changes to simplify the deployment and more control over the function,
I am testing deployment and service files this time, I will add the files below
Opensearch deployment file
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: logging
name: opensearch
labels:
component: opensearch
spec:
selector:
matchLabels:
component: opensearch
replicas: 1
serviceName: opensearch
template:
metadata:
labels:
component: opensearch
spec:
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: opensearch
securityContext:
capabilities:
add:
- IPC_LOCK
image: opensearchproject/opensearch
env:
- name: KUBERNETES_CA_CERTIFICATE_FILE
value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: "cluster.name"
value: "opensearch-cluster"
- name: "network.host"
value: "0.0.0.0"
- name: "discovery.seed_hosts"
value: "[]"
- name: discovery.type
value: single-node
- name: OPENSEARCH_JAVA_OPTS
value: -Xmx512M -Xms512M
- name: "plugins.security.disabled"
value: "false"
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- name: os-mount
mountPath: /data
volumes:
- name: os-mount
persistentVolumeClaim:
claimName: nfs-pvc-os-logging
Opensearch svc file
---
apiVersion: v1
kind: Service
metadata:
name: opensearch
namespace: logging
labels:
service: opensearch
spec:
type: ClusterIP
selector:
component: opensearch
ports:
- port: 9200
targetPort: 9200
Opensearch dashboard deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: open-dash
namespace: logging
spec:
replicas: 1
selector:
matchLabels:
app: open-dash
template:
metadata:
labels:
app: open-dash
spec:
# securityContext:
# runAsUser: 0
containers:
- name: opensearch-dashboard
image: opensearchproject/opensearch-dashboards:latest
ports:
- containerPort: 80
env:
# - name: ELASTICSEARCH_URL
# value: https://opensearch.logging:9200
# - name: "SERVER_HOST"
# value: "localhost"
# - name: "opensearch.hosts"
# value: https://opensearch.logging:9200
- name: OPENSEARCH_HOSTS
value: '["https://opensearch.logging:9200"]'
Opensearch Dashboard svc
---
apiVersion: v1
kind: Service
metadata:
name: opensearch
namespace: logging
labels:
service: opensearch
spec:
type: ClusterIP
selector:
component: opensearch
ports:
- port: 9200
targetPort: 9200
with the above configuration I am able to get the Dashboard UI open but in Dashboard pod logs I can see a 400 code logs can anyone please try to reproduce this issue, Also I need to integrate the logstash with this stack.
{"type":"response","#timestamp":"2023-02-20T05:05:34Z","tags":[],"pid":1,"method":"head","statusCode":400,"req":{"url":"/app/home","method":"head","headers":{"connection":"Keep-Alive","content-type":"application/json","host":"3.108.199.0:30406","user-agent":"Manticore 0.9.1","accept-encoding":"gzip,deflate","securitytenant":"user"},"remoteAddress":"10.244.1.1","userAgent":"Manticore 0.9.1"},"res":{"statusCode":400,"responseTime":2,"contentLength":9},"message":"HEAD /app/home 400 2ms - 9.0B
When deploying a logstash pod I get an error that
[WARN ] 2023-02-20 05:13:52.212 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:xxxxxx#opensearch.logging:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:xxxxxx#opensearch.logging:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}
Can somebody please try to help me with this puzzle
#Benla have made changes as per your recommendation to image now I am getting the following logs in logstash
[2023-02-20T05:18:43,028][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-02-20T05:18:43,147][INFO ][org.reflections.Reflections] Reflections took 70 ms to scan 1 urls, producing 127 keys and 444 values
[2023-02-20T05:18:43,351][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-02-20T05:18:43,370][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3bf49916#/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-02-20T05:18:43,811][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.44}
[2023-02-20T05:18:43,816][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-02-20T05:18:43,821][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-02-20T05:18:43,835][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-02-20T05:18:43,869][INFO ][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] Starting server on port: 5044
I started getting these endless loop of logs
[WARN ] 2023-02-20 05:13:37.191 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:xxxxxx#opensearch.logging:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:xxxxxx#opensearch.logging:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}

Answer 1

I think there is not much difference in deploying logstash<->opensearch pipeline from deploying logstash<->elasticsearch in K8S,
The only differences I've encountered are:
The docker or other container runtime image should be opensearchproject/logstash-oss-with-opensearch-output-plugin:latest rather than logstash.
The output plugin in your logstash config file sould be
opensearch rather than elasticsearch.
Other than those you can go to every Logstash<->ElasticSearch in K8S tutorial, there are many.
Good Luck!