filebeat cannot connect to logstash because of selinux - kubernetes

Question

filebeat cannot connect to logstash because this happen when i upgrade kubernetes to 1.24 the log path is changed, on this version is no longer using docker as a runtime
i only change this path and cannot connect to logstash
paths:
- "/var/log/containers/*.log"
but when i run this on kubernetes with docker runtime is work
paths:
- /var/lib/docker/containers/*/*.log
here my filebeat.yml
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.config:
prospectors:
# Mounted `filebeat-prospectors` configmap:
path: ${path.config}/prospectors.d/*.yml
# Reload prospectors configs as they change:
reload.enabled: false
modules:
path: ${path.config}/modules.d/*.yml
# Reload module configs as they change:
reload.enabled: false
processors:
- add_cloud_metadata:
output.logstash:
hosts: ["log.xxx.com:5044"]
ssl.certificate_authorities: ["/usr/share/filebeat/certs/elk-ca.pem"]
ssl.certificate: "/usr/share/filebeat/certs/beats-client.pem"
ssl.key: "/usr/share/filebeat/certs/beats-client.key"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-prospectors
namespace: kube-system
labels:
k8s-app: filebeat
data:
kubernetes.yml: |-
- type: log
enabled: true
symlinks: true
paths:
- "/var/log/containers/*.log"
processors:
- add_kubernetes_metadata:
in_cluster: true
default_matchers.enabled: false
matchers:
- logs_path:
logs_path: /var/log/containers/
logs
2022-12-19T13:06:35.189Z INFO instance/beat.go:611 Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2022-12-19T13:06:35.190Z INFO instance/beat.go:618 Beat UUID: 68776622-4469-4ff4-bf1a-442d13456c75
2022-12-19T13:06:35.190Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2022-12-19T13:06:35.190Z INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "68776622-4469-4ff4-bf1a-442d13456c75"}}}
2022-12-19T13:06:35.191Z INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "1d55b4bd9dbf106a4ad4bc34fe9ee425d922363b", "libbeat": "6.7.1", "time": "2019-04-02T15:01:15.000Z", "version": "6.7.1"}}}
2022-12-19T13:06:35.191Z INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}}
2022-12-19T13:06:35.193Z INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-12-15T13:22:10Z","containerized":true,"name":"filebeat-2gh9x","ip":["127.0.0.1/8","::1/128","10.220.48.144/32","fe80::74:65ff:fe3e:27f0/64"],"kernel_version":"3.10.0-1160.80.1.el7.x86_64","mac":["02:74:65:3e:27:f0"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2022-12-19T13:06:35.195Z INFO [beat] instance/beat.go:976 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-12-19T13:06:34.150Z"}}}
2022-12-19T13:06:35.195Z INFO instance/beat.go:280 Setup Beat: filebeat; Version: 6.7.1
2022-12-19T13:06:35.197Z INFO [publisher] pipeline/module.go:110 Beat name: filebeat-2gh9x
2022-12-19T13:06:35.197Z WARN [cfgwarn] beater/filebeat.go:89 DEPRECATED: config.prospectors are deprecated, Use `config.inputs` instead. Will be removed in version: 7.0.0
2022-12-19T13:06:35.198Z INFO instance/beat.go:402 filebeat start running.
2022-12-19T13:06:35.198Z INFO registrar/registrar.go:97 No registry file found under: /usr/share/filebeat/data/registry. Creating a new registry file.
2022-12-19T13:06:35.200Z INFO registrar/registrar.go:134 Loading registrar data from /usr/share/filebeat/data/registry
2022-12-19T13:06:35.200Z INFO registrar/registrar.go:141 States Loaded from registrar: 0
2022-12-19T13:06:35.200Z WARN beater/filebeat.go:367 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2022-12-19T13:06:35.200Z INFO crawler/crawler.go:72 Loading Inputs: 0
2022-12-19T13:06:35.245Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2022-12-19T13:06:35.484Z INFO kubernetes/util.go:86 kubernetes: Using pod name filebeat-2gh9x and namespace kube-system to discover kubernetes node
2022-12-19T13:06:35.576Z INFO add_cloud_metadata/add_cloud_metadata.go:345 add_cloud_metadata: hosting provider type detected as ec2, metadata={"availability_zone":"ap-southeast-1a","instance_id":"i-01eae62cd2b63a734","machine_type":"t3.xlarge","provider":"ec2","region":"ap-southeast-1"}
2022-12-19T13:06:35.730Z INFO kubernetes/util.go:93 kubernetes: Using node ip-10-12-xxx.ap-southeast-1.compute.internal discovered by in cluster pod node query
2022-12-19T13:06:35.731Z INFO kubernetes/watcher.go:182 kubernetes: Performing a resource sync for *v1.PodList
2022-12-19T13:06:35.766Z INFO kubernetes/watcher.go:198 kubernetes: Resource sync done
2022-12-19T13:06:35.767Z INFO kubernetes/watcher.go:242 kubernetes: Watching API for resource events
2022-12-19T13:06:35.767Z INFO log/input.go:138 Configured paths: [/var/log/containers/*.log]
2022-12-19T13:06:35.767Z INFO cfgfile/reload.go:150 Config reloader started
2022-12-19T13:06:35.767Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
2022-12-19T13:06:35.768Z INFO cfgfile/reload.go:150 Config reloader started
2022-12-19T13:06:35.768Z INFO cfgfile/reload.go:205 Loading of config files completed.
2022-12-19T13:06:35.770Z INFO kubernetes/util.go:86 kubernetes: Using pod name filebeat-2gh9x and namespace kube-system to discover kubernetes node
2022-12-19T13:06:35.928Z INFO kubernetes/util.go:93 kubernetes: Using node ip-10-12-xxx.ap-southeast-1.compute.interna discovered by in cluster pod node query
2022-12-19T13:06:35.928Z INFO kubernetes/watcher.go:182 kubernetes: Performing a resource sync for *v1.PodList
2022-12-19T13:06:35.934Z INFO kubernetes/watcher.go:198 kubernetes: Resource sync done
2022-12-19T13:06:35.934Z INFO kubernetes/watcher.go:242 kubernetes: Watching API for resource events
2022-12-19T13:06:35.935Z INFO log/input.go:138 Configured paths: [/var/log/containers/*.log]
2022-12-19T13:06:35.935Z INFO input/input.go:114 Starting input of type: log; ID: 12715084902863065571
2022-12-19T13:06:35.935Z INFO cfgfile/reload.go:205 Loading of config files completed.
UPDATE
its selinux problem
any idea fix this instead of change the selinux to permissive ?
type=AVC msg=audit(1671531416.824:408337): avc: denied { read } for pid=10185 comm="filebeat" name="containers" dev="dm-3" ino=524357 scontext=system_u:system_r:container_t:s0:c579,c853 tcontext=system_u:object_r:container_log_t:s0 tclass=dir permissive=0
type=SYSCALL msg=audit(1671531416.824:408337): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=c4202b3b20 a2=80000 a3=0 items=0 ppid=10123 pid=10185 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="filebeat" exe="/usr/share/filebeat/filebeat" subj=system_u:system_r:container_t:s0:c579,c853 key=(null)
type=PROCTITLE msg=audit(1671531416.824:408337): proctitle=66696C6562656174002D63002F6574632F66696C65626561742E796D6C002D65