Fluentd Not able to scrape high volume logs - grafana

Question

I am trying to Use fluentd to push logs to loki using fluentd-loki plugin. I am not able to make fluentd ingest logs in realtime if the logs exceed 24000 lines/sec. I need help in configuration of fluentd to make it scrape logs fastly and in realtime.
my fluentd_deamonset.yaml
This is the deamonset code
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd
namespace: loki-fluentd
labels:
app: fluentd
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
app: fluentd
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
app: fluentd
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.4-debian-forward-1
command:
- /bin/sh
- '-c'
- >
fluent-gem i fluent-plugin-grafana-loki-licence-fix ;
fluent-gem i fluent-plugin-parser-cri --no-document ;
tini /fluentd/entrypoint.sh;
resources:
limits:
memory: 1024Mi
requests:
cpu: 1000m
memory: 1024Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config
mountPath: /fluentd/etc
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: config
configMap:
name: fluentd-config
This is config file
fluentd-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-config
namespace: loki-fluentd
labels:
app: fluentd
data:
fluent.conf: |
<source>
#type tail
#id in_tail_container_logs
path /var/log/containers/loggen-*.log
# exclude_path ["/var/log/containers/fluentd*"]
pos_file /tmp/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
<parse>
#type cri
time_format %Y-%m-%dT%H:%M:%S.%L%z
</parse>
</source>
<match fluentd.**>
#type null
</match>
<match kubernetes.var.log.containers.**fluentd**.log>
#type null
</match>
<filter kubernetes.**>
#type kubernetes_metadata
#id filter_kube_metadata
</filter>
<filter kubernetes.var.log.containers.**>
#type record_transformer
enable_ruby
remove_keys kubernetes, docker
<record>
app ${ record.dig("kubernetes", "labels", "app") }
job ${ record.dig("kubernetes", "labels", "app") }
namespace ${ record.dig("kubernetes", "namespace_name") }
pod ${ record.dig("kubernetes", "pod_name") }
container ${ record.dig("kubernetes", "container_name") }
filename ${ record.dig("kubernetes", "filename")}
workers ${ record.dig("kubernetes", "worker") }
</record>
</filter>
<match kubernetes.var.log.containers.**>
#type copy
<label>
fluentd_worker
</label>
<store>
#type loki
url "http://loki-url"
extra_labels {"env":"dev"}
label_keys "app,job,namespace,pod,container,filename,fluentd_worker,workers"
<buffer>
flush_thread_count "#{ENV['FLUENT_LOKI_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
# flush_interval "#{ENV['FLUENT_LOKI_BUFFER_FLUSH_INTERVAL'] || '1s'}"
flush_mode "#{ENV['FLUENT_LOKI_BUFFER_FLUSH_INTERVAL'] || 'immediate'}"
chunk_limit_size "#{ENV['FLUENT_LOKI_BUFFER_CHUNK_LIMIT_SIZE'] || '512k'}"
queue_limit_length "#{ENV['FLUENT_LOKI_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
retry_max_interval "#{ENV['FLUENT_LOKI_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_forever true
</buffer>
</store>
<store>
#type stdout
</store>
</match>
I want to know whether configuration can help me ingest more than 10000+ lines per second/fluentd node.