I have two sources of data for my logs. One is the beat and one is kafka and I want to create ES indexes based on the source. if kafka -> prefix index_name with kafka, and if beat prefix the index name with beat.
input {
beats {
port => 9300
}
}
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => ["my-topic"]
codec => json
}
}
output {
# if kafka
elasticsearch {
hosts => "http://localhost:9200"
user => "elastic"
password => "password"
index => "[kafka-topic]-my-index"
}
# else if beat
elasticsearch {
hosts => "http://localhost:9200"
user => "elastic"
password => "password"
index => "[filebeat]-my-index"
}
}
Add tags in your inputs and use them to filter the output.
input {
beats {
port => 9300
tags => ["beats"]
}
}
input {
kafka {
bootstrap_servers => "localhost:9092"
topics => ["my-topic"]
codec => json
tags => ["kafka"]
}
}
output {
if "beats" in [tags] {
output for beats
}
if "kafka" in [tags] {
output for kafka
}
}
Related
I am trying to consume a topic from Kafka using Avro Deserializer in Logstash and getting the below error.
Here is my Logstash Config file input
input {
kafka {
bootstrap_servers => "kafka1:9911,kafka2:9911,kafka3.com:9911"
topics => "EMS.Elastic_new"
auto_offset_reset => earliest
group_id => "logstash106"
ssl_truststore_location => "/apps/opt/application/elasticsearch/logstash-7.1.1/kafka_files/kafka.client.truststore.jks"
ssl_truststore_password => "xxxx"
security_protocol => "SSL"
key_deserializer_class => "io.confluent.kafka.serializers.KafkaAvroDeserializer"
value_deserializer_class => "io.confluent.kafka.serializers.KafkaAvroDeserializer"
codec => avro_schema_registry {
endpoint => "https://kafka1:9990"
subject_name => "EMS.Elastic_new"
schema_id => 170
schema_uri => "/apps/opt/application/elasticsearch/logstash-7.1.1/kafka_files/ticketInfo.avsc"
tag_on_failure => true
register_schema => true
}
}
}
output {
elasticsearch {
index => "smd_etms_es2"
document_id => "%{tktnum}%"
action => "update"
doc_as_upsert => "true"
retry_on_conflict => 5
hosts => ["npes1:9200"]
}
stdout { codec => rubydebug }
}
[ERROR][logstash.inputs.kafka ]
Unable to create Kafka consumer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException:
Failed to construct kafka consumer, :cause=>io.confluent.common.config.
ConfigException: Missing required configuration "schema.registry.url"
which has no default value.} [2019-07-26T16:58:22,736][ERROR][logstash.javapipeline ]
A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main
I have provided avro_uri in codec however, the settings is not been read by the logstash.
Missing required configuration "schema.registry.url"
Comes from setting
key_deserializer_class => "io.confluent.kafka.serializers.KafkaAvroDeserializer"
value_deserializer_class => "io.confluent.kafka.serializers.KafkaAvroDeserializer"
Based on the example code, it seems it wants you just to use org.apache.kafka.common.serialization.ByteArraySerializer for both, then I assume avro_codec does the schema management on its own using the endpoint parameter
While getting the output from a kafka stream, logstash is also displaying other characters. (\u0018, \u0000, \u0002, etc.)
I tried adding a key_deserializer_class to the logstash conf file, but that didn't help much.
input {
kafka {
bootstrap_servers => "broker1-kafka.net:9092"
topics => ["TOPIC"]
group_id => "T-group"
jaas_path => "/opt/kafka_2.11-1.1.0/config/kafka_client_jaas.conf"
key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
sasl_mechanism => "SCRAM-SHA-256"
security_protocol => "SASL_PLAINTEXT"
}
}
output { stdout { codec => rubydebug } }
Output
{
"#timestamp" => 2019-04-10T06:09:53.918Z,
"message" => "(TOPIC\u0002U42019-04-10 06:09:47.01739142019-04-10T06:09:53.738000(00000021290065792800\u0002\u0004C1\u0000\u0000\u0002\u001EINC000014418569\u0002\u0010bppmUser\u0002����\v\u0000\u0002\u0010bppmUser\u0002֢��\v\u0002\u0002\u0002\u0002.\u0002\u0018;1000012627;\u0002<AGGAA5V0FEEW7APPOPCYPOR3RPPOLL\u0000\",
"#version" => "1"
}
Is there any way to not get these characters in the output.
this is my logstash conf
input {
kafka {
bootstrap_servers => "127.0.0.1:9092"
topics => ["filebeat", "access"]
group_id => "test-consumer-group"
consumer_threads => 1
decorate_events => true
}
}
I have two topics but I want to use different codec for diffrent topic. how can I do this?
I try to add
if ([topic] == "filebeat") {
codec => "json"
}
in the kafka input conf, the the logstash returns me errors.
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 6, column 8 (byte 143) after input {\n kafka {\n bootstrap_servers => \"127.0.0.1:9092\"\n topics => [\"filebeat\", \"access\"]\n group_id => \"test-consumer-group\"\n if "
You can create 2 separate kafka inputs with each a different codec.
One other option is to add a filter that parses the json object depending on the the topic
filter {
if([topic] == "filebeat") {
json {
source => "message"
}
}
}
for more info check:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html
I am trying to pull data from a mongoDB to Elasticsearch using logstash.
I am using the Logstash-input-mongodb plugin. This is my config file:
input {
mongodb {
uri => 'mongodb://localhost:27017/test'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'student'
batch_size => 202
}
}
filter {
}
output {
elasticsearch {
host => "localhost:9200"
user => elastic
password => changeme
index => "testStudent"
}
stdout { codec => rubydebug }
}
I am having an error saying:
Pipelines YAML file is empty.
Is this because I left the filter part empty?
ELK stack version
Logstash: 5.1.2
Kibana: 5.1.2
Elasticsearch:5.1.2
I have the below logstash configuration to send my router syslog events to elastic search.
My router is configured to send events to port 5514 and I can see the logs in Kibana.
BUT, I would like to to ensure all events send to port 5514 are given the type of syslog-network, which is then filtered by 11-network-filter.conf and send to Elasticsearch logstash-syslog-% index.
At present all the syslog events are falling under the logstash index.
Any ideas why?
03-network-input.conf
input {
syslog {
port => 5514
type => "syslog-network"
tags => "syslog-network"
}
}
11-network-filter.conf
filter {
if [type] == "syslog-network" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
30-elasticsearch-output.conf
output {
if "file-beats" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
else if "syslog-network" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-syslog-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
}