This is my logstash.conf:
input {
http {
host => "127.0.0.1"
port => 31311
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
}
stdout {
codec => "rubydebug"
}
}
As a test, I ran this command in PowerShell:
C:\Users\Me\Downloads\curl-7.64.1-win64-mingw\bin> .\curl.exe -XPUT
'http://127.0.0.1:31311/twitter'
The following output was displayed inside my Logstash terminal:
{
"#timestamp" => 2019-04-09T08:32:09.250Z,
"message" => "",
"#version" => "1",
"headers" => {
"request_path" => "/twitter",
"http_version" => "HTTP/1.1",
"http_user_agent" => "curl/7.64.1",
"request_method" => "PUT",
"http_accept" => "*/*",
"content_length" => "0",
"http_host" => "127.0.0.1:31311"
},
"host" => "127.0.0.1"
}
When I then ran
C:\Users\Me\Downloads\curl-7.64.1-win64-mingw\bin> .\curl.exe -XGET
"http://127.0.0.1:9200/_cat/indices"
inside PowerShell, I saw
yellow open logstash-2019.04.09 1THStdPfQySWl1WPNeiwPQ 5 1 0 0 401b 401b
An index named logstash-2019.04.09 has been created in response to my PUT request, following the ElasticSearch convention.
My question is: If I want the index to have the same value as the {index_name} parameter I pass inside the the command .\curl.exe -XPUT 'http://127.0.0.1:31311/{index_name}', how should I configure the ElasticSearch output inside my logstash.conf file?
EDIT: Just to clarify, I want {index_name} to be read dynamically every single time I make a PUT request to create a new index. Is that even possible?
It is possible with the index output configuration option.
This configuration can be dynamic using the %{foo} syntax. Since you want the value of [headers][request_path] to be in the index configuration, you can do something like this:
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[headers][request_path]}"
}
}
For this to work the value [headers][request_path] field must not contain any of these characters: [ , \", *, \\, <, |, ,, >, /, ?].
I recommend that you use the gsub configuration option of the mutate filter. So, to remove all the forward slashes, you should have something like this:
filter{
mutate{
gsub => ["[headers][request_path]","/",""]
}
}
If the request path has several forward slashes, you could replace them with some character that will be accepted by elasticsearch.
So, your final logstash.conf file should look like this:
input {
http {
host => "127.0.0.1"
port => 31311
}
}
filter{
mutate{
gsub => ["[headers][request_path]","/",""]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[headers][request_path]}"
}
stdout {
codec => "rubydebug"
}
}
You can do so by adding an index configuration setting to your elasticsearch output section. e.g.
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "yourindexnamehere"
}
stdout {
codec => "rubydebug"
}
}
Related
I'm using JDBC input plugin to ingest data from mongodb to ElasticSearch.
My config is:
`input {
jdbc {
jdbc_driver_class => "mongodb.jdbc.MongoDriver"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/mongodb_unityjdbc_free.jar"
jdbc_user => ""
jdbc_password => ""
jdbc_connection_string => "jdbc:mongodb://localhost:27017/pritunl"
schedule => "* * * * *"
jdbc_page_size => 100000
jdbc_paging_enabled => true
statement => "select * from servers_output"
}
}
filter {
mutate {
copy => { "_id" => "[#metadata][id]"}
remove_field => ["_id"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "pritunl"
document_id => "%{[#metadata][_id]}"
}
stdout {}
}`
In Kibana I see only one hitenter image description here, but in stdout I see many records from mongodb collection. What should I do, to see them all?
The problem is, that all your documents are saved with the same "_id", so even though you're sending different records to ES, only one document is being overwritten internally - thus you get 1 hit in Kibana.
There is a typo in your configuration to cause this issue.
You're copying "_id" into "[#metadata][id]"
But you're trying to read "[#metadata][_id]" with an underscore.
Removing the underscore when reading the value for document_id should fix your issue.
output {
elasticsearch {
hosts => "localhost:9200"
index => "pritunl"
document_id => "%{[#metadata][id]}"
}
stdout {}
}`
I'm attempting to do something that should be simple but I cannot get it to work. I've looked and search all over to find detailed doc for perl search::elsticsearch. I can only find CPAN doc and as far as search is concerned it is barely mentioned. I've search here and cannot find a duplicate question.
I have elasticsearch and filebeat. Filebeat is sending syslog to elasticsearch. I just want to search for messages with matching text and date range. I can find the messages but when I try to add date range the query fails. Here is the query from kibana dev tools.
GET _search
{
"query": {
"bool": {
"filter": [
{ "term": { "message": "metrics" }},
{ "range": { "timestamp": { "gte": "now-15m" }}}
]
}
}
}
I don't get exactly what I'm looking for but there isn't an error.
Here is my attempt with perl
my $results=$e->search(
body => {
query => {
bool => {
filter => {
term => { message => 'metrics' },
range => { timestamp => { 'gte' => 'now-15m' }}
}
}
}
}
);
This is the error.
[Request] ** [http://x.x.x.x:9200]-[400]
[parsing_exception]
[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME],
with: {"col":69,"line":1}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__
at ./elasticsearchTest.pl line 15.
With vars: {'body' => {'status' => 400,'error' => {
'root_cause' => [{'col' => 69,'reason' => '[range]
malformed query, expected [END_OBJECT] but found [FIELD_NAME]',
'type' => 'parsing_exception','line' => 1}],'col' => 69,
'reason' => '[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]',
'type' => 'parsing_exception','line' => 1}},'request' => {'serialize' => 'std',
'path' => '/_search','ignore' => [],'mime_type' => 'application/json',
'body' => {
'query' => {
'bool' =>
{'filter' => {'range' => {'timestamp' => {'gte' => 'now-15m'}},
'term' => {'message' => 'metrics'}}}}},
'qs' => {},'method' => 'GET'},'status_code' => 400}
Can someone help me figure out how to search with the search::elasticsearch perl module?
Multiple filter clauses must be passed as separate JSON objects within an array (like in your initial JSON query), not multiple filters in the same JSON object. This maps to how you must create the Perl data structure.
filter => [
{term => { message => 'metrics' }},
{range => { timestamp => { 'gte' => 'now-15m' }}}
]
I am trying to pull data from a mongoDB to Elasticsearch using logstash.
I am using the Logstash-input-mongodb plugin. This is my config file:
input {
mongodb {
uri => 'mongodb://localhost:27017/test'
placeholder_db_dir => '/opt/logstash-mongodb/'
placeholder_db_name => 'logstash_sqlite.db'
collection => 'student'
batch_size => 202
}
}
filter {
}
output {
elasticsearch {
host => "localhost:9200"
user => elastic
password => changeme
index => "testStudent"
}
stdout { codec => rubydebug }
}
I am having an error saying:
Pipelines YAML file is empty.
Is this because I left the filter part empty?
I am trying to input a timestamp field in Logstash and i am getting dateparsefailure message.
My Message -
2014-08-01;11:00:22.123
Pipeline file
input {
stdin{}
#beats {
# port => "5043"
# }
}
# optional.
filter {
date {
locale => "en"
match => ["message", "YYYY-MM-dd;HH:mm:ss.SSS"]
target => "#timestamp"
add_field => { "debug" => "timestampMatched"}
}
}
output {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
stdout { codec => rubydebug }
}
Can someone tell me what i am missing ?
Update 1
I referred to the link - How to remove trailing newline from message field and now it works.
But, in my log message, i have multiple values other than timestamp
<B 2014-08-01;11:00:22.123 Field1=Value1 Field2=Value2
When i give this as input, it is not working. How to read a part of the log and make it as timestamp ?
Update 2
it works now.
Changed the config file as below
filter {
kv
{
}
mutate {
strip => "message"
}
date {
locale => "en"
match => ["timestamp1", "YYYY-MM-dd;HH:mm:ss.SSS"]
target => "#timestamp"
add_field => { "debug" => "timestampMatched"}
}
}
I am posting the answer below and steps i used to solve the issue so that i can help people like me.
Step 1 - I read the message in the form of key and value pair
Step 2 - I trimmed off the extra space that leads to parse exception
Step 3 - I read the timestamp value and other fields in respective fields.
input {
beats {
port => "5043"
}
}
# optional.
filter {
kv { }
date {
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
remove_field => [ "timestamp" ]
}
}
output {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
}
i'm triyng to monitor an irc server. And i'm loot for a way to create a new numeral field (example: Alert_level) only if a message match a specific word inside.
Example: Message: ABC | Alert_level: 1 ; Message: ZYX | Alert_level: 3.
Its the running code
input {
irc {
channels => "#xyz"
host => "a.b.c"
nick => "myusername"
catch_all => true
get_stats => true
}
}
output {
stdout { codec => "rubydebug" }
elasticsearch {
hosts => "localhost"
index => "logstash-irc-%{+YYYY.MM.dd}"
}
}
Thank you!
As #Val suggested above you might need to use the grok filter in order match something from the input. For example your filter could look something like this:
filter {
grok {
match => { "message" => "%{GREEDYDATA:somedata}" }
}
if "ZYX" in [message]{ <-- change your condition accordingly
mutate {
add_field => { "%{Alert_level}" => "12345" } <-- somefield is the field name
convert => { "Alert_level" => "integer" } <-- do the conversion
}
}
}
NOTE that you have to do the conversion in order to create a numeric field through logstash, where you can't directly create one. The above is just a sample so that you can reproduce. Do change the grok match in respect to your requirement. Hope it helps!