I am trying to forward syslogs,
I start the td-agent with read_from_head param set to false.
So on checking logs nothing happens, then I added a line to the target file via
vscode so there wasn't issue with the inode changing.
The issue that I face is that upon any addition of log lines to the target file, the
pos_file gets updated and reads from the top and after adding
another line td-agent collects from the top again.
How does fluentd in_tail works?
<source>
#type tail
format none
path /home/user/Documents/debugg/demo.log
pos_file /var/log/td-agent/demo.log.pos
read_from_head false
follow_inodes true
tag syslog
<source>
i have the follow question. i try connect to eks cluster using a Terraform with Gitlab CI/CD , i receive the error message , but when try it in my compute , this error dont appear, someone had same error ?
$ terraform output authconfig > authconfig.yaml
$ cat authconfig.yaml
<<EOT
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: "arn:aws:iam::503655390180:role/clusters-production-workers"
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
EOT
$ kubectl create -f authconfig.yaml -n kube-system
error: error parsing authconfig.yaml: error converting YAML to JSON: yaml: line 2: mapping values are not allowed in this context
The output is including EOT(EndOfText) marks since it is generated as a multiline string originally.
as documentation suggests (terrafom doc link)
Don't use "heredoc" strings to generate JSON or YAML. Instead, use the
jsonencode function or the yamlencode function so that Terraform can
be responsible for guaranteeing valid JSON or YAML syntax.
use json encoding or yaml encoding before building output.
If you want to continue like this with what you have now then try to give these options with output -json or -raw
terraform output -json authconfig > authconfig.yaml
or
terraform output -raw authconfig > authconfig.yaml
The error message tells you the authconfig.yaml file can not be converted from YAML to JSON, suggesting it's not a valid yaml
The cat authconfig.yaml you're showing us includes some <<EOT and EOT tags. I would suggest to remove those, before running kubectl create -f
Your comment suggests you knew this already - then why didn't you ask about terraform, rather than showing us kubectl create failing? From your post, it really sounded like you copy/pasted the output of your job, without even reading it.
So, obviously, the next step is to terraform output -raw, or -json, there are several mentions in their docs, or knowledge base, a google search would point you to:
https://discuss.hashicorp.com/t/terraform-outputs-with-heredoc-syntax-leaves-eot-in-file/18584/7
https://www.terraform.io/docs/cli/commands/output.html
Last: we could ask why? Why would you terraform output > something, when you can have terraform write a file?
While as a general rule, whenever writing terraform stdout/stderr to files, I strongly suggest going with no-color.
Summary: I am using telegraf to get procstat into InfluxDB. I want to convert the pid from an integer field to a TAG so that I can do group by on it in Influx.
Details:
After a lot of searching I found the following on some site but it seems to be doing the opposite (converts tag into a field). I am not sure how to deduce the opposite conversion syntax from it:
[processors]
[[processors.converter]]
namepass = [ "procstat",]
[processors.converter.tags]
string = [ "cmdline",]
I'm using Influx 1.7.9
The correct processor configuration to convert pid as tag is as below.
[processors]
[[processors.converter]]
namepass = [ "procstat"]
[processors.converter.fields]
tag = [ "pid"]
Please refer the documentation of converter processor plugin
https://github.com/influxdata/telegraf/tree/master/plugins/processors/converter
In the latest version of telegraf pid can be stored as tag by specifying it in the input plugin configuration. A converter processor is not needed here.
Mention pid_tag = true in the configuration. However be aware of the performance impact of having pid as a tag when processes are short lived.
P.S: You should try to upgrade your telegraf version to 1.14.5. There is a performance improvement fix for procstat plugin in this version.
Plugin configuration reference https://github.com/influxdata/telegraf/tree/master/plugins/inputs/procstat
Sample config.
# Monitor process cpu and memory usage
[[inputs.procstat]]
## PID file to monitor process
pid_file = "/var/run/nginx.pid"
## executable name (ie, pgrep <exe>)
# exe = "nginx"
## pattern as argument for pgrep (ie, pgrep -f <pattern>)
# pattern = "nginx"
## user as argument for pgrep (ie, pgrep -u <user>)
# user = "nginx"
## Systemd unit name
# systemd_unit = "nginx.service"
## CGroup name or path
# cgroup = "systemd/system.slice/nginx.service"
## Windows service name
# win_service = ""
## override for process_name
## This is optional; default is sourced from /proc/<pid>/status
# process_name = "bar"
## Field name prefix
# prefix = ""
## When true add the full cmdline as a tag.
# cmdline_tag = false
## Add the PID as a tag instead of as a field. When collecting multiple
## processes with otherwise matching tags this setting should be enabled to
## ensure each process has a unique identity.
##
## Enabling this option may result in a large number of series, especially
## when processes have a short lifetime.
# pid_tag = false
I'm running this implementation of the ELK stack, which is pretty straightforward and easy to configure.
I can push TCP input through the stack using netcat like so:
nc localhost 5000 < /Users/me/path/to/logs/appOne.log
nc localhost 5000 < /Users/me/path/to/logs/appOneStackTrace.log
nc localhost 5000 < /Users/me/path/to/logs/appTwo.log
nc localhost 5000 < /Users/me/path/to/logs/appTwoStackTrace.log
But I cannot get the Logstash to read the file paths I specify in the config:
input {
tcp {
port => 5000
}
file {
path => [
"/Users/me/path/to/logs/appOne.log",
"/Users/me/path/to/logs/appOneStackTrace.log",
"/Users/me/path/to/logs/appTwo.log",
"/Users/me/path/to/logs/appTwoStackTrace.log"
]
type => "log"
start_position => "beginning"
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}
Here is the startup output from the stack regarding logstash input:
logstash_1 | [2019-01-28T17:44:33,206][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enable=>"false"}
logstash_1 | [2019-01-28T17:44:34,037][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_a1605b28f1bc77daf785a8805c32f578", :path=>["/Users/me/path/to/logs/appOne.log", "/Users/me/path/to/logs/appOneStackTrace.log", "/Users/me/path/to/logs/appTwo.log", "/Users/me/path/to/logs/appTwoStackTrace.log"]}
There is no indication the pipeline has any issues starting.
I've also checked that the log files have been updated since the display of TCP input, and they have. The last Logstash-specific log from the ELK stack comes from either startup or the TCP input.
Here is my entire Logstash start-up logging in case that's helpful:
logstash_1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1 | [2019-01-29T13:32:19,391][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
logstash_1 | [2019-01-29T13:32:19,415][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.5.4"}
logstash_1 | [2019-01-29T13:32:23,989][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
logstash_1 | [2019-01-29T13:32:24,648][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash_1 | [2019-01-29T13:32:24,908][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
logstash_1 | [2019-01-29T13:32:25,046][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
logstash_1 | [2019-01-29T13:32:25,051][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
logstash_1 | [2019-01-29T13:32:25,108][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash_1 | [2019-01-29T13:32:25,229][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
logstash_1 | [2019-01-29T13:32:25,276][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
logstash_1 | [2019-01-29T13:32:25,327][INFO ][logstash.inputs.tcp ] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enable=>"false"}
logstash_1 | [2019-01-29T13:32:25,924][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_143c07d174c46eeab78b902edb3b1289", :path=>["/Users/me/path/to/logs/appOne.log", "/Users/me/path/to/logs/appOneStackTrace.log", "/Users/me/path/to/logs/appTwo.log", "/Users/me/path/to/logs/appTwoStackTrace.log"]}
logstash_1 | [2019-01-29T13:32:25,976][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4d1515ce run>"}
logstash_1 | [2019-01-29T13:32:26,088][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1 | [2019-01-29T13:32:26,106][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
logstash_1 | [2019-01-29T13:32:26,432][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
I found the issue - I needed to map the log files from the host into the container(Docker noob). The local paths I was specifying in the Logstash config were fine for TCP, but were unavailable inside the container without volume mapping.
First, I created the container's internal log directories in the Dockerfile for Logstash:
RUN mkdir /usr/share/appOneLogs
RUN mkdir /usr/share/appTwoLogs
Then I volume-mapped my host's log directories into them in the docker-elk/docker-compose.yml file where Logstash is configured:
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
- /Users/me/path/to/appOne/logs:/usr/share/appOneLogs # this bit
- /Users/me/path/to/appTwo/logs:/usr/share/appTwoLogs # and this bit
ports:
- "5000:5000"
...
Finally, I replaced the paths in logstash/pipelines/logstash.config with the directories created in the Dockerfile:
file {
path => [
"/usr/share/appOneLogs",
"/usr/share/appTwoLogs",
]
}
Also of note, I removed start_position => "beginning" from the file input definition as this overrides the default behavior to treat files like live streams and thus start at the end.
I am trying to implement a parser plugin for fluentd. Below are the configuration file and the plugin file.
Fluentd config file.
<source>
type syslog
port 9010
bind x.x.x.x
tag flog
format flog_message
</source>
Plugin file
module Fluent
class TextParser
class ElogParser < Parser
Plugin.register_parser("flog_message", self)
config_param :delimiter, :string, :default => " " # delimiter is configurable with " " as default
config_param :time_format, :string, :default => nil # time_format is configurable
# This method is called after config_params have read configuration parameters
def configure(conf)
if #delimiter.length != 1
raise ConfigError, "delimiter must be a single character. #{#delimiter} is not."
end
# TimeParser class is already given. It takes a single argument as the time format
# to parse the time string with.
#time_parser = TimeParser.new(#time_format)
end
def call(text)
# decode text
# ...
# decode text
yield result_hash
end
end
end
end
However the method call is not executed after running fluentd. Any help is greatly appreciated.
Since v0.12, use parse instead of call.
docs.fluentd.org was outdated so I just updated the article: http://docs.fluentd.org/articles/plugin-development#parser-plugins
Sorry for forget to update the document...