I am trying to forward syslogs,
I start the td-agent with read_from_head param set to false.
So on checking logs nothing happens, then I added a line to the target file via
vscode so there wasn't issue with the inode changing.
The issue that I face is that upon any addition of log lines to the target file, the
pos_file gets updated and reads from the top and after adding
another line td-agent collects from the top again.
How does fluentd in_tail works?
<source>
#type tail
format none
path /home/user/Documents/debugg/demo.log
pos_file /var/log/td-agent/demo.log.pos
read_from_head false
follow_inodes true
tag syslog
<source>
Related
I am trying to collect log files(.csv / .txt) from any folders and send them to kafka server.
To collect log files and print to stdout is working great.
but, when I changed the config file to send the log file to kafka server,
this error message was shown "kafka plugin doesn't exist."
Is there anyone who can help me?
window10 fluent-bit.exe 64bit install
Fluent Bit v2.0.0
config file
[INPUT]
Name tail
Tag my_test
Path C:\Program Files\fluent-bit\logs*.txt
Read_from_Head True
Offset_Key False
Refresh_Interval 1s
[OUTPUT]
name stdout
match *
[OUTPUT]
name kafka
match my_test
Brokers localhost:9092
topics FluentbitMyTest
I tried to copy CMakeLists.txt to default fluent-bit folder and do cmake below.
but this error pops up as well.
C:\Program Files\fluent-bit\build>cmake -DFLB_OUT_KAFKA=On ../
CMake Error at CMakeLists.txt:4 (project):
Running
'nmake' '-?'
failed with:
-- Configuring incomplete, errors occurred!
See also "C:/Program Files/fluent-bit/build/CMakeFiles/CMakeOutput.log".
After shifting from Docker to containerd as docker engine used by our kubernetes, we are not able to show the multiline logs in a proper way by our visualization app (Grafana) as some details prepended to the container/pod logs by the containerd itself (i.e. timestamp, stream & log severity to be specific it is appending something like the following and as shown in the below sample: 2022-07-25T06:43:17.20958947Z stdout F ) which make some confusion for the developers and the application owners.
I am showing here a dummy sample of the logs generated by the application and how it got printed in the nodes of kuberenetes'nodes after containerd prepended the mentioned details.
The following logs generated by the application (kubectl logs ):
2022-07-25T06:43:17,309ESC[0;39m dummy-[txtThreadPool-2] ESC[39mDEBUGESC[0;39m
ESC[36mcom.pkg.sample.ComponentESC[0;39m - Process message meta {
timestamp: 1658731397308720468
version {
major: 1
minor: 0
patch: 0
}
}
when I check the logs in the filesystem (/var/log/container/ABCXYZ.log) :
2022-07-25T06:43:17.20958947Z stdout F 2022-07-25T06:43:17,309ESC[0;39m dummy-[txtThreadPool-2]
ESC[39mDEBUGESC[0;39m
ESC[36mcom.pkg.sample.ComponentESC[0;39m - Process message meta {
2022-07-25T06:43:17.20958947Z stdout F timestamp: 1658731449723010774
2022-07-25T06:43:17.209593379Z stdout F version {
2022-07-25T06:43:17.209595933Z stdout F major: 14
2022-07-25T06:43:17.209598466Z stdout F minor: 0
2022-07-25T06:43:17.209600712Z stdout F patch: 0
2022-07-25T06:43:17.209602926Z stdout F }
2022-07-25T06:43:17.209605099Z stdout F }
I am able to parse the multiline logs with fluentbit but the problem is I am not able to remove the details injected by containerd ( >> 2022-07-25T06:43:17.209605099Z stdout F .......). So is there anyway to configure containerd to not prepend these details somehow in the logs and print them as they are generated from the application/container ?
On the other hand is there any plugin to remove such details from fluentbit side .. as per the existing plugins none of them can manipulate or change the logs (which is logical as the log agent should not do any change on the logs).
Thanks in advance.
This is the workaround I followed to show the multiline log lines in Grafana by applying extra fluentbit filters and multiline parser.
1- First I receive the stream by tail input which parse it by a multiline parser (multilineKubeParser).
2- Then another filter will intercept the stream to do further processing by a regex parser (kubeParser).
3- After that another filter will remove the details added by the containerd by a lua parser ().
fluent-bit.conf: |-
[SERVICE]
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Flush 1
Daemon Off
Log_Level warn
Parsers_File parsers.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
multiline.Parser multilineKubeParser
Exclude_Path /var/log/containers/*_ABC-logging_*.log
DB /run/fluent-bit/flb_kube.db
Mem_Buf_Limit 5MB
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Merge_Parser kubeParser
K8S-Logging.Parser Off
K8S-Logging.Exclude On
[FILTER]
Name lua
Match kube.*
call remove_dummy
Script filters.lua
[Output]
Name grafana-loki
Match kube.*
Url http://loki:3100/api/prom/push
TenantID ""
BatchWait 1
BatchSize 1048576
Labels {job="fluent-bit"}
RemoveKeys kubernetes
AutoKubernetesLabels false
LabelMapPath /fluent-bit/etc/labelmap.json
LineFormat json
LogLevel warn
labelmap.json: |-
{
"kubernetes": {
"container_name": "container",
"host": "node",
"labels": {
"app": "app",
"release": "release"
},
"namespace_name": "namespace",
"pod_name": "instance"
},
"stream": "stream"
}
parsers.conf: |-
[PARSER]
Name kubeParser
Format regex
Regex /^([^ ]*).* (?<timeStamp>[^a].*) ([^ ].*)\[(?<requestId>[^\]]*)\] (?<severity>[^ ]*) (?<message>[^ ].*)$/
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
Time_Offset +0200
[MULTILINE_PARSER]
name multilineKubeParser
type regex
flush_timeout 1000
rule "start_state" "/[^ ]* stdout .\s+\W*\w+\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d\d\d.*$/" "cont"
rule "cont" "/[^ ]* stdout .\s+(?!\W+\w+\d\d\d\d-\d\d-\d\d \d\d\:\d\d\:\d\d,\d\d\d).*$/" "cont"
filters.lua: |-
function remove_dummy(tag, timestamp, record)
new_log=string.gsub(record["log"],"%d+-%d+-%d+T%d+:%d+:%d+.%d+Z%sstdout%sF%s","")
new_record=record
new_record["log"]=new_log
return 2, timestamp, new_record
end
As I mentioned this is a workaround till I can find any other/better solution.
From the configuration options for containerd, it appears there's no way to configure logging in any way. You can see the config doc here.
Also, I checked at the logging code inside containerd and it appears this is prepended to the logs as they are redirected from the stdout of the container. You can see that the testcase here checks for appropriate fields by "splitting" the log line received. It checks for a tag and a stream entry prepended to the content of the log. I suppose that's the way logs are processed in containerd.
The best thing to do would be to open an issue in the project with your design requirement and perhaps the team can develop configurable stdout redirection for you.
This might help. They use a custom regex to capture the message log and the rest of the info in the logs and then use lift in the nest filter to flatten the json.
https://github.com/microsoft/fluentbit-containerd-cri-o-json-log
I have log lines in Kubernetes:
2022-06-12T19:55:19.014511382Z stdout F dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
2022-06-12T19:55:19.014515391Z stdout F Ending HttpMessageHandler cleanup cycle after 0.0016ms - processed: 0 items - remaining: 1 items
2022-06-12T19:55:29.010438412Z stdout F dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[100]
2022-06-12T19:55:29.010512909Z stdout F Starting HttpMessageHandler cleanup cycle with 1 items
2022-06-12T19:55:29.010518914Z stdout F dbug: Microsoft.Extensions.Http.DefaultHttpClientFactory[101]
2022-06-12T19:55:29.010532801Z stdout F Ending HttpMessageHandler cleanup cycle after 0.002ms - processed: 0 items - remaining: 1 items
I'm trying to parse these lines using the multiline plugin. I have the following config:
<source>
#type tail
#id in_tail_container_logs
path /var/log/containers/*namespace*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
<parse>
#type multiline
format_firstline /^(?<logtime>[^\]]*) \w+ \w (?<level>\w+): (?<classpath>(\w+\.?)+\[\d+\])/
format1 /^[^\]]* \w+ \w \s+ (?<message>.+)/
</parse>
</source>
<filter kubernetes.var.log.containers.**.log>
#type kubernetes_metadata
</filter>
<match kubernetes.var.log.containers.**namespace**.log>
#type loggly
loggly_url "https://logs-01.loggly.com/inputs/#{ENV['TOKEN']}/"
</match>
I can see the logs coming into Loggly, but they're missing all the fields I'm adding in the multiline parser's config. I have debug logging enabled and I can see that the loggly plugin is getting the records without the fields before sending them.
I tested the regular expressions against the lines individually using fluentular and they seem to work individually. Is there something I'm missing from my config that could be causing the fields to be missing?
I have a conf file for mongod where I am trying to set log rotation, the log file size, and the number of old instances to be kept.
I have added logRotate: rename to the conf file as follows:
systemLog:
timeStampFormat: ctime
destination: file
logAppend: true
logRotate: rename
path: C:\Program Files\ApplicationName\logs\db.log
But I cannot find any documentation or examples that show how to specify the max size of the log file before it rotates and how many old instances to keep.
Also I don't know how to test this.
Apperently you need setup some kind of external process to send SIGUSR1 signal from OS to trigger rotation.
Any ideas? Has anyone actually done this successfully?
There is nothing available in MongoDB itself. Either you write all logs into one file:
systemLog
logAppend: true
or you create a new log file every time you start MonogDB server:
systemLog
logAppend: true
logRotate: rename
In either case you have to delete/compress old log files manually with a 3rd party tool or by script.
In Linux you can trigger a log rotate by kill -USR1 but I am not aware this would be available in Windows. You need to initiate log rotate from Mongo shell:
use admin
db.runCommand({ logRotate: 1 })
or
db.getSiblingDB('admin').runCommand({ logRotate: 1 })
db.adminCommand({ logRotate: 1 })
You can run it from command line with
mongo --eval "db.adminCommand({ logRotate: 1 })"
Setup such task with Windows Task Scheduler. You may have a look at LogRotateWin which is an implementation of logrotate utility for Windows Platform. I never tried it, but it seems to work as supposed to be. For sample logrotate configuration see How can I disable the logging of MongoDB?
Note, for Linux in order not to lose any logs, you first have to rename the logfile and afterwards initiate the log rotate command. I don't know whether Windows behaves in the same way.
With the help of Wernfried-Domscheit answer, a lot of research, weeks of trial and error and frustration I figured out a working solution on windows.
Prerequisites
You have a MongoDB-Service running with a configuration file mongod.cfg
LogRotateWin is installed. This is a third party package to rotate the logfiles, based on the unix implementation. It provides many customizable functionalities like compression, when rotating is applied, old file deletion and more. You can find a full option list here LogRotateWin Configuration.
Basic knowledge of Windows Taskscheduler.
Very basic knowledge of bat files.
1. Adjust the configuration file of the MongoDB-Service
Stop the service
Open the mongod.cfg and find the code lines where the systemLog is configured:
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: E:\MongoDB\Server\4.4\log\mongod.log
logRotate: reopen
Be sure, that you overwrite the path to your mongod.log. Be also sure, that logAppend: true and logRotate: reopen are set.
Delete the current mongod.log file
Restart the service
2. Configure the logrotate configuration
This is my configuration. You can customize that to your own needs. But dont use copy, copytruncate and create and dont remove the postrotate commands!
E:\MongoDB\Server\4.4\log\mongod.log {
nocompress
daily
size 100m
missingok
rotate 50
postrotate
E:\logrotate\notify_mongodb_service.bat
endscript
}
This configuration rotates the logs uncompressed every day or if the size exceeded 100 megabyte. Maximum of 50 logs are stored, older files will be deleted. The postrotate script will be executed when the rotation succeeded.
3. Create the notify_mongodb_service.bat file
This file sends a command to the MongoDB-Service, that a new file will be used. If you enabled authorization you can add -u username -p password aswell.
E:\MongoDB\Server\4.4\bin\mongo.exe --eval "db.adminCommand({ logRotate: 1 })"
Change the path to mongo.exe to the location on your system.
Save the file and be sure that the path in the logrotate configuration is the same! (the line between postrotate and endscript
The command is stored in an extra file, because the LogRotateWin interprets the Brackets ({}) of the mongo command and throws an Exception.
4. Check that all is working
Open the folder of the logs. You should see a single file mongod.log.
Open a terminal and check that logrotation is working ( -f forces the logrotation even if no trigger was triggered):
logrotate logrotate.conf -f
A new log file should occur. (for me it is mongod.log.1)
The mongod.log should be empty.
Trigger something that will be logged. For example connect with monogdbCompass to your MongoDB.
Check mongod.log. The connection should be logged there.
5. Create a Taskscheduler Job to run logrotation periodically
I wont talk about taskscheduler creation but here is an example of my configuration. You can import that file and modify it to your needs.
Also change the path of the logrotate.exe to your location of your system.
This job runs every hour to check if one or multiple triggers of logrotate triggered:
<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.4" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
<RegistrationInfo>
<Date>2021-12-02T17:57:55.9541897</Date>
<Author>Grayknife</Author>
<Description>Execute Logrotate Hourly</Description>
<URI>\docker\LogRotate</URI>
</RegistrationInfo>
<Triggers>
<CalendarTrigger>
<Repetition>
<Interval>PT1H</Interval>
<StopAtDurationEnd>false</StopAtDurationEnd>
</Repetition>
<StartBoundary>2021-12-02T18:00:00</StartBoundary>
<ExecutionTimeLimit>PT30M</ExecutionTimeLimit>
<Enabled>true</Enabled>
<ScheduleByDay>
<DaysInterval>1</DaysInterval>
</ScheduleByDay>
</CalendarTrigger>
</Triggers>
<Principals>
<Principal id="Author">
<UserId>1234</UserId>
<LogonType>Password</LogonType>
<RunLevel>LeastPrivilege</RunLevel>
</Principal>
</Principals>
<Settings>
<MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
<DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
<StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
<AllowHardTerminate>true</AllowHardTerminate>
<StartWhenAvailable>false</StartWhenAvailable>
<RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
<IdleSettings>
<StopOnIdleEnd>true</StopOnIdleEnd>
<RestartOnIdle>false</RestartOnIdle>
</IdleSettings>
<AllowStartOnDemand>true</AllowStartOnDemand>
<Enabled>true</Enabled>
<Hidden>false</Hidden>
<RunOnlyIfIdle>false</RunOnlyIfIdle>
<DisallowStartOnRemoteAppSession>false</DisallowStartOnRemoteAppSession>
<UseUnifiedSchedulingEngine>true</UseUnifiedSchedulingEngine>
<WakeToRun>false</WakeToRun>
<ExecutionTimeLimit>PT0S</ExecutionTimeLimit>
<Priority>7</Priority>
</Settings>
<Actions Context="Author">
<Exec>
<Command>E:\logrotate\logrotate.exe</Command>
<Arguments>logrotate.conf</Arguments>
<WorkingDirectory>E:\logrotate</WorkingDirectory>
</Exec>
</Actions>
</Task>
I hope i could help someone with that guide.
Edit 2022-05-01
I faced the issue that logrotate throws exception when there are more than 9 files:
E:\logrotate>E:\logrotate\logrotate.exe logrotate.conf -f
logrotate: Force option set to true
logrotate: Exception: Cannot create a file when that file already exists.
logrotate: StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.__Error.WinIOError()
at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
at logrotate.Program.AgeOutRotatedFiles(logrotateconf lrc, FileInfo fi, String rotate_path)
at logrotate.Program.RotateFile(logrotateconf lrc, FileInfo fi)
at logrotate.Program.Main(String[] args)
E:\logrotate>E:\logrotate\logrotate.exe logrotate.conf -f
logrotate: Force option set to true
logrotate: Exception: Access to the path 'E:\MongoDB\Server\4.4\log\mongod.log.10' is denied.
logrotate: StackTrace: at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalDelete(String path, Boolean checkHost)
at System.IO.File.Delete(String path)
at logrotate.Program.DeleteRotateFile(String m_filepath, logrotateconf lrc)
at logrotate.Program.AgeOutRotatedFiles(logrotateconf lrc, FileInfo fi, String rotate_path)
at logrotate.Program.RotateFile(logrotateconf lrc, FileInfo fi)
at logrotate.Program.Main(String[] args)
E:\logrotate>
Quickfix for now is to set a maximum of 9 files.
Create a python file logrotate.py with content:
import os
# path to your mongod.log file
path = 'D:/MongoDB/Server/5.0/log/'
# you can insert a condition here that checks the size of mongod.log first
# ...
# send logrotate command through terminal
# if you don't have mongosh(ell) installed, replace by mongo / mongod
os.system('cmd /c "mongosh --eval "db.adminCommand({ logRotate: 1 })"')
# delete rotated log file
for file in os.listdir(path):
if file.startswith('mongod.log.'):
os.remove(path+file)
quit()
Create a batch file logrotate.bat with content:
python logrotate.py
Create a task in Windows task scheduler that runs this program every X hours.
I have log files on my server as follows
vpn_20191007.log
vpn_20191008.log
vpn_20191009.log
vpn_20191010.log
vpn_20191011.log
vpn_20191012.log
vpn_20191013.log
vpn_20191014.log
vpn_20191015.log
vpn_20191016.log
Is it possible to add log files pattern in fail2ban jail config?
[application]
enabled = false
filter = example
action = iptables
logpath = /var/log/vpn_%D.log
maxretry = 1
Well, conditionally it is possible...
Although wildcards are basically allowed at the moment, so :
logpath = /var/log/vpn_*.log
will do the job, but it is a bit ugly in your case:
fail2ban cumulate the list of files only by start of service, so the list remains obtained in fail2ban (unless it gets reloaded) - this means you should notify fail2ban that the log file name got changed (see https://github.com/fail2ban/fail2ban/issues/1379, the work is in progress).
since only one file will get new messages, the monitoring of other files is unneeded, especially if polling backend is used.
So better create some logrotate rules for that:
in order to rename/compress all previous log-files (to avoid match for obsolete files);
either create hard- or sym-link for last/active file with a fixed name (so fail2ban is always able to find it with the same name, and you'd not need wildcard at all);
or to notify fail2ban to reload the jail if logfile-name got changed(fail2ban-client reload vpn).
Here is an example for logrotate amendment:
postrotate
nfn="/var/log/vpn_$(date +%Y%m%d).log"
touch "$nfn"
ln -fs "$nfn" /var/log/vpn.log
You can add wilcard :
logpath = /var/log/vpn_*.log
and/or you can use multiple lines :
logpath = /var/log/vpn_20191007.log
/var/log/vpn_20191008.log
/var/log/vpn_20191009.log
/var/log/vpn_20191010.log
/var/log/vpn_20191011.log
/var/log/vpn_20191012.log
/var/log/vpn_20191013.log
/var/log/vpn_20191014.log
/var/log/vpn_20191015.log
/var/log/vpn_20191016.log
(You can combine the two)