Log4j - Log file not getting sent to external log viewer - kubernetes

I have a pod that uses a log4j2.xml config file to manage logging configuration and for sending it to an external log viewer (Elastic Search). This pod has two log files that need to sent to Elastic Search. A main log file and a ruletrace log file. This is the log4j config file:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration monitorInterval="30">
<Properties>
<Property name="logFolder">/var/log/svc</Property> <!-- Must NOT end with a slash -->
</Properties>
<appenders>
<Console name="console" target="SYSTEM_ERR">
<PatternLayout pattern="%d{ISO8601};%p;[%t];%logger;%msg\n"/>
</Console>
<RollingRandomAccessFile name="allLogRollingFileAppender"
fileName="${logFolder}/svc.log"
filePattern="${logFolder}/svc-%d{yyyy-MM-dd}T%d{HH-mm}_%i.log.gz"
immediateFlush="false">
<PatternLayout pattern="%d{ISO8601};%p;[%t];%logger;%msg\n"/>
<Policies>
<CronTriggeringPolicy schedule="0 0 0 * * ?"/>
<SizeBasedTriggeringPolicy size="20 MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="${logFolder}" maxDepth="1">
<IfFileName glob="svc-*.gz">
<IfAccumulatedFileSize exceeds="100 MB" />
</IfFileName>
</Delete>
</DefaultRolloverStrategy>
</RollingRandomAccessFile>
<RollingRandomAccessFile name="ruleTraceRollingFileAppender"
fileName="${logFolder}/svc_ruletrace.log"
filePattern="${logFolder}/svc_ruletrace-%d{yyyy-MM-dd}T%d{HH-mm}_%i.log.gz"
immediateFlush="true">
<PatternLayout pattern="%m%n"/>
<Policies>
<CronTriggeringPolicy schedule="0 0 0 * * ?"/>
<SizeBasedTriggeringPolicy size="30 MB" />
</Policies>
<DefaultRolloverStrategy>
<Delete basePath="${logFolder}" maxDepth="1">
<IfFileName glob="svc_ruletrace*.gz">
<IfAccumulatedFileSize exceeds="100 MB" />
</IfFileName>
</Delete>
</DefaultRolloverStrategy>
</RollingRandomAccessFile>
<Syslog name="sysLogAppender"
format="RFC5424"
advertise="false"
host="localhost"
port="5140"
protocol="UDP"
facility="local6"
appName="svc"
enterpriseNumber="193"
mdcId="svcMdcId"
includeMDC="false"
newLine="true"
exceptionPattern="%throwable{full}"
immediateFlush="false">
<MarkerFilter marker="RULETRACE" onMatch="DENY" onMismatch="ACCEPT"/>
</Syslog>
</appenders>
<loggers>
<Logger name="RuleTrace" level="debug" includeLocation="false" additivity="false">
<AppenderRef ref="ruleTraceRollingFileAppender"/>
</Logger>
<Logger name="com.comp" level="info" additivity="true">
<AppenderRef ref="allLogRollingFileAppender"/>
</Logger>
<root level="info">
<AppenderRef ref="console" level="fatal"/>
<AppenderRef ref="sysLogAppender" level="info"/>
</root>
</loggers>
</Configuration>
The main log file (svc.log) gets sent to ElasticSearch with no issues. The ruletrace doesn't, however. There are log messages appearing in the ruletrace log file on the pod but those log messages don't get sent to ElasticSearch. I've tried changing the settings below a few times to different configs but it makes no difference, seemingly.
Is there a limit to the amount of Loggers you can reference? Would the difference in pattern layout make a difference?

Related

Stop log4net (or services) using a file when errors are not being placed in it

I have the following log4net config
<?xml version="1.0" encoding="utf-8" ?>
<log4net xmlns="urn:log4net">
<appender name ="RollingFile" type="log4net.Appender.RollingFileAppender">
<file value="C:\temp\Generator.log" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maximumFileSize value="500KB" />
<maxSizeRollBackups value="10" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date{DATE} %level %thread %logger - %message%newline" />
</layout>
</appender>
<appender name ="GenerationErrors" type="log4net.Appender.FileAppender">
<file type="log4net.Util.PatternString" value="c:\temp\GenerationErrors.log" />
<appendToFile value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%date{DATE} %level %thread %logger - %message%newline" />
</layout>
</appender>
<root>
<appender-ref ref="RollingFile" />
<level value="INFO" />
</root>
<logger name ="GPA.Module.Generate">
<level value="INFO" />
<appender-ref ref="GenerationErrors" />
</logger>
<logger name="PmuManagement">
<level value="INFO" />
<appender-ref ref="GenerationErrors" />
</logger>
</log4net>
The first appender logs all errors for all of my processes and the second only logs a certain group of errors, as required.
I need to be able to delete the contents of the second log file on the fly however this isn't currently possible as the programme needs constant access to it to run (if you try and make any changes you get the "this file is in use by another programme" error.
Is there a way for log4net to only need access to the file when errors need to be written to it?
Or is there a more generic way to stop services/programmes needing access to file temporarily and allowing access again shortly after?
Thanks
I found this question here Intermittent log4net RollingFileAppender locked file issue and the answer is what I needed. I simply added <lockingModel type="log4net.Appender.FileAppender+MinimalLock" /> to my second appender.

Log4Net today's backup file does not have date in file name

Below mentioned Log4Net, having date in file name if date is <= yesterday, however files which are back up today, does not have date in file name.
Is it the specification of Log4Net?
Or
It is because of some setting of Log4Net config file?
File
<?xml version="1.0" encoding="UTF-8"?><configuration>
<log4net>
<logger name="SampleLog">
<level value="FATAL"/>
<level value="ERROR"/>
<level value="WARN"/>
<level value="INFO"/>
<level value="DEBUG"/>
<appender-ref ref="SampleLogFile"/>
</logger>
<appender name="SampleLogFile" type="log4net.Appender.RollingFileAppender">
<file type="log4net.Util.PatternString" value="%property{SampleService}"/>
<appendToFile value="true"/>
<maxSizeRollBackups value="15"/>
<maximumFileSize value="1KB"/>
<rollingStyle value="Composite"/>
<datePattern value="yyyyMMdd"/>
<staticLogFileName value="true"/>
<Encoding value="UTF-8"/>
<layout type="log4net.Layout.PatternLayout">
<!-- %d:DateTime, %t:Thread, %-5p:Categories, %c:LoggerName, %m:Message, %n:line separator-->
<param name="ConversionPattern" value="%p %d{ISO8601} %t %m%n"/>
</layout>
</appender>
</log4net>
</configuration>
File name example :
Application_Sample.log // Currently in progress log
Application_Sample.log.1 // Today's backup
Application_Sample.log20190801.2 // Yesterday's backup
You just specify the rollingStyle as Composite, so RollingFileAppender will roll log files on a date period and within a date period on file size. For each day only the last 10 files of 1MB will be kept. And these rolled files will be named: log.txt.1, log.txt.2, log.txt.3, etc. Youcan find the details via https://logging.apache.org/log4net/release/config-examples.html#RollingFileAppender

RollingFile Appender not using current date

The Rolling File Appender is not putting the most recent log entries in the current date file.
In the log4j2.xml file I have a RollingFile appender which looks like this
<RollingFile
name="viewAppender"
filePattern="/data/view/view%d{yyyy-MM-dd}.log">
<PatternLayout><pattern>%m%n</pattern></PatternLayout>
<Policies>
<TimeBasedTriggeringPolicy />
</Policies>
</RollingFile>
When the new log entry goes, its not going in the current date
Here is the log file. It shows that the most recent entry was from Aug 30 but the log file name was for Aug 28. It should have created a new log file with filePattern view2018-08-30.log
-rw-r--r-- 1 root root 230 Aug 30 10:36 view2018-08-28.log
You can use below log4j2.xml which is a working one and creates new log file everyday
<?xml version="1.0" encoding="UTF-8"?>
<Configuration monitorInterval="60">
<Appenders>
<Console name="console-appender" target="SYSTEM_OUT">
<PatternLayout pattern="[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} -
%msg%n"/>
</Console>
<RollingFile name="rollingFile-appender"
fileName="${sys:user.home}/log/apache-tomcat/file.log"
filePattern="${sys:user.home}/log/apache-tomcat/file-%d{yyyy-MM-dd}.log">
<PatternLayout pattern="[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c{1} -
%msg%n"/>
<Policies>
<TimeBasedTriggeringPolicy/>
<SizeBasedTriggeringPolicy size="50MB"/>
</Policies>
<DefaultRolloverStrategy max="30"/>
</RollingFile>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="rollingFile-appender"/>
<AppenderRef ref="console-appender"/>
</Root>
</Loggers>
</Configuration>

log4j2 highlight first line only in the console

I am using log4j2 to log to the console and file as in this xml file
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%highlight{%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - Line %L: %msg%n}"/>
</Console>
<File name="MyFile" fileName="all.log" immediateFlush="true" append="false">
<PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - Line %L: %msg%n"/>
</File>
</Appenders>
<Loggers>
<Root level="debug">
<AppenderRef ref="Console" level="error"/>
<AppenderRef ref="MyFile" level="info"/>
</Root>
</Loggers>
</Configuration>
The problem is only the first line in the netbeans console appears in red while the remaining lines are black as in the following picture.
Any help will be appreciated
Update:I tried the project with eclipse and all lines appeared in black

Using logback with Akka

I have some problems using Logback with my Akka (2.3.9) application. In order to log to stdout and in the logfile, I specified the logback.xml with all appenders:
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="5 seconds" debug="true">
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<target>System.out</target>
<encoder>
<pattern>%X{akkaTimestamp} %-5level %logger{36} %X{sourceThread} - %msg%n</pattern>
</encoder>
</appender>
<appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>./akka.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>./akka.log.%d{yyyy-MM-dd-HH}</fileNamePattern>
</rollingPolicy>
<encoder>
<pattern>%X{akkaTimestamp} %-5level %logger{36} %X{sourceThread} - %msg%n</pattern>
</encoder>
</appender>
<logger name="proc" level="INFO">
<appender-ref ref="FILE"/>
<appender-ref ref="STDOUT"/>
</logger>
<logger name="akka.actor" level="INFO">
<appender-ref ref="FILE"/>
<appender-ref ref="STDOUT"/>
</logger>
<root level="INFO">
<appender-ref ref="FILE"/>
<appender-ref ref="STDOUT"/>
</root>
</configuration>
After that I use logging in my Actor:
val log = Logging(context.system, classOf[MyActor])
override def receive: Receive = {
case MyEvent(event) =>
log.info("Message received for processing.")
}
The problem is, in SBT everything is fine: I can see all log events in the created logfile.
When I make JAR-File with sbt-assembly and start this jar file (java -jar event-assembly-0.1.1-SNAPSHOT.jar). The app writes log entries to STDOUT but not in the file!
I have no idea how I can fix that. I tried with "lsof", the java process has no open log-files.
I don't know what's wrong but you could do the following:
Check if your logback.xml is in the jar-file
Try to start your application with java -Dlogback.configurationFile=/path/to/logback.xml -jar event-assembly-0.1.1-SNAPSHOT.jar
So, how I found out, the sbt-assembly removed some logback-binaries from the result jar (it was settings in my Build.scala). After I changed the MergingStrategy to preserve the first logback occurrence in the result build, all works correctly.