following is my configration of log4j2:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="trace" name="MyApp" packages="com.swimap.base.launcher.log">
<Appenders>
<RollingFile name="RollingFile" fileName="logs/app-${date:MM-dd-yyyy-HH-mm-ss-SSS}.log"
filePattern="logs/$${date:yyyy-MM}/app-%d{MM-dd-yyyy}-%i.log.gz">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="1 KB"/>
</Policies>
<DefaultRolloverStrategy max="3"/>
</RollingFile>
</Appenders>
<Loggers>
<Root level="trace">
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>
the issue is that each time when start up my service, a new log will be created even the old one has not reached the specific size. If the program restart frequently, i will got many log files end with ‘.log’ which never be compressed.
the logs i got like this:
/log4j2/logs
/log4j2/logs/2017-07
/log4j2/logs/2017-07/app-07-18-2017-1.log.gz
/log4j2/logs/2017-07/app-07-18-2017-2.log.gz
/log4j2/logs/2017-07/app-07-18-2017-3.log.gz
/log4j2/logs/app-07-18-2017-20-42-06-173.log
/log4j2/logs/app-07-18-2017-20-42-12-284.log
/log4j2/logs/app-07-18-2017-20-42-16-797.log
/log4j2/logs/app-07-18-2017-20-42-21-269.log
someone can tell me how can i append log to the exists log file when i start up my program? much thanks whether u can help me closer to the answer!!
I suppose that your problem it that you have fileName="logs/app-${date:MM-dd-yyyy-HH-mm-ss-SSS}.log in your log4j2 configuration file.
This fileName template means that log4j2 will create log file with name that contains current date + hours + minutes + seconds + milliseconds in its name.
You should probably remove HH-mm-ss-SSS section and this will allow you to have daily rolling file and to not create new file every app restart.
You can play with template and choose format that you need.
If you want only one log file forever - then create constant fileName, like fileName=app.log
It's not hard to implement this. There is a interface DirectFileRolloverStrategy, implement below method:
public String getCurrentFileName(RollingFileManager manager)
Mybe someone met same problem and this can help him.
Related
I'm using imply to handle druid's cluster. But my logs files have increased to hundreds of gigabytes of storage. I'm talking about logs files present in imply/var/sv/ directory in which there are these 7 log files, broker.log, historical.log, middleManager.log zk.log, coordinator.log, imply-ui.log, and overlord.log.
Among them, this particular file called coordinator.log has increased to the really massive size of about 560 GBs in a matter of a few months. I have read all those logs, and they don't bother me much. What I'm concerned about is the size of the file which is eating my entire storage. I have tried finding ways to limit the size of those log files but believe me nothing worked for me.
I read in many places that druid uses log4j2 logger so we can limit the size using its configuration from log4j2.xml file. But again a big confusion there are four log4j2.xml files which one shall I modify?
I tried modifying all of them, but still, it didn't work. I'm kind of a fool while handling it seems like... Well so this is my request if anybody could point me in the right direction in limiting the size of these log files
You can setup a simple cron process to truncate these files periodically using truncate -s 0 imply/var/sv/*.log
The default log level in imply distribution is set to info which generates a lots of logs. If they don't bother you much, you can set the log level to error so that logs will be generated only when there is any error during system runtime. To set that, you need to modify logger level inside conf/druid/_common/log4j2.xml file.
<?xml version="1.0" encoding="UTF-8" ?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>
And even after do so, you should periodically truncate log files as #mdeora suggested.
I am developing mule flow wherein mule process has to upload file/files (from source directory) to SFTP server.
IN Folder (This folder holds file to transfer to SFTP)
I have below mule flow
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
<flow name="sftpFlow1">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="30" startDelay="10" timeUnit="SECONDS"/>
<logger message="Hi" level="INFO" doc:name="Logger"/>
</poll>
<flow-ref name="sftpSub_Flow" doc:name="Flow Reference"/>
</flow>
<sub-flow name="sftpSub_Flow">
<file:outbound-endpoint path="D:\IN" responseTimeout="10000" doc:name="File"/>
<sftp:outbound-endpoint exchange-pattern="one-way" host="host" port="port" path="sftppath" user="user" password="password" responseTimeout="10000" doc:name="SFTP"/>
</sub-flow>
</mule>
Issues:
I don't see any error or exceptions but expected files from IN folder are not getting transfer SFTP server. I see ****.dat file every time sub-flow executes.
Any suggestion?
Here you have used both outbound end points. You need file inbound end point component to pick the file from source location.
refer below sample for further
https://dzone.com/articles/anypoint-file-connector-with-mulesoft
you can use the outputPattern property in sftpoutbound endpoint to change the file name with any extension
Because the application doesn't actually show any input, I'll assume that the sample is flawed but the real application has some inbound endpoint (for example a file inbound endpoing) that is reading in streaming mode. The subflow is trying to consume two time the input stream to output to the two outbound endpoints. Because the first one consumes the stream the second one, the SFTP, gets a consumed empty stream as input so it doesn't output anything. A solution is to convert the input to something in memory, like using the object to byte array transformer in the middle. Be warned that for large files you might run out of memory doing this.
The server.log says "signature_status": "UNVERIFIED". Is this a certificate issue?
Also what are the best ways to read the pingfederate logs in windows machine.
That sounds like an issue with signature verification, which could be the cert itself but is more likely a configuration issue. More information is really needed to know which it is.
I assume the issue you are having with reading logs on windows machines is because the files are large or are moving quickly. If the files are too big you can modify the log4j2.xml config file at appdir/pingfed*/pingfed*/server/default/conf/log4j2.xml to reduce the log size to something easier to read in notepad. Here is an example rolling file appender that should leave you with easily maneageable files.
<RollingFile name="FILE" fileName="${sys:pf.log.dir}/server.log"
filePattern="${sys:pf.log.dir}/server.log.%i" ignoreExceptions="false">
<PatternLayout>
<!-- Uncomment this if you want to use UTF-8 encoding instead
of system's default encoding.
<charset>UTF-8</charset> -->
<pattern>%d %X{trackingid} %-5p [%c] %m%n</pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy
size="20000 KB" />
</Policies>
<DefaultRolloverStrategy max="5" />
</RollingFile>
If you issue is that the files are moving too fast to read then you might consider using something like baretail or Get-Content in powershell now that it has a tail switch.
I created a mailstream with following command -------
stream create --name mailstream --definition "mail --host=imap.gmail.com --username=yyyyyyyy12#gmail.com --password=my password | file --dir=/tmp/gmailData" --deploy
Refer -http://docs.spring.io/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#modules
But in the xd-singletone console I get -
Caused by: javax.mail.AuthenticationFailedException: failed to connect, no password specified?
How to resolve this issue.
Also --password=secret - how to keep my password invisible or secret in the XD shell
/shankha
You need to escape "#" with "%40" for both username and password and to specify --port=993 for gmail. Also, it may be possible not to work with the default settings as GMail requires SSL for imap and this needs to be configured as well.
So, I would suggest the following (basically, creating a new source module):
Go to spring-xd-1.0.0.M6\xd\modules\source and make a copy of mail folder and name this copy gmail
Go to spring-xd-1.0.0.M6\xd\modules\source\gmail\config and rename both mail.properties and mail.xml to gmail.properties and gmail.xml respectively
Inside gmail.xml replace everything with:
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/integration"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans"
xmlns:file="http://www.springframework.org/schema/integration/file"
xmlns:int-mail="http://www.springframework.org/schema/integration/mail"
xmlns:context="http://www.springframework.org/schema/context"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="http://www.springframework.org/schema/integration/mail http://www.springframework.org/schema/integration/mail/spring-integration-mail.xsd
http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util.xsd">
<channel id="output" />
<int-mail:mail-to-string-transformer
charset="${charset}" input-channel="transform" output-channel="output" />
<beans:beans profile="use-polling">
<int-mail:inbound-channel-adapter
store-uri="${protocol}://${username:}:${password:}#${host}:${port}/${folder}"
channel="transform" should-mark-messages-as-read="${markAsRead}"
should-delete-messages="${delete}" java-mail-properties="javaMailProperties">
<poller fixed-delay="${fixedDelay}" time-unit="SECONDS">
<advice-chain>
<beans:bean
class="org.springframework.xd.dirt.module.support.ThreadContextClassLoaderSetterAdvice" />
</advice-chain>
</poller>
</int-mail:inbound-channel-adapter>
</beans:beans>
<beans:beans profile="use-idle">
<int-mail:imap-idle-channel-adapter
store-uri="${protocol}://${username:}:${password:}#${host}:${port}/${folder}"
channel="transform" auto-startup="true" mail-filter-expression="${expression}"
should-mark-messages-as-read="${markAsRead}"
should-delete-messages="${delete}" java-mail-properties="javaMailProperties">
</int-mail:imap-idle-channel-adapter>
</beans:beans>
<beans:beans profile="default">
<util:properties id="javaMailProperties">
<beans:prop key="mail.imap.socketFactory.class">javax.net.ssl.SSLSocketFactory</beans:prop>
<beans:prop key="mail.imap.socketFactory.fallback">false</beans:prop>
<beans:prop key="mail.store.protocol">imaps</beans:prop>
<beans:prop key="mail.debug">false</beans:prop>
</util:properties>
</beans:beans>
</beans:beans>
4. In XD shell now you will use something like the following to create your stream:
stream create --name myGmailStream --definition "gmail --host=imap.gmail.com --username=yyyyyyyy12%40gmail.com --password=my_password --port=993 | file --dir=/tmp/gmailData" --deploy
Here, please note the following:
I added --port=993
the username contains "%40" instead of "#"
the definition of the stream starts with "gmail
if your password contains "#" you need to replace that with "%40" as well
What I've done above is to, basically, create a new custom module (a source) which is kind of easy (more details about this you can find in the documentation). The XD single node or the XD Shell doesn't even need to be restarted. Give it a try and let me know how it goes.
Regarding the password that you don't want to appear as part of the stream definition, you can provide it as part of your mail module options, as described here: http://docs.spring.io/spring-xd/docs/1.0.0.BUILD-SNAPSHOT/reference/html/#_module_configuration
ie
<xd_home>/config/modules/source/mail/mail.properties:
password: yourpassword
The Lift wiki page on logging states that a lot of setup is done automatically. Right now my problem is that I already have a running backend with its own logging configuration and a log4j.properties file in my classpath which should be used. There are also dependencies to log4j and SLF4j already in the classpath.
The main problem is that I get complete debug output for everything. Hibernate in particular -- which is very annoying.
I am using Lift 2.3-M1 and tried doing the following in the beginning of boot():
Logger.setup = Full(Log4j.withFile(getClass().getResource("/props/log4j.xml")))
The log4j.xml I am currently using is quickly hacked together to simply suppress DEBUG output.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/doc-files/log4j.dtd">
<log4j:configuration threshold="info" xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="CA" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%p] %c{2} %m%n"/>
</layout>
</appender>
<root>
<level value="info"/>
<appender-ref ref="CA"/>
</root>
</log4j:configuration>
When I create an errornous log4j.xml I also get an error from the SAXParser so it must be parsed. However I am still getting all DEBUG output. A second try was doing the following:
LiftRules.configureLogging = () => ()
Logger.setup = Full(Logback.withFile(getClass().getResource("/props/log4j.xml")))
Since I do not want to have Lift configure logging for me since the backend is already configured I would like to turn it completely off now. Oh and I also tried LogBoot.logSetup = () => false with no luck.
I would greatly appreciate any help on this issue.
The question got answered on the Lift mailing list.
The fix is to remove the logback dependency and to include both log4j and slf4j-log4j. No other configuration in boot() is required besides a valid default.log4j.xml.
I had a similar problem and came upon the following solution:
import net.liftweb.common.{ Empty, Logger }
import net.liftweb.http.Bootable
class BootLoader extends Bootable {
def boot = {
// other boot configuration ...
// prevent Lift from messing up my log4j config
Logger.setup = Empty
}
}