I am having about 15 servers that send their syslog to a common logserver, all servers are redhat7.7, so the rsyslog is a recent version (8.24).
The normal log get transfered to the common server OK, but then there are some application logs that are watched with the imfile module, and my problem is that only some of them are transfered correctly, but others are not transfered, or there are days, where logs are missing for hours, and then are OK the rest of the day.
How can I debug this problem, I can show a small portion of the custom log definition:
$InputFileName /var/log/log-compression.log
$InputFileTag log-compression-log
$InputFileStateFile stat-log-compression-log
$InputFileSeverity info
$InputFileFacility local2
$InputRunFileMonitor
There are about 20 such sections, and some of them work, and others not, and the syntax is exactly the same, it is deployed with ansible.
Related
I've told PostgreSQL to log to syslogd and to tag its messages with "postgresql". These messages appear in /var/log/syslog with that tag.
Now I'd like those messages not to appear in /var/log/syslog but in /var/log/example/postgresql/. I thought I could do that by creating the file /etc/rsyslog.d/25-postgresql.conf with this single line (plus some comments, in real life):
:msg,contains,"postgresql" /var/log/example/postgresql/postgresql.log
Restarting rsyslog doesn't result in postgresql messages going to that file, however: they still go to /var/log/syslog.
Any suggestions what I've done wrong?
I am running an ELK-Stack as a central syslogserver and I set up rsyslog to send logfiles, which are not logging into /var/lib/messages by default, to it.
The setup is working very well but since I made the configuration the external logfiles actually show up in the messages file, which is blowing it out of proportion and makes debugging normal systemlogs difficult.
I want the logs to be send to the syslogserver but not into the messages file.
This is my current configuration:
111-elk-syslog.conf:
*.* ##IP_OF_THE_SYSLOGSERVER:514
101-external-log.conf
$ModLoad imfile
$InputFileName PATH_TO_LOGFILE
$InputFileTag FILE_TAG
$InputFileStateFile FILE_TAG
$InputFileFacility local3
$InputRunFileMonitor
I know, using filebeat, I could circumvent this but rsyslog is working very well in my enviroment and this application is the only one logging so much, that this is an actual problem.
I don't understand in detail your setup, but it may help to know that in general in rsyslogd if you don't want further handling of a message that you have matched, you simply repeat the filter on the next line by using &, and the action stop. For example, you might try
*.* ##IP_OF_THE_SYSLOGSERVER:514
& stop
I am using rsyslog v5 to centralize logs to a server, i see exactly duplicat elogs under /var/log/messages on my log server, although i do not see duplicate lines under distributed servers logs
I am using Amazon AMI-centos
I figuered it out - was monitoring 2 files under each server and was sending them via *.* ##<%= #log_servers %>:514 twice
I provide support for a large application across multiple servers. System has been running live for 6+ months.
8th December: total system failure. iisreset across each of the servers sorted it out. Everything back to normal.
Post failure investigation showed various processes not able to get a response from a particular server which hosts an instance of Dynamics CRM (2011 R11). Specifically it seems the SOAP service was not responding (Organization.svc). 503 - Server Unavailable (really it was just the web service). I suspect it died.
Having the exact time of the error I checked the event logs on the server but these did not have anything of use. The last error prior to the failure was a report rendering error which was 9 minutes before the system actually went down. Surely if web service crashed this would be reflected in the event log?
Fast forward to today, 8th January and the system fails again. The 8th of the month again! iisreset fixes it... again!
Again, completely useless event logs showings no errors prior to failure.
Entertained the idea of Dynamics CRM trace logging but this is out of the question due to the performance hit.
Apart from the event logs where else to look? Are there possible external factors or causes? I'm trying to find the root cause but have run out of ideas!
While this may not address the source of your problem, maybe it can help minimize the symptoms. May I suggest that you configure the IIS server to recycle the application pool at a scheduled interval within your production environment.
http://technet.microsoft.com/en-us/library/cc753179%28v=ws.10%29.aspx
We have an IIS .Net application deployed across several machines. We use IIS log information to do reporting of performance of the web application and navigation by the user. Currently the reporting is only required infrequently (once a day, for the previous day), so we just roll the logs every 24 hours, and move the old logs to our reporting server.
We have a new requirement that means we need much faster turnaround on the IIS log information, say every minute for the sake of the discussion.
There exist Apache tools like Facebook's Scribe to scalably move Apache web server logs across a network of servers.
Are there any similar tools available for IIS?
Is this the right question to ask?
Should we be doing something different, if the timing requirements have changed so much?
I've looked at this question and the answers, and the only one that seems to come close is this one.
Pointers appreciated!
Snare is a little old but worth mentioning.
Snare Agent for IIS Servers
http://www.intersectalliance.com/projects/SnareIIS/index.html
I used this old version a long time ago and it worked well by forwarding/sending/replicating IIS logs over a network via syslog.
Today, they have a newer version called Snare Epilog
http://www.intersectalliance.com/projects/EpilogWindows/index.html
The code is also open source; perhaps you might find it useful.
You might also want to try ...
http://nxlog.org
http://www.syslogserver.com/syslogagent.html
I tend to write a .bat file in conjunction with LOG Parser 2.2. The .Bat file will determine the appropriate file dates and pull the corresponding logs from multiple IIS server log locations into a single local directory. Once the files are across I then run a Log Parser command to query the log content over all log files and then produce a single output file in .csv format. Finally, I run an SSIS job to import the new .csv file into a running log table which I can then query on an ongoing basis.