Transferring logs using syslog-ng `as is` without timestamp and hostname etc - apache-kafka

Question

Background
Apache server running on a machine and producing logs into /var/log/httpd/error_log
Using syslog-ng to send log to a port 5140
Eventually it will be consumed by kafka producer to be send to a topic
Settings
options {
flush_lines (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (no);
};
source s_apache2 {
file("/var/log/httpd/error_log" flags(no-parse));
}
destination loghost {
tcp("*.*.*.*" port(5140));
}
Problem
syslog-ng prepends timestamp and hostname to the log data which is undesirable
<13>Jan 10 11:01:03 hostname [Tue Jan 10 11:01:02 2017] [notice] Digest: generating secret for digest authentication ...
<13>Jan 10 11:01:03 hostname [Tue Jan 10 11:01:02 2017] [notice] Digest: done
<13>Jan 10 11:01:03 hostname [Tue Jan 10 11:01:02 2017] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.30 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
Desired output (Each log line as is from error_log file)
[Tue Jan 10 11:01:02 2017] [notice] Digest: generating secret for digest authentication ...
[Tue Jan 10 11:01:02 2017] [notice] Digest: done
[Tue Jan 10 11:01:02 2017] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.4.30 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
Platform
CentOS release 6.4 (Final)
syslog-ng #version:3.2
PS
Syslog-ng to Kafka Integration : Please let me know if anybody has tried this which will render my java Kafka producer redundant

Answer 1

when you use the flags(no-parse) option in syslog-ng, then syslog-ng does not try to parse the different fields of the message, but puts everything into the MESSAGE field of the incoming log message, and prepends a syslog header. To remove this header, use a template in your syslog-ng destination:
template t_msg_only { template("${MSG}\n"); };
destination loghost {
tcp("*.*.*.*" port(5140) template(t_msg_only) );
}
To use the Kafka destination of syslog-ng, you need a newer version of syslog-ng (I'd recommend 3.8 or 3.9). Peter Czanik has written a detailed post about installing new syslog-ng rpm for CentOS.