Data transformation during load - mongodb

Question

On my server runs an application called 'logstash'. This application receives log entries from multiple servers and uploads them as JSON documents to MongoDB. Works like a charm.
Example:
{
u'syslog_message': u'[10724525.839722] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=1.2.3.4 DST=9.8.7.6 LEN=52 TOS=0x08 PREC=0x20 TTL=50 ID=55384 PROTO=TCP SPT=349 DPT=123 WINDOW=14600 RES=0x00 SYN URGP=0 ',
u'received_from': u'1.3.5.7:1234',
u'#version': u'1',
u'#timestamp': datetime.datetime(2014, 11, 20, 15, 9, 55),
u'syslog_timestamp': u'Nov 20 15:09:55',
u'syslog_facility': u'user-level',
u'syslog_severity': u'notice',
u'host': u'2.4.6.8:2468',
u'syslog_program': u'kernel',
u'syslog_hostname': u'server01',
u'received_at': u'2014-11-20 20:09:55 UTC',
u'message': u'<4>Nov 20 15:09:55 server01 kernel: [10724525.839722] [UFW BLOCK] IN=venet0 OUT= MAC= SRC=1.2.3.4 DST=2.3.4.5 LEN=52 TOS=0x08 PREC=0x20 TTL=50 ID=55384 PROTO=TCP SPT=1234 DPT=543 WINDOW=14600 RES=0x00 SYN URGP=0 ',
u'_id': ObjectId('546e4a93e98673fe8f11a4d2'),
u'type': u'syslog',
u'syslog_severity_code': 5,
u'syslog_facility_code': 1
}
But the data is not exactly how I want it to be. I need to convert string to date, add some elements based on other elements and more 'transformations' on each document that is loaded into a specific collection.
What is the default way to handle this and where is this entire process documented?

Answer 1

Logstash has a number of filter plugins that can be used to add, delete, and modify message fields. The Logstash documentation lists them all. Judging by your example message above I'd say there already are a number of filters in place. It sounds like you'll at least need an additional date filter and a mutate filter to accomplish what you're outlining.