differential osquery query output to "catchall" topic - apache-kafka

Question

I'm using osquery to monitor servers on my network. The following osquery.conf captures snapshots, every minute, of the processes communicating over the network ports and publishes that data to Kafka:
{
"options": {
"logger_kafka_brokers": "cp01.woolford.io:9092,cp02.woolford.io:9092,cp03.woolford.io:9092",
"logger_kafka_topic": "base_topic",
"logger_kafka_acks": "1"
},
"packs": {
"system-snapshot": {
"queries": {
"processes_by_port": {
"query": "select u.username, p.pid, p.name, pos.local_address, pos.local_port, pos.remote_address, pos.remote_port from processes p join users u on u.uid = p.uid join process_open_sockets pos on pos.pid=p.pid where pos.remote_port != '0'",
"interval": 60,
"snapshot": true
}
}
}
},
"kafka_topics": {
"process-port": [
"pack_system-snapshot_processes_by_port"
]
}
}
Here's an example of the output from the query:
osquery> select u.username, p.pid, p.name, pos.local_address, pos.local_port, pos.remote_address, pos.remote_port from processes p join users u on u.uid = p.uid join process_open_sockets pos on pos.pid=p.pid where pos.remote_port != '0';
+--------------------+-------+---------------+------------------+------------+------------------+-------------+
| username | pid | name | local_address | local_port | remote_address | remote_port |
+--------------------+-------+---------------+------------------+------------+------------------+-------------+
| cp-kafka-connect | 13646 | java | 10.0.1.41 | 49018 | 10.0.1.41 | 9092 |
| cp-kafka-connect | 13646 | java | 10.0.1.41 | 49028 | 10.0.1.41 | 9092 |
| cp-kafka-connect | 13646 | java | 10.0.1.41 | 49026 | 10.0.1.41 | 9092 |
| cp-kafka-connect | 13646 | java | 10.0.1.41 | 50558 | 10.0.1.43 | 9092 |
| cp-kafka-connect | 13646 | java | 10.0.1.41 | 50554 | 10.0.1.43 | 9092 |
| cp-kafka-connect | 13646 | java | 10.0.1.41 | 49014 | 10.0.1.41 | 9092 |
| root | 1505 | sssd_be | 10.0.1.41 | 46436 | 10.0.1.89 | 389 |
...
| cp-ksql | 1757 | java | 10.0.1.41 | 56180 | 10.0.1.41 | 9092 |
| cp-ksql | 1757 | java | 10.0.1.41 | 53878 | 10.0.1.43 | 9092 |
| root | 19684 | sshd | 10.0.1.41 | 22 | 10.0.1.53 | 50238 |
| root | 24082 | sshd | 10.0.1.41 | 22 | 10.0.1.53 | 51233 |
| root | 24107 | java | 10.0.1.41 | 56052 | 10.0.1.41 | 9092 |
| root | 24107 | java | 10.0.1.41 | 56054 | 10.0.1.41 | 9092 |
| cp-schema-registry | 24694 | java | 10.0.1.41 | 50742 | 10.0.1.31 | 2181 |
| cp-schema-registry | 24694 | java | 10.0.1.41 | 47150 | 10.0.1.42 | 9093 |
| cp-schema-registry | 24694 | java | 10.0.1.41 | 58068 | 10.0.1.41 | 9093 |
| cp-schema-registry | 24694 | java | 10.0.1.41 | 47152 | 10.0.1.42 | 9093 |
| root | 25782 | osqueryd | 10.0.1.41 | 57700 | 10.0.1.43 | 9092 |
| root | 25782 | osqueryd | 10.0.1.41 | 56188 | 10.0.1.41 | 9092 |
+--------------------+-------+---------------+------------------+------------+------------------+-------------+
Instead of snapshots, I'd like osquery to capture differentials, i.e. to only publish the changes to Kafka.
I tried toggling the snapshot property from true to false. My expectation was that osquery would send the changes. For some reason, when I set "snapshot": false, no data is published to the process-port topic. Instead, all the data is routed to the catchall base_topic.
Can you see what I'm doing wrong?
Update:
I think I'm running into this bug: https://github.com/osquery/osquery/issues/5559
Here's a video walk-through: https://youtu.be/sPdlBBKgJmY
I filed a bug report, with steps to reproduce, in case it's not the same issue: https://github.com/osquery/osquery/issues/5890

Answer 1

Given the context, I can't immediately tell what is causing the issue you are experiencing.
In order to debug this, I would first try using the filesystem logger plugin instead of (or in addition to) the Kafka logger.
Do you get results to the Kafka topic when the query is configured as a snapshot? If so, are you able to verify that the results are actually changing such that a diff should be generate when the query runs in differential mode?
Can you see results logged locally when you use --logger_plugin=filesystem,kafka?