how can I achieve fields grouping between KafkaSpout and the bolts - apache-kafka

Question

I had been sending messages to Kafka topic, these messages are in JSON in the topic and I am using KafkaSpout for fetching the messages from Kafka and sending it to the bolt using shuffle grouping. Now I want to implement fields grouping between KafkaSpout and bolt. Please can anyone help me on this. How can I achieve fields grouping between KafkaSpout and the bolts.

Answer 1

You need to implement the backtype.storm.spout.scheme interface, basically it looks something like this:
public class FooScheme implements Scheme {
public Values deserialize(final byte[] _line) {
try{
Values values = new Values();
JSONObject msg = (JSONObject) JSONValue.parseWithException(new String(_line));
values.add((String) msg.get("a"));
values.add((String) msg.get("b"))
values.add(msg)
}
catch(ParseException e) {
//handle the exception
return null;
}
}
public Fields getOutputFields() {
return new Fields("a", "b", "json");}
}
and you use it with your spout like this:
SpoutConfig spoutConfig = new SpoutConfig( ... your config here ...);
spoutConfig.scheme = new SchemeAsMultiScheme(new FooScheme());
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
topology.setSpout("kafka-spout", 1).setNumTasks(1);
and now you can are ready to use the fields grouping by "a" or "b" or both.
FooBolt bolt = new FooBolt();
topology.setBolt("foo-bolt", new FooBolt(), 1).setNumtasks(1)
.fieldsGrouping("kafka-spout", new Fields("a","b"));
Enjoy