apache beam: accessing metrics during pipeline processing - apache-beam

Question

Using the WordCount.java example to test metrics. The PrintMetricsFn function (see below) prints the metrics correctly once the pipeline is finished, but fails when called while the pipeline is processing. DirectRunner and FlinkRunner both throw exceptions (null input), while SparkRunner prints only the header line (similar to "* Metric Values *").
private static void PrintMetricsFn(final MetricsFilter inputFilter) {
final MetricResults metrics = result.metrics();
final MetricQueryResults returns = metrics.queryMetrics(inputFilter);
final Iterable<MetricResult<Long>> counters = returns.getCounters();
System.out.println("*** Metric Values ***");
for(final MetricResult<Long> s: counters){
System.out.println(s.getName().getNamespace() + ":" +
s.getName().getName() + ":" + s.getStep() + ": count=" +
s.getAttempted()) ;
}
final Iterable<MetricResult<DistributionResult>> dist = returns.getDistributions();
for (final MetricResult<DistributionResult> d: dist) {
System.out.println(d.getName().getNamespace() + ":" +
d.getName().getName() + ":" + d.getStep() + ": sum=" +
d.getAttempted().getSum() + ", count=" +
d.getAttempted().getCount() + ", min=" +
d.getAttempted().getMin() + ", max=" +
d.getAttempted().getMax() + ", mean=" +
d.getAttempted().getMean());
}
final Iterable<MetricResult<GaugeResult>> gauge = returns.getGauges();
for (final MetricResult<GaugeResult> g: gauge) {
System.out.println(g.getName().getNamespace() + ":" +
g.getName().getName() + ":" + g.getStep() + ": gauge=" +
g.getAttempted().getValue());
}
return;
}
Here are the declarations for the external variables:
private static PipelineResult result;
private static MetricQueryResults returns;
private static MetricsFilter mFilter;
And here's where the mFilter and result variables are set:
mFilter = MetricsFilter.builder()
.addNameFilter(MetricNameFilter.named("ExtractWordsFn", "emptyLines"))
.addNameFilter(MetricNameFilter.named("ExtractWordsFn", "lineLenDistro"))
.addNameFilter(MetricNameFilter.named("ExtractWordsFn", "maxLineLen"))
.build();
result = p.run();
Finally, here's the function that calls PrintMetricsFN without success:
#ProcessElement
public void processElement(#Element final String element, final OutputReceiver<String> receiver) {
lineLen = element.length();
lineLenDist.update(lineLen);
if (element.trim().isEmpty()) {
emptyLines.inc();
}
if (lineLen > localMax) {
localMax = lineLen;
maxLineLen.set(localMax);
}
if((lineCnt++ % 10) == 0) {
PrintMetricsFn(mFilter);
}
// Split the line into words.
final String[] words = element.split(ExampleUtils.TOKENIZER_PATTERN, -1);
// Output each word encountered into the output PCollection.
for (final String word : words) {
if (!word.isEmpty()) {
receiver.output(word);
}
}
}
And for completeness, here's the error I get from DirectRunner:
Caused by: java.lang.NullPointerException
at org.apache.beam.examples.WordCount.PrintMetricsFn(WordCount.java:158)
at org.apache.beam.examples.WordCount.access$1(WordCount.java:157)
at org.apache.beam.examples.WordCount$ExtractWordsFn.processElement(WordCount.java:132)