User agent parser (ua-parser) slows down Spark on EMR - scala

Question

I am using ua-parser in my UDFs to parse User agent info. And I noticed that these jobs are very slow compared to the ones without parser. Here is an example:
import org.uaparser.scala.Parser
val parser: Parser = Parser.default
val parseDeviceUDF = udf((ua: String) => Try(parser.parse(ua).device.family).toOption.orNull)
The stange thing is that when I submit job as a EMR step, it is slow, but when I run same code in Zeppelin or Spark shell it works fine. I write data to parquet files. And that is the stage where it gets stuck.

Answer 1

The answer I am about to give is not about an open-source project, but it does provide information that whoever is researching how to parse the user-agent string to obtain device intelligence will want to know about.
WURFL is a time-honored tool to do User-Agent (and more generally HTTP request) analysis and obtain easily consumable device/browser information. ScientiaMobile has recently released a version of WURFL (called WURFL Microservice) that can be obtained from the major marketplaces of AWS, Azure and GCP (in addition to ScientiaMobile itself of course).
In the case at hand, the (Java) code that would bring a Spark user from HTTP logs to device data would look something like this.
JavaDStream enrichedEvents = events.map(evs -> {
WmClient wmClient = WmClientProvider.getOrCreate(wmServerHost, "80");
for (EnrichedEventData evItem : evs) {
...
HttpServletRequestMock request = new HttpServletRequestMock(evItem.getHeaders());
Model.JSONDeviceData device = wmClient.lookupRequest(request);
evItem.setWurflCompleteName(device.capabilities.get("complete_device_name"));
evItem.setWurflDeviceMake(device.capabilities.get("brand_name"));
evItem.setWurflDeviceModel(device.capabilities.get("model_name"));
evItem.setWurflFormFactor(device.capabilities.get("form_factor"));
evItem.setWurflDeviceOS(device.capabilities.get("device_os") + " " + \
device.capabilities.get("device_os_version"));
...
}
return evs;
});
More information about how Spark and WURFL are integrated can be found in this article.
Disclaimer: I am the CTO of ScientiaMobile and original creator of WURFL.