How spark-streaming dealing with snappy compressed data which in kafka - apache-kafka

Question

The kafka producer sample code is:
#!/usr/bin/env python
#-*- coding: utf-8 -*-
import ConfigParser as configparser
from pykafka import KafkaClient
import time
import snappy
config = configparser.ConfigParser()
config.read("conf.ini")
app_name = "test_word_counter"
kafka_hosts = config.get(app_name, 'kafka_hosts')
kafka_topic = config.get(app_name, 'kafka_topic')
print("kafka client: %s" % kafka_hosts)
print("kafka topic: %s" % kafka_topic)
kafka_client = KafkaClient(hosts=kafka_hosts) # Create Kafka client
topic = kafka_client.topics[kafka_topic] # This will create the topic if it does not exist
with topic.get_producer() as producer: # Create Kafka producer on the given topic
while True:
msg = "just a test for snappy compress with kafka and spark"
msg = snappy.compress(msg) # add snappy compress
producer.produce(msg) # Send the message to Kafka
print("send data len(%d)" % len(msg))
print(msg)
time.sleep(5)
Code is very simple, use python snappy, and compressing data, then put it to kafka.
The pyspark code is:
def word_counter(zk_host, topic):
sc = SparkContext(appName="PythonStreamingKafkaWordCounter")
sc = SparkContext(conf=spark_conf)
ssc = StreamingContext(sc, 30)
kvs = KafkaUtils.createStream(ssc, zk_host, "spark-streaming-consumer", {topic: 2})
lines = kvs.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
counts.pprint()
ssc.start()
ssc.awaitTermination()
Then, run spark-commit:
spark-submit --jars /usr/local/services/metrics-spark-analyser/external/spark-streaming-kafka-0-8-assembly_2.11-2.0.2.jar spark_word_counter_consumer.py
I got following spark error msg:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcc in position 1: invalid continuation byte
more detail error code bellow:
16/12/18 13:58:30 ERROR Executor: Exception in task 5.0 in stage 7.0 (TID 30)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in main
process()
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/worker.py", line 167, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2371, in pipeline_func
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2371, in pipeline_func
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 1792, in combineLocally
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/shuffle.py", line 236, in mergeValues
for k, v in iterator:
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 73, in <lambda>
File "/usr/local/services/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py", line 36, in utf8_decoder
return s.decode('utf-8')
File "/usr/local/services/python/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcc in position 1: invalid continuation byte
It seems that spark-streaming can't not uncommpress snappy data from kafka.
Is any configuration should I add in spark?
Thanks~
software detail:
kafka 0.10.1.0
spark 2.0.2 per-build for hadoop 2.7
python-snappy 0.5
ps:
I have wrote a simple kafka consumer to read kafka snappy data, snappy uncompress process is success.