ApacheSpark: Unsupported parquet datatype - scala

Question

I'm trying to read Hive table with SparkSql HiveContext. But, when I submit the job, I get the following error:
Exception in thread "main" java.lang.RuntimeException: Unsupported parquet datatype optional fixed_len_byte_array(11) amount (DECIMAL(24,7))
at scala.sys.package$.error(package.scala:27)
at org.apache.spark.sql.parquet.ParquetTypesConverter$.toPrimitiveDataType(ParquetTypes.scala:77)
at org.apache.spark.sql.parquet.ParquetTypesConverter$.toDataType(ParquetTypes.scala:131)
at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:383)
at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:380)
Column type is DECIMAL(24,7). I've changed column type with HiveQL, but it doesn't work. Also I've tried cast to another Decimal type in sparksql like below:
val results = hiveContext.sql("SELECT cast(amount as DECIMAL(18,7)), number FROM dmp_wr.test")
But, I got same error. My code is like that:
def main(args: Array[String]) {
val conf: SparkConf = new SparkConf().setAppName("TColumnModify")
val sc: SparkContext = new SparkContext(conf)
val vectorAcc = sc.accumulator(new MyVector())(VectorAccumulator)
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
val results = hiveContext.sql("SELECT amount, number FROM dmp_wr.test")
How can i solve this problem? Thank you for your response.
Edit1: I found the Spark source line which thrown exception. It looks like that
if(originalType == ParquetOriginalType.DECIMAL && decimalInfo.getPrecision <= 18)
So, I created new table which has column in DECIMAL(18,7) type and my code works as I expected.
I drop table and create new one which has column in DECIMAL(24,7), after that I changed column type
alter table qwe change amount amount decimal(18,7) and I can see It is changed to DECIMAL(18,7), but Spark
doesn't accept change. It still read column type as DECIMAL(24,7) and give same error.
What can be the main reason?

Answer 1

alter table qwe change amount amount decimal(18,7)
Alter table commands in Hive does not touch the actual data that is stored in Hive. It only changes the metadata in Hive Metastore. This is very different from "alter table" commands in normal databases (like MySQL).
When Spark reads data from Parquet files, it will try to use the metadata in the actual Parquet file to deserialize the data, which will still give it DECIMAL(24, 7).
There are 2 solutions to your problem:
1. Try out a new version of Spark - build from trunk. See https://issues.apache.org/jira/browse/SPARK-6777 which totally changes this part of the code (will only be in Spark 1.5 though), so hopefully you won't see the same problem again.
Convert the data in your table manually. You can use hive query like "INSERT OVERWRITE TABLE new_table SELECT * from old_table") to do it.