Creating hive with multiple parquets - scala

Question

I have parquet folders with their name as "yearquarter" starting from (2007q1 - 2020q3). The hive tables I am creating should pull data from only 2014q1 through 2020q2. How do i achieve this?

Answer 1

You'll have to change the parquet folder names and add a prefix to them like yearquarter=2001q1 (for example) which indicates what column stores these values, so it sits in a hierarchy under a top level folder (named table_name below).
table_name
|
- yearquarter=2001q1
- yearquarter=2001q2
.
.
- yearquarter=2020q3
Hive-based solution:
You would then create an external hive table which is located at the top level folder. You choose external so you can set the location. The table schema should correspond to the column labels in the files.
CREATE EXTERNAL TABLE TABLE_NAME (
col_name1 HIVE_TYPE,
...,
col_nameN HIVE_TYPE)
PARTITIONED BY (yearquarter STRING)
STORED AS PARQUET
LOCATION '/location/to/your/table_name';
After you have a hive table on your folder hierarchy, partitioned by the folders, you create a hive view which uses a WHERE clause to SELECT a subset.
CREATE VIEW view_name
AS SELECT *
FROM table_name
WHERE yearquarter >= "2014q1" AND yearquarter <= "2020q2";
Performing a SELECT from this view will then provide the required range.
Spark-based solution:
You create a DataFrame which reads the top level location. Because you stored the hierarchy like yearquarter=2001q1, these values are automatically read into a column labeled yearquarter.
// Read parquet hierarchy. The schema (if present) is automatically detected.
val df = spark.read.parquet("/location/to/your/table_name")
// Set filter condition to use.
val filterCondition = col("yearquarter") >= "2014q1" && col("yearquarter") <= "2020q2"
// Filter according to condition.
val filtered = df.filter(filterCondition)