HiveContext is not reading schema of an Orcfile - scala

Question

When I run the following:
val df1 = sqlContext.read.format("orc").load(myPath)
df1.columns.map(m => println(m))
The columns are printed as '_col0', '_col1', '_col2' etc. As opposed to their real names such as 'empno', 'name', 'deptno'.
When I 'describe mytable' in Hive it prints the column name correctly, but when I run 'orcfiledump' it shows _col0, _col1, _col2 as well. Do I have to specify 'schema on read' or something? If yes, how do I do that in Spark/Scala?
hive --orcfiledump /apps/hive/warehouse/mydb.db/mytable1
.....
fieldNames: "_col0"
fieldNames: "_col1"
fieldNames: "_col2"
Note: I created the table as follows:
create table mydb.mytable1 (empno int, name VARCHAR(20), deptno int) stored as orc;
Note: This is not a duplicate of this issue (Hadoop ORC file - How it works - How to fetch metadata) because the answer tells me to use 'Hive' & I am already using HiveContext as follows:
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
By the way, I am using my own hive-site.xml, which contains following:
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://sandbox.hortonworks.com:9083</value>
</property>
</configuration>

Answer 1

I figured out what the problem was. It was the way I was creating the test data. I was under the impression that if I run the following commands:
create table mydb.mytable1 (empno int, name VARCHAR(20), deptno int) stored as orc;
INSERT INTO mydb.mytable1(empno, name, deptno) VALUES (1, 'EMP1', 100);
INSERT INTO mydb.mytable1(empno, name, deptno) VALUES (2, 'EMP2', 50);
INSERT INTO mydb.mytable1(empno, name, deptno) VALUES (3, 'EMP3', 200);
Data would be created in the ORC format at: /apps/hive/warehouse/mydb.db/mytable1
Turns out that's not the case. Even though I indicated 'stored as orc' the INSERT statements didn't save the column information. Not sure if that's expected behavior. In any case, it all works now. Apologies for the confusion but hopefully this will help someone in future -:)

Answer 2

#DilTeam
This is the problem, when you are writing data with Hive(version 1.x), it does not store columns' metadata for orc formatted files (it's not same for parquet etc.) , this issue is fixed in new Hive(2.x) to store column info in metadata that allow spark to read metadata from file itself.
Here is another option to load tables written with Hive1 in spark:
val table = spark.table(<db.tablename>)
Here spark is default sparkSession which fetches table's information from hive metastore.
One more option comes with more codeblock and perquisite information:
Create dataframe with defined schema over fetched RDD, this will give u flexibility to change data types,you can read in this link
https://spark.apache.org/docs/2.2.0/sql-programming-guide.html#programmatically-specifying-the-schema
I hope this will help