How to refer to data in Spark SQL using column names? - scala

Question

If I have a file named shades.txt with the following data:
1 | 1 | dark red
2 | 1 | light red
3 | 2 | dark green
4 | 3 | light blue
5 | 3 | sky blue
I can store it in SparkSQL like below:
var shades_part = sc.textFile("shades.txt")
var shades_data = shades_part.map(line => line.split("\\|").map(elem => elem.trim))
shades_data.toDF().createOrReplaceTempView("shades_data")
But when I try to run a query on it
sqlContext.sql("select * from shades_data A where A.shade_name = "dark green")
I get an error:
cannot resolve '`A.shade_name`' given input columns: [A.value];

Answer 1

You can use case class for your code as
case class Shades(id: Int, colorCode:Int, shadeColor: String)
Then modify your code as
var shades_part = sc.textFile("shades.txt")
var shades_data = shades_part.map(line => line.split("\\|")).map(elem => Shades(elem(0).trim.toInt, elem(1).trim.toInt, elem(2).trim))
val df = shades_data.toDF()
You should have dataframe as
+---+---------+----------+
|id |colorCode|shadeColor|
+---+---------+----------+
|1 |1 |dark red |
|2 |1 |light red |
|3 |2 |dark green|
|4 |3 |light blue|
|5 |3 |sky blue |
+---+---------+----------+
Now you can use use filter function as
df.filter($"shadeColor" === "dark green").show(false)
which should give you
+---+---------+----------+
|id |colorCode|shadeColor|
+---+---------+----------+
|3 |2 |dark green|
+---+---------+----------+
Using Schema
You can create schema as
val schema = StructType(Array(StructField("id", IntegerType, true), StructField("colorCode", IntegerType, true), StructField("shadeColor", StringType, true)))
and use the schema in sqlContext as
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("delimiter", "|")
.schema(schema)
.option("ignoreTrailingWhiteSpace", true)
.option("ignoreLeadingWhiteSpace", true)
.load("shades.txt")
Or you can use createDataFrame function as
var shades_part = sc.textFile("shades.txt")
var shades_data = shades_part.map(line => line.split("\\|").map(_.trim)).map(elem => Row.fromSeq(Seq(elem(0).toInt, elem(1).toInt, elem(2))))
sqlContext.createDataFrame(shades_data, schema).show(false)

Answer 2

check the schema of the dataframe, you will know the column names
val dataframe = shades_data.toDF()
dataframe.printSchema()
If there are no column names defined while creating a dataframe then by default it will use _c0, _c1 ... as column names.
or else you can give column names while creating dataframes like below
val dataframe = shades_data.toDF("col1", "col2", "col3")
dataframe.printSchema()
root
|-- col1: integer (nullable = false)
|-- col2: integer (nullable = false)
|-- col3: string (nullable = true)