How to read redis map in spark using spark-redis - scala

Question

I have a normal scala map in Redis (key and value). Now I want to read that map in one of my spark-streaming program and use this as a broadcast variable so that my slaves can use that map to resolve key mapping. I am using spark-redis 2.3.1 library, but now sure how to read that.
Map in redis table "employee" -
name | value
------------------
123 David
124 John
125 Alex
This is how I am trying to read in spark (Not sure if this is correct- please correct me) --
val loadedDf = spark.read
.format("org.apache.spark.sql.redis")
.schema(
StructType(Array(
StructField("name", IntegerType),
StructField("value", StringType)
)
))
.option("table", "employee")
.option("key.column", "name")
.load()
loadedDf.show()
The above code does not show anything, I get empty output.

Answer 1

You could use the below code to your task but you need to utilize Spark Dataset (case Dataframe to case class) to do this task. Below is a full example to read and write in Redis.
object DataFrameExample {
case class employee(name: String, value: Int)
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.appName("redis-df")
.master("local[*]")
.config("spark.redis.host", "localhost")
.config("spark.redis.port", "6379")
.getOrCreate()
val personSeq = Seq(employee("John", 30), employee("Peter", 45)
val df = spark.createDataFrame(personSeq)
df.write
.format("org.apache.spark.sql.redis")
.option("table", "person")
.mode(SaveMode.Overwrite)
.save()
val loadedDf = spark.read
.format("org.apache.spark.sql.redis")
.option("table", "person")
.load()
loadedDf.printSchema()
loadedDf.show()
}
}
Output is below
root
|-- name: string (nullable = true)
|-- value: integer (nullable = false)
+-----+-----+
| name|value|
+-----+-----+
| John| 30 |
|Peter| 45 |
+-----+-----+
You could also check more details in Redis documentation