Cannot connect to Hive metastore from Spark application - scala

Question

I am trying to connect to Hive-metastore from the Spark application but each time it gets stuck on trying to connect and crash with a timeout:
INFO metastore:376 - Trying to connect to metastore with URI thrift://hive-metastore:9083
WARN metastore:444 - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
The application crashes on the line where I create an external Hive table
I run Hive-metastore as well as Spark application (using Spark K8s operator) in Kubernetes cluster. I checked the accessibility of Hive-metastore service outside of the cluster using telnet (node ip: service node port) and curled the service inside of the cluster, the service seems to be assessable. What could be the reason of this error?
This is the configuration of Hive-metastore uri in Spark application
val sparkSession = SparkSession
.builder()
.config(sparkConf)
.config("hive.metastore.uris", "thrift://hive-metastore:9083")
.config("hive.exec.dynamic.partition", "true")
.config("hive.exec.dynamic.partition.mode", "nonstrict")
.enableHiveSupport()
.getOrCreate()
Hive-metastore yaml configuration looks like this:
apiVersion: v1
kind: Service
metadata:
name: hive-metastore-np
spec:
selector:
app: hive-metastore
ports:
- protocol: TCP
targetPort: 9083
port: 9083
nodePort: 32083
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hive-metastore
spec:
replicas: 1
selector:
matchLabels:
app: hive-metastore
template:
metadata:
labels:
app: hive-metastore
spec:
containers:
- name: hive-metastore
image: mozdata/docker-hive-metastore:1.2.1
imagePullPolicy: Always
env:
- name: DB_URI
value: postgresql
- name: DB_USER
value: hive
- name: DB_PASSWORD
value: hive-password
- name: CORE_CONF_fs_defaultFS
value: hdfs://hdfs-namenode:8020
ports:
- containerPort: 9083
UPDATE: When I try to curl hive-metastore:9083, the service is accessible but it returns an empty response which means there might be a problem with hive-metastore K8s definition
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: hive-metastore:9083
> Accept: */*

Answer 1

This error occurs when there's a discrepancy between versions of hive jars in your cluster and the hive jars that Spark uses (which is usually consistent with the Spark version you're using). You need to determine the version of hive jars used in the cluster and add these jars into your Spark image. You can then make your SparkSession use those compatible hive jars by adding the following configurations to your SparkSession:
.conf("spark.sql.hive.metastore.version", "<your hive metastore version>")
.conf("spark.sql.hive.metastore.version", "<your hive version>")
.conf("spark.sql.hive.metastore.jars", "<uri of all the correct hive jars>")