Not able to find spark-warehouse directory - scala

Question

I am a beginner in Spark Programming.
I was practicing the below program:
package practice.spark.examples
import org.apache.log4j.Logger
import org.apache.spark.sql.{SaveMode, SparkSession}
object SparkSQLTableDemo extends Serializable {
#transient lazy val logger: Logger = Logger.getLogger(getClass.getName)
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder()
.appName("SparkSQLTableDemo")
.master("local[3]")
.enableHiveSupport() // to allow the connectivity to a persistent Hive Metastore.
.getOrCreate()
val flightTimeParquetDF = spark.read
.format("parquet")
.option("path", "data/flight*.parquet")
.load()
spark.sql("CREATE DATABASE IF NOT EXISTS AIRLINE_DB")
spark.catalog.setCurrentDatabase("AIRLINE_DB")
flightTimeParquetDF.write
.mode(SaveMode.Overwrite)
.partitionBy("ORIGIN", "OP_CARRIER")
.saveAsTable("flight_data_tbl")
spark.catalog.listTables("AIRLINE_DB").show()
//spark.sql("Select * from flight_data_tbl limit 5").show()
logger.info("Finished.")
spark.stop()
}
}
After running this code we must get 2 directories created right?
1:metastore_db 2:spark-warehouse
But in the current directory, only metastore_db was created.
My spark-warehouse folder was created in the Desktop/ location.
Attaching the pics below:
enter image description here
enter image description here
I tried googling about how to set the spark-warehouse directory but was very confused.
Also every time I run the code I get the below warnings. I thought this has something to do with the error.
22/02/12 21:30:22 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
22/02/12 21:30:22 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
22/02/12 21:30:31 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped
22/02/12 21:30:33 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
22/02/12 21:30:33 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore UNKNOWN#192.168.56.1
22/02/12 21:30:33 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
22/02/12 21:31:02 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
22/02/12 21:31:02 WARN HiveConf: HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
22/02/12 21:31:02 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
22/02/12 21:31:02 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
Can someone please guide me in any way such that I can set the warehouse path to my current practice directory i.e., under metastore_db as shown in Image1 ?
Thank You in Advance :)

Answer 1

You can use below example to specify metastore location.
import java.io.File
import org.apache.spark.sql.{Row, SaveMode, SparkSession}
case class Record(key: Int, value: String)
// warehouseLocation points to the default location for managed databases and tables
val warehouseLocation = new File("spark-warehouse").getAbsolutePath
val spark = SparkSession
.builder()
.appName("Spark Hive Example")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
Details please refer https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html