Creating HiveContext on windows throwing error - scala

Question

I am trying to create HiveContext, but its throwing error.
Is it just because of i dont have winutils.exe? or how can i solve this issue?
The reason i want to create HiveContext is i am planning to use collect_set functions which is created as UDF in Hive.
Error log:
16/11/23 14:12:09 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
16/11/23 14:12:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:10 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:14 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
16/11/23 14:12:15 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/11/23 14:12:15 INFO ObjectStore: Initialized ObjectStore
16/11/23 14:12:15 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/11/23 14:12:16 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/11/23 14:12:16 WARN : Your hostname, INNR90GLH5G resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:1584%net6, but we couldn't find any external IP address!
16/11/23 14:12:17 INFO HiveMetaStore: Added admin role in metastore
16/11/23 14:12:17 INFO HiveMetaStore: Added public role in metastore
16/11/23 14:12:17 INFO HiveMetaStore: No user is added in admin role, since config is empty
16/11/23 14:12:18 INFO HiveMetaStore: 0: get_all_databases
16/11/23 14:12:18 INFO audit: ugi=1554161 ip=unknown-ip-addr cmd=get_all_databases
16/11/23 14:12:18 INFO HiveMetaStore: 0: get_functions: db=default pat=*
16/11/23 14:12:18 INFO audit: ugi=1554161 ip=unknown-ip-addr cmd=get_functions: db=default pat=*
16/11/23 14:12:18 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
Exception in thread "main" java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208)
at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462)
at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461)
at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
at com.scb.cnc.payments.CockpitChannelStatus$.main(CockpitChannelStatus.scala:13)
at com.scb.cnc.payments.CockpitChannelStatus.main(CockpitChannelStatus.scala)
Caused by: java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:482)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:791)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:774)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1097)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:572)
at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:547)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:599)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 12 more

Answer 1

If issue is lack of winutils.exe despite of hadoop and spark on your workstation then try this article in order to solve your problem
solution for spark