On Mac OS X, I compiled Spark from the sources using the following command:
jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true SPARK_GANGLIA_LGPL=true xsbt
...
[info] Set current project to root (in build file:/Users/jacek/oss/spark/)
> ; clean ; assembly
...
[info] Packaging /Users/jacek/oss/spark/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar ...
[info] Done packaging.
[info] Done packaging.
[success] Total time: 1964 s, completed May 9, 2014 5:07:45 AM
When I started ./bin/spark-shell I noticed the following WARN message:
WARN NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
What might be the issue?
jacek:~/oss/spark
$ ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/05/09 21:11:17 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/05/09 21:11:17 INFO SecurityManager: Changing view acls to: jacek
14/05/09 21:11:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jacek)
14/05/09 21:11:17 INFO HttpServer: Starting HTTP Server
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.0.0-SNAPSHOT
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0)
Type in expressions to have them evaluated.
Type :help for more information.
...
14/05/09 21:11:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...
Supported Platforms of the Native Libraries Guide documentation in Apache Hadoop reads:
The native hadoop library is supported on *nix platforms only. The
library does not to work with Cygwin or the Mac OS X platform.
The native hadoop library is mainly used on the GNU/Linus platform and
has been tested on these distributions:
RHEL4/Fedora
Ubuntu
Gentoo
On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.
It appears that the WARN message should be disregarded on Mac OS X as the native library doesn't simply exist for the platform.
In my experience, if you cd into the /sparkDir/conf and rename the spark-env.sh.template to spark-env.sh, and then set the JAVA_OPTSand hadoop_DIR, it works.
You will also have to edit this /etc/profile line:
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native/:$LD_LIBRARY_PATH
Related
Downloaded apache 3.2.0 the latest one as well as the hadoop file
java Java SE Development Kit 17.0.1 is installed too
i am not even able to initialize
input :
import pyspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.sql('''select 'spark' as hello ''')
df.show()
Output#
Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.storage.StorageUtils$
at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:110)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277)
As you can read at https://spark.apache.org/docs/3.2.0/:
Spark 3.2.0 only supports Java version 8-11. I had the same issue on Linux and switching to Java 11 instead of 17 helped in my case.
BTW Spark 3.3.0 supports Java 17.
I faced the same issue today, but fixed it by changing JDK from 17 to 8 (only for spark start) as below.
spark-3.2.1
hadoop3.2
python 3.10
File "D:\sw.1\spark-3.2.1-bin-hadoop3.2\python\lib\py4j-0.10.9.3-src.zip\py4j\protocol.py", line 326, in get_return_value
raise Py4JJavaError(py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.: > java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.storage.StorageUtils$
Env variable was having %JAVA_HOME% to jdk17
Quick fix (incase you want to keep env. variable same but use jdk8 for spark only):
(1) create a batch file (start-pyspark.bat) in d:
(2) add below lines:
set JAVA_HOME=D:\sw.1\jdk1.8.0_332
set PATH=%PATH%;%JAVA_HOME%\bin;%SPARK_HOME%\bin;%HADOOP_HOME%\bin;
pyspark
(3) on cmd, type <start-pyspark.bat> and enter.
d:\>start-pyspark.bat
d:\>set JAVA_HOME=D:\sw.1\jdk1.8.0_332
d:\>set PATH=D:\sw.1\py.3.10\Scripts\;D:\sw.1\py.3.10\;C:\Program Files\Zulu\zulu-17\bin;C:\Program Files\Zulu\zulu-17-jre\bin;C:\windows\system32;....;D:\sw.1\jdk1.8.0_332\bin;D:\sw.1\spark-3.2.1-bin-hadoop3.2\bin;D:\sw.1\hadoop\bin;
d:\>pyspark
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/05/27 18:29:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.2.1
(4) If you close this spark prompt and cmd and restart, it will be in clean state as having JDK-17 set as JAVA_HOME from env.
I'm running spark 2.4.5 in my mac. When I execute spark-submit --version
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.5
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_242
Branch HEAD
Compiled by user centos on 2020-02-02T19:38:06Z
Revision cee4ecbb16917fa85f02c635925e2687400aa56b
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
Note it's using scala version 2.11.12. However, my app is using 2.12.8 and this is throwing me the well known java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V error.
My question is how to make my spark 2.4.5 use scala 2.12 as indicated in their official webiste under Download section: Spark 2.4.5 uses Scala 2.12
I tried brew search apache-spark and got
==> Formulae
apache-spark ✔
and brew info apache-spark returned me
apache-spark: stable 2.4.5, HEAD
Engine for large-scale data processing
https://spark.apache.org/
/usr/local/Cellar/apache-spark/2.4.4 (1,188 files, 250.7MB) *
Built from source on 2020-02-03 at 14:57:17
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-spark.rb
==> Dependencies
Required: openjdk ✔
==> Options
--HEAD
Install HEAD version
==> Analytics
install: 7,150 (30 days), 15,180 (90 days), 64,459 (365 days)
install-on-request: 6,900 (30 days), 14,807 (90 days), 62,407 (365 days)
build-error: 0 (30 days)
Appreciate if any advice is given!
Spark community provides older versions of spark in this website, you can choose any version according your OS, For windows you can use tgz extension file.
https://archive.apache.org/dist/spark/
You can build any custom version of Spark locally.
Clone https://github.com/apache/spark locally
Update pom file, focusing on scala.version, hadoop.version, scala.binary.version, and artifactId in https://github.com/apache/spark/blob/master/pom.xml
mvn -DskipTests clean package (from their README)
After successful build, find all jars in assembly/target/scala-2.11/jars, external/../target, and other external jars you desire, which may be in provided scope of your jars submitted.
Create a new directory and export SPARK_HOME="/path/to/directory_name" so that https://github.com/apache/spark/blob/master/bin/spark-submit will detect it (see the source to see why)
Copy the jars into $SPARK_HOME/jars and make sure there are no conflicting jars
The bin/ scripts should be the same, but if needed, specifically reference those and possibly even unlink the brew ones if you no longer need them
This is a cmd log that I see after running spark-shell command (C:\Spark>spark-shell). As I understand, it's mainly an issue with Hadoop. I use Windows 10. Can somehow please with the below issue?
C:\Users\mac>cd c:\
c:\>winutils\bin\winutils.exe chmod 777 \tmp\hive
c:\>cd c:\spark
c:\Spark>spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/05/14 13:21:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/14 13:21:34 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-rdbms-3.2.9.jar."
17/05/14 13:21:34 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-core-3.2.10.jar."
17/05/14 13:21:34 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-api-jdo-3.2.6.jar."
17/05/14 13:21:48 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.1.9:4040
Spark context available as 'sc' (master = local[*], app id = local-1494764489031).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.1
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
There's no issue in your output. These WARN messages can simply be ignored.
In other words, it looks like you've installed Spark 2.1.1 on Windows 10 properly.
To make sure you installed it properly (so I could remove looks from above sentence) is to do the following:
spark.range(1).show
That by default will trigger loading Hive classes that may or may not end up with exceptions on Windows due to Hadoop's requirements (and hence the need for winutils.exe to work them around).
I am trying to integrate HBase with Spark. I did two types of integration but getting error. First I copied all the HBase lib jars and paste into to Spark jars folder. Some HBase jars conflicted with Spark jars so I kept the spark jars in that case. Also added export SPARK_CLASSPATH=/usr/local/spark/spark-2.1.0/jars/* to spark-env.sh and export HADOOP_USER_CLASSPATH_FIRST=true to bashrc file but got the following IncompatibleClassChangeError error while starting spark shell:
hduser#master:/usr/local/spark/spark-2.1.0/bin$ ./spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
[ERROR] Terminal initialization failed; falling back to unsupported
java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
at jline.TerminalFactory.create(TerminalFactory.java:101)
at jline.TerminalFactory.get(TerminalFactory.java:158)
at jline.console.ConsoleReader.<init>(ConsoleReader.java:229)
at jline.console.ConsoleReader.<init>(ConsoleReader.java:221)
at jline.console.ConsoleReader.<init>(ConsoleReader.java:209)
at scala.tools.nsc.interpreter.jline.JLineConsoleReader.<init>(JLineReader.scala:62)
at scala.tools.nsc.interpreter.jline.InteractiveReader.<init>(JLineReader.scala:34)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at scala.tools.nsc.interpreter.ILoop$$anonfun$scala$tools$nsc$interpreter$ILoop$$instantiater$1$1.apply(ILoop.scala:858)
at scala.tools.nsc.interpreter.ILoop$$anonfun$scala$tools$nsc$interpreter$ILoop$$instantiater$1$1.apply(ILoop.scala:855)
at scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$mkReader$1(ILoop.scala:862)
at scala.tools.nsc.interpreter.ILoop$$anonfun$21$$anonfun$apply$9.apply(ILoop.scala:873)
at scala.tools.nsc.interpreter.ILoop$$anonfun$21$$anonfun$apply$9.apply(ILoop.scala:873)
at scala.util.Try$.apply(Try.scala:192)
at scala.tools.nsc.interpreter.ILoop$$anonfun$21.apply(ILoop.scala:873)
at scala.tools.nsc.interpreter.ILoop$$anonfun$21.apply(ILoop.scala:873)
at scala.collection.immutable.Stream.map(Stream.scala:418)
at scala.tools.nsc.interpreter.ILoop.chooseReader(ILoop.scala:873)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1$$anonfun$apply$mcZ$sp$2.apply(ILoop.scala:914)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:914)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
at org.apache.spark.repl.Main$.doMain(Main.scala:68)
at org.apache.spark.repl.Main$.main(Main.scala:51)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/29 13:02:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/29 13:02:27 WARN spark.SparkConf:
SPARK_CLASSPATH was detected (set to '/usr/local/spark/spark-2.1.0/jars/*').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
17/01/29 13:02:27 WARN spark.SparkConf: Setting 'spark.executor.extraClassPath' to '/usr/local/spark/spark-2.1.0/jars/*' as a work-around.
17/01/29 13:02:27 WARN spark.SparkConf: Setting 'spark.driver.extraClassPath' to '/usr/local/spark/spark-2.1.0/jars/*' as a work-around.
17/01/29 13:02:36 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://10.0.0.1:4040
Spark context available as 'sc' (master = local[*], app id = local-1485720148707).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
The second try was following (both hbase and spark was in initial condition, no copy and paste jars):
hduser#master:~$ HBASE_PATH='/usr/local/hbase/hbase-1.2.4/bin/hbase classpath'
hduser#master:~$ cd /usr/local/spark/spark-2.1.0/bin
hduser#master:/usr/local/spark/spark-2.1.0/bin$ ./spark-shell --driver-class-path $HBASE_PATH
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/01/29 16:40:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/01/29 16:41:03 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://10.0.0.1:4040
Spark context available as 'sc' (master = local[*], app id = local-1485733255875).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.1.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.hadoop.hbase.mapreduce.TableInputFormat
<console>:23: error: object hbase is not a member of package org.apache.hadoop
import org.apache.hadoop.hbase.mapreduce.TableInputFormat
^
scala>
This time I got object hbase is not a member of package org.apache.hadoop.
Please help me to integrate HBase 1.2.4 with Spark 2.1.0.
I get below screen when I run Scala (Using Scala version 2.10.5). I am using it along with Spark version 1.6.1. The book that I am learning from has a screenshot and it doesn't show so many warnings. Why am I getting those warning. Do I need to resolve them for a smooth execution?
c:\spark-1.6.1-bin-hadoop2.6\spark-1.6.1-bin-hadoop2.6>bin\spark-shell
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
16/03/16 08:12:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/03/16 08:13:09 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/spark-1.6.1-bin-hado
p2.6/spark-1.6.1-bin-hadoop2.6/bin/../lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2
6/lib/datanucleus-rdbms-3.2.9.jar."
16/03/16 08:13:09 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark-1.6.1-bin-hadoop2.
/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/c:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2.6/bin/..
lib/datanucleus-api-jdo-3.2.6.jar."
16/03/16 08:13:09 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark-1.6.1-bin-hadoop2.6/spark-
.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/c:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2.6/bin/../lib/datan
cleus-core-3.2.10.jar."
16/03/16 08:13:09 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/16 08:13:09 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/16 08:13:14 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/03/16 08:13:14 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/03/16 08:13:15 WARN : Your hostname, njog-MOBL1 resolves to a loopback/non-reachable address: 192.168.56.1, but we couldn't find any external IP address!
16/03/16 08:13:22 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/spark-1.6.1-bin-hado
p2.6/spark-1.6.1-bin-hadoop2.6/bin/../lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2
6/lib/datanucleus-rdbms-3.2.9.jar."
16/03/16 08:13:22 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark-1.6.1-bin-hadoop2.
/spark-1.6.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/c:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2.6/bin/..
lib/datanucleus-api-jdo-3.2.6.jar."
16/03/16 08:13:22 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/C:/spark-1.6.1-bin-hadoop2.6/spark-
.6.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/c:/spark-1.6.1-bin-hadoop2.6/spark-1.6.1-bin-hadoop2.6/bin/../lib/datan
cleus-core-3.2.10.jar."
16/03/16 08:13:22 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/16 08:13:22 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
SQL context available as sqlContext.