run pyspark locally with pycharm - pyspark

Question

I wrote the following very simple python script with my Pycharm IDE
from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql import SQLContext
from pyspark.sql.types import LongType, FloatType,IntegerType,StringType,DoubleType
from pyspark.sql.functions import udf
from pyspark.sql import functions as F
from pyspark.sql.window import Window
from pyspark.sql.functions import abs
from pyspark.sql import HiveContext
spark = SparkSession.builder.config("requiredconfig").appName("SparkSran").enableHiveSupport().getOrCreate()
sc = spark.sparkContext
when I click on run on my IDE or run the following command
spark-submit --py-files /home/user/PycharmProjects/helloSparkWorld/test.py
I get
/usr/lib/spark/bin/spark-class: line 71: /usr/local/java/jdk10.0.1/bin/java: No such file or directory
my JAVA_HOME and SPARK_HOME are set as follows
echo $SPARK_HOME gives /usr/lib/spark
and
echo $JAVA_HOME gives
/usr/local/java/jdk10.0.1

Answer 1

You can just do a pip install pyspark in your environment that you are using with your pycharm installation to run python programs. You can run your pyspark .py files as python filename.py itself, if you're running it locally.
Basically just provide your pip or python interpreter with the pyspark pip package and you'll be able to run it via pycharm using the same interpreter.