Unable to install PySpark on Google Colab - pyspark

Question

I am trying to install PySpark on Google Colab using the code given below but getting the following error.
tar: spark-2.3.2-bin-hadoop2.7.tgz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
This code has ran successfully once. But it is throwing this error after the notebook restart. I have even tried running this from a different Google account but same error again.
(Also is there any way that we don't need to install PySpark everytime after the notebook re-start?)
code:
--------------------------------------------------------------------------------------------------------------------------------
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q http://apache.osuosl.org/spark/spark-2.3.2/spark-2.3.2-bin-hadoop2.7.tgz
This following line seems to cause the problem as it is not finding the downloaded file.
!tar xvf spark-2.3.2-bin-hadoop2.7.tgz
I have also tried the following two lines (instead of above two lines) suggested somewhere on medium blog. But nothing better.
!wget -q http://mirror.its.dal.ca/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
!tar xvf spark-2.4.0-bin-hadoop2.7.tgz
!pip install -q findspark
-------------------------------------------------------------------------------------------------------------------------------
Any ideas how to get out of this error and install PySpark on Colab?

Answer 1

I am running pyspark on colab by just using
!pip install pyspark
and it works fine.

Answer 2

Date: 6-09-2020
Step 1 : Install pyspark on google colab
!pip install pyspark
Step 2 : Dealing with pandas and spark Dataframe inside spark session
!pip install pyarrow
It facilitates communication between many components, for example, reading a parquet file with Python (pandas) and transforming to a Spark data frame, Falcon Data Visualization or Cassandra without worrying about conversion.
Step 3 : Create Spark Session
from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').getOrCreate()
Done ⭐

Answer 3

you are getting this error because spark-2.3.2-bin-hadoop2.7 is replaced with latest version on official site and mirror sites.
Go to any of this path and get the latest version
http://apache.osuosl.org/spark/
https://www-us.apache.org/dist/spark/
replace spark build version and you are done.
every thing will work smoothly.
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://www-us.apache.org/dist/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz
!tar xf /content/spark-2.4.3-bin-hadoop2.7.tgz
!pip install -q findspark

Answer 4

I had tried to install in the same way but even after checking with proper versions of spark I was getting the same error.
Running below code worked for me!!
!pip install pyspark
!pip install pyarrow
!pip install -q findspark
from pyspark.sql import SparkSession
spark = SparkSession.builder.master('HelloWorld').getOrCreate()

Answer 5

I have used the below setup to run PySpark on Google Colab.
# Installing spark
!apt-get install openjdk-8-jre
!apt-get install scala
!pip install py4j
!wget -q https://downloads.apache.org/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz
!tar xf spark-2.4.8-bin-hadoop2.7.tgz
!pip install -q findspark
# Setting up environment variables
import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-2.4.8-bin-hadoop2.7"
# Importing and initating spark
import findspark
findspark.init()
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[*]").appName("Test Setup").getOrCreate()
sc = spark.sparkContext