Unable to hit breakpoints while debugging Spark with IntelliJ CE - scala

Question

I am running a Spark JAR file using command spark-submit testmysparkfile.jar, after having setup export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005;
The code is written in Scala.
Below is the Spark Session i am creating-
val spark = SparkSession
.builder()
.appName("testmysparkfile")
.config("spark.serializer",classOf[KryoSerializer].getName).master("local[*]")
.getOrCreate()
When I run the JAR, the application is started and it listens to port 5005. Now, when I go back to my IntelliJ, and try to run 'debug', it runs the debugger fine, and the sample output is recieved on the terminal window that was listening to port 5005, however, the breakpoint I have set, is not hit.
Debugger settings-
Debugger mode:Attach to remote JVM
Host:Localhost
Port:5005
command line args for JVM:-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
Use module classpath:
Output from IntelliJ terminal-
Connected to the target VM, address: 'localhost:5005', transport: 'socket'
Disconnected from the target VM, address: 'localhost:5005', transport: 'socket'
I have followed examples that show how to debug Spark using a debugger like this one-http://www.bigendiandata.com/2016-08-26-How-to-debug-remote-spark-jobs-with-IntelliJ/
However, this does not seem to work as I am unable to hit any breakpoints.