Cant find field in Spark dataframe - scala

Question

I am currently working on a scala/spark homework project ibn which I am to read-in a csv file containing a few thousand movie reviews as a dataframe. I am then to analyze these reviews and train a model to detect whether a review is positive or negative. I will be training these models using TF-IDF and Word2Vec. The issue I am having is that the code I have written so far does not find the specified header field named "word" which is output by a regex tokenizer. My code is written below, as well as the console output.
I thank you for your help and appreciate any pointers to how I might do this correctly/better than what I am doing now.
import org.apache.spark._
//import spark.implicits._
import org.apache.spark.sql._
import org.apache.spark.sql.SparkSession
import java.io._
import scala.io._
import scala.collection.mutable.ListBuffer
import org.apache.spark.{Partition, SparkContext, TaskContext}
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType}
import org.apache.spark.sql.Row
import org.apache.spark.sql.functions.rand
import org.apache.spark.rdd.RDD
import org.apache.spark.ml.feature.StopWordsRemover
import org.apache.spark.ml.feature.{RegexTokenizer, Tokenizer}
/*
DONE: step 1: loop through all the positive / negative reviews and label each (1 = Positive, 2 = Negative)
during the loop, place the text that is read as a string into a DF.
step 2: check which words are common among different labels and text with each other (possibly remove stop words)
this will satisfy the TF-IDF requirement
step 3: convert text into vectors and perform regression on the values
step 4: compare accuracy using the actual data (data for the above was using the test folder data)
*/
object Machine {
def main(args: Array[String]) {
val spark = SparkSession.builder.appName("Movie Review Manager").getOrCreate()
println("Reading data...")
val df = spark.read.format("csv").option("header", "true").load("movie_data.csv")
val regexTokenizer = new RegexTokenizer().setInputCol("review").setOutputCol("word").setPattern("\\s")
val remover = new StopWordsRemover().setInputCol("word").setOutputCol("feature")
df.show()
regexTokenizer.transform(df).show(false)
df.collect()
remover.transform(df).show(false)
df.show()
spark.stop()
}
}
And here is the console output:
Exception in thread "main" 2018-03-13 03:41:28 INFO ContextCleaner:54 - Cleaned accumulator 125
java.lang.IllegalArgumentException: Field "word" does not exist.2018-03-13 03:41:28 INFO ContextCleaner:54 - Cleaned accumulator 118
2018-03-13 03:41:28 INFO ContextCleaner:54 - Cleaned accumulator 116
at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267)2018-03-13 03:41:28 INFO ContextCleaner:54 - Cleaned accumulator 102
2018-03-13 03:41:28 INFO ContextCleaner:54 - Cleaned accumulator 110
at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267)2018-03-13 03:41:28 INFO ContextCleaner:54 - Cleaned accumulator 103
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:59)
at org.apache.spark.sql.types.StructType.apply(StructType.scala:266)
at org.apache.spark.ml.feature.StopWordsRemover.transformSchema(StopWordsRemover.scala:111)
at org.apache.spark.ml.feature.StopWordsRemover.transform(StopWordsRemover.scala:91)
at Machine$.main(movieProgram.scala:44)
at Machine.main(movieProgram.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2018-03-13 03:41:28 INFO SparkContext:54 - Invoking stop() from shutdown hook
2018-03-13 03:41:28 INFO AbstractConnector:318 - Stopped Spark#3abfc4ed{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

Answer 1

You are missing to store the transformation of new RegexTokenizer().setInputCol("review").setOutputCol("word").setPattern("\\s") to be used for new StopWordsRemover().setInputCol("word").setOutputCol("feature")
As you missed to save the regex tokenizer algorithm applied dataframe to be used for stop word remover algorithm and used the original df dataframe (where word column is not present) you had the error stating
java.lang.IllegalArgumentException: Field "word" does not exist...
Correct way is to do as below
object Machine {
def main(args: Array[String]) {
val spark = SparkSession.builder.appName("Movie Review Manager").getOrCreate()
println("Reading data...")
val df = spark.read.format("csv").option("header", "true").load("movie_data.csv")
val regexTokenizer = new RegexTokenizer().setInputCol("review").setOutputCol("word").setPattern("\\s")
val remover = new StopWordsRemover().setInputCol("word").setOutputCol("feature")
df.show(false) //original dataframe
val tokenized = regexTokenizer.transform(df)
tokenized.show(false) //tokenized dataframe
val removed = remover.transform(tokenized)
removed.show(false) //stopwords removed dataframe
spark.stop()
}
}
I hope the answer is helpful