I have the following code that constructs a VectorAssembler:
val allColsExceptOceanProximity: Array[String] = dfRaw.drop("ocean_proximity").columns
val assembler = new VectorAssembler()
//.setInputCols(Array("longitude", "latitude", "housing_median_age", "total_rooms", "population", "households", "population", "median_income", "median_house_value"))
.setInputCols(allColsExceptOceanProximity)
.setOutputCol("features")
If I use the Array with the columns statically types, it works. When I pass in the Array dynamically, it kind of fails with the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 1763.0 failed 1 times, most recent failure: Lost task 2.0 in stage 1763.0 (TID 100292) (192.168.0.35 executor driver): org.apache.spark.SparkException: Failed to execute user defined function(VectorAssembler$$Lambda$4540/1415205968: (struct<longitude:double,latitude:double,housing_median_age:double,total_rooms:double,total_bedrooms:double,population:double,households:double,median_income:double,median_house_value:double>) => struct<type:tinyint,size:int,indices:array<int>,values:array<double>>)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.agg_doAggregateWithKeysOutput_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:755)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
at scala.collection.Iterator.foreach(Iterator.scala:941)
Why is this? Any ideas. Is is expecting a Varargs type? I'm on my notebook and do not have the benefits of the IDE.
EDIT: Added the schema:
root
|-- longitude: double (nullable = true)
|-- latitude: double (nullable = true)
|-- housing_median_age: double (nullable = true)
|-- total_rooms: double (nullable = true)
|-- total_bedrooms: double (nullable = true)
|-- population: double (nullable = true)
|-- households: double (nullable = true)
|-- median_income: double (nullable = true)
|-- median_house_value: double (nullable = true)
|-- ocean_proximity: string (nullable = true)
I am trying to use SVMWithSGD to train my model, but I encounter ClassCastException while trying to access my training.
My train_data dataframe schema looks like :
train_data.printSchema
root
|-- label: string (nullable = true)
|-- features: vector (nullable = true)
|-- label_index: double (nullable = false)
I created an LabeledPoint RDD to use it on SVNWithSGD
val targetInd = train_data.columns.indexOf("label_index")`
val featInd = Array("features").map(train_data.columns.indexOf(_))
val train_lp = train_data.rdd.map(r => LabeledPoint( r.getDouble(targetInd),
Vectors.dense(featInd.map(r.getDouble(_)).toArray)))
But When I call
SVMWithSGD.train(train_lp, numIterations)
it gives me :
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched
uler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche
duler.scala:1877)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche
duler.scala:1876)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:
59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.appl
y(DAGScheduler.scala:926)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.appl
y(DAGScheduler.scala:926)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.sc
ala:926)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGSche
duler.scala:2110)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGSchedu
ler.scala:2059)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGSchedu
ler.scala:2048)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1364)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
51)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
12)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.take(RDD.scala:1337)
at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1378)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
51)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:1
12)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
at org.apache.spark.rdd.RDD.first(RDD.scala:1377)
at org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.generateInitia
lWeights(GeneralizedLinearAlgorithm.scala:204)
at org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm.run(Generalize
dLinearAlgorithm.scala:234)
at org.apache.spark.mllib.classification.SVMWithSGD$.train(SVM.scala:217)
at org.apache.spark.mllib.classification.SVMWithSGD$.train(SVM.scala:255)
... 55 elided
Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to org.
apache.spark.mllib.linalg.Vector
My train_data was created based on label (file_name) and features (json file representing images features).
Try using this -
Schema
train_data.printSchema
root
|-- label: string (nullable = true)
|-- features: vector (nullable = true)
|-- label_index: double (nullable = false)
Modify your code as-
val train_lp = train_data.rdd.map(r => LabeledPoint(r.getAs("label_index"), r.getAs("features")))
I have a dataframe with Schema :
root
|-- QUERY: string (nullable = true)
|-- TYPE: string (nullable = true)
|-- DEVICE: string (nullable = true)
|-- PURCHASE_UNITS_SUM: double (nullable = true)
|-- CLICK_SUM: decimal(38,18) (nullable = true)
|-- IMPRESSION_COUNT: long (nullable = false)
|-- CLICK_THROUGH_RATE: decimal(38,2) (nullable = true)
|-- PURCHASE_RATE: double (nullable = true)
I am trying to convert some columns to map (device -> columns) :
val result = df.withColumn("CLICK_THROUGH_RATE_MAP",
map(col("DEVICE"), col("CLICK_THROUGH_RATE")))
.withColumn("PURCHASE_RATE_MAP",
map(col("DEVICE"), col("PURCHASE_RATE")))
.withColumn("PURCHASE_SUM_MAP",
map(col("DEVICE"), col("PURCHASE_UNITS_SUM")))
.withColumn("CLICK_SUM_MAP",
map(col("DEVICE"), col("CLICK_SUM")))
.withColumn("IMPRESSION_SUM_MAP",
map(col("DEVICE"), col("IMPRESSION_COUNT")))
.groupBy("QUERY", "TYPE")
.agg(collect_list("CLICK_THROUGH_RATE_MAP"),
collect_list("PURCHASE_RATE_MAP"),
collect_list("PURCHASE_SUM_MAP"),
collect_list("CLICK_SUM_MAP"),
collect_list("IMPRESSION_SUM_MAP"))
.as[(String, String,
Seq[Map[String, Double]],
Seq[Map[String, Double]],
Seq[Map[String, Double]],
Seq[Map[String, Double]],
Seq[Map[String, Double]])]
.map {
case (query, type, list1, list2, list3, list4, list5) =>
(query, type,
list1.reduce(_ ++ _),
list2.reduce(_ ++ _),
list3.reduce(_ ++ _),
list4.reduce(_ ++ _),
list5.reduce(_ ++ _))
}.
toDF("QUERY",
"TYPE",
"CLICK_THROUGH_RATE",
"PURCHASE_RATE",
"PURCHASE_UNITS",
"CLICKS",
"IMPRESSIONS")
}
This gives me -
root
|-- QUERY: string (nullable = true)
|-- TYPE: string (nullable = true)
|-- CLICK_THROUGH_RATE: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
|-- PURCHASE_RATE: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
|-- PURCHASE_UNITS: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
|-- CLICKS: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
|-- IMPRESSIONS: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
But when I do result.count, I am getting this exception -
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 63.0 failed 4 times, most recent failure: Lost task 0.3 in stage 63.0 (TID 62365, ip-10-0-1-52.ec2.internal, executor 2): java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2347)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2341)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:464)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:490)
at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2232)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2341)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2341)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:464)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2041)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2029)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2028)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2028)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:966)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:966)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:966)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2262)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2211)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2200)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:777)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:401)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3389)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2550)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2550)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2764)
at org.apache.spark.sql.Dataset.getRows(Dataset.scala:254)
at org.apache.spark.sql.Dataset.showString(Dataset.scala:291)
at org.apache.spark.sql.Dataset.show(Dataset.scala:753)
at org.apache.spark.sql.Dataset.show(Dataset.scala:730)
... 53 elided
Caused by: java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2287)
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1417)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2347)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2341)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:464)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at scala.collection.immutable.List$SerializationProxy.readObject(List.scala:490)
at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2232)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2341)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2341)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2265)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2123)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1624)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:464)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
... 3 more
Am I doing something wrong ?
There is the same problem with HashMap.
I found the solution here : https://gist.github.com/ramn/5566596
You have to replace the class ObjectInputStream in your code by a new class : ObjectInputStreamWithCustomClassLoader
class ObjectInputStreamWithCustomClassLoader(
fileInputStream: FileInputStream
) extends ObjectInputStream(fileInputStream) {
override def resolveClass(desc: java.io.ObjectStreamClass): Class[_] = {
try { Class.forName(desc.getName, false, getClass.getClassLoader) }
catch { case ex: ClassNotFoundException => super.resolveClass(desc) }
}
}
I changed your code a little bit and I am getting the result
Created a dataframe with a single record with same schema as yours
val df = Seq(("select * from test", "type1", "device1", "10.0", "20.0", "1234", "23.4567", "10.98")).toDF.selectExpr("_1 as QUERY", "_2 as TYPE", "_3 as DEVICE", "_4 as PURCHASE_UNITS_SUM", "_5 as CLICK_SUM", "_6 as IMPRESSION_COUNT", "_7 as CLICK_THROUGH_RATE", "_8 as PURCHASE_RATE")
Below is the Schema and the sample row
root
|-- QUERY: string (nullable = true)
|-- TYPE: string (nullable = true)
|-- DEVICE: string (nullable = true)
|-- PURCHASE_UNITS_SUM: string (nullable = true)
|-- CLICK_SUM: string (nullable = true)
|-- IMPRESSION_COUNT: string (nullable = true)
|-- CLICK_THROUGH_RATE: string (nullable = true)
|-- PURCHASE_RATE: string (nullable = true)
+------------------+-----+-------+------------------+---------+----------------+------------------+-------------+
| QUERY| TYPE| DEVICE|PURCHASE_UNITS_SUM|CLICK_SUM|IMPRESSION_COUNT|CLICK_THROUGH_RATE|PURCHASE_RATE|
+------------------+-----+-------+------------------+---------+----------------+------------------+-------------+
|select * from test|type1|device1| 10.0| 20.0| 1234| 23.4567| 10.98|
+------------------+-----+-------+------------------+---------+----------------+------------------+-------------+
val result = df.withColumn("CLICK_THROUGH_RATE_MAP", map(col("DEVICE"), col("CLICK_THROUGH_RATE"))).
withColumn("PURCHASE_RATE_MAP", map(col("DEVICE"), col("PURCHASE_RATE"))).
withColumn("PURCHASE_SUM_MAP", map(col("DEVICE"), col("PURCHASE_UNITS_SUM"))).
withColumn("CLICK_SUM_MAP", map(col("DEVICE"), col("CLICK_SUM"))).
withColumn("IMPRESSION_SUM_MAP", map(col("DEVICE"), col("IMPRESSION_COUNT"))).
groupBy("QUERY", "TYPE").
agg(collect_list("CLICK_THROUGH_RATE_MAP"), collect_list("PURCHASE_RATE_MAP"), collect_list("PURCHASE_SUM_MAP"), collect_list("CLICK_SUM_MAP"), collect_list("IMPRESSION_SUM_MAP")).
as[(String, String, Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]])]
result.show
+------------------+-----+------------------------------------+-------------------------------+------------------------------+---------------------------+--------------------------------+
| QUERY| TYPE|collect_list(CLICK_THROUGH_RATE_MAP)|collect_list(PURCHASE_RATE_MAP)|collect_list(PURCHASE_SUM_MAP)|collect_list(CLICK_SUM_MAP)|collect_list(IMPRESSION_SUM_MAP)|
+------------------+-----+------------------------------------+-------------------------------+------------------------------+---------------------------+--------------------------------+
|select * from test|type1| [Map(device1 -> 2...| [Map(device1 -> 1...| [Map(device1 -> 1...| [Map(device1 -> 2...| [Map(device1 -> 1...|
+------------------+-----+------------------------------------+-------------------------------+------------------------------+---------------------------+--------------------------------+
I changed the map function as follows
val finalresultdf = result.map { f => (f._1, f._2, f._3.reduce(_ ++ _), f._4.reduce(_ ++ _), f._5.reduce(_ ++ _), f._6.reduce(_ ++ _), f._7.reduce(_ ++ _)) }.
toDF("QUERY", "TYPE", "CLICK_THROUGH_RATE", "PURCHASE_RATE", "PURCHASE_UNITS", "CLICKS", "IMPRESSIONS")
finalresultdf.show
+------------------+-----+--------------------+--------------------+--------------------+--------------------+--------------------+
| QUERY| TYPE| CLICK_THROUGH_RATE| PURCHASE_RATE| PURCHASE_UNITS| CLICKS| IMPRESSIONS|
+------------------+-----+--------------------+--------------------+--------------------+--------------------+--------------------+
|select * from test|type1|Map(device1 -> 23...|Map(device1 -> 10...|Map(device1 -> 10.0)|Map(device1 -> 20.0)|Map(device1 -> 12...|
+------------------+-----+--------------------+--------------------+--------------------+--------------------+--------------------+
finalresultdf.count
scala> finalresultdf.count
res34: Long = 1
Hope this helps!!!
I am trying to write dataframe data to cassandra table. How do I map the struct in dataframe to column in cassandra table.
Schema of dataframe
root
|-- _1: struct (nullable = true)
| |-- _1: string (nullable = true)
| |-- _2: integer (nullable = false)
| |-- _3: integer (nullable = false)
| |-- _4: integer (nullable = false)
|-- _2: double (nullable = false)
Description of CassandraTable (..I'm trying to use tuple to save struct data)
create table knmi_r (idnkey tuple<text,int,int,int>, logval int, primary key (idnkey) ) ;
below is my code to save to cassandra table
resultdf.printSchema()
resultdf.write
.format("org.apache.spark.sql.cassandra")
.options(Map("table" -> "knmi_r", "keyspace" -> "dbks1"))
.save()
However I am getting below error. Can someone please guide me how to get past this.
Exception in thread "main" java.util.NoSuchElementException: Columns not found in table dbks1.knmi_r: _1, _2
at com.datastax.spark.connector.SomeColumns.selectFrom(ColumnSelector.scala:44)
at com.datastax.spark.connector.writer.TableWriter$.apply(TableWriter.scala:385)
at com.datastax.spark.connector.RDDFunctions.saveToCassandra(RDDFunctions.scala:35)
at org.apache.spark.sql.cassandra.CassandraSourceRelation.insert(CassandraSourceRelation.scala:76)
at org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.scala:90)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
at nl.rug.sc.SparkExample.testExample(SparkExample.scala:286)
at nl.rug.sc.app.SparkBootcamp$class.run(SparkBootcamp.scala:19)
at nl.rug.sc.app.SparkLocalMain$.run(SparkLocalMain.scala:6)
at nl.rug.sc.app.SparkLocalMain$.delayedEndpoint$nl$rug$sc$app$SparkLocalMain$1(SparkLocalMain.scala:18)
at nl.rug.sc.app.SparkLocalMain$delayedInit$body.apply(SparkLocalMain.scala:6)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at nl.rug.sc.app.SparkLocalMain$.main(SparkLocalMain.scala:6)
at nl.rug.sc.app.SparkLocalMain.main(SparkLocalMain.scala)
The error is because of mismatch between schema names and cassandra column names. I simply renamed them and it worked
val newNames = Seq("idnkey", "logval")
val resultdf1 = resultdf.toDF(newNames: _*)
resultdf1.printSchema
now the schema of dataframe matches with cassandra table
root
|-- idnkey: struct (nullable = true)
| |-- _1: string (nullable = true)
| |-- _2: integer (nullable = false)
| |-- _3: integer (nullable = false)
| |-- _4: integer (nullable = false)
|-- logval: double (nullable = false)
I try to apply kmeans algorithm.
Code
val dfJoin_products_items = df_products.join(df_items, "product_id")
dfJoin_products_items.createGlobalTempView("products_items")
val weightFreight = spark.sql("SELECT cast(product_weight_g as double) weight, cast(freight_value as double) freight FROM global_temp.products_items")
case class Rows(weight:Double, freight:Double)
val rows = weightFreight.as[Rows]
val assembler = new VectorAssembler().setInputCols(Array("weight", "freight")).setOutputCol("features")
val data = assembler.transform(rows)
val kmeans = new KMeans().setK(4)
val model = kmeans.fit(data)
Values
dfJoin_products_items
scala> dfJoin_products_items.printSchema
root
|-- product_id: string (nullable = true)
|-- product_category_name: string (nullable = true)
|-- product_name_lenght: string (nullable = true)
|-- product_description_lenght: string (nullable = true)
|-- product_photos_qty: string (nullable = true)
|-- product_weight_g: string (nullable = true)
|-- product_length_cm: string (nullable = true)
|-- product_height_cm: string (nullable = true)
|-- product_width_cm: string (nullable = true)
|-- order_id: string (nullable = true)
|-- order_item_id: string (nullable = true)
|-- seller_id: string (nullable = true)
|-- shipping_limit_date: string (nullable = true)
|-- price: string (nullable = true)
|-- freight_value: string (nullable = true)
weightFreight
scala> weightFreight.printSchema
root
|-- weight: double (nullable = true)
|-- freight: double (nullable = true)
Error
2019-02-03 20:51:41 WARN BlockManager:66 - Putting block rdd_126_1 failed due to exception org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<weight:double,freight:double>) => struct<type:tinyint,size:int,indices:array<int>,values:array<double>>).
2019-02-03 20:51:41 WARN BlockManager:66 - Block rdd_126_1 could not be removed as it was not found on disk or in memory
2019-02-03 20:51:41 WARN BlockManager:66 - Putting block rdd_126_2 failed due to exception org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<weight:double,freight:double>) => struct<type:tinyint,size:int,indices:array<int>,values:array<double>>).
2019-02-03 20:51:41 ERROR Executor:91 - Exception in task 1.0 in stage 16.0 (TID 23)
org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<weight:double,freight:double>) => struct<type:tinyint,size:int,indices:array<int>,values:array<double>>)
I don't understand this error, someone can explain me please ?
Thanks a lot!
UPDATE 1 : Full stacktrace
The stacktrace is huge, so you can find it here : https://pastebin.com/PhmZPtDk