AnalysisException: Failure when resolving conflicting references in Join: 'Join Inner - scala

Question

I have this simple code
var count = event_stream
.groupBy("value").count()
event_stream.join(count,"value").printSchema() //get error on this line
event_stream and count schemas are as follows
root
|-- key: binary (nullable = true)
|-- value: binary (nullable = true)
|-- topic: string (nullable = true)
|-- partition: integer (nullable = true)
|-- offset: long (nullable = true)
|-- timestamp: timestamp (nullable = true)
|-- timestampType: integer (nullable = true)
root
|-- value: binary (nullable = true)
|-- count: long (nullable = false)
two questions
Why do I get this error and how to fix?
Why does groupby.count drops all other columns?
The error is as follows
Exception in thread "main" org.apache.spark.sql.AnalysisException:
Failure when resolving conflicting references in Join:
'Join Inner
:- AnalysisBarrier
: +- StreamingRelationV2 org.apache.spark.sql.kafka010.KafkaSourceProvider#7f2c57fe, kafka, Map(startingOffsets -> latest, failOnDataLoss -> false, subscribe -> events-identification-carrier, kafka.bootstrap.servers -> svc-kafka-pre-c1-01.jamba.net:9092), [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], StreamingRelation DataSource(org.apache.spark.sql.SparkSession#3dbd7107,kafka,List(),None,List(),None,Map(startingOffsets -> latest, failOnDataLoss -> false, subscribe -> events-identification-carrier, kafka.bootstrap.servers -> svc-kafka-pre-c1-01.jamba.net:9092),None), kafka, [key#0, value#1, topic#2, partition#3, offset#4L, timestamp#5, timestampType#6]
+- AnalysisBarrier
+- Aggregate [value#8], [value#8, count(1) AS count#46L]
+- StreamingRelationV2 org.apache.spark.sql.kafka010.KafkaSourceProvider#7f2c57fe, kafka, Map(startingOffsets -> latest, failOnDataLoss -> false, subscribe -> events-identification-carrier, kafka.bootstrap.servers -> svc-kafka-pre-c1-01.jamba.net:9092), [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13], StreamingRelation DataSource(org.apache.spark.sql.SparkSession#3dbd7107,kafka,List(),None,List(),None,Map(startingOffsets -> latest, failOnDataLoss -> false, subscribe -> events-identification-carrier, kafka.bootstrap.servers -> svc-kafka-pre-c1-01.jamba.net:9092),None), kafka, [key#0, value#1, topic#2, partition#3, offset#4L, timestamp#5, timestampType#6]
Conflicting attributes: value#8
EDIT: yes! the changing the name of the columns works.
But now, If I use the join, I have to use OutputMode.Append and for that, I need to add Watermarks to the stream.
What I want is extract the count and topic(from the above printed schema) in the resultingDF and write that to some Sink.
Two questions
Is there any other way/better way to do this?
Can I do multiple aggs like count() and then also add another column which is of String type i.e. topic is this case?

Answer 1

Why do I get this error and how to fix?
I think you are getting the error because the final joined schema contains two value fields, one from each side of the join. To fix this rename the "value" field on one of the two joined Streams like this:
var count = event_stream.
groupBy("value").count().
withColumnRenamed("value", "join_id")
event_stream.join(count, $"value" === $"join_id").
drop("join_id").
printSchema()
Why does groupby.count drops all other columns?
groupBy operations basically are dividing your fields up into two lists. A list of fields to use as the key, and a list of fields to aggregate. The key fields just show up as is in the final result, but any fields not in the list need to have an aggregate operation defined to show up in the result. Otherwise spark has no way to know how you want to combine multiple values of that field! Did you want to just count it? Did you want the max value? Did you want to see all distinct values? To specify how to rollup a field, you can define it in a .agg(..) call.
Example:
val input = Seq(
(1, "Bob", 4),
(1, "John", 5)
).toDF("key", "name", "number")
input.groupBy("key").
agg(collect_set("name") as "names",
max("number") as "maxnum").
show
+---+-----------+------+
|key|name |maxnum|
+---+-----------+------+
| 1|[Bob, John]| 5|
+---+-----------+------+

Answer 2

The reason for error is the column name which is used for joining .
You can use the operation like .
var count = event_stream
.groupBy("value").count()
event_stream.join(count,Seq("value"))