Finding most populated cities per country - scala

Question

I need to write code that gives the most populated cities per country with the population.
Here is a input data:
DataFrame = {
/** Input data */
val inputDf = Seq(
("Warsaw", "Poland", "1 764 615"),
("Cracow", "Poland", "769 498"),
("Paris", "France", "2 206 488"),
("Villeneuve-Loubet", "France", "15 020"),
("Pittsburgh PA", "United States", "302 407"),
("Chicago IL", "United States", "2 716 000"),
("Milwaukee WI", "United States", "595 351"),
("Vilnius", "Lithuania", "580 020"),
("Stockholm", "Sweden", "972 647"),
("Goteborg", "Sweden", "580 020")
).toDF("name", "country", "population")
println("Input:")
inputDf.show(false)
My solution was:
val topPopulation = inputDf
// .select("name", "country", "population")
.withColumn("population", regexp_replace($"population", " ", "").cast("Integer"))
// .agg(max($"population").alias(("population")))
// .withColumn("population", regexp_replace($"population", " ", "").cast("Integer"))
// .withColumn("country", $"country")
// .withColumn("name", $"name")
// .cast("Integer")
.groupBy("country")
.agg(
max("population").alias("population")
)
.orderBy($"population".desc)
// .orderBy("max(population)")
topPopulation
But i have troubke, because "Except can only be performed on tables with the same number of columns, but the first table has 2 columns and the second table has 3 columns;;
"
Input:
+-----------------+-------------+----------+
|name |country |population|
+-----------------+-------------+----------+
|Warsaw |Poland |1 764 615 |
|Cracow |Poland |769 498 |
|Paris |France |2 206 488 |
|Villeneuve-Loubet|France |15 020 |
|Pittsburgh PA |United States|302 407 |
|Chicago IL |United States|2 716 000 |
|Milwaukee WI |United States|595 351 |
|Vilnius |Lithuania |580 020 |
|Stockholm |Sweden |972 647 |
|Goteborg |Sweden |580 020 |
+-----------------+-------------+----------+
Expected:
+----------+-------------+----------+
|name |country |population|
+----------+-------------+----------+
|Warsaw |Poland |1 764 615 |
|Paris |France |2 206 488 |
|Chicago IL|United States|2 716 000 |
|Vilnius |Lithuania |580 020 |
|Stockholm |Sweden |972 647 |
+----------+-------------+----------+
Actual:
+-------------+----------+
|country |population|
+-------------+----------+
|United States|2716000 |
|France |2206488 |
|Poland |1764615 |
|Sweden |972647 |
|Lithuania |580020 |
+-------------+----------+

Answer 1

Try this-
Load the test data
val inputDf = Seq(
("Warsaw", "Poland", "1 764 615"),
("Cracow", "Poland", "769 498"),
("Paris", "France", "2 206 488"),
("Villeneuve-Loubet", "France", "15 020"),
("Pittsburgh PA", "United States", "302 407"),
("Chicago IL", "United States", "2 716 000"),
("Milwaukee WI", "United States", "595 351"),
("Vilnius", "Lithuania", "580 020"),
("Stockholm", "Sweden", "972 647"),
("Goteborg", "Sweden", "580 020")
).toDF("name", "country", "population")
println("Input:")
inputDf.show(false)
/**
* Input:
* +-----------------+-------------+----------+
* |name |country |population|
* +-----------------+-------------+----------+
* |Warsaw |Poland |1 764 615 |
* |Cracow |Poland |769 498 |
* |Paris |France |2 206 488 |
* |Villeneuve-Loubet|France |15 020 |
* |Pittsburgh PA |United States|302 407 |
* |Chicago IL |United States|2 716 000 |
* |Milwaukee WI |United States|595 351 |
* |Vilnius |Lithuania |580 020 |
* |Stockholm |Sweden |972 647 |
* |Goteborg |Sweden |580 020 |
* +-----------------+-------------+----------+
*/
find the city in the country having max population
val topPopulation = inputDf
.withColumn("population", regexp_replace($"population", " ", "").cast("Integer"))
.withColumn("population_name", struct($"population", $"name"))
.groupBy("country")
.agg(max("population_name").as("population_name"))
.selectExpr("country", "population_name.*")
topPopulation.show(false)
topPopulation.printSchema()
/**
* +-------------+----------+----------+
* |country |population|name |
* +-------------+----------+----------+
* |France |2206488 |Paris |
* |Poland |1764615 |Warsaw |
* |Lithuania |580020 |Vilnius |
* |Sweden |972647 |Stockholm |
* |United States|2716000 |Chicago IL|
* +-------------+----------+----------+
*
* root
* |-- country: string (nullable = true)
* |-- population: integer (nullable = true)
* |-- name: string (nullable = true)
*/