How to optimize withColumn in Spark scala? - scala

Question

I am new at Spark and Scala and I want to optimize a request that I wrote on Spark which is very very heavy and slow (my database is huge and it contains a lot of data).
I have a first table "city_ID" :
ID City_CODE
1 10
2 11
3 12
And a second table "city_name" that has a common field with the first one :
City_Code City_NAME_CRYPT
10 ADFKJV - Paris
11 AGHDFBNLKFJ - London
12 AGZEORIUYG- Madrid
What I want to have in my final result is the city id and its proper name (which I can compute with a regex on the city_name field) WITHOUT ANY OTHER DATA. So, it should look like this :
ID NAME
10 Paris
11 London
12 Madrid
Here is my current code :
val result = spark.sql(""" SELECT t1.id, t2.city_name_crypt AS name_crypt
FROM table1 t1
INNER JOIN table2
on t1.city_code = t2.city_code""").withColumn("name", regexp_extract($"name_crypt", ".*?(\\d+)\\)$", 1)).drop($"name_crypt").show()
The big problem for me is that I just want to have 2 columns, not 3! But since I did an inner join I am obliged to keep this third column on my dataframe while it's useless in my case. It's why I used the drop after the with column method.
Can you please help me to fix this problem?
Thank you in advance!

Answer 1

I think that's not what is making it slow. But you can use withColumnRenamed like so...
result.withColumnRenamed("name", regexp_extract($"name_crypt", ".*?(\\d+)\\)$", 1))
If you are new to Spark a lot of people don't parallelize the tasks at first. Perhaps you should make sure that the parallelization of your tasks is good. Check the num-executors and executor-memory
https://spark.apache.org/docs/latest/configuration.html
Here is an example spark-submit command...
spark-submit \
--class yourClass \
--master yarn \
--deploy-mode cluster \
--executor-memory 8G \
--num-executors 40 \
/path/to/myJar.jar