Substring with delimiters with Spark Scala - scala

Question

I am new at Spark and Scala and I want to ask you a question :
I have a city field in my database (that I have already loaded it in a DataFrame) with this pattern : "someLetters" + " - " + id + ')'.
Example :
ABDCJ - 123456)
AGDFHBAZPF - 1234567890)
The size of the field is not fixed and the id here can be an integer of 6 or 10 digits. So, what I want to do is to extract that id in a new column called city_id.
Concretely, I want to start by the last character of the digit which is ')', ignore it and extract the integer until I find a space. Then break.
I already tried to do this using withColumn or a regex or even subString index but I got confused since they are based on the index which I can't use here.
How can I fix this?

Answer 1

start by the last character of the digit which is ')', ignore it and
extract the integer until I find a space
This can be done with regex pattern .*?(\\d+)\\)$, where \\)$ matches the ) at the end of the string, and then capture the digits with \\d+, and extract it as a new column; Notice .*? lazily (due to ?) matches a string until the pattern (\\d+)\\)$ is found:
df.withColumn("id", regexp_extract($"city", ".*?(\\d+)\\)$", 1)).show
+--------------------+----------+
| city| id|
+--------------------+----------+
| ABDCJ - 123456)| 123456|
|AGDFHBAZPF - 1234...|1234567890|
+--------------------+----------+

Answer 2

import org.apache.spark.sql.functions._
val df=tempDF.withColumn("city_id",rtrim(element_at(split($"city"," - "),2),")"))

Answer 3

Assuming that the input is in the format in your example.
In order to get the number after the - without the trailing ) you can execute the following command:
split(" - ")(1).dropRight(1)
The above split by the - sign and takes the second element (i.e. the number), and remove the last char (the )).
You can create udf which execute the above command, and create a new column using withColumn command

Answer 4

I would go for regex_extract, but there are many alternatives : You can also do this using 2 splits :
df
.withColumn("id",
split(
split($"city"," - ")(1),"\\)"
)(0)
)
First, you split by - and take the second element, then split by ) and take the first element
Or another alternative, split by - and then drop ) :
df
.withColumn("id",
reverse(
substring(
reverse(split($"city"," - ")(1)),
2,
Int.MaxValue
)
)
)

Answer 5

You can use 2 regexp_replace functions also.
scala> val df = Seq(("ABDCJ - 123456)"),("AGDFHBAZPF - 1234567890)")).toDF("cityid")
df: org.apache.spark.sql.DataFrame = [citiid: string]
scala> df.withColumn("id",regexp_replace(regexp_replace('cityid,""".*- """,""),"""\)""","")).show(false)
+------------------------+----------+
|cityid |id |
+------------------------+----------+
|ABDCJ - 123456) |123456 |
|AGDFHBAZPF - 1234567890)|1234567890|
+------------------------+----------+
scala>
Since the id seems to be an integer, you can cast it to long as
scala> val df2 = df.withColumn("id",regexp_replace(regexp_replace('cityid,""".*- """,""),"""\)""","").cast("long"))
df2: org.apache.spark.sql.DataFrame = [cityid: string, id: bigint]
scala> df2.show(false)
+------------------------+----------+
|cityid |id |
+------------------------+----------+
|ABDCJ - 123456) |123456 |
|AGDFHBAZPF - 1234567890)|1234567890|
+------------------------+----------+
scala> df2.printSchema
root
|-- cityid: string (nullable = true)
|-- id: long (nullable = true)
scala>