how to split scala string with regular expression - scala

I come up a pattern like
val pattern = "(\\w+)\\|(.*)\\|\\[(.*)\\]\\|\"(.*)\"\\|\"(.*)\"\\|\\[(.*)\\]\\|\\[(.*)\\]\\|(.*)\\|\\[(.*)\\]\\|\\[(.*)\\]".r
and I have a original string
var str = """AuthLogout|vmlxapp21a|[13/Jan/2016:16:33:15 +0100]|"66.77.444.44 uid=XXXXX,ou=People,o=Bank,o=External,dc=xxxx,dc=com"|"abcd_123_portalweb_w "|[]|[41]||[]|[]"""
then apply pattern to the string, but it is always empty.
val items = pattern.findAllIn(str).toList

If I understand what you're trying to do, perhaps using a giant regex isn't the easiest way: You can split by | and get rid of the unwanted separators ([, ], ") using replaceAll:
val str = """AuthLogout|vmlxapp21a|[13/Jan/2016:16:33:15 +0100]|"66.77.444.44 uid=XXXXX,ou=People,o=Bank,o=External,dc=xxxx,dc=com"|"abcd_123_portalweb_w "|[]|[41]||[]|[]"""
val withoutBoundaries = str.replaceAll("[\"\\]\\[]","")
val result = withoutBoundaries.split("\\|")
result.foreach(println)
Which prints:
AuthLogout
vmlxapp21a
13/Jan/2016:16:33:15 +0100
66.77.444.44 uid=XXXXX,ou=People,o=Bank,o=External,dc=xxxx,dc=com
abcd_123_portalweb_w
41
If you do want to use a regex here, I'd create sub-regex vars representing the different text parts that you're after, to make this somewhat manageable:
val plain = "(.*)" // no boundary characters
val boxed = s"\\[$plain\\]" // same, encapsulated by square brackets
val quoted = '"' + plain + '"' // same, encapsulated by double quotes
// the whole thing, separated by pipes:
val r = s"$plain\\|$plain\\|$boxed\\|$quoted\\|$quoted\\|$boxed\\|$boxed\\|$plain\\|$boxed\\|$boxed".r
val result = r.findAllIn(str).toList // this list has one item, as expected.
Now, if you want to see how this regex looks like, here it is - but I don't recommend having this in your code...:
val r = """(.*)\|(.*)\|\[(.*)\]\|"(.*)"\|"(.*)"\|\[(.*)\]\|\[(.*)\]\|(.*)\|\[(.*)\]\|\[(.*)\]""".r

Related

How do you expand one literal of regex into multiple lines?

For example, I have a regex string:
val myRegex:Regex = "blahblah".r
but if the 'blahblah' is like more than thousand characters long, I want to split them into multiple lines so I can read easier. like so:
val myRegex:Regex = "blah".r
+ "blah".r
this does not work because value unary_+ is not a member of scala.util.matching.Regex.
is there a proper way?
One possible solution:
val myRegex:Regex =
"""a
|very
|long
|pattern
|"""
.stripMargin
.replaceAll("\n", "")
.r

Apply a text-preprocessing function to a dataframe column in scala spark

I want to create a function to handle the text-prepocessing in a problem I am facing with text data. I am familiar with Python and pandas dataframe and my usual thought process of solving the problem is to use a function and then using pandas apply method to apply the function to all the elements in a column. However I don't know where to begin to accomplish this.
So, I created two functions to handle the replacements. The problem is that I don't know how to put more than one replace inside this method. I need to make about 20 replacements for three separate dataframes so to solve it with this method it would take me 60 lines of code. Is there a way to do all the replacements inside a single function and then apply it to all the elements in a dataframe column in scala?
def removeSpecials: String => String = _.replaceAll("$", " ")
def removeSpecials2: String => String = _.replaceAll("?", " ")
val udf_removeSpecials = udf(removeSpecials)
val udf_removeSpecials2 = udf(removeSpecials2)
val consolidated2 = consolidated.withColumn("product_description", udf_removeSpecials($"product_description"))
val consolidated3 = consolidated2.withColumn("product_description", udf_removeSpecials2($"product_description"))
consolidated3.show()
Well you can simply add every replacement next to the previous one like this :
def removeSpecials: String => String = _.replaceAll("$", " ").replaceAll("?", " ")
But in this case where the replacement character is the same, it would be better to use regular expressions to avoid multiple replaceAll.
def removeSpecials: String => String = _.replaceAll("\\$|\\?", " ")
Note that \\ is used as escape character.

Strip margin of indented triple-quote string in Purescript?

When using triple quotes in an indented position I for sure get indentation in the output js string too:
Comparing these two in a nested let
let input1 = "T1\nX55.555Y-44.444\nX52.324Y-40.386"
let input2 = """T1
X66.324Y-40.386
X52.324Y-40.386"""
giving
// single quotes with \n
"T1\x0aX55.555Y-44.444\x0aX52.324Y-40.386"
// triple quoted
"T1\x0a X66.324Y-40.386\x0a X52.324Y-40.386"
Is there any agreed upon thing like stripMargin in Scala so I can use those without having to unindent to top level?
Update, just to clarify what I mean, I'm currently doing:
describe "header" do
it "should parse example header" do
let input = """M48
;DRILL file {KiCad 4.0.7} date Wednesday, 31 January 2018 'AMt' 11:08:53
;FORMAT={-:-/ absolute / metric / decimal}
FMAT,2
METRIC,TZ
T1C0.300
T2C0.400
T3C0.600
T4C0.800
T5C1.000
T6C1.016
T7C3.400
%
"""
doesParse input header
describe "hole" do
it "should parse a simple hole" do
doesParse "X52.324Y-40.386" hole
Update:
I was asked to clarify stripMargin from Scala. It's used like so:
val speech = """T1
|X66.324Y-40.386
|X52.324Y-40.386""".stripMargin
which then removes the leading whitespace. stripMargin can take any separator, but defaults to |.
More examples:
Rust has https://docs.rs/trim-margin/0.1.0/trim_margin/
Kotlin has in stdlib: https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.text/trim-margin.html
I guess it might sound like asking for left-pad ( :) ) but if there's something there already I'd rather not brew it myself…
I'm sorry you didn't get a prompt response to this one, but I have implemented this function here. In case the pull request isn't merged, here's an implementation that just depends on purescript-strings:
import Data.String (joinWith, split) as String
import Data.String.CodeUnits (drop, dropWhile) as String
import Data.String.Pattern (Pattern(..))
stripMargin :: String -> String
stripMargin =
let
lines = String.split (Pattern "\n")
unlines = String.joinWith "\n"
mapLines f = unlines <<< map f <<< lines
in
mapLines (String.drop 1 <<< String.dropWhile (_ /= '|'))

Remove white spaces in scala-spark

I have sample file record like this
2018-01-1509.05.540000000000001000000751111EMAIL#AAA.BB.CL
and the above record is from a fixed length file and I wanted to split based on the lengths
and when I split I am getting a list as shown below.
ListBuffer(2018-01-15, 09.05.54, 00000000000010000007, 5, 1111, EMAIL#AAA.BB.CL)
Everything looks fine until now . But I am not sure why is there extra-space adding in each field in the list(not for the first field).
Example : My data is "09.05.54",But I am getting as" 09.05.54" in the list.
My Logic for splitting is shown below
// Logic to Split the Line based on the lengths
def splitLineBasedOnLengths(line: String, lengths: List[String]): ListBuffer[Any] = {
var splittedLine = line
var split = new ListBuffer[Any]()
for (i <- lengths) yield {
var c = i.toInt
var fi = splittedLine.take(c)
split += fi
splittedLine = splittedLine.drop(c)
}
split
}
The above code take's the line and list[String] which are nothing but lengths as input and gives the listbuffer[Any] which has the lines split according to the length.
Can any one help me why am I getting extra space before each field after splitting ?
There are no extra spaces in the data. It's just adding some separation between the elements when printing them (using toString) to make them easier to read.
To prove this try the following code:
split.foreach(s => println(s"\"$s\""))
You will see the following printed:
"2018-01-15"
"09.05.54"
"00000000000010000007"
"5"
"1111"
"EMAIL#AAA.BB.CL"

Scala program to replace words in an alphabetical order with in a string

I am learning Scala and have been trying to create a program which should replace characters in each word with in a string in an alphabetical order. For example, the string is "Where are you" so program should change it to "Eehrw aer ouy". I googled search and found some examples but I am not able to write an error free program. I think I am far from having a working program.
def main(args:Array[String]){
val r = "Where are you"
val newstr = r.map(x=>(x,_) match {
case ' ' = ' '
case y => {
val newchar = (x.toByte).toChar
if newchar.toByte.toChar > (newchar + 1).toByte.toChar
x = newchar
else
x
}
})
}
The tricky part is restoring the original capitalization. Add punctuation to the mix and it turns into a fun little challenge.
val str = "Where, aRe yoU?"
val sortedLowerCase = str.toLowerCase.split("(?=\\W)").map(_.sorted).mkString
val capsAt = str.indices.filter(str(_).isUpper)
capsAt.foldLeft(sortedLowerCase)((s,x) => s.patch(x,Seq(s(x).toUpper),1))
// res0: String = Eehrw, aEr ouY?
Time spent studying the Standard Library will be richly rewarded.
r.split(" ").map(word => word.toLowerCase.sorted)
To keep the capital letters, instead of .toLowerCase.sorted, used .sortWith and implement the sort comparison function according to how you want to sort characters.
Let me expand on Ren's answer:
compare based on lowercase and then capitalize only if there's an uppercase letter
r.split(" ").map(word => word.sortWith(_.toLower < _.toLower))
.map(x => if (x.exists(_.isUpper)) x.toLowerCase.capitalize else x )