For example, I have a regex string:
val myRegex:Regex = "blahblah".r
but if the 'blahblah' is like more than thousand characters long, I want to split them into multiple lines so I can read easier. like so:
val myRegex:Regex = "blah".r
+ "blah".r
this does not work because value unary_+ is not a member of scala.util.matching.Regex.
is there a proper way?
One possible solution:
val myRegex:Regex =
"""a
|very
|long
|pattern
|"""
.stripMargin
.replaceAll("\n", "")
.r
Related
I have a spark scala dataframe which has column "Name"
I have extracted the values of that column in to scala array[string]
org_name: Array[String] = Array(SARATOGA SENIOR HIGH SCHOOL)
I want to replace whitespaces with _ and encode that value in to utf-8 (any encoding is fine as long as it replaces special chars with something else)
so if there are any special chars those will be removed. later i want to use those in file path .
var org_name = orgsFlatDF.rdd.collect
.map( _.getString(2))
This is how i am extracting those vals ^^. I haven't found any method which I can use to do that. Replace or replaceall doesn't work on array
I tried this :
org_name.replace("\\s", "")
That didn't work .
Expected output : SARATOGA_SENIOR_HIGH_SCHOOL
if name is : new $ high school it should gets converted to new_$_high_school then encoded to new_%24_high_school
There are a couple of issues with what you are asking.
Java/Scala Arrays don't have a replace method. Even if they did have a replace method, would they replace the values they hold or the characters in a String they hold?
Let's assume this line org_name.replace("\\s", "") didn't compiled and org_name is indeed a an Array[String] holding one element.
scala> val org_name=Array("SARATOGA SENIOR HIGH SCHOOL")
val org_name: Array[String] = Array(SARATOGA SENIOR HIGH SCHOOL)
scala> org_name(0).replace(" ","_")
val res15: String = SARATOGA_SENIOR_HIGH_SCHOOL
replace("\\s","_") wouldn't work because it represents a \s string. "\" represents \. That's only way you'd be able to define strings containing other escape codes like \n or \t.
PS: to transform all the string in the array use org_name.map(_.replace(" ","_")), this gives you back another another array.
I want to create a function to handle the text-prepocessing in a problem I am facing with text data. I am familiar with Python and pandas dataframe and my usual thought process of solving the problem is to use a function and then using pandas apply method to apply the function to all the elements in a column. However I don't know where to begin to accomplish this.
So, I created two functions to handle the replacements. The problem is that I don't know how to put more than one replace inside this method. I need to make about 20 replacements for three separate dataframes so to solve it with this method it would take me 60 lines of code. Is there a way to do all the replacements inside a single function and then apply it to all the elements in a dataframe column in scala?
def removeSpecials: String => String = _.replaceAll("$", " ")
def removeSpecials2: String => String = _.replaceAll("?", " ")
val udf_removeSpecials = udf(removeSpecials)
val udf_removeSpecials2 = udf(removeSpecials2)
val consolidated2 = consolidated.withColumn("product_description", udf_removeSpecials($"product_description"))
val consolidated3 = consolidated2.withColumn("product_description", udf_removeSpecials2($"product_description"))
consolidated3.show()
Well you can simply add every replacement next to the previous one like this :
def removeSpecials: String => String = _.replaceAll("$", " ").replaceAll("?", " ")
But in this case where the replacement character is the same, it would be better to use regular expressions to avoid multiple replaceAll.
def removeSpecials: String => String = _.replaceAll("\\$|\\?", " ")
Note that \\ is used as escape character.
Suppose I have the text file like this:
Apple#mango&banana#grapes
The data needs to be split on multiple delimiters before performing the word count.
How to do that?
Use split method:
scala> "Apple#mango&banana#grapes".split("[#&#]")
res0: Array[String] = Array(Apple, mango, banana, grapes)
If you just want to count words, you don't need to split. Something like this will do:
val numWords = """\b\w""".r.findAllIn(string).length
This is a regex that matches start of a word (\b is a (zero-length) word boundary, \w is any "word" character (letter, number or underscore), so you get all the matches in your string, and then just check how many there are.
If you are looking to count each word separately, and do it across multiple lines, then, split is, probably, a better option:
source
.getLines
.flatMap(_.split("\\W+"))
.filterNot(_.isEmpty)
.groupBy(identity)
.mapValues(_.size)
I come up a pattern like
val pattern = "(\\w+)\\|(.*)\\|\\[(.*)\\]\\|\"(.*)\"\\|\"(.*)\"\\|\\[(.*)\\]\\|\\[(.*)\\]\\|(.*)\\|\\[(.*)\\]\\|\\[(.*)\\]".r
and I have a original string
var str = """AuthLogout|vmlxapp21a|[13/Jan/2016:16:33:15 +0100]|"66.77.444.44 uid=XXXXX,ou=People,o=Bank,o=External,dc=xxxx,dc=com"|"abcd_123_portalweb_w "|[]|[41]||[]|[]"""
then apply pattern to the string, but it is always empty.
val items = pattern.findAllIn(str).toList
If I understand what you're trying to do, perhaps using a giant regex isn't the easiest way: You can split by | and get rid of the unwanted separators ([, ], ") using replaceAll:
val str = """AuthLogout|vmlxapp21a|[13/Jan/2016:16:33:15 +0100]|"66.77.444.44 uid=XXXXX,ou=People,o=Bank,o=External,dc=xxxx,dc=com"|"abcd_123_portalweb_w "|[]|[41]||[]|[]"""
val withoutBoundaries = str.replaceAll("[\"\\]\\[]","")
val result = withoutBoundaries.split("\\|")
result.foreach(println)
Which prints:
AuthLogout
vmlxapp21a
13/Jan/2016:16:33:15 +0100
66.77.444.44 uid=XXXXX,ou=People,o=Bank,o=External,dc=xxxx,dc=com
abcd_123_portalweb_w
41
If you do want to use a regex here, I'd create sub-regex vars representing the different text parts that you're after, to make this somewhat manageable:
val plain = "(.*)" // no boundary characters
val boxed = s"\\[$plain\\]" // same, encapsulated by square brackets
val quoted = '"' + plain + '"' // same, encapsulated by double quotes
// the whole thing, separated by pipes:
val r = s"$plain\\|$plain\\|$boxed\\|$quoted\\|$quoted\\|$boxed\\|$boxed\\|$plain\\|$boxed\\|$boxed".r
val result = r.findAllIn(str).toList // this list has one item, as expected.
Now, if you want to see how this regex looks like, here it is - but I don't recommend having this in your code...:
val r = """(.*)\|(.*)\|\[(.*)\]\|"(.*)"\|"(.*)"\|\[(.*)\]\|\[(.*)\]\|(.*)\|\[(.*)\]\|\[(.*)\]""".r
I need to make some Windows file paths that contain spaces into string literals in Scala. I have tried wrapping the entire path in double quotes AND wrapping the entire path in double quotes with each directory name that has a space with single quotes. Now it is wanting an escape character for "\Jun" in both places and I don't know why.
Here are the strings:
val input = "R:\'Unclaimed Property'\'CT State'\2015\Jun\ct_finderlist_2015.pdf"
val output = "R:\'Unclaimed Property'\'CT State'\2015\Jun"
Here is the latest error:
The problem is with the \ character, that has to be escaped.
This should work:
val input = "R:\\Unclaimed Property\\CT State\\2015\\Jun.ct_finderlist_2015.pdf"
val output = "R:\\Unclaimed Property\\CT State\\2015\\Jun"
A cleaner way to create string literals is to use triple quotes.
You can wrap your string directly in triple quotes without escaping special characters. And you can put multiple lines string in it.
It's much easier to code and read.
For example
val input =
"""
|R:\Unclaimed Property\CT State\2015\Jun.ct_finderlist_2015.pdf
"""
To add a variable to the string, do it like the following by adding "$variableName".
val input =
s"""
|R:\Unclaimed Property\$variablePath\CT State\2015\Jun.ct_finderlist_2015.pdf
"""