I have this String:
val str = "9617 / 20634"
So after split i want to parse only the 2 values 9617 & 20634 in order to calculate percentage.
So instead of trim after the split can i do it before ?
It is easier to remove spaces after the split than before. Unless you are expected to trim it before, here is a simpler way of doing it.
val Array(x, y) = "9617 / 20634".split("/").map(_.trim.toFloat)
val p = x / y * 100
Values are converted to Float to prevent integer division leading to 0.
The val Array(x,y) = ... statement is just another way of calling
unapplySeq of the Array object.
For scalable solution one might use regexps:
scala> val r = """(\d+) */ *(\d+)""".r
r: scala.util.matching.Regex = (\d+) +/ +(\d+)
scala> "111 / 222" match {
case r(a, b) => println(s"Got $a, $b")
case _ =>
}
Got 111, 222
This is useful if you need different usage patterns.
Here you define correspondence between variables from you input (111, 222) and match groups ((\d+), (\d+)).
Related
I have an array
val a = "((x1,x2),(y1,y2),(z1,z2))"
I want to parse this into a scala array
val arr = Array(("x1","x2"),("y1","y2"),("z1","z2"))
Is there a way of directly doing this with an expr() equivalent ?
If not how would one do this using split
Note : x1 x2 x3 etc are strings and can contain special characters so key would be to use () delimiters to parse data -
Code I munged from Dici and Bogdan Vakulenko
val x2 = a.getString(1).trim.split("[\()]").grouped(2).map(x=>x(0).trim).toArray
val x3 = x2.drop(1) // first grouping is always null dont know why
var jmap = new java.util.HashMap[String, String]()
for (i<-x3)
{
val index = i.lastIndexOf(",")
val fv = i.slice(0,index)
val lv = i.substring(index+1).trim
jmap.put(fv,lv)
}
This is still suceptible to "," in the second string -
Actually, I think regex are the most convenient way to solve this.
val a = "((x1,x2),(y1,y2),(z1,z2))"
val regex = "(\\((\\w+),(\\w+)\\))".r
println(
regex.findAllMatchIn(a)
.map(matcher => (matcher.group(2), matcher.group(3)))
.toList
)
Note that I made some assumptions about the format:
no whitespaces in the string (the regex could easily be updated to fix this if needed)
always tuples of two elements, never more
empty string not valid as a tuple element
only alphanumeric characters allowed (this also would be easy to fix)
val a = "((x1,x2),(y1,y2),(z1,z2))"
a.replaceAll("[\\(\\) ]","")
.split(",")
.sliding(2)
.map(x=>(x(0),x(1)))
.toArray
I have a string as
something,'' something,nothing_something,op nothing_something,'' cat,cat
I want to achieve my output as
'' something,op nothing_something,cat
Is there any way to achieve it?
If I understand your requirement correctly, here's one approach with the following steps:
Split the input string by "," and create a list of indexed-CSVs and convert it to a Map
Generate 2-combinations of the indexed-CSVs
Check each of the indexed-CSV pairs and capture the index of any CSV which is contained within the other CSV
Since the CSVs corresponding to the captured indexes are contained within some other CSV, removing these indexes will result in remaining indexes we would like to keep
Use the remaining indexes to look up CSVs from the CSV Map and concatenate them back to a string
Here is sample code applying to a string with slightly more general comma-separated values:
val str = "cats,a cat,cat,there is a cat,my cat,cats,cat"
val csvIdxList = (Stream from 1).zip(str.split(",")).toList
val csvMap = csvIdxList.toMap
val csvPairs = csvIdxList.combinations(2).toList
val csvContainedIdx = csvPairs.collect{
case List(x, y) if x._2.contains(y._2) => y._1
case List(x, y) if y._2.contains(x._2) => x._1
}.
distinct
// csvContainedIdx: List[Int] = List(3, 6, 7, 2)
val csvToKeepIdx = (1 to csvIdxList.size) diff csvContainedIdx
// csvToKeepIdx: scala.collection.immutable.IndexedSeq[Int] = Vector(1, 4, 5)
val strDeduped = csvToKeepIdx.map( csvMap.getOrElse(_, "") ).mkString(",")
// strDeduped: String = cats,there is a cat,my cat
Applying the above to your sample string something,'' something,nothing_something,op nothing_something would yield the expected result:
strDeduped: String = '' something,op nothing_something
First create an Array of words separated by commas using split command on the given String, and do other operations using filter and mkString as below:
s.split(",").filter(_.contains(' ')).mkString(",")
In Scala REPL:
scala> val s = "something,'' something,nothing_something,op nothing_something"
s: String = something,'' something,nothing_something,op nothing_something
scala> s.split(",").filter(_.contains(' ')).mkString(",")
res27: String = '' something,op nothing_something
As per Leo C comment, I tested it as below with some other String:
scala> val s = "something,'' something anything anything anything anything,nothing_something,op op op nothing_something"
s: String = something,'' something anything anything anything anything,nothing_something,op op op nothing_something
scala> s.split(",").filter(_.contains(' ')).mkString(",")
res43: String = '' something anything anything anything anything,op op op nothing_something
I want to split following string which takes the form
val str = "X|blnk_1|blnk_2|blnk_3|blnk_4|time1|time2|blnk_5|blnk_6|blnk_7|blnk_8| |Z01|Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|]|Z01|Str2|02|002|NE]|[HEX3|HEX4]|[NA|002:1001|234:456|[01]|]|Z02|02|z2|Str|Str|"
This string always start with X and split positions are Z01,Z02,...,Z0D
str.split("""\|Z0[1|2|3|4|5|6|7|8|9|A|B|C|D]{1}\|""") foreach println
Here there could be no ordering of the position of Z01,...,Z0D appear in the string.
The split gives the desired result :
X|blnk_1|blnk_2|blnk_3|blnk_4|time1|time2|blnk_5|blnk_6|blnk_7|blnk_8|
Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|]
Str2|02|002|NE]|[HEX3|HEX4]|[NA|002:1001|234:456|[01]|]
02|z2|Str|Str|
However I want to map X, Z01,... to case classes. Since there is no ordering there is no way of identifying to which case class split would need to map(can't use length of individual splits ).
I expect my split to have following output :
X|blnk_1|blnk_2|blnk_3|blnk_4|time1|time2|blnk_5|blnk_6|blnk_7|blnk_8|
Z01|[Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|]|
Z01|[Str2|02|002|NE]|[HEX3|HEX4]|[NA|002:1001|234:456|[01]|]|
Z01|02|z2|Str|Str|
so that the result could be mapped to case class with the help of the preappended pattern.
For example:
case class X( ....)
case class Z01(val1: String, val2: String, val3: String)
case class Z02(val1: Int, val2: String, val3: String,val4:String)
.................
X|blnk_1|blnk_2|blnk_3|blnk_4|time1|time2|blnk_5|blnk_6|blnk_7|blnk_8| maps to case class X
and
Z01|[Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|]| maps to case class Z01
and in the end result should be in the form of ordered and similar groups to be taken as a array of particular case class.
X
Array[Z01]
Array[Z02]
......
......
As an alternative it might be an option to get your values by matching them:
(?:Z0[1-9A-D]|^X).*?(?=\|Z0[1-9A-D]|$)
This matches:
(?:Z0[1-9A-D]|X\|) In a non capturing group match Z0 and list the possible options in a character class or | X at the start ^of the line
.*? Match any character one or more times non greedy
(?=\|Z0[1-9A-D]|$) Positive lookahead which asserts that what follows is a pipe | followed by Z0 and a character from the character list or | the end of the line $
For example:
val re = """(?:Z0[1-9A-D]|^X).*?(?=\|Z0[1-9A-D]|$)""".r
val str = "X|blnk_1|blnk_2|blnk_3|blnk_4|time1|time2|blnk_5|blnk_6|blnk_7|blnk_8| |Z01|Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|]|Z01|Str2|02|002|NE]|[HEX3|HEX4]|[NA|002:1001|234:456|[01]|]|Z02|02|z2|Str|Str|"
re.findAllIn(str) foreach println
That will result in:
X|blnk_1|blnk_2|blnk_3|blnk_4|time1|time2|blnk_5|blnk_6|blnk_7|blnk_8|
Z01|Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|]
Z01|Str2|02|002|NE]|[HEX3|HEX4]|[NA|002:1001|234:456|[01]|]
Z02|02|z2|Str|Str|
Demo
How about this idea?
val x = str.split("""\|Z0[1|2|3|4|5|6|7|8|9|A|B|C|D]{1}\|""") // actual string splits
val y = """\|Z0[1|2|3|4|5|6|7|8|9|A|B|C|D]{1}\|""".r.findAllIn(str).toArray // delimiters Array
val final_data = x.slice(1, x.size).zip(y).map(x => x._2+x._1).toList // taking actual splits except first one .... and then zipping and concatenating with delimiters like below.
/*
List(|Z01|Str1|01|001|NE]|[HEX1|HEX2]|[NA|001:1000|123:456|[00]|], |Z01|Str2|02|002|NE]|[HEX3|HEX4]|[NA|002:1001|234:456|[01]|], |Z02|02|z2|Str|Str|) */
the first | in the final_data can be removed with subString
Following is my code that I have used to convert Iterator[char] to Seq[String].
val result = IOUtils.toByteArray(new FileInputStream (new File(fileDir)))
val remove_comp = result.grouped(11).map{arr => arr.update(2, 32);arr}.flatMap{arr => arr.update(3, 32); arr}
val convert_iter = remove_comp.map(_.toChar.toString).toSeq.mkString.split("\n")
val rdd_input = Spark.sparkSession.sparkContext.parallelize(convert_iter)
val fileDir:
12**34567890
12##34567890
12!!34567890
12¬¬34567890
12
'34567890
I am not happy with this code as the data size is big and converting to string would end up with heap space.
val convert_iter = remove_comp.map(_.toChar)
convert_iter: Iterator[Char] = non-empty iterator
Is there a better way of coding?
By completely disregarding corner cases about empty Strings etc I would start with something like:
val test = Iterable('s','f','\n','s','d','\n','s','v','y')
val (allButOne, last) = test.foldLeft( (Seq.empty[String], Seq.empty[Char]) ) {
case ((strings, chars), char) =>
if (char == '\n')
(strings :+ chars.mkString, Seq.empty)
else
(strings, chars :+ char)
}
val result = allButOne :+ last.mkString
I am sure it could be made more elegant, and handle corner cases better (once you define you want them handled), but I think it is a nice starting point.
But to be honest I am not entirely sure what you want to achieve. I just guessed that you want to group chars divided by \n together and turn them into Strings.
Looking at your code, I see that you are trying to replace the special characters such as **, ## and so on from the file that contains following data
12**34567890
12##34567890
12!!34567890
12¬¬34567890
12
'34567890
For that you can just read the data using sparkContext textFile and use regex replaceAllIn
val pattern = new Regex("[¬~!##$^%&*\\(\\)_+={}\\[\\]|;:\"'<,>.?` /\\-]")
val result = sc.textFile(fileDir).map(line => pattern.replaceAllIn(line, ""))
and you should have you result as RDD[String] which also an iterator
1234567890
1234567890
1234567890
1234567890
12
34567890
Updated
If there are \n and \r in between the texts at 3rd and 4th place and if the result is all fixed length of 10 digits text then you can use wholeTextFiles api of sparkContext and use following regex as
val pattern = new Regex("[¬~!##$^%&*\\(\\)_+={}\\[\\]|;:\"'<,>.?` /\\-\r\n]")
val result = sc.wholeTextFiles(fileDir).flatMap(line => pattern.replaceAllIn(line._2, "").grouped(10))
You should get the output as
1234567890
1234567890
1234567890
1234567890
1234567890
I hope the answer is helpful
I have an array of strings. Each string has 2 parts and is separated by white space. Looks like:
x <white space> y
I want to turn it into an array of Tuples where each tuple has (x, y)
How can I write this in scala? I know it will need something similar to:
val results = listOfStrings.collect { str => (str.left, str.right) }
not sure how i can break up each str to the left and right sides needed...
You could take advantage of the fact that in Scala, Regular expressions are also "extractors".
scala> var PairWithSpaces = "(\\w+)\\s+(\\w+)".r
PairWithSpaces: scala.util.matching.Regex = (.+)\s+(.+)
scala> val PairWithSpaces(l, r) = "1 17"
l: String = 1
r: String = 17
Now you can build your extractor into a natural looking "map":
scala> Array("a b", "1 3", "Z x").map{case PairWithSpaces(x,y) => (x, y) }
res10: Array[(String, String)] = Array((a,b), (1,3), (Z,x))
Perhaps overkill for you, but can really help readability if your regex gets fancy. I also like how this approach will fail fast if an illegal string is given.
Warning, not sure if the regex matches exactly what you need...
You could (assuming that you want to drop without complaint any string that doesn't fit the pattern):
val results = listOfStrings.map(_.split("\\s+")).collect { case Array(l,r) => (l,r) }