I need some test company-names, like "rnd_company_blah23haf9", "rnd_company_g356fhg57" etc.
Is it possible to do something like
import scala.util.Random
val company = s"rnd_company_${Random.alphanumeric take 10 ?????}"
provided someone can fill out ????? of course.
Use .mkString("") to create a String from the Stream :
scala> val company = s"rnd_company_${Random.alphanumeric take 10 mkString}"
company: String = rnd_company_BbesF0EY1o
You have an example here
scala> val x = Random.alphanumeric
x: scala.collection.immutable.Stream[Char] = Stream(Q, ?)
scala> x take 10 foreach println
Q
n
m
x
S
Q
R
e
P
B
So you can try this:
import scala.util.Random
val company = s"rnd_company_${(xx take 10).mkString}"
Something verbose than the above answers but this one helps you to constrain the alphabet:
def randomText(textLength: Int = 10, alphabet: List[Char] = ('a' to 'd').toList) = {
(1 to textLength).toList.map { charPos =>
val randomIndex = (Math.random() * alphabet.length).floor.toInt
alphabet(randomIndex)
}.mkString("")
}
Related
I hava a class with custom Operators
case class Num(var value:Int) {
def add(x:Num) = Num(value + x.value)
def mul(x:Num) = Num(value * x.value)
}
So I can call them like this
val a = Num(2)
val b = Num(3)
val c = Num(4)
val m1 = a add b mul c
But how can I execute mul before add? I saw a solution like +| instead of add, but I want include letters in my Operator and +add and *mul not working. Also I want to include a pow function, so this needs an higher precidence than mul
You can use Parenthesis after add.
val m1 = a add (b mul c) = 14
val m1 = a add b mul c = 20
Update
you do not have any restrictions in naming your methods. For example, you can define methods +, -, * and etc. for a class.
case class Num(var value:Int) {
def + (x:Num) = Num(value + x.value)
def *(x:Num) = Num(value * x.value)
}
object Num extends App {
val a = Num(2)
val b = Num(3)
val c = Num(4)
val m1 = a + b * c
println(m1)
}
Output
Num(14)
I want to append an element to the end of an iterator like this:
val a = Iterator(3, 4)
val b = a + 5 // expect b == Iterator(3,4,5), but there is no + method
val b2 = a ++ Iterator(5) // That works, but not concise.
Is there a better way than b2 to make this?
You can always just hide the not-concise syntax behind something that you like better.
implicit class IterPlus[A](itr: Iterator[A]) {
def +(elem: A) = itr ++ Iterator(elem)
}
val a = Iterator(3, 4)
val b = a + 5 //Iterator(3, 4, 5)
How to find and filter unique values from a text file.
I tried like below, its not working.
val spark = SparkSession.builder().master("local").appName("distinct").getOrCreate()
var data = spark.sparkContext.textFile("text/file/opath")
val uniqueval = data.map { rec => (rec.split(",")(3).distinct) }
var fils = data.filter(line => line.split(",")(3).equals(uniqueval)).map(x => (x)).foreach { println }
Sample Data:
ID | Name
1 john
1 john
2 david
3 peter
4 steve
Required Output:
1 john
2 david
3 peter
4 steve
You almost have it right. .distinct() must just be called on the RDD.
I'd replace statement 3 with:
val uniqueval = data.distinct().map...
This assumes that similar records will have identical lines in the text file.
Is core scala allowed?
scala> val text = List ("single" , "double", "mono", "double")
text: List[String] = List(single, double, mono, double)
scala> val u = text.distinct
u: List[String] = List(single, double, mono)
scala> val d = text.diff(u)
d: List[String] = List(double)
scala> val s = u.diff (d)
s: List[String] = List(single, mono)
your code can be something like:
sparkContext.textFile("sample-data.txt").distinct()
.saveAsTextFile("sample-data-dist.txt");
distinct method can do the action you want.
I've defined a case class and a value:
scala> case class N(a:Int, b:Int)
defined class N
scala> val nnn = N(2,3)
nnn: N = N(2,3)
I would like to modify a field based on an optional value, t1 and t2 with type Option[Int], this is what i did:
val nnn1 = t1.map( x => nnn.copy( a = x)).getOrElse(nnn)
val nnn2 = t2.map( x => nnn1.copy( b = x)).getOrElse(nnn1)
Theres a Lens/Monocle/Scalaz way to do it generic?
A good approach colud be
def someF(i:Int) = // a great computation with i :)
val na = t1.map(someF).getOrElse(nnn.a)
val newnnn = nnn.copy(a = na, b = t2.getOrElse(nnn.b))
In a case where you have to apply someF to t1 is probably a good idea separate it.
No lens for you.
I'm working on Spark in Scala and I want to transform
Array[(Double, Double)] = Array((0.9398785848878621,1.0), (0.25788885483788343,1.0), (0.6093264774118677,1.0), (0.19736451516248585,0.0), (0.9952925254744414,1.0), (0.6920511147023924,0.0...
into something like
Array[Double]=Array(0.9398785848878621, 0.25788885483788343, 0.6093264774118677, 0.19736451516248585, 0.9952925254744414 , 0.6920511147023924 ...
How can I do it?
Then how can I use this Array[Double] to create a diagonal matrix ?
Just take the first part of your tuple :
val a = Array((0.9398785848878621,1.0), (0.25788885483788343,1.0))
val result = a.map(_._1)
Try this:
import org.apache.spark.mllib.linalg.distributed.{CoordinateMatrix, MatrixEntry}
val a = Array((0.9398785848878621,1.0), (0.25788885483788343,1.0), ...)
val res1 = a.map(_._1)
val entries = New Array[Double](res1.length)
for (i <- 0 to res1.length - 1){
entries(i) = MatrixEntry(i,i,res1(i))
}
val mat = CoordinateMatrix(sc.parallelize(res1))