Converting string features to numeric features: algorithm efficiency - scala

Question

I'm converting several columns of strings to numeric features I can use in a LabeledPoint. I'm considering two approaches:
Create a mapping of strings to doubles, iterate through the RDD and lookup each string and assign the appropriate value.
Sort the RDD by the column, iterate through the RDD with a counter, assign each string to the current counter value until the string changes at which time the counter value is incremented and assigned. Since we never see a string twice (thanks to sorting) this will effectively assign a unique value to each string.
In the first approach we must collect the unique values for our map. I'm not sure how long this takes (linear time?). Then we iterate through the list of values and build up a HashMap - linear time and memory. Finally we iterate and lookup each value, N * eC (effective constant time).
In the second approach we sort (n log n time) and then iterate and keep track of a simple counter and a few variables.
What approach is recommended? There are memory, performance, and coding style considerations. The first feels like 2N + eC * N with N * (String, Double) memory and can be written in a functional style. The second is N log N + N but feels imperative. Will Spark need to transfer the static map around? I could see that being a deal breaker.

Answer 1

The second method unfortunately won't work the reason is you can not read form counter you can only increment it. What is even worst you dont really know when value changes you dont have state to remember previous vector. I guess you could use something like mapPartition and total order partitioner. You would have to know that your partitions are processed in order and there cant be same keys in more then one partition but this feels really hacky (and i dont know if it would work).
I dont think its possible to do this in one pass. But you can do it in two. In your first method you can use for example set accumulator put all you values in it then number them in driver and use in second pass to replace them. The complexity would be 2N (assuming that number of values << N).
Edit:
implicit object SetAcc extends AccumulatorParam[Set[String]] {
def zero(s: Set[String]) = Set()
def addInPlace(s1: Set[String], s2: Set[String]) = s1 ++ s2
}
val rdd = sc.parallelize(
List((1, "a"), (2, "a"), (3, "b"), (4, "a"), (5, "c"), (6, "b"))
)
val acc: Accumulator[Set[String]] = sc.accumulator(Set())
rdd.foreach(p => acc += Set(p._2))
val encoding = acc.value.zipWithIndex.toMap
val result = rdd map {p => (p._1, encoding(p._2))}
If you feel like this dictionary is too big you can of course brodcast it. If you have to many features and values in them and you dont want to create so many big accumulators then you can just use reduce function to process them all together and collect on driver. Just my thoughts about it. I guess you just have to try and see whats suits the best your usecase.
Edit:
In mllib there is class meant for this purpose HashingTF. It allows you to translate you data set in one pass. The drawback is that it uses hashing modulo specified parameter to map Objects to Doubles. This can lead to collisions if parameter is too small.
val tf = new HashingTF(numFeatures = 10000)
val transformed = data.map(line => tf.transform(line.split("""\s+"""))
Ofc you can do the same thing by hand without using HashingTF class.