Fuzzy Compare between two hive columns using apache spark with scala - scala

Question

I am reading the data from 2 hive tables. Token table has the tokens that needs to be matched with the input data. Input data will have description column along with other columns. I need to split input data and need to compare each splitted element with all the elements from the token table.
currently I am using me.xdrop.fuzzywuzzy.FuzzySearch library for fuzzy match.
below is my code snippet-
val tokens = sqlContext.sql("select token from tokens")
val desc = sqlContext.sql("select description from desceriptiontable")
val desc_tokens = desc.flatMap(_.toString().split(" "))
Now i need to iterate desc_tokens and each element of desc_tokens should be fuzzy matched with each element of tokens and it it exceeds 85% match i need to replace element from desc_tokens by element from the tokens.
Example --
My token list is
hello
this
is
token
file
sample
and my input description is
helo this is input desc sampl
code should return
hello this is input desc sample
as hello and helo are fuzzy matched > 85% so helo will be replaced by hello. Similarly for sampl.

Answer 1

I make a test with this library : https://github.com/rockymadden/stringmetric
Other idea (Not optimized) :
//I change order tokens
val tokens = Array("this","is","sample","token","file","hello");
val desc_tokens = Array("helo","this","is","token","file","sampl");
val res = desc_tokens.map(str => {
//Compute score beetween tokens and desc_tokens
val elem = tokens.zipWithIndex.map{ case(tok,index) => (tok,index,JaroMetric.compare(str, tok).get)}
//Get token has max score
val emax = elem.maxBy{case(_,_,score) => score}
//if emax have a score > 0.85 get It. Else keep input
if(emax._3 > 0.85) tokens(emax._2) else str
})
res.foreach { println }
My Output :
hello
this
is
token
file
sample