Spark : subtractByKey issue (pyspark) - pyspark

Question

I have some kinds of issues with subtractByKey.
I have 2 files :
First one is like : (Client ID + Client Mail)
client_id emails
4A85FD8E-197D-2AE3-B939-A527AFF16A04 imperdiet.non.vestibulum#mon***tur.com
D48D530C-CF68-DAF1-18F0-E0A0A03F3E06 rutrum.urna#estm***ncus.net:facilisis#i****m.ca
40815230-25DC-9EA0-01D1-2706B4B56958 iaculis.nec.eleifend#gr****nc.net
...
and the second one : (Only Mail)
pharetra#P****s.com
ut.aliquam#o****m.org
erat#a****e.edu
....
Some lines in the first file can have 2 (or more) mails with this format :
mail:mail
What did i do :
*test1=sc.textFile("file1")
*test2=sc.textFile("file2")
*test3=test1.subtractByKey(test2)
and the result is ... :
[(u'A', u'B'), (u'A', u'D'), (u'A', u'1'), (u'A', u'D'), (u'A', u'D'), (u'A', u'B'), (u'A', u'F'), (u'A', u'E'), (u'A', u'9'), (u'A', u'5'), (u'A', u'9'), (u'A', u'6'), (u'c', u'l'), (u'E', u'8'), (u'E', u'4'), (u'E', u'6'), (u'E', u'6'), (u'E', u'7'), (u'E', u'5'), (u'E', u'5'), (u'E', u'5'), (u'E', u'2'), (u'E', u'8'), (u'C', u'2'), (u'C', u'5'), (u'C', u'6'), (u'C', u'C'), (u'C', u'E'), (u'C', u'3'), (u'C', u'F'), (u'C', u'4'), (u'C', u'B'), (u'C', u'F'), (u'C', u'F'), (u'C', u'8'), (u'C', u'0'), (u'1', u'D'), (u'1', u'2'), (u'1', u'3'), (u'1', u'8'), (u'1', u'0'), (u'1', u'F'), ... ]
I wanted to delete clients in the first file who had their mails in the second file but it did not work.

Answer 1

note: I am not very familiar with pyspark but the spark api should be the
same.
first you should make email as the key
rdd1=sc.textFile("file1").map(lambda line: (line.split(" ")[0], line.split(" ")[1]))
this will give you a rdd of
[(4A85FD8E-197D-2AE3-B939-A527AFF16A04,imperdiet.non.vestibulum#mon***tur.com)]
then as there may be multi email, you should do a flatMapValues()
rdd2 = rdd1.flatMapValues(lambda email: email.split(":"))
this will give you a pair rdd and each one just contains one email
now you can switch the key and value
rdd3=rdd2.map(lambda kv: (kv[1], kv[0]))
now you get a rdd that using user email as a key and the UUID as a value
such as
[(imperdiet.non.vestibulum#mon***tur.com, 4A85FD8E-197D-2AE3-B939-A527AFF16A04)]
now you should find which UUID's email is contained in file2, to do that you should load the second file as a rdd:
secondRdd = sc.textFile("file2").map(lambda line: (line, 1))
and do a join and tweak the join result rdd.
rdd4 = rdd3.join(secondRdd).map(lambda kv: (kv[1][0], kv[0]))
if everything is right now you should get a rdd which is as the format of (UUID, email) which represents all the users whose email occurs in file2,
then you can do a subtractByKey() with rdd1 which we originally got.