Comparing two email address lists anonymously - email

Question

Given two lists:
Company A:
user1#example.com
user2#example.com
user3#example.com
user4#example.com
Company B:
user2#example.com
user4#example.com
user5#example.com
Is there a way to anonymously compare them to get the number of email addresses in common (i.e., 2) without either company knowing which addresses were the ones in common?
Background:
Let's say that company A and company B want to know what portion of their userbase is common. For simplicity, they are just going to base it on email address and not concern themselves with people who use multiple addresses or different address variations (user+misc#example.com).
For the sake of privacy, neither company can give the other the plain list of email addresses. If they used the same simple hash, e.g. MD5, each company could easily know which members were in common (not desired). If they used a hash salted with a company specific secret, the addresses wouldn't be comparable any longer so the question couldn't be answered.
Is there some trick using key encryption or some other mathematical way to accomplish what I'm looking to do?

Answer 1

I believe this question could be understood better in the realm of cryptography.
It is a problem of secure multi-party computation.
I'm not aware of any bullet proof solution for this problem but I can think of the following:
Choose a commutative hash function (H):
H(H(string, seed1), seed2) = H(H(string, seed2), seed1)
Each party (Company A and Company B) has to choose a secret seed:
SEED_A, SEED_B
Company A hashes all email addresses using SEED_A, Company B hashes all email addresses using SEED_B.
They interchange the hashes.
Each company applies the hash function again on the set received from the opposing party.
At this point the data should already be garbled and the companies should not be able to recognize their own email addresses (since they've been already hashed twice - the second time with an unknown key).
All the email addresses should be laid out openly and those that have the same hash should be counted as the email addresses that belong to both companies (except that neither company can tell the source of the hash).
This is the theory. Hopefully I didn't miss anything and there are no flaws in the algorithm.
As for the implementation, here's the most trivial PHP script that I could come with:
$a = array("user1#example.com", "user2#example.com", "user3#example.com", "user4#example.com");
$b = array("user2#example.com", "user4#example.com", "user5#example.com");
function enc($str, $seed) {
for ($i = strlen($str) - 1; $i >= 0; $i--) {
$str[$i] = $str[$i] ^ $seed[$i % strlen($seed)];
}
return $str;
}
/* Company A */
$hashesForB = array();
$SEED_A = 'SALT FOR COMPANY A';
foreach ($a as $address) {
$hashesForB[] = enc($address, $SEED_A);
}
/* Company B */
$hashesForA = array();
$SALT_B = 'THIS IS THE SALT FOR COMPANY B';
foreach ($b as $address) {
$hashesForA[] = enc($address, $SALT_B);
}
/* Company A */
$hashesForB_2 = array();
foreach ($hashesForA as $hash) {
$hashesForB_2[] = enc($hash, $SEED_A);
}
/* Company B */
$hashesForA_2 = array();
foreach ($hashesForB as $hash) {
$hashesForA_2[] = enc($hash, $SALT_B);
}
$common = count(array_intersect($hashesForA_2, $hashesForB_2));
print $common; // it will output 2
Click here for the DEMO
As you can see in the code above, I used the XOR algorithm for (pseudo) hashing (actually, any addition based hash function should do the job).
Obviously, this is not the best choice for many reasons:
XOR will return the original input upon a new call with the same salt
the entropy is not the best you could hope for
the data is not truncated
Still, you could implement your own hashing function using the suggestions here, here, here or here.

Answer 2

Is the privacy concern that privacy agreement prohibit sharing of email addresses? or is it a competitive concern?
If you just want to get an idea of percentage of overlap, then I'd think a simple encoding of the email addresses might work. For example, de-dupe each list, Base64 encode each email address, then run the comparison to get overlap, then report on the numbers.
A simple NDA could make this a less technical problem.

Answer 3

It depends the language you want to use.
In python, you could use this script :
listA = ('user1#example.com', 'user2#example.com', 'user3#example.com')
listB = ('user1#example.com', 'user2#example.com')
result = [x for x in listA if x in listB]
print(len(result))
For security, you could host this script in an external server where both companies just can put in their lists and then check the result.