How to cluster based on ssdeep? - perl

Question

Hi I am trying to find the groups out of files based on ssdeep.
I have generated ssdeep of files and kept it in csv file.
I am parsing the file in perl script as follows:
foreach( #all_lines )
{
chomp;
my $line = $_;
my #split_array = split(/,/, $line);
my $md5 = $split_array[1];
my $ssdeep = $split_array[4];
my $blk_size = (split(/:/, $ssdeep))[0];
if( $blk_size ne "")
{
my $cluster_id = check_In_Cluster($ssdeep);
print WFp "$cluster_id,$md5,$ssdeep\n";
}
}
This also checks whether the ssdeep is present in previously clustered group and if not creates new group.
Code for chec_In_Cluster
my $ssdeep = shift;
my $cmp_result;
if( $cluster_cnt > 0 ) {
$cmp_result = ssdeep_compare( $MRU_ssdeep, $ssdeep );
if( $cmp_result > 85 ) {
return $MRU_cnt;
}
}
my $d = int($cluster_cnt/4);
my $thr1 = threads->create(\&check, 0, $d, $ssdeep);
my $thr2 = threads->create(\&check, $d, 2*$d, $ssdeep);
my $thr3 = threads->create(\&check, 2*$d, 3*$d, $ssdeep);
my $thr4 = threads->create(\&check, 3*$d, $cluster_cnt, $ssdeep);
my ($ret1, $ret2, $ret3, $ret4);
$ret1 = $thr1->join();
$ret2 = $thr2->join();
$ret3 = $thr3->join();
$ret4 = $thr4->join();
if($ret1 != -1) {
$MRU_ssdeep = $ssdeep;
$MRU_cnt = $ret1;
return $MRU_cnt;
} elsif($ret2 != -1) {
$MRU_ssdeep = $ssdeep;
$MRU_cnt = $ret2;
return $MRU_cnt;
} elsif($ret3 != -1) {
$MRU_ssdeep = $ssdeep;
$MRU_cnt = $ret3;
return $MRU_cnt;
} elsif($ret4 != -1) {
$MRU_ssdeep = $ssdeep;
$MRU_cnt = $ret4;
return $MRU_cnt;
} else {
$cluster_base[$cluster_cnt] = $ssdeep;
$MRU_ssdeep = $ssdeep;
$MRU_cnt = $cluster_cnt;
$cluster_cnt++;
return $MRU_cnt;
}
and the code for chech:
sub check($$$) {
my $from = shift;
my $to = shift;
my $ssdeep = shift;
for( my $icnt = $from; $icnt < $to; $icnt++ ) {
my $cmp_result = ssdeep_compare( $cluster_base[$icnt], $ssdeep );
if( $cmp_result > 85 ) {
return $icnt;
}
}
return -1;
}
But this process takes very much time( for 20-30MB csv file it takes 8-9Hours).
I have also tried using multithreading while checking in Cluster but not much help i got from this.
Since their is no need of csv parser like Text::CSV (because of less operation on csv) i didn't used it.
can anybody please solve my issue? Is it possible to use hadoop or some other frameworks for grouping based on ssdeep?

Answer 1

There is a hint from Optimizing ssDeep for use at scale (2015-11-27).
Depends on your purpose, loop and match SSDEEP in different chunk size will create a N x (N-1) hash comparison. Unless you need to find partial contents, otherwise, avoid it.
It is possible to breakdown of the hash index in step 1 as suggested in the article. This is a better way for partial contents match with different chunk size.
It is possible to reduce SSDEEP hash by grouping similar hash by generate a "distance cousin" hash.