Large volume of queries on a large CSV file - mongodb

Question

I am trying to query a large csv file(100gb,+-1.1billion records) for partial matches in the url column of a csv. I aim to query for about 23000 possible matches.
Example input:
url,answer,rrClass,rrType,tlp,firstSeenTimestamp,lastSeenTimestamp,minimumTTLSec,maximumTTLSec,count
maps.google.com.,173.194.112.106,in,a,white,1442011301000,1442011334000,300,300,2
drive.google.com.,173.194.112.107,in,a,white,1442011301000,1442011334000,300,300,2
nokiantires.com.,185.53.179.22,in,a,white,1529534626596,1529534626596,600,600,1
woodpapersilk.,138.201.32.142,in,a,white,1546339972354,1553285334535,3886,14399,2
xn--c1yn36f.cn.,167.160.174.76,in,a,white,1501685257255,1515592226520,14400,14400,38
maps.google.com.malwaredomain.com.,118.193.165.236,in,a,white,1442148766000,1442148766000,600,600,1
whois.ducmates.blogspot.com.,216.58.194.193,in,a,white,1535969280784,1535969280784,44,44,1
Queries are of the following pattern: /^.*[someurl].*$/ each of these [someurls] come from a different file and can be assumed as an array of size 23000.
Matching Queries:
awk -F, '$1 ~ /^.*google\.com\.$/' > file1.out
awk -F, '$1 ~ /^.*nokiantires\.com\.$/' > file2.out
awk -F, '$1 ~ /^.*woodpapersilk\.com\.$/' > file3.out
awk -F, '$1 ~ /^.*xn--.*$/' > file4.out
Queries that match nothing:
awk -F, '$1 ~ /^.*seasonvintage\.com\.$/' > file5.out
awk -F, '$1 ~ /^.*java2s\.com\.$/' > file6.out
file1.out:
maps.google.com.,173.194.112.106,in,a,white,1442011301000,1442011334000,300,300,2
drive.google.com.,173.194.112.107,in,a,white,1442011301000,1442011334000,300,300,2
file2.out:
nokiantires.com.,185.53.179.22,in,a,white,1529534626596,1529534626596,600,600,1
file3.out:
woodpapersilk.,138.201.32.142,in,a,white,1546339972354,1553285334535,3886,14399,2
file4.out:
xn--c1yn36f.cn.,167.160.174.76,in,a,white,1501685257255,1515592226520,14400,14400,38
file 5.out and file6.out are both empty as nothing matches
I have also uploaded these inputs and outputs as a gist.
Essentially each query extracts a partial match in the url column.
Currently I use the following code with awk to search for possible matches:
awk -F, '$1 ~ /^.*xn--.*$/' file.out > filter.csv
This solution returns a valid response but it takes 14 minutes to query for one example. Unfortunately I am looking to query for 23000 possible matches.
As such I am looking for a more workable and efficient solution.
I have thought of/tried the following
Can i include all the tags in a huge regex or does this increase the inefficiency?
I have tried using MongoDB but it does not work well with just a single machine.
I have an AWS voucher that has about $30 left. Is there any particular AWS solution that could help here?
What would be a more workable solution to process these queries on said csv file?
Many thanks

Answer 1

Given what we know so far and guessing at the answers to a couple of questions, I'd approach this by separating the queries into "queries that can be matched by a hash lookup" (which is all but 1 of the queries in your posted example) and "queries that need a regexp comparison to match" (just xn--.*$ in your example) and then evaluating them as such when reading your records so that any $1 that can be matched by an almost instantaneous hash lookup against all hash-able queries will be done like that and only the few that need a regexp match will be handled sequentially in a loop:
$ cat ../queries
google.com.$
nokiantires.com.$
woodpapersilk.com.$
xn--.*$
seasonvintage.com.$
java2s.com.$
$ cat ../records
url,answer,rrClass,rrType,tlp,firstSeenTimestamp,lastSeenTimestamp,minimumTTLSec,maximumTTLSec,count
maps.google.com.,173.194.112.106,in,a,white,1442011301000,1442011334000,300,300,2
drive.google.com.,173.194.112.107,in,a,white,1442011301000,1442011334000,300,300,2
nokiantires.com.,185.53.179.22,in,a,white,1529534626596,1529534626596,600,600,1
woodpapersilk.,138.201.32.142,in,a,white,1546339972354,1553285334535,3886,14399,2
xn--c1yn36f.cn.,167.160.174.76,in,a,white,1501685257255,1515592226520,14400,14400,38
maps.google.com.malwaredomain.com.,118.193.165.236,in,a,white,1442148766000,1442148766000,600,600,1
whois.ducmates.blogspot.com.,216.58.194.193,in,a,white,1535969280784,1535969280784,44,44,1
.
$ cat ../tst.awk
BEGIN { FS="," }
NR==FNR {
query = $0
outFile = "file" ++numQueries ".out"
printf "" > outFile; close(outFile)
if ( query ~ /^[^.]+[.][^.]+[.][$]$/ ) {
# simple end of field string, can be hash matched
queriesHash[query] = outFile
}
else {
# not a simple end of field string, must be regexp matched
queriesRes[query] = outFile
}
next
}
FNR>1 {
matchQuery = ""
if ( match($1,/[^.]+[.][^.]+[.]$/) ) {
fldKey = substr($1,RSTART,RLENGTH) "$"
if ( fldKey in queriesHash ) {
matchType = "hash"
matchQuery = fldKey
outFile = queriesHash[matchQuery]
}
}
if ( matchQuery == "" ) {
for ( query in queriesRes ) {
if ( $1 ~ query ) {
matchType = "regexp"
matchQuery = query
outFile = queriesRes[matchQuery]
break
}
}
}
if ( matchQuery != "" ) {
print "matched:", matchType, matchQuery, $0, ">>", outFile | "cat>&2"
print >> outFile; close(outFile)
}
}
.
$ ls
$
$ tail -n +1 *
tail: cannot open '*' for reading: No such file or directory
.
$ awk -f ../tst.awk ../queries ../records
matched: hash google.com.$ maps.google.com.,173.194.112.106,in,a,white,1442011301000,1442011334000,300,300,2 >> file1.out
matched: hash google.com.$ drive.google.com.,173.194.112.107,in,a,white,1442011301000,1442011334000,300,300,2 >> file1.out
matched: hash nokiantires.com.$ nokiantires.com.,185.53.179.22,in,a,white,1529534626596,1529534626596,600,600,1 >> file2.out
matched: regexp xn--.*$ xn--c1yn36f.cn.,167.160.174.76,in,a,white,1501685257255,1515592226520,14400,14400,38 >> file4.out
.
$ ls
file1.out file2.out file3.out file4.out file5.out file6.out
$
$ tail -n +1 *
==> file1.out <==
maps.google.com.,173.194.112.106,in,a,white,1442011301000,1442011334000,300,300,2
drive.google.com.,173.194.112.107,in,a,white,1442011301000,1442011334000,300,300,2
==> file2.out <==
nokiantires.com.,185.53.179.22,in,a,white,1529534626596,1529534626596,600,600,1
==> file3.out <==
==> file4.out <==
xn--c1yn36f.cn.,167.160.174.76,in,a,white,1501685257255,1515592226520,14400,14400,38
==> file5.out <==
==> file6.out <==
$
The initial printf "" > outFile; close(outFile) is just to ensure you get an output file per query even if that query oesn't match, just like you asked for in your example.
If you're using GNU awk then it can manage the multiple open output files for you and then you can make these changes:
printf "" > outFile; close(outFile) -> printf "" > outFile
print >> outFile; close(outFile) -> print > outFile
which will be more efficient because then the output file isn't being opened+closed on every print.