performance reading a file in SCALA - scala

Question

I am reading a file that contain 277272 lines with Int triples (s,p,o) like:
2,126088,113358
2,126101,126102
2,126106,126107
2,126111,126112
2,126128,126129
2,126136,126137
2,126141,126142
2,126146,126147
4,1,3
4,7,8
4,15,16
4,26,27
4,41,94825
4,63,15764
4,68,69
4,89,94836
4,90,91
4,93,94
4,105,94844
this triples are order by s and then by p. I write the following code that search a value by s, and once find the value I write a conditional to filter by p.
def SP_O (yi:Int, zi:Int):List[Vector[Int]]={
val file = new RandomAccessFile("patSPO.dat", "r")
var lengthfile = (file.length().asInstanceOf[Int])
var initpointer = lengthfile/2
while (band == true){
var p = bs.firstvalue(initpointer,"patSPO.dat") // p es la lista del valor del predicado con su respectivo puntero
var newpointer = p(0)
var subj = p(1)
//Comienzo el analisis para saber si el predicado es igual al del BGP
if (subj == yi){
// Hay que imprimir los valores de la presente linea
if (firsttime == true){
val fields = bs.getvalueSP_O(newpointer,zi)
if (fields.size != 0){
S_POlistbuffer += fields
//yld(Record(fields, schema))
}
firsttime = false
pointerad = newpointer
pointerat = newpointer
} // Ahora tengo que analizar si hay valores atras o adelante
while (boolad == true || boolat == true){
if (boolat == true) {
//Valores para atras
pointerat -= 6
if (pointerat <= 0) {
pointerat = 0
boolat = false
}
p = bs.firstvalue(pointerat, "patSPO.dat") // p ds la lista del valor del sujeto en pointerat y da su respectivo pointer
if (p(1) == yi) {
val fields = bs.getvalueSP_O(p(0),zi)
if (fields.size != 0){
S_POlistbuffer += fields
//yld(Record(fields, schema))
}
} else {
boolat = false
}
}
if (boolad == true){ //valores par adelante
p = bs.nextvalue(pointerad,"patSPO.dat")
if (p(1)==yi){
val fields = bs.getvalueSP_O(p(0),zi)
pointerad = p(0)
if (fields.size != 0){
S_POlistbuffer += fields
//yld(Record(fields, schema))
}
}else{
boolad = false
}
}
if (boolad == false && boolat == false){
band = false
}
}
}
else if ( subj > yi ){
initpointer = initpointer/2
}
else if ( subj < yi) {
initpointer = (initpointer*6)/4
}
}
val listf:List[Vector[Int]] = S_POlistbuffer.toList
listf
}
The general idea is that the code start looking the value from the half of the file, with the filerandomaccess, then I should extract the first value of the line, and compare with the value that I need, once I find the correct value I must analyze if the next line also has the correct value, in paralell I analyze the line before to the line that I choose in order to see if that line also is a correct value. For each line that match I analyze if the second value is the correct. If the line match I extract the o value, and I store it in a List.
The problem is when I am printing the result, this take so much time. However I developed this another solution that runs over the whole file:
while (in.hasNext) {
val s = in.next(',').toInt
val p = in.next(',').toInt
val o = in.next('\n').toInt
//val fields = schema map (n => in.next(if (n == schema.last) '\n' else ','))
if (p == yi && s == xi){
val fields = schema map (n => o)
yld(Record(fields, schema))
}
}
With this code I run over the whole file and I get the results faster than the first code. My big question is why if the first code in the best of the cases just run a portion of the file is to slow than the second code that run over the whole file? Is there another way to write this code with a better performance?
The time of the first code execution is like 750 seconds, the second code takes like 10 seconds.

Answer 1

There's two important pieces of information that I think you missed here :
There's no such thing as reading a file "in parallel". Even nowadays, a lot of disk reading is spinning the damn disk until it's in the correct position, and read. So, you can multi thread all you want, the more concurrent disk access you do, the worst read performance you get. A simplified view is that every time you go "backwards", you actually make your disk do a full spin forward to get to that previous line !
Reading from disk sequentially is multiple orders of magnitude faster than random disk access. This is correlated to 1, but really, reading a file line by line is the perfect way to access a disk ! And as it turns out, it's what your naive version do !
Which leads me to believe those are the logical next steps :
Apply your smart algorithm to the file having previously loaded it in memory
Compare again with the naive version
Find no significant difference
Conclude that your workflow is definitely I/O bound, and stop trying to optimize the part that doesn't matter in it !
Note that your mileage may vary if you have a SSD (I assumed not, since there's really no way I can think of a SSD would need 750 seconds to random access lines in it).