How can I process this big data in scala IntelliJ? - scala

Question

It has been a few days and i started learning Scala on IntelliJ and I am learning all by myself. Please bear my rookie mistakes. I have a csv file with more than 10,000 rows and 13 columns.
The heading of of the columns are:
Category | Rating | Reviews | Size | Installs | Type | Price | Content Rating | Genres | Last updated | Current Version | Android Version
I did manage to read and display the the csv file with the following code:
import scala.io.Source
object task {
def main(args: Array[String]): Unit = {
for(line <- Source.fromFile("D:/data.csv"))
{
println(line)
}
}
}
The problem with this is that this code displays one alphabet or digit, moves onto the next line and displays the next alphabet or digit. It does not display a row in one line.
I want to find out the best app for each category (ART_AND_DESIGN, AUTO_AND_VEHICLES, BEAUTY…,) based on its assigned priorities of reviews and ratings. The priorities are defined as 60 % for “reviews” and 40% for “rating” columns respectively. Calculate a value for each category (ART_AND_DESIGN, AUTO_AND_VEHICLES, BEAUTY…,) by using these assigned values of priorities. This value will help us out to find the best app in each category. You can use Priority formula equation as follows.
Priority = ( (((rating/max_rating) * 100) * 0.4) + (((reviews/max_reviews) * 100) * 0.6) )
Here max_rating is maximum rating of given data in same category like category(“ART_AND_DESIGN”) maximum rating is “4.7”, max_reviews is maximum reviews of app in same category like category(“ART_AND_DESIGN”) maximum reviews is “295221”. So priority value will be for first data record of category(“ART_AND_DESIGN”) is:
Rating= 4.1, reviews= 159,
max_rating= 4.7, max_reviews= 295221
My question is, how can i store every column in an array? That is how i plan on computing the data. If there is any other way to solve the above problem, i am open to suggestions.
I can upload a small chunk of the data if anyone wants to.

Answer 1

Source gives you a byte Iterator by default. To iterate through lines, use .getLines:
Source.fromFile(fileName)
.getLines
.foreach(println)
To split lines into arrays, use split (assuming the column values do not include separator):
val arrays = Source.fromFile(fileName).getLines.map(_.split("|"))
It is better to avoid using raw arrays though. Creating a case class makes for much better, readable code:
case class AppData(
category: String,
rating: Int,
reviews: Int,
size: Int,
installs: Int,
`type`: String,
price: Double,
contentRating: Int,
generes: Seq[String],
lastUpdated: Long,
version: String,
androidVersion: String
) {
def priority(maxRating: Int, maxReview: Int) =
if(maxRatings == 0 || maxReviews == 0) 0 else
(rating * 0.4 / maxRating + reviews * 0.6 /maxReview) * 100
}
object AppData {
def apply(str: String) = {
val fields = str.split("|")
assert(fields.length == 12)
AppData(
fields(0),
fields(1).toInt,
fields(2).toInt,
fields(3).toInt,
fields(4).toInt,
fields(5),
fields(6).toDouble,
fields(7).toInt,
fields(8).split(",").toSeq,
fields(9).toLong,
fields(10),
fields(11)
)
}
}
Now you can do what you want pretty neatly:
// Read the data, parse it and group by category
// This gives you a map of categories to a seq of apps
val byCategory = Source.fromFile(fileName)
.map(AppData)
.groupBy(_.category)
// Now, find out max ratings and reviews for each category
// This could be done even nicer with another case class and
// a monoid, but tuple/fold will do too
// It is tempting to use `.mapValues` here, but that's not a good idea
// because .mapValues is LAZY, it will recompute the max every time
// the value is accessed!
val maxes = byVategory.map { case (cat, data) =>
cat ->
data.foldLeft(0 -> 0) { case ((maxRatings, maxReviews), in) =>
(maxRatings max in.rating, maxReviews max in.reviews)
}
}.withDefault( _ => (0,0))
// And finally go through your categories, and find best for each,
// that's it!
val bestByCategory = byCategory.map { case(cat, apps) =>
cat -> apps.maxBy { _.priority.tupled(maxes(cat)) }
}