Data analysis on a subset with scala - scala

Question

I'm new to learning Scala and exploring the ways it can do things, and am now trying to learn to implement some slightly more sophisticated data analysis.
I have weather data for various cities in different countries in a text file loaded into the program. I have so far figured out how to calculate simple things like the average temperature across a country per day, or the average temperature of each city grouped by country across the whole file, using Maps/Mapvalues to bind keys to the values I'm looking for.
Now would like to be able to specify a time window (say, a week) and, from there, grouped by country, figure out things like the average temperature of each city in that time window. For simplicity, I've made the dates simple INTs rather than go with MM/DD/YY format.
In another language I would likely go for loops to do this, but I'm not quite sure the best "Scala" way to do this. At first I thought maybe "sliding" or "grouped," but have found this would split the list entirely and thefore I could not specify an arbitrary day to calculate the week from. I've included example code for my method which calculates the average temperature per city over the whole time period
def citytempaverages(): Map[String, Map[String, Double]] = {
weatherpatterns.groupBy(_.country)
.mapValues(_.groupBy(_.city)
.mapValues(cityavg => cityavg.map(_.temperature).sum /cityavg.length))
Does it even still make sense to use maps for this new problem, or perhaps another method in the collections API is more suited?
UPDATE #1: so I've built a collection like so:
def dailycities(): Map[Int, Map[String,Map[String, List[Double]]]] = {
weatherpatterns.groupBy(_.day)
.mapValues(_.groupBy(_.country).mapValues(_.groupBy(_.city)
.mapValues(_.map(_.temperature))))
}
And then created a new map using filterKeys and the Set function to give me back just a list of the days I'm looking for. So I suppose now it's just a matter of formatting to get the averages out correctly.

Answer 1

I would't call it a best way in scala to do this. Rather any way to minimize iteration is the best imo in that case:
def averageOfDay(country: String, city: String, day: Int) = {
val temps = weatherPatterns.collect {
case WeatherPattern(`day`, `country`, `city`, temp) => temp
}
temps.sum / temps.length
}
Edit
I just noticed you mainly need an operation that calculates avgs for all cities and countries. In that case I'd say instead of forming the hierarchical relationship of country -> city -> temp in every operation, you'd rather opt for building the hierarchy once beforhand then operate on that:
case class DailyTemperature(day: Int, temperature: Double)
object DailyTemperature {
def sequence(patterns: List[WeatherPattern]): List[DailyTemperature] =
patterns.map(p => DailyTemperature(p.day,p.temperature))
}
case class CityTempInfo(city: String, dailyTemperatures: List[DailyTemperature])
object CityTempInfo {
def sequence(patterns: List[WeatherPattern]): List[CityTempInfo] =
patterns.groupBy(_.city).map {
case (city, ps) => CityTempInfo(city,DailyTemperature.sequence(ps))
}.toList
}
case class CountryTempInfo(country: String, citiesInfo: List[CityTempInfo])
object CountryTempInfo {
def sequence(patterns: List[WeatherPattern]) =
patterns.groupBy(_.country).map {
case (country, ps) => CountryTempInfo(country, CityTempInfo.sequence(ps))
}.toList
}
now to have your tree of country -> city -> temp you call the CountryTempInfo.sequence and feed it your list of WeatherPatterns. any other method you want to have operate on DailyTemperature,CityTempInfo, of CountryTempInfo can be defined on their respective classes.

Answer 2

I am not sure what exactly you mean when you say that you use "simple ints" for the date, but if it is something sensible, like, for instance "days since epoch", you could fairly easily come up with a grouping function, that maps to weeks:
def weakOf(n: Int, start: Int) = (start + n) / 7
patterns
.groupBy { p => (weakOf(p.day, startDay), p.country, p.city) }
.mapValues(_.map(_.temperature))
.mapValues { s => s.sum / s.size }