Given this case classs:
case class DateX (year: Int, month: Int, day: Int)
And this sequence
val dates = Seq(DateX(2001,1,1),DateX(2002,2,2),DateX(2003,3,3))
I need to convert the sequence of dates into LocalDate, I tried this but doesn't work:
val list = dates.map { x => (x.year, new LocalDate(x.year,x.month,x.day)) }
It says that LocalDate does not have a constructor. How to fix this?
This should get you started. Adjust as needed.
dates.map{ case DateX(y,m,d) => java.time.LocalDate.of(y,m,d) }
// res0: Seq[java.time.LocalDate] = List(2001-01-01, 2002-02-02, 2003-03-03)
Related
case class Transaction(
transactionId: String,
accountId: String,
transactionDay: Int,
category: String,
transactionAmount: Double)
I created a list like this:
val transactions: List[Transaction] = transactionslines.map { line =>
val split = line.split(',')
Transaction(split(0), split(1), split(2).toInt, split(3), split(4).toDouble)
}.toList
Below command gives the maximum transaction amount by considering all the days but I want to use a filter so that it will consider only previous 5 days of the transaction from the current date and display the maximum value. Can anyone help me in implementing the filter function in the below command
val a = transactions.groupBy(_.accountId).mapValues(trans => trans.map(amount => amount.transactionAmount).max).foreach(println)
Output:
(A3,928.88)
(A14,990.14)
(A44,990.11)
(A7,924.59)
(A25,979.22)
(A48,797.19)
Not sure I understand how your transactionDay column of value 1-29 is correlated to the current date. Assuming that it represents the number of days before the current day, your filter can be something like the following:
val a = transactions.
filter(trans => trans.transactionDay <= 5).
groupBy(_.accountId).
mapValues(trans => trans.map(amount => amount.transactionAmount).max)
I actually want to compute the duration and sort the results by descending order.
I have tried,
db.run(totalSuccess.sortBy(ele => computeDuration(ele.startedOn, ele.completedOn).desc))
private def computeDuration(d1: Option[DateTime], d2: Option[DateTime]) = (d1, d2) match {
case (Some(d1),Some(d2)) => new Period(d1, d2, PeriodType.hours())
case _ => None
}
Getting a type mismatch error where ele.startedOn is Rep[Option[DateTime] and computeDuration expects Option[DateTime].
Will it work ? or is there any other simple way to do this.
Do it at the database level using database functions.
def diff(d1: Rep[DateTime], d2: Rep[DateTime]): Rep[Long] = {
SimpleFunction.binary[DateTime, DateTime, Long]("datediff").apply(d1, d2)
}
replace datediff with appropriate database function for your database solution
My input file is below . It contains some purchase details for each Customer.
Input:
100,Surender,2015-01-23,PHONE,20000
100,Surender,2015-01-24,LAPTOP,25000
101,Ajay,2015-02-21,LAPTOP,40000
101,Ajay,2015-03-10,MUSIC_SYSTEM,50000
102,Vikram,2015-07-20,WATCH,60000
My requirement is I would like to find out the latest Purchase details for each Customer .
So the expected output is
Expected OutPut:
List(101,Ajay,2015-03-10,MUSIC_SYSTEM,50000)
List(100,Surender,2015-01-24,LAPTOP,25000)
List(102,Vikram,2015-07-20,WATCH,60000)
I tried the below code and it is giving me the expected output..
But this below logic is somewhat similar to java .
My Scala code :
package pack1
import scala.io.Source
import scala.collection.mutable.ListBuffer
object LatestObj {
def main(args:Array[String])=
{
var maxDate ="0001-01-01"
var actualData:List[String] =List()
var resultData:ListBuffer[String] = ListBuffer()
val myList=Source.fromFile("D:\\Scala_inputfiles\\records.txt").getLines().toList;
val myGrped = myList.groupBy { x => x.substring(0,3) }
//println(myGrped)
for(mappedIterator <- myGrped)
{
// println(mappedIterator._2)
actualData =mappedIterator._2
maxDate=findMaxDate(actualData)
println( actualData.filter { x => x.contains(maxDate) })
}
}
def findMaxDate( mytempList:List[String]):String =
{
var maxDate ="0001-01-01"
for(m <- mytempList)
{
var transDate= m.split(",")(2)
if(transDate > maxDate)
{
maxDate =transDate
}
}
return maxDate
}
}
Could some one help me on trying the same approach in a simpler way using scala?
Or The above code is the only way to achieve that logic?
Even simpler version, also using a case class with coincidentally the same name. Doesn't remove bad records like Tzach's, though, and I leave everything as String.
case class Record(id: String, name: String, dateString: String, item: String, count: String)
myList.map { line =>
val Array(id, name, dateString, item, count) = line.split(",")
Record(id, name, dateString, item, count)
}
.groupBy(_.id)
.map(_._2.maxBy(_.dateString))
.toList
Here's a simple version using groupBy and reduce, plus using a convenient case class to elegantly represent records:
case class Record(id: Int, username: String, date: Date, product: String, cost: Double)
val dateFormat: SimpleDateFormat = new SimpleDateFormat("yyyy-MM-dd")
val stringList = Source.fromFile("./records.txt").getLines().toList
// split by comma and parse into case class - while REMOVING bad records
val records = stringList.map(_.split(",")).collect {
case Array(id, username, date, product, cost) => Record(id.toInt, username, dateFormat.parse(date), product, cost.toDouble)
}
// group by key, and reduce each group to latest record
val result = records.groupBy(_.id).map { _._2.reduce {
(r1: Record, r2: Record) => if (r1.date.after(r2.date)) r1 else r2
}}
result.foreach(println)
// prints:
// Record(101,Ajay,Tue Mar 10 00:00:00 IST 2015,MUSIC_SYSTEM,50000.0)
// Record(100,Surender,Sat Jan 24 00:00:00 IST 2015,LAPTOP,25000.0)
// Record(102,Vikram,Mon Jul 20 00:00:00 IDT 2015,WATCH,60000.0)
Note that this implementation does not make any use of mutable variables or collections, which often simplifies the code significantly, and is considered more idiomatic for functional languages like Scala.
I've got a Scalding data flow that starts with a bunch of Pipe separated value files. The first column is a DateTime in a slightly non-standard format. I want to use the strongly typed TypedPipe API, so I've specified a tuple type and a case class to contain the data:
type Input = (DateTime, String, Double, Double, String)
case class LatLonRecord(date : DateTime, msidn : String, lat : Double, lon : Double, cellname : String)
however, Scalding doesn't know how to coerce a String into a DateTime, so I tried adding an implicit function to do the dirty work:
implicit def stringToDateTime(dateStr: String): DateTime =
DateTime.parse(dateStr, DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.S"))
However, I still get a ClassCastException:
val lines: TypedPipe[Input] = TypedPipe.from(TypedPsv[Input](args("input")))
lines.map(x => x._1).dump
//cascading.flow.FlowException: local step failed at java.lang.Thread.run(Thread.java:745)
//Caused by: cascading.pipe.OperatorException: [FixedPathTypedDelimite...][com.twitter.scalding.RichPipe.eachTo(RichPipe.scala:509)] operator Each failed executing operation
//Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.joda.time.DateTime
What do I need to do to get Scalding to call my conversion function?
So I ended up doing this:
case class LatLonRecord(date : DateTime, msisdn : String, lat : Double, lon : Double, cellname : String)
object dateparser {
val format = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.S")
def parse(s : String) : DateTime = DateTime.parse(s,format);
}
//changed first column to a String, yuck
type Input = (String, String, Double, Double, String)
val lines: TypedPipe[Input] = TypedPipe.from( TypedPsv[Input]( args("input")) )
val recs = lines.map(v => LatLonRecord(dateparser.parse(v._1), v._2, v._3,v._4, v._5))
But I feel like its a sub-optimal solution. I welcome better answers from people who have been using Scala for more than, say, 1 week, like me.
I am trying to make an List of all dates between 2 given dates in scala. This is the program that I have written:
object testingscala
{
def main(args: Array[String])
{
println(calculateDates(LocalDate.of(2014,1,1), LocalDate.of(2014,2,5)))
}
def calculateDates(from: LocalDate, until: LocalDate): List[LocalDate] =
{
var arr = List[LocalDate]()
var dateList = calculateDatesRecur(from, until) // forward reference extends over definition of variable
dateList
def calculateDatesRecur(from: LocalDate, until: LocalDate): List[LocalDate] =
{
if (from.compareTo(until) > 1) {return arr}
else
{ arr = arr :+ from; calculateDatesRecur(from.plusDays(1), until)}
}
return dateList
}
}
I am relatively new at Scala so I am not able to figure out what is wrong with the implementation.
The function just takes two parameters and prints out all the dates between the two dates. I have used recursion.
That is because Scala have no literal syntax for arrays/lists. You have to go with
var arr = List.empty[LocalDate]
or
var arr = List[LocalDate]()
Not that list isn't array in scala.
Although you haven't asked, I do believe this code could be written in a much more concise way:
object testingscala extends App
{
type LD = LocalDate
println(calculateDatesRecur(LocalDate.of(2014,1,1), LocalDate.of(2014,2,5)))
#annotation.tailrec
def calculateDatesRecur(from: LD, until: LD, xs: List[LD] = List.empty[LD]): List[LD] =
if (from.compareTo(until) > 1) xs.reverse
else calculateDatesRecur(from.plusDays(1), until, from::xs)
}
Haven't tested it though.
And answering your comment:
swap recursive function and dateList variable, so function definition comes first.
...
def calculateDatesRecur(from: LocalDate, until: LocalDate): List[LocalDate] =
{
if (from.compareTo(until) > 1) {return arr}
else
{ arr = arr :+ from; calculateDatesRecur(from.plusDays(1), until)}
}
var dateList = calculateDatesRecur(from, until)
...
you're trying to assign function result, but since it is used like a nested function it is not known apriory so compiler don't know what to set to dateList (actually it do knows, but prevents you from doing nasty things).
Similar to Given a range, getting all dates within that range in Scala
With Lamma Date simply (2014, 5, 5) to (2014, 5, 10) foreach println