I am trying to calculae the interval of 'n' days from start date to end date.Function signature would have start_date,end_date, interval as argument which return a map with list of start, end days of given intervals.
Example: start_date:2018-01-01 , End_date : 2018-02-20 interval: 20
Expected Output:
2018-01-01 , 2018-01-20 (20 days)
2018-01-21 , 2018-02-09 (20 days)
2018-02-09 , 2018-01-20 (remaining)
I tried to write in scala but i dont feel it's a proper functional style of writing.
case class DateContainer(period: String, range: (LocalDate, LocalDate))
def generateDates(startDate: String, endDate: String,interval:Int): Unit = {
import java.time._
var lstDDateContainer = List[DateContainer]()
var start = LocalDate.parse(startDate)
val end = LocalDate.parse(endDate)
import java.time.temporal._
var futureMonth = ChronoUnit.DAYS.addTo(start, interval)
var i = 1
while (end.isAfter(futureMonth)) {
lstDDateContainer = DateContainer("P" + i, (start, futureMonth)):: lstDDateContainer
start=futureMonth
futureMonth = ChronoUnit.DAYS.addTo(futureMonth, interval)
i += 1
}
lstDDateContainer= DateContainer("P" + i, (start, end))::lstDDateContainer
lstDDateContainer.foreach(println)
}
generateDates("2018-01-01", "2018-02-20",20)
Could anyone help me to write in a functional style.
I offer a solution that produces a slightly different result than given in the question but could be easily modified to get the desired answer:
//Preliminaries
val fmt = DateTimeFormatter.ofPattern("yyyy-MM-dd")
val startDate ="2018-01-01"
val endDate = "2018-02-21"
val interval = 20L
val d1 = LocalDate.parse(startDate, fmt)
val d2 = LocalDate.parse(endDate, fmt)
//The main code
Stream.continually(interval)
.scanLeft((d1, d1.minusDays(1), interval)) ((x,y) => {
val finDate = x._2.plusDays(y)
if(finDate.isAfter(d2))
(x._2.plusDays(1), d2, ChronoUnit.DAYS.between(x._2, d2))
else
(x._2.plusDays(1), x._2.plusDays(y), y)
}).takeWhile(d => d._3 > 0).drop(1).toList
Result:
(2018-01-01,2018-01-20,20)
(2018-01-21,2018-02-09,20)
(2018-02-10,2018-02-21,12)
The idea is to scan a 3-tuple through a stream of interval and stop when no more days are remaining.
Use the java.time library to generate the dates and Stream.iterate() to generate the sequence of intervals.
import java.time.LocalDate
def generateDates( startDate :LocalDate
, endDate :LocalDate
, dayInterval :Int ) :Unit = {
val intervals =
Stream.iterate((startDate, startDate plusDays dayInterval-1)){
case (_,lastDate) =>
val nextDate = lastDate plusDays dayInterval
(lastDate plusDays 1, if (nextDate isAfter endDate) endDate
else nextDate)
}.takeWhile(_._1 isBefore endDate)
println(intervals.mkString("\n"))
}
usage:
generateDates(LocalDate.parse("2018-01-01"), LocalDate.parse("2018-02-20"), 20)
// (2018-01-01,2018-01-20)
// (2018-01-21,2018-02-09)
// (2018-02-10,2018-02-20)
Something like (Untested):
def dates(startDate: LocalDate, endDate: LocalDate, dayInterval: Int): List[(LocalDate, LocalDate, Int)] = {
if(startDate.isAfter(endDate)) {
Nil
}
else {
val nextStart = startDate.plusDays(dayInterval)
if(nextStart.isAfter(startDate)) {
List((startDate, endDate, ChronoUnit.DAYS.between(startDate, endDate)))
}
else {
(startDate, nextStart, dayInterval) :: dates(nextStart, endDate, dayInterval)
}
}
}
If your'e open to using Joda for date-time manipulations, here's what I use
import org.joda.time.{DateTime, Days}
// given from & to dates, find no of days elapsed in between (integer)
def getDaysInBetween(from: DateTime, to: DateTime): Int = Days.daysBetween(from, to).getDays
def getDateSegments(from: DateTime, to: DateTime, interval: Int): Seq[(DateTime, DateTime)] = {
// no of days between from & to dates
val days: Int = DateTimeUtils.getDaysInBetween(from, to) + 1
// no of segments (date ranges) between to & from dates
val segments: Int = days / interval
// (remaining) no of days in last range
val remainder: Int = days % interval
// last date-range
val remainderRanges: Seq[(DateTime, DateTime)] =
if (remainder != 0) from -> from.plusDays(remainder - 1) :: Nil
else Nil
// all (remaining) date-ranges + last date-range
(0 until segments).map { segment: Int =>
to.minusDays(segment * interval + interval - 1) -> to.minusDays(segment * interval)
} ++ remainderRanges
}
Related
i am trying to pass a list of date ranges needs to be in the below format.
val predicates =
Array(
“2021-05-16” → “2021-05-17”,
“2021-05-18” → “2021-05-19”,
“2021-05-20” → “2021-05-21”)
I am then using map to create a range of conditions that will be passed to the jdbc method.
val predicates =
Array(
“2021-05-16” → “2021-05-17”,
“2021-05-18” → “2021-05-19”,
“2021-05-20” → “2021-05-21”
).map { case (start, end) =>
s"cast(NEW_DT as date) >= date ‘$start’ AND cast(NEW_DT as date) <= date ‘$end’"
}
The process will need to run daily and i need to dynamically populate these values as i cannot use the hard coded way.
Need help in how i can return these values from a method with incrementing start_date and end_date tuples that can generate like above.I had a mere idea like below but as i am new to scala not able to figure out. Please help
def predicateRange(start_date: String, end_date: String): Array[(String,String)] = {
// iterate over the date values and add + 1 to both start and end and return the tuple
}
This assumes that every range is the same duration, and that each date range starts the next day after the end of the previous range.
import java.time.LocalDate
import java.time.format.DateTimeFormatter
def dateRanges(start: String
,rangeLen: Int
,ranges: Int): Array[(String,String)] = {
val startDate =
LocalDate.parse(start, DateTimeFormatter.ofPattern("yyyy-MM-dd"))
Array.iterate(startDate -> startDate.plusDays(rangeLen), ranges){
case (_, end) => end.plusDays(1) -> end.plusDays(rangeLen+1)
}.map{case (s,e) => (s.toString, e.toString)}
}
usage:
dateRanges("2021-05-16", 1, 3)
//res0: Array[(String, String)] = Array((2021-05-16,2021-05-17), (2021-05-18,2021-05-19), (2021-05-20,2021-05-21))
You can use following method to generate your tuple array,
import java.time.LocalDate
import java.time.format.DateTimeFormatter
def generateArray3(startDateString: String, endDateString: String): Array[(String, String)] = {
val dateFormatter = DateTimeFormatter.ISO_LOCAL_DATE
val startDate = LocalDate.parse(startDateString)
val endDate = LocalDate.parse(endDateString)
val daysCount = startDate.until(endDate).getDays
val dateStringTuples = Array.tabulate(daysCount)(i => {
val firstDate = startDate.plusDays(i)
val secondDate = startDate.plusDays(i + 1)
(dateFormatter.format(firstDate), dateFormatter.format(secondDate))
})
dateStringTuples
}
Usage:
println("--------------------------")
generateArray("2021-02-27", "2021-03-02").foreach(println)
println("--------------------------")
generateArray("2021-05-27", "2021-06-02").foreach(println)
println("--------------------------")
generateArray("2021-12-27", "2022-01-02").foreach(println)
println("--------------------------")
output :
--------------------------
(2021-02-27,2021-02-28)
(2021-02-28,2021-03-01)
(2021-03-01,2021-03-02)
--------------------------
(2021-05-27,2021-05-28)
(2021-05-28,2021-05-29)
(2021-05-29,2021-05-30)
(2021-05-30,2021-05-31)
(2021-05-31,2021-06-01)
(2021-06-01,2021-06-02)
--------------------------
(2021-12-27,2021-12-28)
(2021-12-28,2021-12-29)
(2021-12-29,2021-12-30)
(2021-12-30,2021-12-31)
(2021-12-31,2022-01-01)
(2022-01-01,2022-01-02)
--------------------------
I need to find all the year weeks between the given weeks.
201824 is an example of an year week. It means 24th week of the year 2018.
Assuming that there are 52 weeks in a year, The year weeks of 2018 start with 201801 and end with 201852. After that, it continues with 201901.
I was able to find the range of all year weeks between 2 weeks if the start week and the end week are in the same year like below
val range = udf((i: Int, j: Int) => (i to j).toArray)
The above code works only works when the start week and end week are in the same year, for example 201912 - 201917
How do I make it work if the start week and the end week belongs to different years.
Example: 201849 - 201903
The above weeks should give the output as:
201849,201850,201851,201852,201901,201902,201903
Well there is still a lot of optimizations to do, but for the general direction you could use:
I am using org.joda.time.format here, but java.time should also fit.
def rangeOfYearWeeks(weeksRange: String): Array[String] = {
try {
val left = weeksRange.split("-")(0).trim
val right = weeksRange.split("-")(1).trim
val leftPattern = s"${left.substring(0, 4)}-${left.substring(4)}"
val rightPattern = s"${right.substring(0, 4)}-${right.substring(4)}"
val fmt = DateTimeFormat.forPattern("yyyy-w")
val leftDate = fmt.parseDateTime(leftPattern)
val rightDate = fmt.parseDateTime(rightPattern)
//if (leftDate.isAfter(rightDate))
val weeksBetween = Weeks.weeksBetween(leftDate, rightDate).getWeeks
val dates = for (one <- 0 to weeksBetween) yield {
leftDate.plusWeeks(one)
}
val result: Array[String] = dates.map(date => fmt.print(date)).map(_.replaceAll("-", "")).toArray
result
} catch {
case e: Exception => Array.empty
}
}
Example:
val dates = Seq("201849 - 201903", "201912 - 201917").toDF("col")
val weeks = udf((d: String) => rangeOfYearWeeks(d))
dates.select(weeks($"col")).show(false)
+-----------------------------------------------------+
|UDF(col) |
+-----------------------------------------------------+
|[201849, 201850, 201851, 201852, 20181, 20192, 20193]|
|[201912, 201913, 201914, 201915, 201916, 201917] |
+-----------------------------------------------------+
Here's a solution with an UDF that uses the java.time API:
def weeksBetween = udf{ (startWk: Int, endWk: Int) =>
import java.time.LocalDate
import java.time.format.DateTimeFormatter
import scala.util.{Try, Success, Failure}
def formatYW(yw: Int): String = {
val pattern = "(\\d{4})(\\d+)".r
s"$yw" match { case pattern(y, w) => s"$y-$w-1"}
}
val formatter = DateTimeFormatter.ofPattern("YYYY-w-e") // week-based year
Try(
Iterator.iterate(LocalDate.parse(formatYW(startWk), formatter))(_.plusWeeks(1)).
takeWhile(_.isBefore(LocalDate.parse(formatYW(endWk), formatter))).
map{ s =>
val a = s.format(formatter).split("-")
(a(0) + f"${a(1).toInt}%02d").toInt
}.
toList.tail
) match {
case Success(ls) => ls
case Failure(_) => List.empty[Int] // return an empty list
}
}
Testing the UDF:
val df = Seq(
(1, 201849, 201903), (2, 201908, 201916), (3, 201950, 201955)
).toDF("id", "start_wk", "end_wk")
df.withColumn("weeks_between", weeksBetween($"start_wk", $"end_wk")).show(false)
// +---+--------+------+--------------------------------------------------------+
// |id |start_wk|end_wk|weeks_between |
// +---+--------+------+--------------------------------------------------------+
// |1 |201849 |201903|[201850, 201851, 201852, 201901, 201902] |
// |2 |201908 |201916|[201909, 201910, 201911, 201912, 201913, 201914, 201915]|
// |3 |201950 |201955|[] |
// +---+--------+------+--------------------------------------------------------+
I have 3 scala functions, working well, I have an errors just in the function
I did another function equals: compute the number of hours between 2 dates by calling the 3 functions above: toEnd, ToStart, jourOuvree.
I wrote all the function in a scala Object. But when I run the function equals, it gives me an errors, this following:
^
I should use the function and apply it on my dataframe to compute the number of hours between 2 dates.
I'm not excellent in java.time API.
Someone can help me please to resolve these errors ?
I rewrote your solution. See comments inline.
import java.time._
import java.time.temporal.ChronoUnit
import scala.annotation.tailrec
import scala.concurrent.duration._
object Demo extends App {
val start = LocalTime.of(9, 0)
val midEnd = LocalTime.of(13, 0)
val midStart = LocalTime.of(14, 0)
val end = LocalTime.of(18, 0)
val firstHalf = start.until(midEnd, ChronoUnit.MILLIS).millis
val secondHalf = midStart.until(end, ChronoUnit.MILLIS).millis
implicit class formatter(d: FiniteDuration) {
def withMinutes = {
val l = d.toMinutes
s"${l / 60}:${l % 60}"
}
def withSeconds = s"${d.toHours}:${d.toMinutes % 60}:${d.toSeconds % 60}"
}
// check if day is working. Could be easily adjusted to check not only weekend but holidays as well
def isWorkingDay(d: LocalDate) = !Set(DayOfWeek.SATURDAY, DayOfWeek.SUNDAY).contains(d.getDayOfWeek)
// check if current day is working day and if not switch to next start of day/previous end of day
// depending on ask parameter
#tailrec
def adjust(t: LocalDateTime, asc: Boolean = true): LocalDateTime = {
if (!isWorkingDay(t.toLocalDate) && asc) adjust(t.toLocalDate.plusDays(1).atTime(start), asc)
else if (!isWorkingDay(t.toLocalDate) && !asc) adjust(t.toLocalDate.minusDays(1).atTime(end), asc)
else t
}
def toEnd(t: LocalTime) = {
if (t.isBefore(start)) firstHalf + secondHalf
else if (t.isBefore(midEnd)) t.until(midEnd, ChronoUnit.MILLIS).millis + secondHalf
else if (t.isBefore(midStart)) secondHalf
else if (t.isBefore(end)) t.until(end, ChronoUnit.MILLIS).millis
else 0.hours
}
def toStart(t: LocalTime) = {
if (t.isBefore(start)) 0.hours
else if (t.isBefore(midEnd)) start.until(t, ChronoUnit.MILLIS).millis
else if (t.isBefore(midStart)) firstHalf
else if (t.isBefore(end)) firstHalf + midStart.until(t, ChronoUnit.MILLIS).millis
else firstHalf + secondHalf
}
// count amount of working days between two dates.
// if dates are the same - means 0 days
def jourOuvree(d1: LocalDate, d2: LocalDate): Int = {
#tailrec
def count(n: LocalDate, acc: Int): Int = {
if (n.isEqual(d2) || n.isAfter(d2)) acc
else if (!isWorkingDay(n)) count(n.plusDays(1), acc)
else count(n.plusDays(1), acc + 1)
}
count(d1, 0)
}
// evaluate duration of working time between to date/times
// take into account that start/end could be non-working and
// adjust
def toEquals(rd1: LocalDateTime, rd2: LocalDateTime) = {
val d1 = adjust(rd1)
val d2 = adjust(rd2, asc = false)
// if we are in the same day ==> I should compute just a difference between hours
if (d1.isAfter(d2)) 0.hours
else if (d1.toLocalDate.isEqual(d2.toLocalDate)) {
toEnd(d1.toLocalTime) - toEnd(d2.toLocalTime)
}
else {
toEnd(d1.toLocalTime) + jourOuvree(d1.toLocalDate.plusDays(1), d2.toLocalDate.minusDays(1)) * 8.hours + toStart(d2.toLocalTime)
}
}
val thursdayStart = LocalDate.of(2018, 3, 1).atTime(start)
val thursdayEnd = LocalDate.of(2018, 3, 1).atTime(end)
val fridayStart = LocalDate.of(2018, 3, 2).atTime(start)
val fridayEnd = LocalDate.of(2018, 3, 2).atTime(end)
val saturdayStart = LocalDate.of(2018, 3, 3).atTime(start)
val saturdayEnd = LocalDate.of(2018, 3, 3).atTime(end)
val sundayStart = LocalDate.of(2018, 3, 4).atTime(start)
val sundayEnd = LocalDate.of(2018, 3, 4).atTime(end)
val mondayStart = LocalDate.of(2018, 3, 5).atTime(start)
val mondayEnd = LocalDate.of(2018, 3, 6).atTime(end)
println(toEquals(thursdayStart, sundayStart).withSeconds)
println(toEquals(thursdayStart, saturdayStart).withSeconds)
println(toEquals(thursdayStart, mondayStart).withSeconds)
println(toEquals(thursdayStart, mondayEnd).withSeconds)
println(toEquals(thursdayStart, thursdayEnd).withSeconds)
// formatter to use to parse string input
import java.time.format.DateTimeFormatter
val DATE_TIME_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")
println(toEquals(LocalDateTime.parse("2018-03-01 11:22:33", DATE_TIME_FORMATTER), LocalDateTime.parse("2018-03-02 11:22:33", DATE_TIME_FORMATTER)).withSeconds)
}
I have some examples:
case1. val date1 = new DateTime("2015-08-03T04:59:00.000")
output: new DateTime("2015-08-03T04:00:00.000")
case2. val date2 = new DateTime("2015-08-03T04:15:00.000")
output: new DateTime("2015-08-03T04:00:00.000")
means for any datetime if the time is more the 1 minute output should be start of hour. Example for day: datetime.withTimeAtStartOfDay.
I'm assuming that you are using the joda time DateTime. If you are, then you can use the following method
def dateTimeAtStartOfHour(s: String) = {
new DateTime(s)
.withMinuteOfHour(0)
.withSecondOfMinute(0)
.withMillisOfSecond(0)
}
val date1 = dateTimeAtStartOfHour("2015-08-03T04:59:00.000")
val date2 = dateTimeAtStartOfHour("2015-08-03T04:15:00.000")
val date3 = dateTimeAtStartOfHour("2015-08-03T04:59:13.000")
Output is
date1: org.joda.time.DateTime = 2015-08-03T04:00:00.000-04:00
date2: org.joda.time.DateTime = 2015-08-03T04:00:00.000-04:00
date3: org.joda.time.DateTime = 2015-08-03T04:00:00.000-04:00
You need to use "truncate" analogue, which is called roundFloorCopy in joda:
https://stackoverflow.com/a/8510936/1349366
Given a start and an end date I would like to iterate on it by day using a foreach, map or similar function. Something like
(DateTime.now to DateTime.now + 5.day by 1.day).foreach(println)
I am using https://github.com/nscala-time/nscala-time, but I get returned a joda Interval object if I use the syntax above, which I suspect is also not a range of dates, but a sort of range of milliseconds.
EDIT: The question is obsolete. As advised on the joda homepage, if you are using java 8 you should start with or migrate to java.time.
You may use plusDays:
val now = DateTime.now
(0 until 5).map(now.plusDays(_)).foreach(println)
Given start and end dates:
import org.joda.time.Days
val start = DateTime.now.minusDays(5)
val end = DateTime.now.plusDays(5)
val daysCount = Days.daysBetween(start, end).getDays()
(0 until daysCount).map(start.plusDays(_)).foreach(println)
For just iterating by day, I do:
Iterator.iterate(start) { _ + 1.day }.takeWhile(_.isBefore(end))
This has proven to be useful enough that I have a small helper object to provide an implicit and allow for a type transformation:
object IntervalIterators {
implicit class ImplicitIterator(val interval: Interval) extends AnyVal {
def iterateBy(step: Period): Iterator[DateTime] = Iterator.iterate(interval.start) { _ + step }
.takeWhile(_.isBefore(interval.end))
def iterateBy[A](step: Period, transform: DateTime => A): Iterator[A] = iterateBy(step).map(transform)
def iterateByDay: Iterator[LocalDate] = iterateBy(1.day, { _.toLocalDate })
def iterateByHour: Iterator[DateTime] = iterateBy(1.hour)
}
}
Sample usage:
import IntervalIterators._
(DateTime.now to 5.day.from(DateTime.now)).iterateByDay // Iterator[LocalDate]
(30.minutes.ago to 1.hour.from(DateTime.now)).iterateBy(1.second) // Iterator[DateTime], broken down by second
Solution with java.time API using Scala
Necessary import and initialization
import java.time.temporal.ChronoUnit
import java.time.temporal.ChronoField.EPOCH_DAY
import java.time.{LocalDate, Period}
val now = LocalDate.now
val daysTill = 5
Create List of LocalDate for sample duration
(0 to daysTill)
.map(days => now.plusDays(days))
.foreach(println)
Iterate over specific dates between start and end using toEpochDay or getLong(ChronoField.EPOCH_DAY)
//Extract the duration
val endDay = now.plusDays(daysTill)
val startDay = now
val duration = endDay.getLong(EPOCH_DAY) - startDay.getLong(EPOCH_DAY)
/* This code does not give desired results as trudolf pointed
val duration = Period
.between(now, now.plusDays(daysTill))
.get(ChronoUnit.DAYS)
*/
//Create list for the duration
(0 to duration)
.map(days => now.plusDays(days))
.foreach(println)
This answer fixes the issue of mrsrinivas answer, that .get(ChronoUnits.DAYS) returns only the days part of the duration, and not the total number of days.
Necessary import and initialization
import java.time.temporal.ChronoUnit
import java.time.{LocalDate, Period}
Note how above answer would lead to wrong result (total number of days is 117)
scala> Period.between(start, end)
res6: java.time.Period = P3M26D
scala> Period.between(start, end).get(ChronoUnit.DAYS)
res7: Long = 26
Iterate over specific dates between start and end
val start = LocalDate.of(2018, 1, 5)
val end = LocalDate.of(2018, 5, 1)
// Create List of `LocalDate` for the period between start and end date
val dates: IndexedSeq[LocalDate] = (0L to (end.toEpochDay - start.toEpochDay))
.map(days => start.plusDays(days))
dates.foreach(println)
you can use something like that:
object Test extends App {
private val startDate: DateTime = DateTime.now()
private val endDate: DateTime = DateTime.now().plusDays(5)
private val interval: Interval = new Interval(startDate, endDate)
Stream.from(0,1)
.takeWhile(index => interval.contains(startDate.plusDays(index)))
.foreach(index => println(startDate.plusDays(index)))
}
In this case, the Scala way is the Java way:
When running Scala on Java 9+, we can use java.time.LocalDate::datesUntil:
import java.time.LocalDate
import collection.JavaConverters._
// val start = LocalDate.of(2019, 1, 29)
// val end = LocalDate.of(2018, 2, 2)
start.datesUntil(end).iterator.asScala
// Iterator[java.time.LocalDate] = <iterator> (2019-01-29, 2019-01-30, 2019-01-31, 2019-02-01)
And if the last date is to be included:
start.datesUntil(end.plusDays(1)).iterator.asScala
// 2019-01-29, 2019-01-30, 2019-01-31, 2019-02-01, 2019-02-02
import java.util.{Calendar, Date}
import scala.annotation.tailrec
/** Gets date list between two dates
*
* #param startDate Start date
* #param endDate End date
* #return List of dates from startDate to endDate
*/
def getDateRange(startDate: Date, endDate: Date): List[Date] = {
#tailrec
def addDate(acc: List[Date], startDate: Date, endDate: Date): List[Date] = {
if (startDate.after(endDate)) acc
else addDate(endDate :: acc, startDate, addDays(endDate, -1))
}
addDate(List(), startDate, endDate)
}
/** Adds a date offset to the given date
*
* #param date ==> Date
* #param amount ==> Offset (can be negative)
* #return ==> New date
*/
def addDays(date: Date, amount: Int): Date = {
val cal = Calendar.getInstance()
cal.setTime(date)
cal.add(Calendar.DATE, amount)
cal.getTime
}