I was trying to calculate the number of days between 2 dates in scala. I tried using compareTo and import java.time.Period function. But it is not giving the exact value of days when comparing two dates from different months.
val date1 = "2022-04-01"
date1: String = 2022-04-01
val date2 = "2022-04-04"
date2: String = 2022-04-04
date2.compareTo(date1)
res37: Int = 3
val date2 = "2022-05-04"
date2: String = 2022-05-04
date2.compareTo(date1)
res38: Int = 1
val date1 = LocalDate.parse("2022-04-01")
date1: java.time.LocalDate = 2022-04-01
val date2 = LocalDate.parse("2022-04-04")
date1: java.time.LocalDate = 2022-04-04
val p = Period.between(date1, date2)
p: java.time.Period = P3D
p.getDays
res39: Int = 3
val date2 = LocalDate.parse("2022-05-04")
date2: java.time.LocalDate = 2022-05-04
val p = Period.between(date1, date2)
p: java.time.Period = P1M3D
p.getDays
res40: Int = 3
I want to get the difference as 33 days while comparing the dates 2022-04-01 and 2022-05-04. Is there a different way to achieve this?
What you are looking for is the LocalDate.until() method with unit set to ChronoUnit.DAYS:
import java.time.LocalDate
import java.time.temporal.ChronoUnit
val date1 = LocalDate.parse("2022-04-01")
val date2 = LocalDate.parse("2022-05-03")
val daysBetween = date1.until(date2, ChronoUnit.DAYS)
// scala> date1.until(date2, ChronoUnit.DAYS)
// res1: Long = 32
Note that your first attempt does simple lexicographical string comparison which is oblivious to the fact that your strings are dates. The return value of String.compareTo() (and Comparable<T>.compareTo() in general) is to be interpreted only on the basis whether it is 0, < 0, or > 0. The actual value is meaningless in most cases. The fact that it matches the actual number of days in your first example is due to how your dates are formatted and to how String.compareTo() works:
In this case, compareTo returns the difference of the two character values at position k in the two string [sic]
Here, k refers to the position where the two strings first differ.
Your second attempt with Period doesn't work either because Period represents conceptual differences. Adding P1M3D to some date means incrementing the month by one and the day by three, therefore the length in days is dependent on the month to which it is being applied:
2022-04-01 + P1M3D = 2022-05-04 -> 33 days
2022-05-01 + P1M3D = 2022-06-04 -> 34 days
Thus, a freestanding Period object has no length in days (unless it only has days but not months or years) and the getDays accessor returns only the days component, not the full number of days.
Related
maybe it is a duplicated question, I tried to find a solution but no way.
Problem : convert a DateTime instance to Timestamp.
My DateTime instance is created as :
import org.joda.time.DateTime
val start = (new DateTime).withYear(2016)
.withMonthOfYear(12)
.withDayOfMonth(1)
.withMinuteOfHour(0)
.withHourOfDay(0)
with println of this date, I got :
2016-12-01T00:00:18.856+01:00
What I need is like Epoch timestamp in picture bellow :
IMHO one of the fastest way to calculate the Epoch timestamp is:
scala> :paste
// Entering paste mode (ctrl-D to finish)
def epochDayForYear(year: Int): Long =
365L * year + (((year + 3) >> 2) - (if (year < 0) {
val century = year * 1374389535L >> 37 // divide int by 100 (a sign correction is not required)
century - (century >> 2)
} else ((year + 99) * 1374389535L >> 37) - ((year + 399) * 1374389535L >> 39))) // divide int by 100 and by 400 accordingly (a sign correction is not required)
def dayOfYearForYearMonth(year: Int, month: Int): Int =
((month * 1050835331877L - 1036518774222L) >> 35).toInt - // == (367 * month - 362) / 12
(if (month <= 2) 0
else if (isLeap(year)) 1
else 2)
def isLeap(year: Int): Boolean = (year & 3) == 0 && {
val century = (year * 1374389535L >> 37).toInt - (year >> 31) // divide int by 100
century * 100 != year || (century & 3) == 0
}
def secondOfDay(hour: Int, month: Int, day: Int): Int =
hour * 3600 + month * 60 + day
val (year, month, day, hour, minute, second, timeZoneOffsetHour, timeZoneOffsetMinute) =
(2016, 12, 1, 0, 0, 2, 0, 0)
val epochSecond =
(epochDayForYear(year) + (dayOfYearForYearMonth(year, month) + day - 719529)) * 86400 +
secondOfDay(hour + timeZoneOffsetHour, minute + timeZoneOffsetMinute, second) // where 719528 is days 0000 to 1970
// Exiting paste mode, now interpreting.
epochDayForYear: (year: Int)Long
dayOfYearForYearMonth: (year: Int, month: Int)Int
isLeap: (year: Int)Boolean
secondOfDay: (hour: Int, month: Int, day: Int)Int
year: Int = 2016
month: Int = 12
day: Int = 1
hour: Int = 0
minute: Int = 0
second: Int = 2
timeZoneOffsetHour: Int = 0
timeZoneOffsetMinute: Int = 0
epochSecond: Long = 1480550402
BTW, most of this code is borrowed from the jsoniter-scala project.
Using java.time. Does this help?
scala> val x = java.time.LocalDateTime.ofEpochSecond(System.currentTimeMillis/1000,0,java.time.ZoneOffset.UTC)
x: java.time.LocalDateTime = 2018-11-21T18:41:29
scala> java.time.LocalDateTime.parse("2018-11-21T18:41:29.123").toEpochSecond(java.time.ZoneOffset.UTC)
res41: Long = 1542825689
scala> java.time.LocalDateTime.parse("2018-11-21T18:41:29.123").format(java.time.format.DateTimeFormatter.ofPattern("eeee, MMMM dd, yyyy hh:mm:ss a"))
res61: String = Wednesday, November 21, 2018 06:41:29 PM
scala>
This link https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html might be helpful to know the symbols used in the format options
First: Scala gets compiled into Java, so when you are asking a question about a Java library you can use examples from that.
Second: Do you have any default Timezone in your application? I would put this in a logical place (i.e. when you start you application / in some config)
DateTimeZone.setDefault(DateTimeZone.UTC);
Now you can just create a new java.sql.Timestamp like this:
import java.sql.Timestamp;
import java.time.Instant
val start = (new DateTime).withYear(2016)
.withMonthOfYear(12)
.withDayOfMonth(1)
.withMinuteOfHour(0)
.withHourOfDay(0)
val timestamp = new Timestamp(start.getMillis()) : Timestamp
// To get the epoch timestamp, take a look at java.time.Instant
val epochTimestamp = timestamp.toInstant() : Instant
More information here: https://www.jeejava.com/conversion-of-joda-date-time-to-sql-timestamp-and-vice-versa/
My data-frame has a DateId (i.e. an integer column defining a date as a number of days since 1993-06-25). Objective is to calculate date id of the last day of month prior to each date in the column:
DateId -> _intermittent calc Date_ -> _result LastDayOfPriorMonthId_
9063 -> 2018-04-18 -> 9045 (i.e. 2018-03-31)
8771 -> 2017-06-30 -> 8741 (i.e. 2017-05-31)
9175 -> 2018-08-08 -> 9167 (i.e. 2018-07-31)
Solution would be really easy, but I'm running into issues with type conversion:
val a = Seq(9063, 8771, 9175).toDF("DateId")
val timeStart = to_date(lit("1993-06-25"))
val dateIdAdd : (Column) => Column = x => {x - date_add(timeStart, x).DATE_OF_MONTH}
The function compilation is failing with following error:
notebook:2: error: type mismatch;
found : org.apache.spark.sql.Column
required: Int
x - date_add(timeStart, x).DATE_OF_MONTH
Expressions like .cast(IntegerType) do not change the outcome (x is still a spark Column type and .cast(Int) is not applicable.
Please note: similar problem was addressed in this SO question, but the same approach is failing when the timeStart constant is applied here. Also using function would be preferred over expression, because the same calculation is used multiple columns with real data.
Can you translate from Java? Sorry, I don’t code Scala (yet).
private static final LocalDate baseDate = LocalDate.of(1993, Month.JUNE, 25);
public static long dateIdAdd(long dateId) {
LocalDate date = baseDate.plusDays(dateId);
LocalDate lastOfPrevMonth = YearMonth.from(date).minusMonths(1).atEndOfMonth();
return ChronoUnit.DAYS.between(baseDate, lastOfPrevMonth);
}
Edit: according to you (Dan, the asker), the Scala version is:
val baseDate = LocalDate.of(1993, Month.JUNE, 25)
val lastDayIdOfPriorMonth = udf((dateId : Long) => {
val date = baseDate.plusDays(dateId)
val lastOfPrevMonth = YearMonth.from(date).minusMonths(1).atEndOfMonth()
ChronoUnit.DAYS.between(baseDate, lastOfPrevMonth)
})
Let’s try it with your example dates (Java again):
System.out.println("9063 -> " + dateIdAdd(9063));
System.out.println("8771 -> " + dateIdAdd(8771));
System.out.println("9175 -> " + dateIdAdd(9175));
This prints:
9063 -> 9045
8771 -> 8741
9175 -> 9167
In your question you gave 9176 as desired result in the last case, but I believe that was a typo?
And please enjoy how clear and self-explanatory the code is.
After testing many options with Scala conversion function, hack based on UDF with Java string and SimpleDateFormat the only thing I could figure out:
val dateIdAdd = udf((dateId : Long) => {
val d = new SimpleDateFormat("yyyy-MM-dd")
val ts = d.parse("1993-06-25")
val tc = d.format(new Date(ts.getTime() + (24 * 3600 * 1000 * dateId)))
dateId - Integer.parseInt(tc.substring(tc.length()-2))
})
After adding another support function for validation and a simple select:
val dateIdToDate = udf((dateId : Long) => {
val d = new SimpleDateFormat("yyyy-MM-dd")
val ts = d.parse("1993-06-25")
d.format(new Date(ts.getTime() + (24 * 3600 * 1000 * dateId)))
})
val aa = a.select($"*"
, dateIdToDate($"DateId") as "CalcDateFromId"
, dateIdAdd($"DateId") as "CalcLastDayOfMonthId")
display(aa)
Expected results are generated (but I doubt this is the most efficient way available):
DateId CalcDateFromId CalcLastDayOfMonthId
9063 4/18/2018 9045
8771 6/30/2017 8741
9175 8/8/2018 9167
I have a number in minutes, for example:
Int totalminutes= 342
I want to dispaly it like this:
5h:42m:..s
I did a scala function to convert the minute to hour and return HH:MM:SS
scala> def convertSeconds(input:Int):String ={
| val totalMinutes = input/60
| val seconds = input%60
| val minutes = totalMinutes%60
| val hours = totalMinutes%60
|
| return "%sh:%sm:%ss".format(hours,minutes,seconds)
| }
convertSeconds: (input: Int)String
I did my tests:
scala> convertSeconds(120) // 120 minutes
res22: String = 2h:2m:0s // why it add 2 minutes
scala> convertSeconds(60) // 60 minutes
res23: String = 1h:1m:0s // adding 1 minute
scala> convertSeconds(36) // 36 minutes
res24: String = 0h:0m:36s // I want to return 0h:30m:06s
The solution is:
scala> def convertSeconds(input:Int):String ={
//val totalMinutes = input/60
//val seconds = input/60%60
val minutes = input%60
val hours = input/60
return "%sh:%sm".format(hours,minutes)
}
convertSeconds: (input: Int)String
scala> convertSeconds(36)
res57: String = 0h:36m
scala> convertSeconds(120)
res58: String = 2h:0m
scala> convertSeconds(122)
res59: String = 2h:2m
scala> convertSeconds(2166)
res60: String = 36h:6m
Your function is supposed to be like this and you pass seconds not minutes:
def convertSeconds(input:Int):String ={
val hours = input/3600
val minutes = (input-hours*3600)/60
val seconds = input%3600-minutes*60
"%sh:%sm:%ss".format(hours,minutes,seconds)
}
If you want to pass minutes, the function should be something like this, I named it convertMinutes, and your seconds will zero, and you can get in your prescribed display by this code:
def convertMinutes(input:Int):String ={
val hours = input/60
val minutes = input-hours*60
"%sh:%sm:%ss".format(hours,minutes,"..")
}
In Scala REPL:
scala> convertSeconds(7523)
res4: String = 2h:5m:23s
scala> convertSeconds(9523)
res5: String = 2h:38m:43s
scala> convertSeconds(3724)
res6: String = 1h:2m:4s
scala> convertMinutes(342)
res10: String = 5h:42m:..s
I am not sure if you need a separate function to achieve this.
Scala already gives a library to do that
scala> val secs = 62
secs: Int = 62
scala> import scala.concurrent.duration._
scala> val inTime = Duration(secs,SECONDS)
inTime: scala.concurrent.duration.FiniteDuration = 62 seconds
scala> "%02d:%02d:%02d".format(inTime.toHours,inTime.toMinutes%60,inTime.toSeconds%60)
res8: String = 00:01:02
EDIT-1: What this does not handle is if you pass a number >= 86400 (total seconds in a day)
In Java, trusting you to translate yourself:
int totalminutes = 342;
Duration dur = Duration.ofMinutes(totalminutes);
String display = String.format("%dh:%02dm:%02ds",
dur.toHours(), dur.toMinutesPart(), dur.toSecondsPart());
System.out.println(display);
This prints:
5h:42m:00s
If had a number of seconds instead, use Duration.ofSeconds instead.
Don’t do the calculation yourself. It’s error-prone and it will require your reader to take some time to convince herself/himself that your calculation is correct. Instead let the library methods do the calculations, even when they seem trivial.
What went wrong in your code?
It’s been said already: You used
val hours = totalMinutes%60
This should have been a division:
val hours = totalMinutes / 60
It’s easier than you might expect to make such an error. For comparison you don’t that easily call toMinutesPart when you intended toHours, and if you still do, it’s easier to spot the bug.
I'm trying to calculate the hour difference between two joda dates.
val d1 = getDateTime1()
val d2 = getDateTime2()
Here is what I did:
# attemp1
val hours = Hours.hoursBetween(d1, d2) // error - types mismatch
# attempt2
val p = Period(d1, d2, PeriodType.hours()) // error - there is such a constructor
p.getHours
So how do I do this?
The type of getDateTime1 should be org.joda.time.DateTime.
This woks fine:
val d1: org.joda.time.DateTime = DateTime.now
val d2: org.joda.time.DateTime = DateTime.nextMonth
val hours = Hours.hoursBetween(d1, d2)
// org.joda.time.Hours = PT672H
There is no factory method apply for Period, you should use new to create Period:
val p = new Period(d1, d2, PeriodType.hours())
// org.joda.time.Period = PT672H
If you are using nscala-time you could also use method to to get Interval and than convert it to Period:
d1 to d2 toPeriod PeriodType.hours
// org.joda.time.Period = PT672H
(d1 to d2).duration.getStandardHours
// Long = 672
Also try sbt clean.
I am trying to split a number, like
20130405
into three parts: year, month and date.One way is to convert it to string and use regex. Something like:
(\d{4})(\d{2})(\d{2}).r
A better way is to divide it by 100. Something like:
var date = dateNumber
val day = date % 100
date /= 100
val month = date % 100
date /= 100
val year = date
I get itchy while using 'var' in Scala. Is there a better way to do it?
I would go with the former:
scala> val regex = """(\d{4})(\d{2})(\d{2})""".r
regex: scala.util.matching.Regex = (\d{4})(\d{2})(\d{2})
scala> val regex(year, month, day) = "20130405"
year: String = 2013
month: String = 04
day: String = 05
This is probably not much better than your own solution, but it doesn't use var and doesn't require transforming the number to a string. On the other hand, it's not very safe - if you're not 100% sure that your number is going to be well formatted, better use a SimpleDateFormat - granted, it's more expensive, but at least you're safe from illegal input.
val num = 20130405
val (date, month, year) = (num % 100, num / 100 % 100, num / 10000)
println(date) // Prints 5
println(month) // Prints 4
println(year) // Prints 2013
I'd personally use a SimpleDateFormat even if I were sure the input will always be legal. The only certainty there is is that I'm wrong and the input will someday be illegal.
Better than substring would be to use the java Date and SimpleDateFormat classes, see:
https://stackoverflow.com/a/4216767/88588
Not very scala-ish but...
scala> import java.util.Calendar
import java.util.Calendar
scala> val format = new java.text.SimpleDateFormat("yyyyMMdd")
format: java.text.SimpleDateFormat = java.text.SimpleDateFormat#ef87e460
scala> format.format(new java.util.Date())
res0: String = 20131025
scala> val d=format.parse("20130405")
d: java.util.Date = Fri Apr 05 00:00:00 CEST 2013
scala> val calendar = Calendar.getInstance
calendar: java.util.Calendar = [cut...]
scala> calendar.setTime(d)
scala> calendar.get(Calendar.YEAR)
res1: Int = 2013