I'm using Scala along with play framework. I want to parse the String with simple date and say it's in UTC. So if I have 2018-10-04 I want to get 2018-10-04T00:00:00.000Z
With this:
DateTime.parse("2018-10-04", DateTimeFormat.forPattern("yyyy-mm-dd")).withZone(DateTimeZone.UTC)
I keep getting 2018-10-03T22:00:00.000Z if I have +2 timezone. How to just say that it's already in UTC?
One way is to use LocalDatetime to initially ignore the timezone:
scala> LocalDateTime.parse("2018-10-04", DateTimeFormat.forPattern("yyyy-MM-dd"))
res11: org.joda.time.LocalDateTime = 2018-10-04T00:00:00.000
then you can use toDateTime to make this UTC:
scala> LocalDateTime.parse("2018-10-04", DateTimeFormat.forPattern("yyyy-MM-dd")).toDateTime(DateTimeZone.UTC)
res12: org.joda.time.DateTime = 2018-10-04T00:00:00.000Z
Also: You should use MM (month) rather than mm (minute).
First the imports.
import org.joda.time.{DateTime, DateTimeZone}
import org.joda.time.format.DateTimeFormat
Since the time part is fixed, we can use a constant suffix for the time part.
val timeString = "T00:00:00.000Z"
We use string-interpolation to append this to the incoming dates.
DateTime.parse(s"2018-10-04$timeString", DateTimeFormat.forPattern("yyyy-mm-dd'T'HH:mm:ss.SSSZ")).withZone(DateTimeZone.UTC)
Related
I have a scala / spark dataframe, with one column named "utcstamp" with values of the following format: 2018-12-12 21:15:00
I want to obtain a new column with the week day, and inspired by this question in the forum, used the following code:
import java.util.Calendar
import java.text.SimpleDateFormat
val dowText = new SimpleDateFormat("E")
df = df.withColumn("weekday" , dowText.format(df.select(col("utcstamp"))))
However, I get the following error:
<console>:58: error: type mismatch;
found : String
required: org.apache.spark.sql.Column
When I try this applied to a specific date (like in the link provided) it works, I just can't apply it to the whole column.
Can anyone help me with this? If you have an alternative way of converting an utc column into weekday that'll also do for me.
You can use dayofweek function of Spark SQL, which gives you a number from 1-7, for Sunday to Saturday:
val df2 = df.withColumn("weekday", dayofweek(col("utcstamp").cast("timestamp")))
Or if you want words (Sun-Sat) instead,
val df2 = df.withColumn("weekday", date_format(col("utcstamp").cast("timestamp"), "EEE"))
You can simply get the day of week with date format as "E" or EEEE (eg. for Sun and Sunday)
df.withColumn("weekday", date_format(to_timestamp($"utcstamp"), "E"))
If you want day of week as numeric value use dayofweek function which is availabe from spark 2.3+
Is there any way to get current timestamp in Date format in scala. I needed to create a date histogram and new Date() gives time in seconds and not in dd-mm-yyyy hh:mm format
Try to use the classes from the java.time package:
import java.time._
import java.time.format._
val format = DateTimeFormatter.ofPattern("dd-MM-yyyy HH:mm:ss");
LocalDateTime.now().format(format)
How can I convert a Date and a Time to a DateTime?
Say I have the following date and time
iex> date = ~D[2018-01-01]
iex> time = ~T[00:00:01.000]
How can I combine these to output the datetime: #DateTime<2018-01-01 00:00:01Z> in a clean way?
The best I come up with is using the Timex library:
Timex.add(Timex.to_datetime(date), Timex.Duration.from_time(time))
but I feel that surely there is a nicer, more readable way to combine this.
If you are using the calendar library, you can use either the Calendar.DateTime.from_date_and_time_and_zone function or the Calendar.NaiveDateTime.from_date_and_time function:
iex(4)> Calendar.DateTime.from_date_and_time_and_zone(~D[2018-10-01], ~T[12:22:22], "Australia/Melbourne")
{:ok, #DateTime<2018-10-01 12:22:22+10:00 AEST Australia/Melbourne>}
iex(5)> Calendar.NaiveDateTime.from_date_and_time(~D[2018-10-01], ~T[12:22:22])
{:ok, ~N[2018-10-01 12:22:22]}
There are also from_date_and_time! and from_date_and_time! variants.
You can use NaiveDateTime.new/2:
iex> NaiveDateTime.new(date, time)
{:ok, ~N[2018-01-01 00:00:01.000]}
The reason it is a "naive datetime" instead of a datetime is that the Time struct doesn't contain time zone information. If you know the time zone, you can add that information using DateTime.from_naive/2:
iex> DateTime.from_naive(~N[2018-01-01 00:00:01.000], "Etc/UTC")
{:ok, #DateTime<2018-01-01 00:00:01.000Z>}
While the answer by #legoscia is perfectly valid, here is how you deal with date and time pair (without Timex, just pure Elixir standard library):
date = ~D[2018-01-01]
time = ~T[00:00:01.000]
{Date.to_erl(date), Time.to_erl(time)}
|> NaiveDateTime.from_erl!()
|> DateTime.from_naive("Etc/UTC")
#⇒ {:ok, #DateTime<2018-01-01 00:00:01Z>}
I'd like to get the day from a date object as an integer. This is my code so far.
val dateString = "2015-11-24 23:23:09"
val format = new java.text.SimpleDateFormat("yyyy-MM-dd H:m:s")
val date = format.parse(dateString)
print(date) # this gives Tue Nov 24 23:23:09 CST 2015
Now, from date, I want to get the day of the month as an integer. How do I do that?
Thanks.
Just use the Java Calendar class (although I would recommend moving over to the Joda library if you are doing much serious work with dates/times):
val cal = Calendar.getInstance()
cal.setTime(date)
val dayOfMonth = cal.get(Calendar.DAY_OF_MONTH)
tl;dr
LocalDateTime.parse(
"2015-11-24 23:23:09".replace( " " , "T" )
).getDayOfMonth()
java.time
The modern approach uses the java.time classes.
Using Java syntax here as I don't know Scala. Note that java.time uses immutable objects.
Convert your string to comply with ISO 8601 standard format with a T in the middle.
String input = "2015-11-24 23:23:09".replace( " " , "T" ) ;
Parse as an LocalDateTime as your input lacks any time zone or offset-from-UTC.
LocalDateTime ldt = LocalDateTime.parse( input ) ;
Interrogate for the day of month.
int dom = ldt.getDayOfMonth() ;
Using Joda,
org.joda.time.DateTime.now().getDayOfMonth()
or equivalently,
import java.util.Date
new org.joda.time.DateTime(new Date()).getDayOfMonth
I generate an ISO datetime string (without time zone) like this:
val dateTime: LocalDateTime = LocalDateTime.now
val dateTimeStr: String = ISODateTimeFormat.dateTime.withZone(DateTimeZone.UTC).print(dateTime)
The code above produces the following string:
2014-04-10T06:13:19.283
Now I need to convert this string back to a LocalDateTime...
val dateTime = LocalDateTime.parse(dateTimeStr, ISODateTimeFormat.dateTime().withZone(DateTimeZone.UTC))
... and compare it with the current time:
val isBefore = dateTime.isBefore(LocalDateTime.now)
The code above doesn't work and produces the following error:
Invalid format: \"2014-04-27T17:51:06.780\" is too short
To fix the problem, I need to append a Z to dateTimeStr:
val dateTime = LocalDateTime.parse(s"${dateTimeStr}Z", ISODateTimeFormat.dateTime().withZone(DateTimeZone.UTC))
Is there a way to generate the ISO datetime string with the Z at the end?
A LocalDateTime has NO timezone. So you cannot associate a timezone-aware format (Z stands for UTC timezone) with this zoneless data type.
If you insist on having Z-formats then you have to work with a global type like DateTime. So you have two different steps. One step is object conversion between local and global type:
LocalDateTime ldt = ...;
DateTime dt = ldt.toDateTime(DateTimeZone.UTC); // or another timezone
// reverse
DateTime dt = ...;
LocalDateTime ldt = dt.toLocalDateTime();
Second step is conversion between formatted string form and global timestamp:
LocalDateTime ldt = ...;
DateTime dt = ldt.toDateTime(DateTimeZone.UTC); // or another timezone
String iso = ISODateTimeFormat.dateTime().print(dt);
// reverse
String iso = ...; // with trailing Z or another timezone information
DateTime dt = IsoDateTimeFormat.parseDateTime(iso);
LocalDateTime ldt = dt.toLocalDateTime();
Finally you can combine these two steps for conversion between ISO-format with timezone information and a LocalDateTime as shown in the second step.
If you only need formatted strings without any timezone or offset information then you can stick the global type DateTime completely and just use localDateOptionalTimeParser() as #Sebastian has correctly mentioned.
Try to use
val dateTime: DateTime = DateTime.now
instead of
val dateTime: LocalDateTime = LocalDateTime.now
Or if you want to stick to LocalDateTime change the parsing:
val dateTime = LocalDateTime.parse(dateTimeStr) // this uses the default localDateOptionalTimeParser