The question is kind of similar with the problem: Change the timestamp to UTC format in Pyspark
Basically, it is convert timestamp string format ISO8601 with offset to UTC timestamp string(2017-08-01T14:30:00+05:30 -> 2017-08-01T09:00:00+00:00 ) using scala.
I am kind of new to scala/java, I checked spark library which they dont have a way to convert without knowing the timezone, which I dont have a idea of timezone unless (I parse it in ugly way or using java/scala lib?) Can someone help?
UPDATE: The better way to do this: setup timezone session in spark, and use df.cast(DataTypes.TimestampType) to do the timezone shift
org.apache.spark.sql.functions.to_utc_timestamp:
def to_utc_timestamp(ts: Column, tz: String): Column
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.
You can use the java.time primitives to parse and convert your timestamp.
scala> import java.time.{OffsetDateTime, ZoneOffset}
import java.time.{OffsetDateTime, ZoneOffset}
scala> val datetime = "2017-08-01T14:30:00+05:30"
datetime: String = 2017-08-01T14:30:00+05:30
scala> OffsetDateTime.parse(datetime).withOffsetSameInstant(ZoneOffset.UTC)
res44: java.time.OffsetDateTime = 2017-08-01T09:00Z
Related
I have a date string like this - 190515(YYMMDD) format. How to get a populate a exact one year previous date from this? ie, Expected Answer - 180515 (YYMMDD). I am using Scala for this.
I have tried the below. But am getting the below exception - Text '190515' could not be parsed
import java.time.LocalDateTime
import java.time.format.{ DateTimeFormatter, DateTimeParseException }
val d1 =LocalDateTime.parse("190515",DateTimeFormatter.ofPattern("yyMMdd"))
d1.minusYears(1)
LocalDateTime is used for points of time having both a date and a time precision. Yours only have a date, so you should be using LocalDate
Is there an "instance-local" version of Joda's DateTimeZone.setDefault? I'm trying to achieve something like this:
val parser = new ParserWithDefaultTimezone("GMT+1");
parser.parse("1970-01-01 00:00:00").getMillis // -3600000L
parser.parse("1970-01-01 00:00:00 UTC").getMillis // 0L
Without polluting anything global. All I can find in the Joda docs is something that (ironically) modifies global state.
If there is a non-Joda solution, I am interested in that, too.
EDIT: Forgot to mention that if there is no ready-to-use class that does this, I'll settle for: "what is the easiest way to see if a time string contains an explicit timezone?" I can't distinguish between an explicit one and a timezone that Joda set by default.
EDIT 2: I don't have a format string to feed; I'm looking for something that infers the format at runtime.
You can use withZone to alter the date/time zone used:
val fmt = DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss ZZZ").withZone(DateTimeZone.forID("Europe/Berlin"))
val date = fmt.parseDateTime(???);
The set-up required to make the time zone optional is a little bit more complicated:
val tz = new DateTimeFormatterBuilder().appendLiteral(" ").appendTimeZoneId().toFormatter()
val fmt = new DateTimeFormatterBuilder()
.append(DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss"))
.appendOptional(tz.getParser())
.toFormatter().withZone(DateTimeZone.forID("Europe/Berlin"))
println(fmt.parseDateTime("1970-01-01 12:00:00 UTC"))
println(fmt.parseDateTime("1970-01-01 12:00:00 Europe/Berlin"))
println(fmt.parseDateTime("1970-01-01 12:00:00"))
As long as your remark
I don't have a format string to feed; I'm looking for something that infers the format at runtime.
applies only with respect to the time zone, solution 2 might do what you want. If, on the other hand, you really don't know, in wich format the dates are provided (dd/mm/yyyy vs. mm/dd/yyyy vs. yyyy/mm/dd vs. whatever), then I think you are out of luck: such a conversion would be ambiguous at best. Is 01/03/2015 the 1st of March or the 3rd of January?
I'm using joda time to format my ISO Date input string, but I'm getting an exception that my ISO Date is malformed:
Invalid format: "2014-06-20T11:41:08+02:00" is malformed at "+02:00"
This is my code:
val formatter: DateTimeFormatter = ISODateTimeFormat.dateTime.withZone(DateTimeZone.getDefault)
val date: DateTime = formatter.parseDateTime("2014-06-20T11:41:08+02:00")
What's wrong here?
The error comment is slightly misleading here, as Joda formatter you derive from ISODateTimeFormat expects the millisecond part of the date/time string to be present, therefore the following will work fine:
val formatter: DateTimeFormatter = ISODateTimeFormat.dateTime().withZone(DateTimeZone.getDefault())
val date: DateTime = formatter.parseDateTime("2014-06-20T11:41:08.0+02:00")
The answer by Radyk is correct.
ISO 8601 Formats Built-In
However, you needn't specify a formatter at all. The DateTime class has a built-in parser for your ISO 8601 compliant format, used automatically by the constructor.
DateTime dateTime = new DateTime( "2014-06-20T11:41:08+02:00", timeZone );
While the second argument is optional, I suggest you assign a DateTimeZone object to be assigned to the DateTime if you know such a time zone. The input string has an offset-from-UTC, but a time zone is more than just an offset. A time zone includes rules for Daylight Saving Time and other anomalies. Use proper time zone names, never 3 or 4 letter codes like EST or IST.
Other Formats
You can apply many other formats:
Built-in ISO 8601 formatters
Built-in localized (short, medium, long, and full formats, Locale-sensitive)
Custom specified by you.
For example, if you want only the date portion without the time-of-day in your String representation, call ISODateTimeFormat.date() to access a built-in formatter.
Example code in Joda-Time 2.8.
String output = ISODateTimeFormat.date().print( dateTime ); // Format: yyyy-MM-dd
Search StackOverflow for hundreds of other Questions and Answers about formatting date-time values.
I am having the time-stamped value in nanoseconds for example, 1126732882247990. I am trying to convert into standard utc format but I am not able to convert it as most of the functions for standard utc conversion in MATLAB are limited to microsecond value.
Can someone of you help me to figure out the conversion in MATLAB.
First of all, you don't have nanoseconds as you seem to believe.
The date 1126732882247990 can be decomposed as follows :
1126732882247990 microseconds
1126732882247 milliseconds
1126732882 seconds.
So, If you need a milliseconds precision it is quite simple:
// keep only the milliseconds
long date = Long.parseLong("1126732882247990".substring(0,13));
2005-09-14 23:21:22.247
// then apply whatever conversion you want
DateFormat df = new SimpleDateFormat("yyyy-MM-dd hh:mm:ss.SSS");
df.setTimeZone(TimeZone.getTimeZone("France"));
2005-09-14 09:21:22.247
If the only finality is to display it, you also can add the microseconds:
String timeStamp = df.format(dateObj).concat(".").concat("1126732882247990".substring(13,16));
2005-09-14 09:21:22.247.990
I am trying to get the current timestamp and convert it into a UTC date for an XML file.
I am using this
import java.util.Date;
import java.util.TimeZone;
import com.google.gwt.i18n.client.DateTimeFormat;
DateTimeFormat.format( new Date(), TimeZone.getTimeZone("UTC"));
but am getting the following error
The method format(Date, TimeZone) in
the type DateTimeFormat is not
applicable for the arguments (Date,
TimeZone)
I need the output as "yyyy-mm-ddThh:mm:ssZ"
Use com.google.gwt.i18n.client.TimeZone.createTimeZone(0) to create a UTC TimeZone object, and then use that in DateTimeFormat.format(Date, TimeZone).
You can use apostrophe to indicate literals in a DateTimeFormat pattern.
eg. "HH'o''clock'"
So, the formatter you need would look something like this:
DateTimeFormat formatter = DateTimeFormat.getFormat("yyyy-mm-dd'T'HH:mm:ssZ");
I tried it out. It gave me an output in format 2010-16-29T08:16:23+0530
Is this what you are looking for?
You should provide a com.google.gwt.i18n.client.TimeZone instead of java.util.TimeZone