How to convert unix time format to timestamp in spark [duplicate] - scala

This question already has answers here:
Convert timestamp to date in Spark dataframe
(7 answers)
How to convert unix timestamp to date in Spark
(7 answers)
How do I convert column of unix epoch to Date in Apache spark DataFrame using Java?
(3 answers)
Closed 8 months ago.
Hi I wanted to get the timestamp, appId, and name of the sparkjob that I'm running and log it in a parquet file. I have this line of code:
AuditLog.application_id = spark.sparkContext.applicationId
AuditLog.job_name = spark.sparkContext.appName
AuditLog.execution_timestamp = spark.sparkContext.startTime:Long
For the execution_timestamp, the result is in Unix format and I need to convert it to timestamp with format: yyyy/MM/dd HH:mm:ss

Related

Postgres ERROR: cannot cast type double precision to date [duplicate]

This question already has answers here:
Convert Unix timestamp to timestamp without time zone
(1 answer)
How to convert Unix epoch to a timestamp
(1 answer)
Closed 9 months ago.
I have a table with creation timestamps as decimals and am trying to convert to date data type. When I go to cast as date I get the following error message: "ERROR: cannot cast type double precision to date". []
cast(creation_timestamp as date)

I want to generate current epoch time in scala [duplicate]

This question already has answers here:
Scala UTC timestamp in seconds since January 1st, 1970
(2 answers)
Closed 2 years ago.
I am trying to generate a new epoch timestamp(eg 1435655706000) in scala.
I've used the below function but got time format.
Val timestamp: Timestamp = new Timestamp(System.currentTimeMillis())

Use java.time instances when you work with time
java.time.Instant.now.toEpochMilli

spark dataframe column string to date [duplicate]

This question already has answers here:
Convert pyspark string to date format
(6 answers)
Closed 2 years ago.
I would like to convert a spark dataframe string column of 'yyyyMMdd' to date format with spark session (spark) - not spark context.
Since I'm not working with spark context (sc), I cannot use the following code, although it would precisly do what I'd like it to do:
.withColumn("column1",DF.to_date(F.col("column1"),"yyyyMMdd"))
As I do not want to convert the column to timestamp, I also don't want to use the following code:
.withColumn("column1", unix_timestamp(col("column1"), "yyyyMMdd").cast("timestamp"))
The final goal is to replace the former string column by the date formatted column.
Many thanks in advance!

the following code works fine:
.withColumn("column1", to_date(DF["column1"], 'yyyyMMdd'))
Thanks for your attention!

Spark SQL is not converting timezone correctly [duplicate]

This question already has answers here:
Spark Strutured Streaming automatically converts timestamp to local time
(4 answers)
Closed 4 years ago.
Using Scala 2.10.4 and spark 1.5.1 and spark 1.6
sqlContext.sql(
"""
|select id,
|to_date(from_utc_timestamp(from_unixtime(at), 'US/Pacific')),
|from_utc_timestamp(from_unixtime(at), 'US/Pacific'),
|from_unixtime(at),
|to_date(from_unixtime(at)),
| at
|from events
| limit 100
""".stripMargin).collect().foreach(println)
Spark-Submit options:
--driver-java-options '-Duser.timezone=US/Pacific'
result:
[56d2a9573bc4b5c38453eae7,2016-02-28,2016-02-27 16:01:27.0,2016-02-28 08:01:27,2016-02-28,1456646487]
[56d2aa1bfd2460183a571762,2016-02-28,2016-02-27 16:04:43.0,2016-02-28 08:04:43,2016-02-28,1456646683]
[56d2aaa9eb63bbb63456d5b5,2016-02-28,2016-02-27 16:07:05.0,2016-02-28 08:07:05,2016-02-28,1456646825]
[56d2aab15a21fa5f4c4f42a7,2016-02-28,2016-02-27 16:07:13.0,2016-02-28 08:07:13,2016-02-28,1456646833]
[56d2aac8aeeee48b74531af0,2016-02-28,2016-02-27 16:07:36.0,2016-02-28 08:07:36,2016-02-28,1456646856]
[56d2ab1d87fd3f4f72567788,2016-02-28,2016-02-27 16:09:01.0,2016-02-28 08:09:01,2016-02-28,1456646941]
The time in US/Pacific should be 2016-02-28 00:01:27 etc but some how it subtracts "8" hours twice

after reading for sometime following are the conclusions:
Spark-Sql doesn't support date-time, and nor timezones
Using timestamp is the only solution
from_unixtime(at) parses the epoch time correctly, just that the printing of it as a string changes it due to timezone. It is safe to assume that the from_unixtime will convert it correctly ( although printing it might show different results)
from_utc_timestamp will shift ( not just convert) the timestamp to that timezone, in this case it will subtract 8 hours to the time since (-08:00)
printing sql results messes up the times with respect to timezone param

For the record, here we convert Long values like that using an UDF.
For our purpose, we are interested in only the Date string representation of the timestamp (in ms since epoch in UTC)
val udfToDateUTC = udf((epochMilliUTC: Long) => {
val dateFormatter = java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd").withZone(java.time.ZoneId.of("UTC"))
dateFormatter.format(java.time.Instant.ofEpochMilli(epochMilliUTC))
})
This way, we control the parsing as well as the rendering of the dates.

How to change the date format of the input parameters in jaspersoft report (with class type java.util.date) from MM/DD/YY to DD/MM/YY [duplicate]

This question already has answers here:
How to display date in HH:mm:ss format in JasperReports?
(6 answers)
Create a calendar to pick dates (and time) in Jasper server
(1 answer)
Closed 7 years ago.
I have a jasperreport with datefrom ,dateto, rundate as input parameters.Class type java.util.date. I want to change date format to DD/MM/YY from MM/DD/YY.So what changes do i need to do?

I suggest to use the xml view and try something like this:
new SimpleDateFormat("DD/MM/YY").format($P{datefrom})