I have a timestamp pulled from MongoDB
example: 2007-01-01 01:00:00
I need it to be a simple date: 2007-01-01
Been looking at: Convert UNIX epoch to Date object
Having a hard time formatting
Using sources:
Convert column in data.frame to date
I had to assess if the class that I was trying to convert was a class that as.Date was able to take.
Turns out it was a data.frame class
Used the following to convert it:
dat_dump %>%
mutate( date = as.Date(date, format = "%Y-%m-%d"))
Related
I have a field in a dataframe that has a column with date like 1632838270314 as an example
I want to convert it to date like 'yyyy-MM-dd' I have this so far but it doesn't work:
date = df['createdOn'].cast(StringType())
df = df.withColumn('date_key',unix_timestamp(date),'yyyy-MM-dd').cast("date"))
createdOn is the field that derives the date_key
The method unix_timestamp() is for converting a timestamp or date string into the number seconds since 01-01-1970 ("epoch"). I understand that you want to do the opposite.
Your example value "1632838270314" seems to be milliseconds since epoch.
Here you can simply cast it after converting from milliseconds to seconds:
from pyspark.sql import functions as F
df = sql_context.createDataFrame([
Row(unix_in_ms=1632838270314),
])
(
df
.withColumn('timestamp_type', (F.col('unix_in_ms')/1e3).cast('timestamp'))
.withColumn('date_type', F.to_date('timestamp_type'))
.withColumn('string_type', F.col('date_type').cast('string'))
.withColumn('date_to_unix_in_s', F.unix_timestamp('string_type', 'yyyy-MM-dd'))
.show(truncate=False)
)
# Output
+-------------+-----------------------+----------+-----------+-----------------+
|unix_in_ms |timestamp_type |date_type |string_type|date_to_unix_in_s|
+-------------+-----------------------+----------+-----------+-----------------+
|1632838270314|2021-09-28 16:11:10.314|2021-09-28|2021-09-28 |1632780000 |
+-------------+-----------------------+----------+-----------+-----------------+
You can combine the conversion into a single command:
df.withColumn('date_key', F.to_date((F.col('unix_in_ms')/1e3).cast('timestamp')).cast('string'))
I am using Scala spark.I have two similar CSV files with 10 columns.One difference is with the Date column format.
1st file Date format yyyy-MM-dd
2nd file Date format dd-MM-yyyy
Objective is to: create seperate schema rdd for each file and finally merge both the Rdds.
For the first case class, I have used Date.valueOf [java.sql.Date] in the case class mapping.No issues here..
Am finding issue with the 2nd file Date format..
I have used the same Date.valueOf mapping..but it's throwing error in the date format...
How can I map the date format in the second file as like the 1st format yyyy-MM-dd? Please assist
Use java.util.Date:
val sDate1="31/12/1998"
val date1=new SimpleDateFormat("dd/MM/yyyy").parse(sDate1)
import java.text.SimpleDateFormat
Result:
sDate1: String = 31/12/1998
date1: java.util.Date = Thu Dec 31 00:00:00 CET 1998
to change the output format as a common string format.
val date2=new SimpleDateFormat("yyyy/MM/dd")
date2.format(date1)
Result:
res1: String = 1998/12/31
I need to convert the date string of type WEEKDAY DATE MONTHNAME, Example: from "Monday 5 October" to date object.
I have tried with
Utilities.formatDate(new Date("Monday 5 October"), "GMT", "yyyy-MM-dd'T'HH:mm:ss'Z'")
How do I convert it, I am ok using V8 apps script engine
The Date constructor accepts timestamp strings formatted according to IETF-compliant RFC 2822 timestamps and ISO8601.
There are many ways to convert your string to date, but probably one of the simplest is appending the current year to your string, using getFullYear():
const source = "Monday 5 October";
const date = new Date(`${source} ${new Date().getFullYear()}`);
Reference:
Date() constructor
IETF-compliant RFC 2822 timestamps
I am trying to convert a long utc value into "yyyy-MM-dd HH:mm:ss" formatted pattern. I am expecting my data to be converted on 24 hours range scale and in GMT. My code passes all the test cases, I push the data into database using the jar that is newly built with this code -
dbRecord("order_dt_utc") = if (orderTs.isDefined) Some(new DateTime(orderTs.get, DateTimeZone.UTC).toString("yyyy-MM-dd HH:mm:ss")) else None
and now, when I query my database, I find that the data is still converting on 12 hours range. The query -
SELECT order_id, order_dt, order_dt_utc, order_ts_utc, from_unixtime(order_ts_utc/1000) FROM order_items where order_dt >= '2018-08-01' AND order_dt <= '2018-08-02' ORDER BY order_dt_utc LIMIT 1000;
And you can see the the values are not matching in the columns from_unixtime(order_ts_utc/1000) and order_dt_utc -
I am not able to figure the reason for this behaviour.
To convert Time Zone use the function first:
CONVERT_TZ (dateobj, oldtz, newtz)
After that use the date_format function:
date_format(from_unixtime(order_ts_utc), '%Y-%m-%d %H:%i:%s');
to format your time to 00-23 format.
I am using Databricks and SparkR, trying to extract the month from a Date field but keep getting the following error, when trying to using the function month():
Error in as.POSIXlt.default(x, tz = tz(x)) :
Error in as.POSIXlt.default(x, tz = tz(x)) :
do not know how to convert 'x' to class “POSIXlt”
If I use dayofmonth()it runs perfectly so I am not sure what i am doing wrong.
Here is a sample of the fields i am trying to work with.
$ Net_due_date : Date 2017-10-06 2017-10-05 2018-01-17 2017-12-23 2017-08-20 2018-01-17
$ Clearing_Date : Date 2017-10-06 2017-10-17 1900-01-01 2017-12-26 2017-08-24 2018-01-19
Any ideas?
Thanks
I can't be sure without a reproducible example, but try casting the column to timestamp first.
df$Net_due_date_month <- month(cast(df$Net_due_date, 'timestamp'))
You can provide a small subset of your data so that the error is replicated. I suspect that your date column are not date format.
You can try any of these below:
library(lubridate)
df$Net_due_date_month <- month(as.Date(df$Net_due_date, format = '%Y-%m-%d'))
or
df$Net_due_date_month <- month(as.POSIXlt(df$Net_due_date, format = '%Y-%m-%d'))