I am using Databricks and SparkR, trying to extract the month from a Date field but keep getting the following error, when trying to using the function month():
Error in as.POSIXlt.default(x, tz = tz(x)) :
Error in as.POSIXlt.default(x, tz = tz(x)) :
do not know how to convert 'x' to class “POSIXlt”
If I use dayofmonth()it runs perfectly so I am not sure what i am doing wrong.
Here is a sample of the fields i am trying to work with.
$ Net_due_date : Date 2017-10-06 2017-10-05 2018-01-17 2017-12-23 2017-08-20 2018-01-17
$ Clearing_Date : Date 2017-10-06 2017-10-17 1900-01-01 2017-12-26 2017-08-24 2018-01-19
Any ideas?
Thanks
I can't be sure without a reproducible example, but try casting the column to timestamp first.
df$Net_due_date_month <- month(cast(df$Net_due_date, 'timestamp'))
You can provide a small subset of your data so that the error is replicated. I suspect that your date column are not date format.
You can try any of these below:
library(lubridate)
df$Net_due_date_month <- month(as.Date(df$Net_due_date, format = '%Y-%m-%d'))
or
df$Net_due_date_month <- month(as.POSIXlt(df$Net_due_date, format = '%Y-%m-%d'))
Related
I have a timestamp pulled from MongoDB
example: 2007-01-01 01:00:00
I need it to be a simple date: 2007-01-01
Been looking at: Convert UNIX epoch to Date object
Having a hard time formatting
Using sources:
Convert column in data.frame to date
I had to assess if the class that I was trying to convert was a class that as.Date was able to take.
Turns out it was a data.frame class
Used the following to convert it:
dat_dump %>%
mutate( date = as.Date(date, format = "%Y-%m-%d"))
I have a set of data stored in neo4j in milliseconds.
I'm trying to get the data then change it to date format and compare it to today's date in where clause to get Media posts for today only.
I have tried with these
MATCH (media:Media)
RETURN date(datetime({millisecond:media.dateCreated}))
and it returns
Neo.ClientError.Statement.ArgumentError: year must be specified
next, I've tried
MATCH (media:Media)
RETURN apoc.date.field(media.dateCreated)
and it returns
Neo.ClientError.Statement.SyntaxError: Unknown function
'apoc.date.field' (line 2, column 28 (offset: 45)) "MATCH
(media:Media) RETURN apoc.date.field(media.dateCreated)" ^
I have tried multiple ways that more or less return the same kind of error
I expect the data to be shown in date format instead of milliseconds.
You can use epochMillis to create the date from milliseconds.
MATCH (media:Media)
RETURN date(datetime({epochMillis:media.dateCreated}))
This returns the date in the format as shown in the following screenshot:
This query:
RETURN datetime({epochMillis: 1475292465000});
returns the datetime that corresponds to the epoch timestamp 1475292465000:
╒════════════════════════════════════════╕
│"datetime({epochMillis: 1475292465000})"│
╞════════════════════════════════════════╡
│"2016-10-01T03:27:45Z" │
└────────────────────────────────────────┘
If you actually want a date, you can use this query:
RETURN date(datetime({epochMillis: 1475292465000}));
I am trying to convert a long utc value into "yyyy-MM-dd HH:mm:ss" formatted pattern. I am expecting my data to be converted on 24 hours range scale and in GMT. My code passes all the test cases, I push the data into database using the jar that is newly built with this code -
dbRecord("order_dt_utc") = if (orderTs.isDefined) Some(new DateTime(orderTs.get, DateTimeZone.UTC).toString("yyyy-MM-dd HH:mm:ss")) else None
and now, when I query my database, I find that the data is still converting on 12 hours range. The query -
SELECT order_id, order_dt, order_dt_utc, order_ts_utc, from_unixtime(order_ts_utc/1000) FROM order_items where order_dt >= '2018-08-01' AND order_dt <= '2018-08-02' ORDER BY order_dt_utc LIMIT 1000;
And you can see the the values are not matching in the columns from_unixtime(order_ts_utc/1000) and order_dt_utc -
I am not able to figure the reason for this behaviour.
To convert Time Zone use the function first:
CONVERT_TZ (dateobj, oldtz, newtz)
After that use the date_format function:
date_format(from_unixtime(order_ts_utc), '%Y-%m-%d %H:%i:%s');
to format your time to 00-23 format.
I'm currently parsing a time string and saving it to the db (Postgresql):
event.Time, _ := time.Parse("3:04 PM", "9:00 PM")
// value of event.Time now is: 0000-01-01 21:00:00 +0000 UTC
db.Create(&event)
It's giving me this error: pq: R:"DateTimeParseError" S:"ERROR" C:"22008" M:"date/time field value out of range: \"0000-01-01T21:00:00Z\"" F:"datetime.c" L:"3540"
event.Time's type is time.Time.
I also tried setting event.Time's type to string and using time data type in postgresql:
type Event struct {
Time string `gorm:"type:time
}
But now I'm getting an error when fetching records in the db:
sql: Scan error on column index 4: unsupported driver -> Scan pair: time.Time -> *string
Investigated this issue further. Currently, there's no support in GORM for any Date/Time types except timestamp with time zone
See this part of code from dialect_postgres.go:
case reflect.Struct:
if _, ok := dataValue.Interface().(time.Time); ok {
sqlType = "timestamp with time zone"
}
So basically I see two options for you:
Either use varchar(10) in DB, and string in Go, an simply save it as "9:00 PM" (where 10 is some number that suits you)
Or use timestamp with time zone in DB, time.Time in Go, and format your date part as a constant date, 01/01/1970, for example:
time.Parse("2006-01-02 3:04PM", "1970-01-01 9:00PM")
In that case you'll have to omit the date part in your presentation, but if you plan to select by date range, that could work better for you.
You can set an arbitrary database-specific type with Gorm using sql tag
type Event struct {
Time time.Time `sql:"type:timestamp without time zone"`
}
When updating the DATETIME field in SQL, the Go string must be in this format: time.Now().Format(time.RFC3339).
From Postgres perspective the error stems from there being no year 0000. If you don't the date you may just be able to add 1 year to the converted timestamp giving '0001-01-01T21:00:00+00' which is a valid Postgres timestamp.
select '0000-01-01T21:00:00+00'::timestamptz at time zone 'UTC'
ERROR: date/time field value out of range: "0000-01-01T21:00:00+00"
Gives he same error. And just as a demonstration 1 day before 0001-01-01 gives:
select '0001-01-01T21:00:00+00'::timestamptz at time zone 'UTC' - interval '1 day' "day_before_1/1/1";
--day_before_1/1/1
--0001-12-31 21:00:00 BC
Currently I'm storing it as a String, but got problems using it when it comes to querying by date with GQL.
Date date = Calendar.getInstance().getTime();
DateFormat formatter = new SimpleDateFormat("hh:mm:ss, z");
String todayDate = formatter.format(date);
The query:
"SELECT FROM SomeTable p WHERE date = 01/01/2011"
Error:
Exception:
org.datanucleus.store.appengine.query.DatastoreQuery$UnsupportedDatastoreFeatureException:
Problem with query : Right side of expression is
composed of unsupported components.
Left:
org.datanucleus.query.expression.Literal,
Op: / , Right:
DyadicExpression{Literal{5} /
Literal{11}}
How can I search by date?
Always as a date - the database should check on the types of data that it stores thus making sure the data is what it says. the information about what type is known on the insert so check it then rather than later when none has any idea of what the correct value should have been.
the error you are getting is probably not due to this but because the date in the query needs to be made a date if types are correct or if a string needs to be as a single-quoted string see GQL Reference
e.g.
"SELECT FROM SomeTable p WHERE date = '01/01/2011'"
but what date is 06/07/2011 ? I think it is today 6th July, others 7th June. So even if you use strings use the ISO format 2011-07-06 A date type will hide this detail making it easier. Note that GQL only uses the ISO form so even if you got the command to have the date in a string it would fail.
but better as
SELECT FROM SomeTable p WHERE date = DATE( 2011, 1, 1)