I m having some issues with converting a regular timestamp in cassandra with Apache Nifi.
My use case is following:
I have a csv file with a date in it looking like this ('2015010109') and I want to put it in cassandra by converting this string ('2015010109') to an proper format: 2015-01-01 09:00 -> yyyy/MM/dd HH:mm (I dont exactly need the minutes, but I guess it is more useful for later usage)
So far I got this propertie in my UpdateAttribute processor when trying to convert this string to a timestamp:
date : ${csvfiledate:toDate("yyyyMMddHH","GMT"):format("yyyy-MM-dd-HH")}
but then there is an error occuring in my PutCassandraQL processor: Unable to coerce '2015-01-01-09' to a formatted date (long).
I tried something along
date : ${csvfiledate:toDate("yyyyMMddHH","GMT"):format("yyyy-MM-dd HH-mmZ")} aswell, but the same error is occuring.
It seems like you need to have a specific timestamp type for cassandra as you can see here:
http://docs.datastax.com/en/archived/cql/3.0/cql/cql_reference/timestamp_type_r.html
But it isnt working so far, maybe you got some tipps.
Thanks in advance.
You've got it backwards..
toDate parameters are used to describe how to parse the date. format function is used to describe how the date output should be. So the expression should be:
${csvfiledate:toDate('yyyy-MM-dd-HH','GMT'):format('yyyyMMddHH')}
Related
Please help me to convert string to timestamp.
source data is in Excel
Need to convert it as below timestamp
2019-12-15T16:35:53.663-04:00
I tried with.
select from_unixtime(unix_timestamp('12/15/2019 21:18','mm/dd/yyyy'),'YYYY-MM-DDT00:00:00-00:00')
Got below error
Both source pattern and target pattern are wrong in your query. See SimpleDateFormat for reference. Also initial string does not contain the timezone and it is not clear how are you going to derive it as -04:00. In such case it will be UTC timezone used, you can convert to other timezone using from_utc_timestamp.
Timestamp string conversion demo:
select from_unixtime(unix_timestamp('12/15/2019 21:18','MM/dd/yyyy HH:mm'),"yyyy-MM-dd'T'HH:mm:ss.SSSZ")
Result:
2019-12-15T21:18:00.000+0000
I am trying to change the format of a timestamp field to a date field in a certain format with AWS Redshift.
I googled a lot and the common "best practice" that I found was to cast the timestamp to a date and then use to_char to bring it into the right format. In the end I want to use the date field in a Qlik Sense dashboard where it is input for a date picker extension that apparently requires the format DD/MM/YYYY to work.
Current date format in the DB:
9/2/2019 6:38:00 AM (which I would describe as M/D/YYYY H:MM:SS ZZ)
Desired output:
DD/MM/YYYY, resulting in the value 02/09/2019
Current status:
to_char(cast(timestamp_field as date), 'DD/MM/YYYY') --> result: 02/09/2019
However, the date picker extension in Qlik Sense still does not work and I guess that's because the output is a string and not a date. Casting the string to a date returns in an error.
cast(to_char(cast(timestamp_field as date), 'DD/MM/YYYY') as date) as date_picker_date
Connector reply error: SQL##f - SqlState: 57014, ErrorCode: 30,
ErrorMsg: [Amazon][Amazon Redshift] (30) Error occurred while trying
to execute a query: [SQLState 57014] ERROR: Error converting text to
date
I am new to Redshift and would have expected to be able to cast to date with a format string as parameter, but apparently that's not a thing. Can someone enlighten me on how to solve this?
You can try with the timestamp at the beginning of the script of the QlikSense.
It will be at the start of the script. in the Main section as shown below:
SET TimestampFormat='M/D/YYYY h:mm:ss[.fff] TT';
here SET is being used to define the time stamp in that application.
You can try changing here to your requirements.
Else 2 options 1 use a stamp which you have already used or create your own extension from the lib
Stamp use can be like this:
Datestamp_Table:
Load *,
Date(Date#(StringDate,'M/D/YY'),'DD/MMM/YY') as Date;
LOAD * INLINE [
StringDate
8/7/97
8/6/97];
Output will be like this:
Stringdate Date
8/7/97 07/Aug/97
8/6/97 06/Aug/97
I've looked everywhere and can't find this answer. It's a pretty simple query, but I can't for the life of me figure out how to change the date.
I have a date coming in as a string, but it's not being picked up. The date is being brought in as 20170601 but I need it to be in a date format to be picked up in Tableau. I'm using Standard SQL and have tried to PARSE_DATE("%x", date) as parsed, cast(date as date), etc. and I keep getting Error: Failed to parse input string "20170918" or some variation of that error.
#standardSQL
SELECT
visitorid,
parse_DATE("%x", date) AS parse
FROM google.com:analytics-bigquery.LondonCycleHelmet.ga_sessions_20130910
The table is within `
Please advise!!
You could try doing a regex replacement to build the date string which you require:
SELECT
REGEXP_REPLACE('20170601', r"^([0-9]{4})([0-9]{2})([0-9]{2})", "\\1/\\2/\\3")
This would output 2017/06/01, which perhaps is the format you require. Actually, I don't know what format Tableau is expecting, but YYYYMMDD is usually the correct order for a date, because it will sort correctly as text. You may use any replacement you want, using the above query as an example.
I am using Tableau 9.3 to do a preliminary data analysis on one of my log file, the log file is like below:
"199.72.81.55",01/Jul/1995:00:00:01,/history/apollo/,200,6245,Sat
As you can see, there is a datetime for timestamp
In Tableau, initially it is recognized as a string like below:
That's fine, I want to make the field into datetime, and Tableau seems failed on it:
Why? How do I fix it?
Thank you very much.
UPDATED: after applying the formula suggested below, Tableau still cannot recognize the timestamp, here is the screenshot:
UPDATED AGAIN: after tested by nick, it is confirmed his first script is correct and working on his Tableau, why it fails on mine, I don't know, you are welcome to share any clue please, thank you.
Tableau implicit conversions are limited to more standard formats. You can still create a DATETIME field from your timestamp string using a calculated field with the following formula:
DATEPARSE('dd/MMM/yyyy:HH:mm:ss',[timestamp])
Using the above will transform a string like 01/Jul/1995:00:00:01 to a date and time of 7/1/1995 12:00:01 AM
Output using example data:
Sometimes the "date parse" function in Tableau doesn't quite do the job.
When this happens it is worth testing manual string manipulation with your timestamp field to put it into ISO-standard format and only then trying to convert it into a date. ISO format is yyyy-mm-dd hh:mm:ss (eg 2012-02-28 13:04:30). It is common to find that the original string has spurious characters or spaces that throw dateparse. But these are usually easy to manipulate away with suitable text manipulations. This can sometimes be longwinded, but it always works.
It turned out to be the region setting issue, it works after I switch it to USA
So I have one big file (13 million rows) and date formatted as:
2009-04-08T01:57:47Z. Now I would like to split it into 2 columns now,
one with just date as dd-MM-yyyy and other with time only hh:MM.
How do I do it?
You can simply use tMap and parseDate/formatDate to do what you want. It is neither necessary nor recommended to implement your own date parsing logic with regexes.
First of all, parse the timestamp using the format yyyy-MM-dd'T'HH:mm:ss'Z'. Then you can use the parsed Date to output the formatted date and time information you want:
dd-MM-yyyy for the date
HH:mm for the time (Note: you mixed up the case in your question, MM stands for the month)
If you put that logic into a tMap:
you will get the following:
Input:
timestamp 2009-04-08T01:57:47Z
Output:
date 08-04-2009
time 01:57
NOTE
Note that when you parse the timestamp with the mentioned format string (yyyy-MM-dd'T'HH:mm:ss'Z'), the time zone information is not parsed (having 'Z' as a literal). Since many applications do not properly set the time zone information anyway but always use 'Z' instead, so this can be safely ignored in most cases.
If you need proper time zone handling and by any chance are able to use Java 7, you may use yyyy-MM-dd'T'HH:mm:ssXXX instead to parse your timestamp.
I'm guessing Talend is falling over on the T and Z part of your date time stamp but this is easily resolved.
As your date time stamp is in a regular pattern we can easily extract the date and time from it with a tExtractRegexFields component.
You'll want to use "^([0-9]{4}-[0-9]{2}-[0-9]{2})T([0-9]{2}:[0-9]{2}):[0-9]{2}Z" as your regex which will capture the date in yyyy-MM-dd format and the time as mm:HH (you'll want to replace the date time field with a date field and a time field in the schema).
Then to format your date to your required format you'll want to use a tMap and use TalendDate.formatDate("dd-MM-yyyy",TalendDate.parseDate("yyyy-MM-dd",row7.date)) to return a string in the dd-MM-yyyy format.