Attempting to use a built-in Cloudwatch Log function like "date_floor" with a 8601 timestamp string like "2020-03-24T19:03:34.645Z" gives back an "Invalid date" error in Cloudwatch Log Insights.
Does anyone know what is required to parse/cast timestamps that appear within your logs so they can be used with the built-in date/time functions?
My end goal is to compare a timestamp that appears within my logs with the built-in #ingestionTime field.
Related
I'm attempting to call the Azure DevOps API to determine the outcome for test runs using the URL, narrowing the results to a single releaseEnvId:
https://dev.azure.com/organisation/project/_apis/test/runs?api-version=5.0&releaseEnvIds=12345&minLastUpdatedDate=2019-05-17T14:00:00.910Z&maxLastUpdatedDate=2019-05-15T14:00:00.910Z
The API request requires two mandatory date-time fields, but the doc at https://learn.microsoft.com/en-us/rest/api/azure/devops/test/runs/query?view=azure-devops-rest-5.0 does not specify the format:
maxLastUpdatedDate
minLastUpdatedDate
The date-time format I am specifying is wrong because the error comes up:
{"$id":"1","innerException":null,"message":"Invalid dates specified.","typeName":"Microsoft.TeamFoundation.TestManagement.WebApi.InvalidPropertyException, Microsoft.TeamFoundation.TestManagement.WebApi","typeKey":"InvalidPropertyException","errorCode":0,"eventId":3000}
As I understand the doc, the min date must be less than 7 days from the max. if I add garbage to the date formats then it produces a proper date parsing error.
The format you provided it's good but in your example the minLastUpdatedate (5/17) it's after the maxLastUpadeDate (5/15) in it should be the opposite. try to replace the values and it should work.
BTW - you specified the date in the full format, the API should works also with date only - 2019-05-15 or with a time - 2019-05-15T14:00:00.
Is it a requirement that you must use a time or datetime column in postgres to pull metric's on a Grafana dashboard?
I ask because I have a column just with a date only and I'm unable to show metrics base on dates only. Unless I am missing something in the documentation. Postgres in Grafana
If anyone has any helpful information, it would be greatly appreciated.
All I need is just a starting point.
I am a new user to Grafana and I'm trying to figure this out.
According to documentation:
If you set Format as to Time series, for use in Graph panel for example, then the query must return a column named time that returns either a sql datetime or any numeric datatype representing unix epoch in seconds.
So, all you need is to convert your column with date to unix epoch and name it as time. You can do it yourself or use macros provided by grafana (see docs) for convertion which are expanded into native postgres expressions. It is also helpful to look at 'Generated SQL' to see the actual query sent to database.
I m having some issues with converting a regular timestamp in cassandra with Apache Nifi.
My use case is following:
I have a csv file with a date in it looking like this ('2015010109') and I want to put it in cassandra by converting this string ('2015010109') to an proper format: 2015-01-01 09:00 -> yyyy/MM/dd HH:mm (I dont exactly need the minutes, but I guess it is more useful for later usage)
So far I got this propertie in my UpdateAttribute processor when trying to convert this string to a timestamp:
date : ${csvfiledate:toDate("yyyyMMddHH","GMT"):format("yyyy-MM-dd-HH")}
but then there is an error occuring in my PutCassandraQL processor: Unable to coerce '2015-01-01-09' to a formatted date (long).
I tried something along
date : ${csvfiledate:toDate("yyyyMMddHH","GMT"):format("yyyy-MM-dd HH-mmZ")} aswell, but the same error is occuring.
It seems like you need to have a specific timestamp type for cassandra as you can see here:
http://docs.datastax.com/en/archived/cql/3.0/cql/cql_reference/timestamp_type_r.html
But it isnt working so far, maybe you got some tipps.
Thanks in advance.
You've got it backwards..
toDate parameters are used to describe how to parse the date. format function is used to describe how the date output should be. So the expression should be:
${csvfiledate:toDate('yyyy-MM-dd-HH','GMT'):format('yyyyMMddHH')}
I am using Tableau 9.3 to do a preliminary data analysis on one of my log file, the log file is like below:
"199.72.81.55",01/Jul/1995:00:00:01,/history/apollo/,200,6245,Sat
As you can see, there is a datetime for timestamp
In Tableau, initially it is recognized as a string like below:
That's fine, I want to make the field into datetime, and Tableau seems failed on it:
Why? How do I fix it?
Thank you very much.
UPDATED: after applying the formula suggested below, Tableau still cannot recognize the timestamp, here is the screenshot:
UPDATED AGAIN: after tested by nick, it is confirmed his first script is correct and working on his Tableau, why it fails on mine, I don't know, you are welcome to share any clue please, thank you.
Tableau implicit conversions are limited to more standard formats. You can still create a DATETIME field from your timestamp string using a calculated field with the following formula:
DATEPARSE('dd/MMM/yyyy:HH:mm:ss',[timestamp])
Using the above will transform a string like 01/Jul/1995:00:00:01 to a date and time of 7/1/1995 12:00:01 AM
Output using example data:
Sometimes the "date parse" function in Tableau doesn't quite do the job.
When this happens it is worth testing manual string manipulation with your timestamp field to put it into ISO-standard format and only then trying to convert it into a date. ISO format is yyyy-mm-dd hh:mm:ss (eg 2012-02-28 13:04:30). It is common to find that the original string has spurious characters or spaces that throw dateparse. But these are usually easy to manipulate away with suitable text manipulations. This can sometimes be longwinded, but it always works.
It turned out to be the region setting issue, it works after I switch it to USA
I have imported some data into SAS from some Excel spreadsheets sent to me. When I view the output from the imported table, the date appears as "01APR2014" and maintains chronological order. When I view the column properties the type is "Date" and the length is 8. Both the format and informat are DATE9.
I need to be able to convert this date to week-year and month-year, but no matter what I try I always get Jan, 1960.
Using proc sql, I used the below to get the week-year,
"(put(datepart(a.fnlz_date),weeku3.))|| "-" ||(put(datepart(a.fnlz_date),year.)) as FNLZD_WK_YR,"
but all I got was "W00-1960". I've used the formula above successfully many times before with SAS datetime values.
For month-yr, using proc sql, I tried
"datepart(a.fnlz_date) as DT_FNLZD format=monyy.,"
but the only value returned is "JAN60".
I also tried using SUBSTR, but got an error saying it requires a character argument, so SAS must see it as a number at least.
My question; does anyone know a way to get the week-yr and/or month-yr from this format? If so, how? I'm not opposed to using a data step, but I haven't been able to get that to work either.
Thanks in advance for any help or insight provided.
datepart converts datetimes to dates. Not helpful here.
If you're just displaying this, then you have a few options, particularly for month. You can just change the format of the variable (This changes what's displayed, but not the underlying value; consider this a value label).
When you use this like this (again, it looks like you got most of the way there):
proc sql;
select datevar format=monyy5. from table;
quit;
Just don't include that datepart function call as that's not appropriate unless you have a datetime. (Date=# of days since 1/1/1960, Datetime = # of seconds since 1/1/1960:00:00:00).
That will display it with MONYY5. format, which would be MAY10 for May, 2010. You have some other format options, see the documentation on formats by category for more details.
I can't think of a Week format that matches what you want (there are week formats, like WEEKW., as you clearly found, but I don't know that they do exactly what you want. So, if you want to build one yourself, you can either build a custom picture format, or you can make a string.
Building a custom picture format isn't too hard; see the documentation on Picture formats or google SAS Date Picture Format.
proc format;
picture weekyear (default=8)
low-high = 'W%0U-%Y' (datatype=date) ;
quit;
Now you can use that as a normal format.
To get at the week/etc. to build values, you can also use functions week(), month(), etc., if that's easier.
Since the data was already in a date format, I only needed to drop the DATEPART function that only works with datetime values. So, for month-yr,
"a.fnlz_date as fnlz_mnth format=monyy.,"
gives me the results I'm looking for.
Cheers!