I am consuming some external data from postgres (using google data studio):
The column is 'costperhour', why the data type appears as 'hour'? I think is a mistake...
i.e.
could it be that it just converted automatically 15, 10, 12, 8 and 10 to those values?
At the Data Source, it can be achieved by changing the Type of the costperhour field to the required Type, for example:
From:
Date > Hour
To:
Currency > USD - US Dollar ($)
Google Data Studio Report and a GIF to demonstrate
Related
I've pulled in data in Qlik Sense (cloud) from the Google Analytics connector. I'm trying to convert the standard date [ga_dateHourMinute] field to a more eligible date format.
The current format is YearMonthDayhHourMinute, for instance: 201810250004, I would like to convert this to the standard date format 2018-10-25 00:04:00. How do I do this? Answers concerning methods in the data load script or master formula's or variables are all welcome.
Click to see screenshot: left = current date format and right = desired date format
This is one approach:
DateAlter:
LOAD * INLINE [
Date
201810250004
];
DateAlter2:
LOAD
Date(Date#([Date], 'YYYYMMDDhhmm') ,'YYYY-MM-DD hh:mm') AS [MyDateField]
Resident
DateAlter;
If you haven't played around with the data manager, it will allow you to edit fields and handle dates in a fairly easy manner:
Qlik Sense Data Manager
I ended up creating a text fields that combined the different digits from the one field into a new order, then pushed the data format into a date/time. Quite a bit of work, but fairly solid.
I have IOT data with a Unix time. I need to show this to the second in a Data Studio table report. Cannot make this work. In the PubSub function I have:
d = new Date();
d.setTime(res[0] * 1000); // 1000 is for JS time
console.log( d.toISOString());
and later set the field with:
timestamp:d.toISOString(),
Using BigQuery the table shows the field as:
2018-07-06 23:44:49 UTC
which is correct. The 'timestamp' field in Data Studio appears in the table as just:
July 6, 2018
I need to get more resolution, down to the second, for the table and eventually graphs. I've tried custom queries for the data source but cannot get Data Studio to show better resolution.
I tried creating a new field using TODATE(timestamp, 'RFC_3339', "%Y%m%d%H") but cannot that also just shows the July 7, 2018 format. Cannot get it to go to greater resolution.
Yes, I've tried all the other approaches suggested in other questions but none match what I am trying, or they don't succeed.
From what I can see on the DataStudio documentation, this product doesn't support seconds for their date output formats, it only supports up to the minute for output formats and for the 'mm' date type.
I suggest to vote on the related feature requests to support more specific time resolution. 1 2
I have Excel data and trying to insert the data into MongoDB using Talend Big Data for Open Studio. This is my job,
tFileInputExcel --> tMap --> tMongoDBOutput
In excel sheet, i have a date value column in this format 7/13/2017(MM/dd/yyyy) as string type and I am trying to insert this column value as ISO format ISODate("2017-07-13T00:00:00.000Z") in MongoDB.
This is my Job:
tFileInputExcel:
tMap:
tMongoDBOutput:
When execute this job, I'm getting the below error.
Error:
When i change the parse format like this TalendDate.parseDate("MM/dd/yyyy",row1.ClosingDate) , I'm getting SimpleDateFormat error. Simple Date Format Error
How to resolve this issue?
you can do simply if you mongodb column schema is date:
TalendDate.parseDate("MM/dd/yyyy",row3.newColumn)
That will automatically convert the date in the date model that your mongoDB column have.
You can change in your schema in Talend the date Model like "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'".
This is a very common mistake doing in reading data without understanding the underlying data types.
I have blogged about this especially for Talend: https://www.tobiasmaasland.de/2017/07/20/using-date-in-talend-etl-jobs/
But let me explain a bit.
Sometimes Excel tries to convert data in the cell even if one might think the cell type is set to String. Insted, it is set to Date. As such, no conversion is needed and the type needs to be Date in the input component.
If it is a String and an error occurs, the the structure of the String is either not the same everywhere or some cells are empty (null). So you might be lucky with
TalendDate.parseDate("MM/dd/yyyy", (row1.ClosingDate == null), "01/01/1970", row1.ClosingDate)
I just assumed you might want to use a placeholder date insted of having null.
This heavily depends on the actual data type in the cells, if every cell has the same data type and if all the data is formatted correctly.
To sum up one of the facts in my blog post: Don't use String for dates. Use Date for dates in Excel. It makes everything easier.
I have a decimal date field (TDDATR) that is in the YYYYMMDD format.
I would like to create a field that is TDDATR + 30 days but I am unable to.
Using 'Define Results Field' I have tried a few things;
Simply doing this;
TDDATR + 30 DAYS
But it returned this error: Labeled duration not used correctly.
I tried using the DIGITS and SUBSTR commands to create a field in the DDMMYYYY format and then +30 days but got the same error.
Same as above but in the DD/MM/YYYY format - same error.
Using DATE(TDDATR) but all I see is +'s in the field.
Using DATE( ) on the fields created in step 2 and 3 - still get +'s
I've ran out of ideas - any help would be greatly appreciated.
Query/400 lacks a lot of the features that an SQL based interface has.
I'd urge you to consider switching to Query Manager (STRQM) which is a fully SQL based product. You can even convert Query/400 queries to Query Manager queries with the RTVQMQRY command by having the ALWQRYDFN parm set to *YES.
The other option that IBM is pushing is Web Query. Again, fully SQL based and you can convert Query/400 queries into it.
Having said that, the problem is that FLD + 30 DAYS only works when FLD is a DATE data type. Query/400 includes a DATE() function to convert non-date types into date. But it's very limited in that it only works with character fields formatted according to your job defaults. Assuming you're in the US, it'd only work with a character value of '07/01/15'.
You could do a lot of manipulation in Query/400 and end up with a result field that meets DATE()'s requirements. But a better solution would be to create an SQL view over your table and have your numeric date converted into a date data type in the view.
You can find code examples that show how to convert a numeric YYYYMMDD to a actual date data type in the view. However, I'd recommend create a user defined function (UDF) that will do the conversion for you. That will make it much easier to use in the view and to reuse in other places.
If you'd like, there's an open source package called iDate, that includes all the code required for convert to/from date data types.
Download that, install/compile it and your SQL view becomes
select ... idate(TDDATR,'*CCYMD') as TD_DATE
from myfile
The use of days is as follow
Field Expression
CURDATE_30 days(current(date)) + 30
The solution to your problem is: given the field A dec(8,0)
Field Expression
YYYYMMDD_ date(substr(digits(a),5,2)||'/'||
substr(digits(a),7,2)||'/'||
substr(digits(a),3,2))
NEXT_MONTH DAYS(YYYYMMDD_) + 30
Remember to check the date format in your job description. In the example the format is MDY or MM/DD/YY.
More info here
Based on the information here, I created the below 2 fields;
TDDIGI DIGITS(TDDATR)
TDDAT1 SUBSTR(TDDIGI,7,2)||'/'||
SUBSTR(TDDIGI,5,2)||'/'||
SUBSTR(TDDIGI,3,2)
From here I was able to create a date field;
TDDAT2 DATE(TDDAT1)
Which allowed me to perform the necessary calculations.
The format of TDDAT1 is based on your job description which can be found by;
WRKJOB
Option 2
Page down
Date format..: X
Mine was *DMY, so TDDAT1 was formatted based on this.
Running a SELECT against the AS400 using the IBMDA400 OleDb provider appears to return dates as string values, SSRS just laughs at you when you try and apply a date format to the field. I've tried a simple CAST in the SELECT to no avail.
How can I get an actual DBTYPE_DBDATE struct back from the iSeries OleDb provider?
I should mention that the dates in question are all being returned by a UDF with a type of DATE. IBM appears to map DATE type into a DBTYPE_STR OleDb type.
The field(s) in the table(s) are probably not defined as a date type. You will need to convert them using the DATE function as part of the query.
You can use the DSPFFD command, the Navigator, or query the SYSIBM.SQLCOLUMNS table to view the field definitions.
UPDATE
After further testing with the IBMDA400 provider I found the Convert Date Time To Char property hidden away in the OLE DB Technical Reference installed as part of the Programmer's Toolkit with Access. The default value is TRUE. Set Convert Date Time To Char=FALSE in the connection string or properties to disable this 'feature'.
Here's a quick VBA test:
Set cn = CreateObject("ADODB.Connection")
cn.Open "Provider=IBMDA400;Data Source=...;User ID=...;Password=...;Convert Date Time To Char=FALSE"
Set rs = cn.Execute("SELECT DATE(NOW()) FROM SYSIBM.SYSDUMMY1")
MsgBox "Returned ADO type: " & rs.Fields(0).Type
See MSDN: DataTypeEnum for the possible ADO data types. adDBDate is 133.
IBM i Access for Windows OLE DB Technical Reference
IBM i OLE DB provider functions > Special Properties
Convert Date Time To Char
Specifies conversion of DB2 for IBM i Date, Time, and Timestamp data types to corresponding PC data types and vice versa.
Settings and Return Values
Sets or returns one of the following string values. The default value is "TRUE".
"TRUE"
DB2 for IBM i Date, Time, and Timestamp data types are treated as character strings. When reading IBM i data, these values are converted to character strings. When writing data to the system, character strings are expected as input for these values. The supported character string format for the Date data type is ISO format: yyyy-mm-dd. The supported character string format for the Time data type is the earlier version of the ISO format: hh.mm.ss. The supported character string format for the Timestamp data type is: yyyy-mm-dd-hh.mm.ss.nnnnnn.
"FALSE"
DB2 for IBM i Date, Time, and Timestamp data types will be converted to PC Date, Time, and Timestamp data types. Care should be taken when using this value in an environment that only supports the Variant Date data type (such as Visual Basic). You may encounter unexpected errors due to truncation or overflow caused by the limitations of the Variant Date data type.
Following are additional considerations when Convert Date Time To Char is FALSE.
The variant Date data type, which is actually a timestamp, does not support micro-seconds - the precision of the DB2 for IBM i timestamp. The OLE DB provider will truncate the fractional timestamp without reporting an error. For example, 1990-03-02-08.30.00.100517 will become 1990-03-02-08.30.00.000000. All updated or inserted timestamp values will have 0 micro-seconds.
Leap second overflow error. The OLE DB timestamp allows up to two leap seconds (a value of 60 or 61). DB2 for IBM i supports a maximum value of 59. An overflow error is returned if leap seconds are set.
The variant Date data type does not support the data limits of an ISO date or timestamp. The value "0001-01-01", used as a default date in many databases, including DB2 for IBM i, will cause an overflow.
DB2 for IBM i supports a time value of 24:00:00 for certain older formats of the TIME data type. The OLE DB provider will convert values of 24:00:00 to 00:00:00 without any error message or warning.
Typically for VB variant, a date value of 1899-12-30 (which is a 0 date) is used to imply a Time only variant date. A time of Midnight (00:00:00) is used to imply a Date only variant date.
Remarks
This custom property is available on the ADO connection object. The property is read/write when the connection is closed and read-only when the connection is open.
Delphi example
<connection>.Provider := 'IBMDA400';
<connection>.Properties('Convert Date Time To Char') := "TRUE";
OR
<connection>.Open('Provider=IBMDA400;Data Source=SystemA;Convert Date Time To Char =TRUE', 'Userid', 'Password');
PowerBuilder example
<connection>.Provider = "IBMDA400"
SetProperty(<connection>), "Convert Date Time To Char", "TRUE")
OR
<connection>.Open("Provider=IBMDA400;Data Source=SystemA;Convert Date Time To Char=TRUE", "Userid", "Password")
Visual Basic example
<connection>.Provider = "IBMDA400"
<connection>.Properties("Convert Date Time To Char") = "TRUE"
AND/OR
<connection>.Open "Provider=IBMDA400;Data Source=SystemA;Convert Date Time To Char=TRUE", "Userid", "Password")
It appears the correct answer is, you can't. No way, no how, does the IBMDA400 provider map any type into a DBTYPE_DBDATE.
What you can do is use the DateValue() SSRS function to convert the returned DBTYPE_STR value to a date/time serial. From there the format functions will work on it.
I didn't have a problem, here in North America, with the DateValue() function directly interpreting the returned DBTYPE_STR value, however, this could be an issue in other locales due to date format differences.