following 1 I started creating a Spreadsheet which reads data from BigQuery, but I'm having an issue handling parameters related to date values.
In the first sheet, I created 2 cells with 2 parameters, the start and the end of a date interval, with proper values. Both cells are formatted as "Date" value.
In the second sheet I configured BigQuery connector, for this example, I'm using a public dataset with dates. bigquery-public-data.utility_eu.date_greg
From the BigQuery connector wizard I added:
"STARTDATE" as "PARAMETERS!B1"
"ENDDATE" as "PARAMETERS!B2"
After this configuration, this is the resulting query:
SELECT
date,
date_str,
date_int
FROM `bigquery-public-data.utility_eu.date_greg`
WHERE date > DATE(#STARTDATE) AND date < DATE(#ENDDATE)
LIMIT 10
I'm getting an error directly from the editor with this message:
> Error BigQuery: No matching signature for function DATE for argument types: INT64. Supported signatures: DATE(TIMESTAMP, [STRING]); DATE(DATETIME); DATE(INT64, INT64, INT64) at [8:14]
As far as I can understand, the "date" cells are retrieved as a number, so the direct parse is not working. After a couple of tests, I understood the that given int value is the number I can obtain change cell format to "number".
If you convert cell value from DATE to NUMBER you get this value:
01/05/2019 -> 43.586
31/05/2019 -> 43.616
What is this number? It is not milliseconds, it increases by 1 every next day. In order to create the proper query that can parse this int, I need to understand what is this int (of course I can handle the cell as "text" and writing the timestamp value directly, but I would prefer to have the native date format so I can use the built-in calendar.
My consideration (with simple math) is that this number refers to a number of days since 30/12/1899, but it is very odd (also, every date BEFORE this days is always 0), so I'm asking you directly how to handle this value. Basing on my understanding of when the number counter starts (30/12/1899), I created this query which add the number retrieved from the cell:
SELECT *
FROM `bigquery-public-data.utility_eu.date_greg`
WHERE
date >= DATE_ADD(DATE("1899-12-30"), INTERVAL #DATAINIZIO DAY)
AND date <= DATE_ADD(DATE("1899-12-30"), INTERVAL #DATAFINE DAY)
It is working... but I think I'm doing a workaround that is not the proper way of doing this.
Also, is there any full documentation related to this BigQuery connection provided by Spreadsheet? Besides presentation in 1 I'm unable to find any specific documentation.
Spreadsheets (Google, Excel, ...) store the dates as days passed since a starting date with a fractional day representing time.
From here: "Excel stores dates and times as a number representing the number of days since 1900-Jan-0, plus a fractional portion of a 24 hour day: ddddd.tttttt . This is called a serial date, or serial date-time."
Now, you have to ways to filter by date on your Query:
In the query, you can use DATE_ADD to add your number of days (cell value) to the base date. (Carefull, DATE_ADD takes INT, and the date value is float so needs prior casting).
(preferred) on your spreadsheet you use TEXT(cell, "yyyy-mm-dd") so you can then use DATE() in the BigQuery query.
I use the second method as, though you need that extra cell (unless you directly store the date as YYYY-MM-DD; keeps the query cleaner than having a cast and date_add in there. Also would save you from the "1904 problem" explained in the link above.
What is this number? It is not milliseconds, it increases by 1 every next day.
This is so called serial number which represent number of days since "very beginning"
Google's Spreadsheet date calendar starts from 1900-01-01 - which is treated as a "very beginning"
In order to create the proper query that can parse this int, I need to understand what is this int
Armed with above info you can adjust you dates calculation to be in sync with what BigQuery expects
You mentioned that your fields are already in Date format, maybe you are doing an extra parsing in your query.
Try to do it without the DATE functions.
Also, I found this other doc, not merely related to connection, but might be helpful: Getting info from Spreadsheets with BigQuery.
Related
I am very new to Azure Data Factory. I have created a simple Pipeline using the same source and target table. The pipeline is supposed to take the date column from the source table, apply an expression to the column date (datatype date as shown in the schema below) in the source table, and it is supposed to either load 1 if the date is within the last 7 days or 0 otherwise in the column last_7_days (as in schema).
The schema for both source and target tables look like this:
Now, I am facing a challenge to write an expression in the component DerivedColumn. I have managed to find out the date which is 7 days ago with the expression: .
In summary, the idea is to load last_7_days column in target Table with value '1' if date >= current date - interval 7 day and date <= current date like in SQL.I would be very grateful, if anyone could help me with any tips and suggestions. If you require further information, please let me know.
Just for more information: source/target table column date is static with 10 years of date from 2020 till 2030 in yyyy-mm-dd format. ETL should run everyday and only put value 1 to the last 7 days: column last_7_days looking back from current date. Other entries must recieve value 0.
You currently use the expression bellow:
case ( date == currentDate(),1, date >= subDays(currentDate(),7),1, date <subDays(currentDate(),7,0, date > currentDate(),0)
If we were you, we will also choose case() function to build the expression.
About you question in comment, I'm afraid no, there isn't an another elegant way for. To achieve our request, Data Flow expression can be complex. It may be comprised with many functions. case() function is the best one for you.
It's very clear and easy to understand.
I am extracting three values (server, region, max(date)) from my postgresql> But I want to extract an additional 4th field which should be the numerical addition of 1 to 3rd field. I am unable to use date add function as in the database date field is defined as an integer.
date type in DB
date|integer|not null
tried using cast and date add function
MAX(s.date)::date + cast('1 day' as interval)
Error Received
ERROR: cannot cast type integer to date
Required output
select server, region, max(alarm_date), next date from table .....
testserver, europe, 20190901, 20190902
testserver2, europe, 20191001, 20191002
next date value should be the addition to alarm_date
To convert an integer like 20190901 to a date, use something like
to_date(CAST(s.date AS text), 'YYYYMMDD')
It is a bad idea to store dates as integers like that. Using the date data type will prevent corrupted data from entering the database, and it will make all operations natural.
First solution that came to my mind:
select (20190901::varchar)::date + 1
Which output 2019-09-02 as type date.
Other solutions can be found here.
First of all im a starter,iam using eclipse.
I want to add current date and time of login in to db and search a day and find out the time between login and logout.
°What is the data type for the date colum in mysql ?
°Is it necessary separate column for date and time ?
°which one i want to import, java.util.date or java.sql.date ?
°In Java code simple date format or calender is better ?
Advanced thanks.....
You might want to read this:
Should I use field 'datetime' or 'timestamp'?
For example, if you have mysql populate the log record's date/time (using "DEFAULT CURRENT_TIMESTAMP" in your field definition), you will want to use timestamp. For certain situations where you fill a date value from your application, you may wish to use datetime. Be careful with timezones.
Here are the date functions in mysql:
https://dev.mysql.com/doc/refman/5.7/en/date-and-time-functions.html
DATEDIFF(), for example, will calculate the number of days between two datetimes. If you use unix timestamps, you can use standard arithmetic between the values to find the number of seconds between them, and then calculate minutes, hours, days, etc. by dividing appropriately.
This answer is focused on how to handle the dates in mysql. Not enough info to provide guidance on java.
Tableau is reading my dates wrong. I have 2 columns, Date and number for each day.
The date format is “yyyymmdd” i.e. (20160617) and per day number is integer. I am fetching this data directly from SQL server and my problem is, tableau is reading my dates wrong.
So I tried DATEPARSE() to convert my date.
My DATEPARSE function is : DATEPARSE(“yyyymmdd”,”Date”) , now after using DATEPARSE function, I get NULL for my dates.
Can anyone please help me why I get NULL for dates, my query returns 30-day data which is divided into per day count.
Sample after running the query on SQL
Date Per day number
20160617 215674
Tableau does not accept this date format and I applied DateParse(), which I guess is returning string since my date is null. I would ideally like to get the correct date so I can apply a trend line on my data.
Thanks in advance.
Cheers!
You aren't using DateParse() correctly. The second parameter, which you have as "Date", should be the name of the field you want parsed. So for example, if you store 20160617 in a field called my_date_as_integer, your function should be DateParse("yyyymmdd", [my_date_as_integer])
I have a decimal date field (TDDATR) that is in the YYYYMMDD format.
I would like to create a field that is TDDATR + 30 days but I am unable to.
Using 'Define Results Field' I have tried a few things;
Simply doing this;
TDDATR + 30 DAYS
But it returned this error: Labeled duration not used correctly.
I tried using the DIGITS and SUBSTR commands to create a field in the DDMMYYYY format and then +30 days but got the same error.
Same as above but in the DD/MM/YYYY format - same error.
Using DATE(TDDATR) but all I see is +'s in the field.
Using DATE( ) on the fields created in step 2 and 3 - still get +'s
I've ran out of ideas - any help would be greatly appreciated.
Query/400 lacks a lot of the features that an SQL based interface has.
I'd urge you to consider switching to Query Manager (STRQM) which is a fully SQL based product. You can even convert Query/400 queries to Query Manager queries with the RTVQMQRY command by having the ALWQRYDFN parm set to *YES.
The other option that IBM is pushing is Web Query. Again, fully SQL based and you can convert Query/400 queries into it.
Having said that, the problem is that FLD + 30 DAYS only works when FLD is a DATE data type. Query/400 includes a DATE() function to convert non-date types into date. But it's very limited in that it only works with character fields formatted according to your job defaults. Assuming you're in the US, it'd only work with a character value of '07/01/15'.
You could do a lot of manipulation in Query/400 and end up with a result field that meets DATE()'s requirements. But a better solution would be to create an SQL view over your table and have your numeric date converted into a date data type in the view.
You can find code examples that show how to convert a numeric YYYYMMDD to a actual date data type in the view. However, I'd recommend create a user defined function (UDF) that will do the conversion for you. That will make it much easier to use in the view and to reuse in other places.
If you'd like, there's an open source package called iDate, that includes all the code required for convert to/from date data types.
Download that, install/compile it and your SQL view becomes
select ... idate(TDDATR,'*CCYMD') as TD_DATE
from myfile
The use of days is as follow
Field Expression
CURDATE_30 days(current(date)) + 30
The solution to your problem is: given the field A dec(8,0)
Field Expression
YYYYMMDD_ date(substr(digits(a),5,2)||'/'||
substr(digits(a),7,2)||'/'||
substr(digits(a),3,2))
NEXT_MONTH DAYS(YYYYMMDD_) + 30
Remember to check the date format in your job description. In the example the format is MDY or MM/DD/YY.
More info here
Based on the information here, I created the below 2 fields;
TDDIGI DIGITS(TDDATR)
TDDAT1 SUBSTR(TDDIGI,7,2)||'/'||
SUBSTR(TDDIGI,5,2)||'/'||
SUBSTR(TDDIGI,3,2)
From here I was able to create a date field;
TDDAT2 DATE(TDDAT1)
Which allowed me to perform the necessary calculations.
The format of TDDAT1 is based on your job description which can be found by;
WRKJOB
Option 2
Page down
Date format..: X
Mine was *DMY, so TDDAT1 was formatted based on this.