I have some files that using different date types:
some only have year "2020", others have month "2020-12", and still, some have date:"2020-12-21"
So I want to use interval.
Like this:
(define-table test-date ()
((date :col-type :interval
:initarg :date
:accessor date)))
(create-dao 'test-date
:date "2020 years")
(date (find-dao 'test-date))=>((:MONTHS 24240) (:DAYS 0) (:SECONDS 0) (:USECONDS 0))
Q1. Is this is how interval supposed to use?
Q2. How to get year?
Your use of interval is smelly. Look, you store a year and you are getting a duration in months. interval is used to store durations, and is particularly used in date arithmetic: date + INTERVAL expr unit. So I don't think you want to use it here to store a date format.
You could:
either store a date and transform an incomplete date to a full one: "2020" would become "2020-00-00" if that is acceptable for your logic,
either store a string and use a validator function of your own (you could use Mito's inflation/deflation mechanism to retrieve your data in the right data format),
either use more date fields (year, month, day…) that are nullable.
Recipes for dates and time manipulation: https://lispcookbook.github.io/cl-cookbook/dates_and_times.html
Related
following 1 I started creating a Spreadsheet which reads data from BigQuery, but I'm having an issue handling parameters related to date values.
In the first sheet, I created 2 cells with 2 parameters, the start and the end of a date interval, with proper values. Both cells are formatted as "Date" value.
In the second sheet I configured BigQuery connector, for this example, I'm using a public dataset with dates. bigquery-public-data.utility_eu.date_greg
From the BigQuery connector wizard I added:
"STARTDATE" as "PARAMETERS!B1"
"ENDDATE" as "PARAMETERS!B2"
After this configuration, this is the resulting query:
SELECT
date,
date_str,
date_int
FROM `bigquery-public-data.utility_eu.date_greg`
WHERE date > DATE(#STARTDATE) AND date < DATE(#ENDDATE)
LIMIT 10
I'm getting an error directly from the editor with this message:
> Error BigQuery: No matching signature for function DATE for argument types: INT64. Supported signatures: DATE(TIMESTAMP, [STRING]); DATE(DATETIME); DATE(INT64, INT64, INT64) at [8:14]
As far as I can understand, the "date" cells are retrieved as a number, so the direct parse is not working. After a couple of tests, I understood the that given int value is the number I can obtain change cell format to "number".
If you convert cell value from DATE to NUMBER you get this value:
01/05/2019 -> 43.586
31/05/2019 -> 43.616
What is this number? It is not milliseconds, it increases by 1 every next day. In order to create the proper query that can parse this int, I need to understand what is this int (of course I can handle the cell as "text" and writing the timestamp value directly, but I would prefer to have the native date format so I can use the built-in calendar.
My consideration (with simple math) is that this number refers to a number of days since 30/12/1899, but it is very odd (also, every date BEFORE this days is always 0), so I'm asking you directly how to handle this value. Basing on my understanding of when the number counter starts (30/12/1899), I created this query which add the number retrieved from the cell:
SELECT *
FROM `bigquery-public-data.utility_eu.date_greg`
WHERE
date >= DATE_ADD(DATE("1899-12-30"), INTERVAL #DATAINIZIO DAY)
AND date <= DATE_ADD(DATE("1899-12-30"), INTERVAL #DATAFINE DAY)
It is working... but I think I'm doing a workaround that is not the proper way of doing this.
Also, is there any full documentation related to this BigQuery connection provided by Spreadsheet? Besides presentation in 1 I'm unable to find any specific documentation.
Spreadsheets (Google, Excel, ...) store the dates as days passed since a starting date with a fractional day representing time.
From here: "Excel stores dates and times as a number representing the number of days since 1900-Jan-0, plus a fractional portion of a 24 hour day: ddddd.tttttt . This is called a serial date, or serial date-time."
Now, you have to ways to filter by date on your Query:
In the query, you can use DATE_ADD to add your number of days (cell value) to the base date. (Carefull, DATE_ADD takes INT, and the date value is float so needs prior casting).
(preferred) on your spreadsheet you use TEXT(cell, "yyyy-mm-dd") so you can then use DATE() in the BigQuery query.
I use the second method as, though you need that extra cell (unless you directly store the date as YYYY-MM-DD; keeps the query cleaner than having a cast and date_add in there. Also would save you from the "1904 problem" explained in the link above.
What is this number? It is not milliseconds, it increases by 1 every next day.
This is so called serial number which represent number of days since "very beginning"
Google's Spreadsheet date calendar starts from 1900-01-01 - which is treated as a "very beginning"
In order to create the proper query that can parse this int, I need to understand what is this int
Armed with above info you can adjust you dates calculation to be in sync with what BigQuery expects
You mentioned that your fields are already in Date format, maybe you are doing an extra parsing in your query.
Try to do it without the DATE functions.
Also, I found this other doc, not merely related to connection, but might be helpful: Getting info from Spreadsheets with BigQuery.
I am using RapidMiner to deal with a dataset that has a timestamp attribute. I am using the Generate Attribute operator to obtain the date from the timestamp:
date = date_parse([vr_timestamp])
However, I need to obtain the Day of the week (Monday, Tuesday, ...).
Any idea of how can this be achieved?
You can use the following within Generate Attributes.
dayofweek = date_str_custom(aDate, "E")
In this case, aDate is an attribute of type Date time and dayofweek is your new attribute.
A full reference to all possible values that control this is given here.
Morning experts. I have a very simple query that I can not seem to get working as it should.
I have a table that has invoiceNumber, CustNum, Datetime, and Grand_total
I want to pull all transactions between a set of dates in this example all transactions between 4/10/2014 and 4/23/2014. I was using:
SELECT Invoice_number
,CustNum
,DATETIME
,Grand_Total
FROM invoice_totals
WHERE custnum = '10014877'
AND DATETIME BETWEEN '2014-04-10'
AND '2014-04-23'
ORDER BY DATETIME DESC
I just realized if there are any dates on 4/23 that this does not show them.. I have tried to substuite using:
WHERE custnum = '10014877'
AND DATETIME >= '2014-04-10' AND DATETIME <='2014-04-23'
But it is still giving me the same results (ignoring any transactions that occured on 4/23)
the last record pulling up has a datetime stamp of 2014-04-22 12:26:08.000. There ARE 2 transactions on the 23'ed I am trying to include.
Thank you very much.
The part:
AND DATETIME <='2014-04-23'
is actually(according to TSQL):
AND DATETIME <='2014-04-23 00:00:000'
So you're quering from midnight and missing all the transactions from 00:01 to 23:59 on the 23rd.
Try:
AND DATETIME <='2014-04-23 23:59:999'
Or
AND DATETIME < '2014-04-24'
Both should include all the transactions for the day of the 23rd.
If you're working with continuous data, such as datetimes, it's usually better to switch to using a semi-open interval - an inclusive start date and an exclusive end date. Exclusive end dates are usually easier to calculate:
WHERE custnum = '10014877'
AND DATETIME >= '20140410' AND DATETIME <'20140424'
I've also switched to a safe, unambiguous date format.
The alternative, using <= or BETWEEN (which is just shorthand for a pair of >= and <= comparisons, so your two queries were identical) requires an inclusive end date. Which depending on the exact data type you're using may be 2014-04-23T23:59:59.997 or 2014-04-23T23:59:59.9999 or any number of other possibilities - if you get it wrong and overspecify the value, it'll get rounded to be 20140424 and then an inclusive comparison is incorrect.
And even if you get it right today, it's a pain to find all usage of this pattern if the data type of the column changes later.
BETWEEN is inclusive with respect to the boundaries.
However, I'm guessing that the type of your (poorly named) DATETIME field is .. DATETIME, so in the comparison, the date '2014-04-23' gets converted to a DATETIME as follows: '2014-04-23:00:000'. Thus, all records with DATETIME greater than that first moment on 04-23 are rejected.
Date calculations are not my strong point and I need a little help.
I'm trying to check if a date (which is a timestamp) from a selected field is before 12pm of the current day. Thanks in advance.
Scenario: if an order is placed before 12pm that day, it will qualify for x otherwise it gets y. So my create date (including time) of that order is what I get in my select statement.
The DATE type doesn't carry time information, so it is up to you to define at what point in time the date is. You should use TIMESTAMP type if the time information is also important.
Anyway, lets say that the field stores the date at 12pm time, then you use
WHERE date_field <= CURRENT_DATE;
CURRENT_DATE is so called context variable which returns, obviously, current date. CURRENT_TIMESTAMP and CURRENT_TIME are also available. You can use DATEADD and DATEDIFF builtin functions to do some date calculations.
So if the field is actually timestamp, you could do it like
WHERE date_field < DateAdd(12 HOUR to cast(CURRENT_DATE as timestamp));
One of my customers would like to have a custom date column, where he could store the year only, a combination of month and year (without the day), or a classic date with day, month and year.
It should be possible to use this field for sorting the data. A "month-year" date should be considered as "01-month-year" for the sort, and a "year" date should be treated as "01-01-year" for the sort.
I could imagine two solutions to that:
Store the date in the standard "day-month-year" format, and keep in a separate column how the date was entered ("year", "month-year", "day-month-year"), so the approximate date can be displayed exactly how it was entered.
Use some sort of custom date column in the postgresql database.
Has anyone experience with that?
You could use date-time functions to extract date components. I don't think it has any sense to create additional columns. Also, some databases allow to create indexes by functions.