In TimescaleDB I have a continuous aggregate which contains daily averages, so bucket size is 1 day:
CREATE MATERIALIZED VIEW sensors_daily
WITH (timescaledb.continuous, timescaledb.materialized_only=true)
AS
SELECT time_bucket_gapfill('1d', time) AS time,
id,
average(time_weight('LOCF', time, sensor1)) AS sensor1,
average(time_weight('LOCF', time, sensor2)) AS sensor2,
FROM sensors
GROUP BY time_bucket('1d', time), id;
I also created a continuous aggregate policy to keep the last 30 days and update it daily. It looks like this:
SELECT add_continuous_aggregate_policy('sensors_daily',
start_offset => INTERVAL '30 days',
end_offset => INTERVAL '1 day',
schedule_interval => INTERVAL '1 day');
Now here is my problem: I do not get data up to and including yesterday.
I also do not get data for the day before yesterday. The view is always 3 days behind.
When I query latest time on 2022-02-17 in the afternoon
select max(time) from sensors_daily;
I get the 14th
2022-02-14 01:00:00.000 +0100
Querying the job via
SELECT * FROM timescaledb_information.job_stats;
I see it runs succesfully every day shortly after midnight.
last_run_started_at: 2022-02-17 00:12:07.208 +0100
last_successful_finish: 2022-02-17 00:12:51.699 +0100
last_run_status: Success
last_run_duration: 00:00:44.491458
next_start: 2022-02-18 00:12:51.699 +0100
What do I need to change to get daily data up to and including yesterday?
Edit 2022-02-18 Maybe important: timestamps in the sensors source hyper-table are TIMESTAMPTZ (timestamp with timezone, German).
I finally solved this by reducing end_offset and schedule_interval from 1 day to 6 hours in the continuous aggregate policy.
Related
Here I'm with another unusual requirement.
Ok, so I have BO webi report (db2 database), and the report is supposed to run on 5th of every month and then it should have only data between a certain billing cycle (26th to 25th of last month)
So basically if I run the report on 5th of March, it should have data of billing cycle** 26th Jan - 25th Feb**.
I know I can manually put the dates in the query every month and run the report, but this report is going to be a schedule. So has to run automatically.
Any ideas ? Any date functions that can particularly tells a query to run between those dates ?
have not tried anything yet, but trying to come up a syntax. no clue :(
If you have whatever date of the same month as 2022-03-05, then you may get your date intervals as follows.
WITH PAR (DT) AS (VALUES '2022-03-05'::DATE)
SELECT
DT - (DAY (DT) - 1) - 2 MONTH + 25 AS DATE_FROM
, DT - (DAY (DT) - 1) - 1 MONTH + 24 AS DATE_TO
FROM PAR;
DATE_FROM
DATE_TO
2022-01-26
2022-02-25
I'm trying to get Weekly MTD and YTD values based of hourly data, but I'm having difficulties achieving this.
This is the data I'm working with:
max(Date) - Last day of the week
ISOWeek - Week in question
Value - The data I'm trying to sum
SELECT MAX(ISOWeek) AS [ISOWeek]
,MAX(Date) AS [Date]
,SUM(Value1) AS [MTD]
FROM Table1
GROUP BY ISOWeek, FORMAT(Date,'yyMM')
ORDER BY ISOWeek DESC
This is what that query returns:
ISOWeek Date MTD
29 2020-07-19 367529
28 2020-07-12 367138
27 2020-06-30 103290
27 2020-07-05 266755
26 2020-06-28 346588
25 2020-06-21 337168
This is what I would like to get:
ISOWeek Date MTD
29 2020-07-19 261515
28 2020-07-12 184104
27 2020-07-05 103414
26 2020-06-28 432114
25 2020-06-21 346588
The data has to be grouped by ISOWeek, if it's a week that dips into two months, I'm only interested in the MTD of the month in which the week ends. We have hundreds of values, so the plan is to create a MTD view and a YTD view. If I can get some help with the MTD one, I can get the other one done.
I'm nearly sure that what I'm after has to do with a WHERE clause and DATEADD but I'm not too sure what it should say.
Thank you for taking the time.
I don't really follow the rules you would like to apply, but per dates apply the formula to get weekstart/monthend or what you need. Place the date instead of the current date in the example.
Then group by the modified date.
You could build a date dimension where you have the required dates in some columns (first day of month, first day of week,etc.). This way you get a table with all the dates and the matching result for each.
It might be easier/faster to join it on the requried column.
declare #monthstart date,
#monthend date,
#weekstart date
;
select #monthstart=datefromparts(year(current_timestamp),month(current_timestamp),1);
select #monthend=EOMONTH(getdate(),0);
select #monthstart,#monthend,EOMONTH(getdate(),1) as next_month, EOMONTH(getdate(),-1) as previous_month;
select cast(DATEADD(d,1-DATEPART(WEEKDAY,current_timestamp),CURRENT_TIMESTAMP) as date) as Sunday,
cast(DATEADD(d,2-case when DATEPART(WEEKDAY,current_timestamp)=1 then 8 else DATEPART(WEEKDAY,current_timestamp) end,CURRENT_TIMESTAMP) as date) as Monday
;
I'm used to do the following syntax when analysing weekly data:
select week(creation_date)::date as week,
count(*) as n
from table_1
where creation_date > current_date - 30
group by 1
However, by doing this I will get just part of the first week.
Is there any smart way to alway get a whole week in the beginning?
Like get the first day of the week I would get half of.
First off you need to define what you mean by "week". This is more difficult than it appears. While humans have an intuitive since of a week, computers are just not that smart. There are 2 common conventions: the ISO-8601 Standard and, for lack of a better term, Traditional. ISO-8601 defines a week as always beginning on Monday and always containing 7 days. Traditional weeks begin on Sunday (usually) but may have weeks with less than 7 days. This results from having the 1st week of the year beginning on 1-Jan regardless of day of week. Thus the 1st and/or last weeks may have less than 7 days. ISO-8601 throws it own curve into the mix: the 1st week of the year begins on the week containing 4-Jan. Thus the last days of Dec may be in week 1 of the next year and the first days Jan may be in week 52/53 of the prior year.
All the below assume the ISO-8061.
Secondly there is no week function in Postgres. In you need extract function. So for this particular case:
select extract(week from creation_date)::integer as week, ...
Finally, your predicate (current_date - 30) ensures you will unusually not begin on the 1st of the week. To get the correct date take that result back 1 week, then go forward to the next Monday.
with days_to_monday (day_adj) as
( values ('{7,6,5,4,3,2,1}'::int[]) )
select current_date - 30
, current_date - 30 - 7 + day_adj[extract (isodow from current_date - 30 )]
from table_1 cross join days_to_monday;
The CTE establishes an array which for a given day of the week contains the number of days need to the next Monday. That main query extracts the day of week of current date and uses that to index the array. The corresponding value is added to get the proper date.
Putting that together with your original query to arrive at:
with next_week (monday) as
( values (current_date - 30 - 7
+ ('{7,6,5,4,3,2,1}'::int[])[extract (isodow from current_date - 30 )])
)
select extract(week from creation_date) as week,
count(*) as n
from table_1
where creation_date >= (select monday from next_week)
group by 1
order by 1;
For full example see fiddle.
I have 12 monthly tables, one for each month of the year 2019, records are order by an identifier (mmsi) and datetime (timestamp). I have calculated the interval (linetime2) between two consecutive rows. see below (August 2019):
Now I need to convert interval in hours in a new column. How can I do it? Can I run this:
SELECT EXTRACT(epoch FROM linetime2)/3600
or I have to take into account the number of days in a month and year?
I need to generate a report on 28th of every month .
So for that I need to run an autosys job.
In that I have a query with the condition
validation_date >= (number of days since last run)
Could you please help me on this .How can I achieve this condition in DB2 ?
This is a monthly job.So I don't want to hard code my previous run date in the query .At the same time I need to get a condition which satisfies for all the months .
Note :
If the query is running on feb 28th ,then feb 28th is not included. I need to get data from january 28th(included) till feb 27th(included)
similarly for march 28th run ,I need to get data from feb 28th(included) till march 27th(included)...Thanks in advance.Please help
Consider putting your report generation in a procedure, and parameterizing the start and end dates. In other words, have something like this:
create procedure monthly_report(
start_date date,
end_date date
)
language sql
begin
... report queries here ...
end
Now you potentially have something much more flexible (depending on the report requirements). If, in the future, you want to run a report on a different day, or for a different length of time, you will be able to do that.
Once you design it this way, it also may be easier to set the dates in your job scheduling script, rather than in SQL. If you did it in SQL, you could do something like this:
call monthly_report(
(select
year(current timestamp - 2 months) ||'-'||
month(current timestamp - 2 months) ||'-'||
'28' from sysibm.sysdummy1
),
(select
year(current timestamp - 1 month) ||'-'||
month(current timestamp - 1 month) ||'-'||
'27' from sysibm.sysdummy1
)
)
You may need to tweak it to handle some edge cases (I'm not exactly sure if you care what happens if it runs on the 29th of the month, and if so, how to handle it). But you get the basic approach.
You can use DAY() function that extracts day of month from date and you can use it for triggering job. for example where day(param)=28.
other two parameters can be calculated with date calculation , here is example for trigger , date_to value and date_from value
select day(timestamp_format(20170228,'yyyyMMdd') ),timestamp_format(20170228,'yyyyMMdd')- 1 DAY,timestamp_format(20170228,'yyyyMMdd') -1 month from sysibm.sysdummy1;
if your parameter/column is date/timestamp you can remove timestamp_format(20170228,'yyyyMMdd') function and just put your column/parameter