Year over year monthly sales - tsql

I am using SQL Server 2008 R2. Here is the query I have that returns monthly sales totals by zip code, per store.
select
left(a.Zip, 5) as ZipCode,
s.Store,
datename(month,s.MovementDate) as TheMonth,
datepart(year,s.MovementDate) as TheYear,
datepart(mm,s.MovementDate) as MonthNum,
sum(s.Dollars) as Sales,
count(*) as [TxnCount],
count(distinct s.AccountNumber) as NumOfAccounts
from
dbo.DailySales s
inner join
dbo.Accounts a on a.AccountNumber = s.AccountNumber
where
s.SaleType = 3
and s.MovementDate > '1/1/2016'
and isnull(a.Zip, '') <> ''
group by
left(a.Zip, 5),
s.Store,
datename(month, s.MovementDate),
datepart(year, s.MovementDate),
datepart(mm, s.MovementDate)
Now I'd like to add columns that compare sales, TxnCount, and NumOfAccounts to the same month the previous year for each zip code and store. I also would like each zip code/store combo to have a record for every month in the range; so zeros if null.
I do have a calendar table that I tried to use to get all months, but I ran into problems because of my "where" statements.
I know that both of these issues (comparing to previous year and including all dates in a date range) have been asked and answered before, and I've gotten them to work before myself, but this particular one has me running in circles. Any help would be appreciated.
I hope this is clear enough.
Thanks,
Tim

Treat the Query you have above as a data source. Run it as a CTE for the period you want to report, plus the period - 12 months (to get the historic data). (SalesPerMonth)
Then do a query that gets all the months you need from your calendar table as another CTE. This is the reporting months, not the previous year. (MonthsToReport)
Get a list of every valid zip code / Store combo - probably a select distinct from the SalesPerMonth CTE this would give you only combos that have at least one sale in the period (or historical period - you probably also want ones that sold last year, but not this year). Another CTE - StoreZip
Finally, your main query cross joins the StoreZip results with the MonthsToReport - this gives you the one row per StoreZip/Month combos you are looking for. Left join twice to the SalesPerMonth data, once for the month, once for the 1 year previous data. Use ISNULL to change any null records (no data) to zero.
Instead of CTEs, you could also do it as separate queries, storing the results in Temp tables instead. This may work better for large amounts of data.

Related

I need to add up lots of values between date ranges as quickly as possible using PostgreSQL, what's the best method?

Here's a simple example of what I'm trying to do:
CREATE TABLE daily_factors (
factor_date date,
factor_value numeric(3,1));
CREATE TABLE customer_date_ranges (
customer_id int,
date_from date,
date_to date);
INSERT INTO
daily_factors
SELECT
t.factor_date,
(random() * 10 + 30)::numeric(3,1)
FROM
generate_series(timestamp '20170101', timestamp '20210211', interval '1 day') AS t(factor_date);
WITH customer_id AS (
SELECT generate_series(1, 100000) AS customer_id),
date_from AS (
SELECT
customer_id,
(timestamp '20170101' + random() * (timestamp '20201231' - timestamp '20170101'))::date AS date_from
FROM
customer_id)
INSERT INTO
customer_date_ranges
SELECT
d.customer_id,
d.date_from,
(d.date_from::timestamp + random() * (timestamp '20210211' - d.date_from::timestamp))::date AS date_to
FROM
date_from d;
So I'm basically making two tables:
a list of daily factors, one for every day from 1st Jan 2017 until today's date;
a list of 100,000 "customers" all who have a date range between 1st Jan 2017 and today, some long, some short, basically random.
Then I want to add up the factors for each customer in their date range, and take the average value.
SELECT
cd.customer_id,
AVG(df.factor_value) AS average_value
FROM
customer_date_ranges cd
INNER JOIN daily_factors df ON df.factor_date BETWEEN cd.date_from AND cd.date_to
GROUP BY
cd.customer_id;
Having a non-equi join on a date range is never going to be pretty, but is there any way to speed this up?
The only index I could think of was this one:
CREATE INDEX performance_idx ON daily_factors (factor_date);
It makes a tiny difference to the execution time. When I run this locally I'm seeing around 32 seconds with no index, and around 28s with the index.
I can see that this is a massive bottleneck in the system I'm building, but I can't think of any way to make things faster. The ideas I did have were:
instead of using daily factors I could largely get away with monthly ones, but now I have the added complexity of "whole months and partial months" to work with. It doesn't seem like it's going to be worth it for the added complexity, e.g. "take 7 whole months for Feb to Aug 2020, then 10/31 of Jan 2020 and 15/30 of September 2020";
I could pre-calculate every average I will ever need, but with 1,503 factors (and that will increase with each new day), that's already 1,128,753 numbers to store (assuming we ignore zero date ranges and that my maths is right). Also my real world system has an extra level of complexity, a second identifier with 20 possible values, so this would mean having c.20 million numbers to pre-calculate. Also, every day the number of values to store grows exponentially;
I could take this work out of the database, and do it in code (in memory), as it seems like a relational database might not be the best solution here?
Any other suggestions?
The classic way to deal with this is to store running sums of factor_value, not (or in addition to) individual values. Then you just look up the running sum at the two end points (actually at the end, and one before the start), and take the difference. And of course divide by the count, to turn it into an average. I've never done this inside a database, but there is no reason it can't be done there.

Showing this year and last year's data in one row in PostgreSQL

I have a table of sales grouped by week. I want to write a query that creates a new table giving the sales of the week in question AND the sales of that item from this time last year, but my attempts either give blank cells for the this-time-last-year (TTLY) values or duplicates.
I've tried writing a subquery that takes the date, subtracts 52 weeks, and shows the value for that week, then joining that subquery to my main query.
However, that subquery isn't working: the query shows the date of a year ago correctly, but doesn't then pull the SALES for that TTLY week, only the current week.
with ttyl as
(select
date::date as date,
(sales.date - interval '52 weeks') as date_ttly,
ID,
value
from sales
where country = 'uk' and date > '2019-08-01' and ID = '12345678')
In this example the subquery generates the previous year's date in the date_ttly column but pulls 2019 data in the value column.
All the WHERE conditions are just temporary so as to make building the query easier.
Thank you!
Assuming that the sales are grouped by date and by country only, a join on the same table should work:
SELECT sales1.id,
sales1.date,
sales1.value,
sales2.date,
sales2.value
FROM sales AS sales1
JOIN sales AS sales2 ON sales1.date - interval '52 weeks' = sales2.date
AND sales1.country = sales2.country
However, this also assumes that your date is always the same day of the week, e.g. Monday.

Why are these two SQL date functions not providing the same answer

I've actually already solved the problem, but I'm trying to understand why the problem occurred because as far as I can see it has no reason to happen.
I have a rather large query that I run to prepare a table with some often used combinations. Generally, it only contains 2 years of data. Occasionally I will reconstruct it. While doing this I tweaked the query to add more information, but suddenly the result no longer matched up to the old query. Comparing the old to the new I noticed several missing orders. Amazingly, even after removing the tweaked parts the results still didn't match up.
I ultimately tracked the problem down to my WHERE clause, which was different from how I did it last time.
The type of the orderdate column I go over has type (datetime, null)
One of the orders that was omitted had this as date:
2018-12-23 20:58:52.383
An order that was included had this as date:
2019-01-28 15:20:49.107
It looks exactly the same to me.
The entire query is the same, except for the WHERE clause. My original where was:
WHERE DATEPART(yyyy,tbOrder.[OrderDate]) >= DATEPART(yyyy,GETDATE()-2)
My new where is now:
WHERE tborder.[OrderDate] >= DATEADD(yy, DATEDIFF(yy, 0, GETDATE())-2, 0)
Any help in understanding why the original where clause drops some lines would be greatly appreciated.
Because you are doing two different things. First predicate,
WHERE DATEPART(yyyy,tbOrder.[OrderDate]) >= DATEPART(yyyy,GETDATE()-2)
Take all order dates that are bigger than the year for the day before yesterday or two days before. Notice that, -2 is inside the brackets.
Second predicate,
WHERE tborder.[OrderDate] >= DATEADD( yy, DATEDIFF( yy, 0, GETDATE() ) - 2, 0)
Take all order dates bigger than two years before, i.e., datediff(yy,startdate,enddate) will return the year result of the difference between today and the initial value for date datatype, which is 1900-01-01. Then, add this, -2, to 1900-01-01. The second expression is the form of:
1900 + ( 201X - 1898 )
I simplified 1900 - 2 = 1898.
The two expressions return completely different things, so it shouldn't be a surprise the results are different. The first one returns the current year as a number (or the year of the day before yesterday to be precise). The second one returns January 1st two years ago.
You can put both expressions in a SELECT query to see what they return :
select DATEPART(yyyy,GETDATE()-2), DATEADD(yy, DATEDIFF(yy, 0, GETDATE()) - 2, 0)
The result is :
2019 2017-01-01 00:00:00.000
Both expressions are more complex than they need to be. The first condition will also harm performance because DATEPART(yyyy,tbOrder.[OrderDate]) prevents the server from using any indexes that cover OrderDate.
The question doesn't explain what you actually want to return. If you wanted to return all rows in the current year you can use :
Where
OrderDate >=DATEFROMPARTS( YEAR(GETDATE()) ,1,1) and
OrderDate < DATEFROMPARTS( YEAR(GETDATE()) + 1,1,1)
The same can be used to find rows two years in the past :
Where
OrderDate >= DATEFROMPARTS( YEAR(GETDATE()) -2 ,1,1) and
OrderDate < DATEFROMPARTS(YEAR(GETDATE()) - 1,1,1)
All rows since January 1st two years ago :
Where OrderDate >= DATEFROMPARTS( YEAR(GETDATE()) -2 ,1,1)
All those queries can take advantage of indexes that cover OrderDate.
Date range queries become a lot easier if you use a Calendar table. A Calendar table is a table that contains eg 50 or 100 years' worth of dates with extra columns for month, month day, week number, day of week, quarter, semester, month and day names, holidays, business reprorint periods, formatted short, long dates etc.
This makes yearly, monthly or weekly queries as easy as joining with the Calendar table and filtering based on the month or period you want.
In this case, retrieving rows two yeas in the past would look like :
From Orders inner Join Calendar on OrderDate=Calendar.Date
Where Calendar.Year=YEAR(GETDATE())-2
That may not looks so impressive but what about Q2 two years ago?
From Orders inner Join Calendar on OrderDate=Calendar.Date
Where Calendar.Year=YEAR(GETDATE())-2 and Quarter=2
Two years ago, same quarter
From Orders inner Join Calendar on OrderDate=Calendar.Date
Where Calendar.Year=YEAR(GETDATE())-2 and Quarter=DATEPART(q,GETDATE())
Retrieving totals for the current quarter for the last two years :
SELECT Year,Quarter,SUM(Total) QuarterTotal
From Orders inner Join Calendar on OrderDate=Calendar.Date
Where Calendar.Year > YEAR(GETDATE())-2 and Quarter=DATEPART(q,GETDATE())
GROUP BY Calendar.Year

Optimizing a WHERE clause with a dateadd function

For the business that I am working in I would like to get information on our customers. The base information I have on these customers is as follows:
Activation_Date stored in a Loans table, datatype is datetime.
ActivityDate stored in a CustomerDailyLoanActivity_Information table (a daily loans table to those interested, it is part of a datamart and stores for each day that a customer has been active with our company how much they have paid into their loan, so if a customer has an Activation_Date of 15-03-2017, it has ActivityDates in the CustomerDailyLoanActivity_Information table from 15-03-2017 up until now whereby each ActivityDate has a record in another column Sum_Paid_To_Date how much has been paid up until that ActivityDate). Datatype of ActivityDate is date.
What I would like to know is the following, I would like to know how much each customer has paid on 1, or 2, or 3, etc. months after his Activation_Date. So the query would look something like the following (slightly pseudo-code, the more important part is the WHERE clause).
SELECT
cldai.Sum_Paid_To_Date,
cldai.ActivityDate,
cldai.Customer_Account_Number
FROM
CustomerLoanDailyActivity_Information cldai
INNER JOIN
Loans l ON l.Customer_Account_Number = cldai.Customer_Account_Number
WHERE
(cldai.ActivityDate = CAST(l.Activation_Date AS date)
OR
cldai.ActivityDate = DATEADD(month, 1, CAST(l.Activation_Date AS date))
OR
cldai.ActivityDate = DATEADD(month, 2, CAST(l.Activation_Date AS date))
OR
cldai.ActivityDate = DATEADD(month, 3, CAST(l.Activation_Date AS date))
)
ORDER BY
l.Customer_Account_Number, cldai.ActivityDate ASC
So the problem is that this query is really really slow (because of the WHERE clause and because the cldai table is big (~6 GB)) and exits before any data is retrieved. A couple of problems that I have heard, and possible solutions, but haven't worked so far.
The CAST function makes the query really slow because it does a comparison with the ActivityDate column, which is indexed. I used CONVERT before but that was also really slow. I feel like I need to do the convert/cast though, because the ActivityDate is of date type and the Activation_Date is of datetime type, so there is a possibility that the time part of the datetime in Activation_Date will cause there to be no matches with the ActivityDate (e.g. Activation_Date for a given customer is 15-03-2017 09:00:00 so it will never match with ActivityDate 15-03-2017 because this might be converted to datetime 15-03-2017 00:00:00, which will never be equal because of the time part).
I have to use "DateTime" evaluations, which has been suggested as a solution, but I have no clue on how to apply this correctly.
I can't look at the execution plan because the DBA has blocked me from seeing that.
Any ideas on how to make this query perform more quickly? Any help would be greatly appreciated.
So a massive speedup was obtained by using a LEFT JOIN instead of an INNER JOIN and by not ordering the data on the server but on the client side. This reduced the query time from about an hour and 10 minutes to about 1 minute. It seems unbelievable but it's what happened.
Regards,
Tim.
If you are guaranteed to have a record for each day, you could apply use the row_number() function to apply row numbers for each group of customer loan repayment records, and then retrieve rows 1,31,61 and 91? This would avoid any date manipulation.
How about splitting this up into two steps? Step one - build a table with the four dates for each customer. Then step two, join this to your main CustomerLoanDailyActivity_Information table on date and customer account number. The second step would have a much simpler join, just an = between the ActivityDate and date entry in the table you have built.

PostgreSQL 8.2 extract week number from a a date field

This might be a simple one but I haven't got a solution yet. I have a create_date field which is a date type, and a revenue number. I want to see weekly break down of revenue.
I can get the numbers easily in tableau because of built in functionality but doing it in PostgreSQL is where I need some help.
If you want the revenue by week, you'll need to group and aggregate:
select extract (week from create_date) as week, sum(revenue) from table group by week