Monday this week (26th August) was a public holiday in the UK. I need to exclude this date from some of my reports.
Simple enough but I have one database where the date is broken into 15 minute segments, for example:
2019-08-26 08:30:00.000
2019-08-26 16:15:00.000
The only way I can work out to exclude all these dates from the report would be to use NOT IN, for example:
AND a.mydate NOT IN ('2019-08-26 08:30:00.000', '2019-08-26 16:15:00.000')
Which seems quite a ponderous way of doing it to me.
There's also:
and a.mydate NOT BETWEEN '2019-08-26 08:30:00.000' and '2019-08-26 19:30:00.000'
but any advance on that would be useful to know about, if there is an advance on this, as there will be more, perhaps many more, public holidays to exclude, moving forwards.
I do not have a separate table with holiday dates in it, so I need to script as per the above.
Thank you
If you have discreet values, you could add them to a table and do a left join and put a filter in WHERE saying the exclusion table's Id should be null (meaning the join failed).
Sth like:
SELECT
mydata
FROM
mytable mt LEFT JOIN Holidays h ON mt.mydate = h.date
WHERE
h.Id IS NULL
This is just an alternative way made possible because you have discrete values, I'm not sure if it's actually faster or not...
This worked better:
and cast(a.mydate as date) not in
( '2019-08-26',
'2019-12-25',
'2019-12-26',
'2020-01-01',
. . . )
Related
I am using SQL Server 2008 R2. Here is the query I have that returns monthly sales totals by zip code, per store.
select
left(a.Zip, 5) as ZipCode,
s.Store,
datename(month,s.MovementDate) as TheMonth,
datepart(year,s.MovementDate) as TheYear,
datepart(mm,s.MovementDate) as MonthNum,
sum(s.Dollars) as Sales,
count(*) as [TxnCount],
count(distinct s.AccountNumber) as NumOfAccounts
from
dbo.DailySales s
inner join
dbo.Accounts a on a.AccountNumber = s.AccountNumber
where
s.SaleType = 3
and s.MovementDate > '1/1/2016'
and isnull(a.Zip, '') <> ''
group by
left(a.Zip, 5),
s.Store,
datename(month, s.MovementDate),
datepart(year, s.MovementDate),
datepart(mm, s.MovementDate)
Now I'd like to add columns that compare sales, TxnCount, and NumOfAccounts to the same month the previous year for each zip code and store. I also would like each zip code/store combo to have a record for every month in the range; so zeros if null.
I do have a calendar table that I tried to use to get all months, but I ran into problems because of my "where" statements.
I know that both of these issues (comparing to previous year and including all dates in a date range) have been asked and answered before, and I've gotten them to work before myself, but this particular one has me running in circles. Any help would be appreciated.
I hope this is clear enough.
Thanks,
Tim
Treat the Query you have above as a data source. Run it as a CTE for the period you want to report, plus the period - 12 months (to get the historic data). (SalesPerMonth)
Then do a query that gets all the months you need from your calendar table as another CTE. This is the reporting months, not the previous year. (MonthsToReport)
Get a list of every valid zip code / Store combo - probably a select distinct from the SalesPerMonth CTE this would give you only combos that have at least one sale in the period (or historical period - you probably also want ones that sold last year, but not this year). Another CTE - StoreZip
Finally, your main query cross joins the StoreZip results with the MonthsToReport - this gives you the one row per StoreZip/Month combos you are looking for. Left join twice to the SalesPerMonth data, once for the month, once for the 1 year previous data. Use ISNULL to change any null records (no data) to zero.
Instead of CTEs, you could also do it as separate queries, storing the results in Temp tables instead. This may work better for large amounts of data.
I have a datetime list (which for some reason I call it column date) containing over 1k datetime.
adates:2017.10.20T00:02:35.650 2017.10.20T01:57:13.454 ...
For each of these dates I need to select the data from some table, then pivotize by a column t i.e. expiry, add the corresponding date datetime as column to the pivotized table and stitch together the pivotization for all the dates. Note that I should be able to identify which pivotization corresponds to a date and that's why I do it one by one:
fPivot:{[adate;accypair]
t1:select from volatilitysurface_smile where date=adate,ccypair=accypair;
mycols:`atm`s10c`s10p`s25c`s25p;
t2:`t xkey 0!exec mycols#(stype!mid) by t:t from t1;
t3:`t xkey select distinct t,tenor,xi,volofvol,delta_type,spread from t1;
result:ej[`t;t2;t3];
:result}
I then call this function for every datetime adates as follows:
raze {[accypair;adate] `date xcols update date:adate from fPivot[adate;accypair] }[`EURCHF] #/: adates;
this takes about 90s. I wonder if there is a better way e.g. do a big pivotization rather than running one pivotization per date and then stitching it all together. The big issue I see is that I have no apparent way to include the date attribute as part of the pivotization and the date can not be lost otherwise I can't reconciliate the results.
If you havent been to the wiki page on pivoting then it may be a good start. There is a section on a general pivoting function that makes some claims to being somewhat efficient:
One user reports:
This is able to pivot a whole day of real quote data, about 25 million
quotes over about 4000 syms and an average of 5 levels per sym, in a
little over four minutes.
As for general comments, I would say that the ej is unnecessary as it is a more general version of ij, allowing you to specify the key column. As both t2 and t3 have the same keying I would instead use:
t2 ij t3
Which may give you a very minor performance boost.
OK I solved the issue by creating a batch version of the pivotization that keeps the date (datetime) table field when doing the group by bit needed to pivot i.e. by t:t from ... to by date:date,t:t from .... It went from 90s down to 150 milliseconds.
fBatchPivot:{[adates;accypair]
t1:select from volatilitysurface_smile where date in adates,ccypair=accypair;
mycols:`atm`s10c`s10p`s25c`s25p;
t2:`date`t xkey 0!exec mycols#(stype!mid) by date:date,t:t from t1;
t3:`date`t xkey select distinct date,t,tenor,xi,volofvol,delta_type,spread from t1;
result:0!(`date`t xasc t2 ij t3);
:result}
For the business that I am working in I would like to get information on our customers. The base information I have on these customers is as follows:
Activation_Date stored in a Loans table, datatype is datetime.
ActivityDate stored in a CustomerDailyLoanActivity_Information table (a daily loans table to those interested, it is part of a datamart and stores for each day that a customer has been active with our company how much they have paid into their loan, so if a customer has an Activation_Date of 15-03-2017, it has ActivityDates in the CustomerDailyLoanActivity_Information table from 15-03-2017 up until now whereby each ActivityDate has a record in another column Sum_Paid_To_Date how much has been paid up until that ActivityDate). Datatype of ActivityDate is date.
What I would like to know is the following, I would like to know how much each customer has paid on 1, or 2, or 3, etc. months after his Activation_Date. So the query would look something like the following (slightly pseudo-code, the more important part is the WHERE clause).
SELECT
cldai.Sum_Paid_To_Date,
cldai.ActivityDate,
cldai.Customer_Account_Number
FROM
CustomerLoanDailyActivity_Information cldai
INNER JOIN
Loans l ON l.Customer_Account_Number = cldai.Customer_Account_Number
WHERE
(cldai.ActivityDate = CAST(l.Activation_Date AS date)
OR
cldai.ActivityDate = DATEADD(month, 1, CAST(l.Activation_Date AS date))
OR
cldai.ActivityDate = DATEADD(month, 2, CAST(l.Activation_Date AS date))
OR
cldai.ActivityDate = DATEADD(month, 3, CAST(l.Activation_Date AS date))
)
ORDER BY
l.Customer_Account_Number, cldai.ActivityDate ASC
So the problem is that this query is really really slow (because of the WHERE clause and because the cldai table is big (~6 GB)) and exits before any data is retrieved. A couple of problems that I have heard, and possible solutions, but haven't worked so far.
The CAST function makes the query really slow because it does a comparison with the ActivityDate column, which is indexed. I used CONVERT before but that was also really slow. I feel like I need to do the convert/cast though, because the ActivityDate is of date type and the Activation_Date is of datetime type, so there is a possibility that the time part of the datetime in Activation_Date will cause there to be no matches with the ActivityDate (e.g. Activation_Date for a given customer is 15-03-2017 09:00:00 so it will never match with ActivityDate 15-03-2017 because this might be converted to datetime 15-03-2017 00:00:00, which will never be equal because of the time part).
I have to use "DateTime" evaluations, which has been suggested as a solution, but I have no clue on how to apply this correctly.
I can't look at the execution plan because the DBA has blocked me from seeing that.
Any ideas on how to make this query perform more quickly? Any help would be greatly appreciated.
So a massive speedup was obtained by using a LEFT JOIN instead of an INNER JOIN and by not ordering the data on the server but on the client side. This reduced the query time from about an hour and 10 minutes to about 1 minute. It seems unbelievable but it's what happened.
Regards,
Tim.
If you are guaranteed to have a record for each day, you could apply use the row_number() function to apply row numbers for each group of customer loan repayment records, and then retrieve rows 1,31,61 and 91? This would avoid any date manipulation.
How about splitting this up into two steps? Step one - build a table with the four dates for each customer. Then step two, join this to your main CustomerLoanDailyActivity_Information table on date and customer account number. The second step would have a much simpler join, just an = between the ActivityDate and date entry in the table you have built.
I have found similar help, but the issue was more complex, I am only good with the basics of SQL and am striking out here. I get a handful of columns a,b,c,startdate,enddate and i need to parse that data out into multiple rows depending on how many months are within the range.
Eg: a,b,c,1/1/2015, 3/15,2015 would become:
a,b,c,1/1/2015,value_here_doesnt_matter
a,b,c,2/1/2015,value_here_doesnt_matter
a,b,c,3/1/2015,value_here_doesnt_matter
Does not matter if the start date or end date is on a specific day, the only thing that matters is month and year. So if the range included any day in a given month, I'd want to output start days for each month in the range, with the 1st as a default day.
Could I have any advice on which direction to begin? I'm attempting generate_series, but am unsure if this is the right approach or how to make it work with keeping the data in the first few arbitrary columns consistent.
I think generate_series is the way to go. Without knowing what the rest of your data looks like, I would start with something like this:
select
a, b, c, generate_series(startdate, enddate, interval '1 month')::date
from
my_table
Description
Hi,
I have a query using the same table twice in a left join with a inequality, but it does not produce any records, even though I am using a left join. I use MS Access 2013.
Code
The code is:
SELECT DCT01A.*,
DCT01B.*
FROM utb_DCT_01_DailyConversionTrends AS DCT01A
LEFT JOIN utb_DCT_01_DailyConversionTrends AS DCT01B
ON DCT01A.[Hour] = DCT01B.[Hour]
AND DCT01A.[WeekDay] = DCT01B.[WeekDay]
AND DCT01A.[Specification] = DCT01B.[Specification]
AND INT(DCT01A.[Date]) > INT(DCT01B.[Date])
Data
I am expecting (at the moment, though this will change later) that an inner join would result in no records produced. This is due to that this is only test data and the dates only span two days.
Hour and WeekDay refers to the current the hour of the day and the week day number in the week. Specification is an ID. Date is that date, which include time.
Goal
My goal of this query is to find all previous data on the same week day and hour, without picking the current record again (hence an inequality).
I realise I can simply run the inequality as >= instead, and then remove the = records afterwards. So I do have a simple workaround, I just can't understand why it won't work when written as above.
Thank you all for looking at this.
Would this work for you?
SELECT * FROM
(
SELECT DCT01A.*,
DCT01B.*
FROM utb_DCT_01_DailyConversionTrends AS DCT01A
LEFT JOIN utb_DCT_01_DailyConversionTrends AS DCT01B
ON DCT01A.[Hour] = DCT01B.[Hour]
AND DCT01A.[WeekDay] = DCT01B.[WeekDay]
AND DCT01A.[Specification] = DCT01B.[Specification]
AND INT(DCT01A.[Date]) >= INT(DCT01B.[Date])
)
WHERE
DCT01A.[Date] <> DCT01B.[Date]
Kindest regards..