I asked a question yesterday about a procedure we're trying to re-write/optimize in our application. It's off of a search form with a bunch of criteria the user can specify. 40 parameters, 3 of which are long strings of Guids that I am passing into a UDF that returns a Table variable, all 3 of which we JOIN into our main FROM statement.
We did much of this query using Dynamic SQL, one of the main reasons we're re-writing the whole thing is because it's Dynamic SQL. Everything I ever read about Dynamic SQL is bad, especially for execution plans and optimization. Then I start coming across articles like these two....
Sometimes the Simplest Solution isn't the Best Solution
Erland Sommarskog - Dynamic SQL Conditions in T-SQL
I've always though Dynamic SQL was bad for security and optimization, we've tried removing it form our system wherever possible. Now we're restructuring the most executed query in our system (main search query) and we thought stripping all the Dynamic SQL was going to help.
Basically replacing
IF(#Param1 IS NULL)
#SQLString = #SQLString + " AND FieldX = #Param1"
...execute the #SQLString
with one large SQL block that has
WHERE (#Param1 IS NOT NULL AND FieldX = #Param1)
Reading those two articles it seems like this is going to work against me. I can't use RECOMPILE because we're in 2k5 still and even if we could this stored procedure is very high-use. Do I really want to write this Query in Dynamic SQL? How can it be faster if no execution plans can be stored?
Related
Most common answers are: NO, don't use cursors, don't use dynamic SQL
But this question is to solicit feedback from a coding style that seems nifty but can be bad practice, heavier processing speeds, and SQL injections(?).
I learned this style due to an annoyance with copy-pasting set queries of which only one or two items change between each query. I find this style easier to perform code-reviews due to having only one code block, removing the need to scroll up and down.
An example use case is: a big slow table with historical data from 20 different insurance companies that is 1.7 billion rows by 200 columns. To do productive analysis, 10 columns are retrieved into separate tables for each of the 20 insurance companies.
Before the cursor/dynamic combination, a query was built and code-reviewed for one plan, then copied 19 times, each time retrieving a different plan.
After utilizing cursor/dynamic combo, there is one 20 item cursor and one dynamic SQL block. Review wise, it seems more consistent and less prone to human error.
A code example of the combo is below:
Declare #company_name varchar(10)
,#SQL_STATEMENT varchar(100)
Declare company_cursor cursor fast_forward for (
SELECT * FROM (VALUES('APPLE'),('GOOGLE'),('AMAZON')) AS TABLE_NAME(COLUMN_NAME)
)
open company_cursor
fetch next from company_cursor into #company_name
while ##fetch_status = 0
begin
set #SQL_STATEMENT = 'select * from database.schema.'+#company_name
print (#SQL_STATEMENT)
fetch next from company_cursor into #company_name
end
close company_cursor
deallocate company_cursor
I also noticed using a PRINT instead of EXEC instantaneously prints the query statement and acts as a code generator if the PRINT results are copied into the SQL editor.
Can somebody offer opinions, advice, or general practice rules surrounding this style of T-SQL coding? (bracing for downvotes...sniff)
The question pretty much sums it up. I've got to replace text in a large number for store procedures. Its not so many that doing it manually is impossible, but enough that I'm asking the question. I also prefer automation as it reduces the change of user error when we make the change in production.
I can Identify them like this:
select OBJECT_DEFINITION(object_id), *
from sys.procedures
where OBJECT_DEFINITION(object_id) like '%''MyExampleLiteral''%'
order by name
Is there any way to mass update them all to change 'MyExampleLiteral' to 'MyOtherExampleLiteral'?
I'd even settle for a way to open all the stored procs. Just Finding these store procs in a larger list will take some time.
I thought about generating alter statements using the above select statements, but then I lose line breaks.
Thanks in advance,
This is a Microsoft SQL Server.
There are different tools to use depending on the database in question. For example, Microsoft SQL Server Data Tools integrates with Visual Studio, and allows you to do these types of operations fairly easily. The database is stored in your solution as scripts, which you can then search and replace any keyword you wish. I'm assuming there would be similar tools available for other platforms.
You could do this with dynamic sql. Query the system tables to get all the SPs containing your "MyExampleLiteral":
SELECT [object_id] FROM sys.objects o
WHERE type_desc = 'SQL_STORED_PROCEDURE'
AND is_ms_shipped = 0
AND OBJECT_DEFINITION(o.[object_id]) LIKE '%<search string>%'
Then, write a while loop to go through those object_ids. In the while loop, get the OBJECT_DEFINITION() into a string and replace the "MyExampleLiteral", then replace CREATE PROCEDURE with ALTER PROCEDURE and execute the string using sp_executesql.
Doing something this crazy, make sure you backup the database first.
I have a generic code that is used to retrieve DDL information from a Firebird database (FB2.1). It generates SQL code like
SELECT * FROM MyTable where 'c' <> 'c'
I cannot change this code. Actually, if that matters, it is inside Report Builder 10.
The fact is that some tables from my database are becoming a litle too populated (>1M records) and that query is starting to take too long to execute.
If I try to execute
SELECT * FROM MyTable where SomeIndexedField = SomeImpossibleValue
it will obviously use that index and run very quickly.
Well, it wouldn´t be that hard to the database find out that that is an impossible matcher and make some sort of optimization and avoid testing it against each row.
Is there any way to make my firebird database to optimize that search?
As the filter condition is a negative proposition (and also doesn't refer a column to search, but only a value to compare to another value), Firebird need to do a full table scan (without use any index) to confirm that aren't any record that meet your criteria.
If you can't change you need to wait for the upcoming 3.0 version, that will implement the Boolean data type, and therefore should start to evaluate "constant" fake comparisons in advance (maybe the client library will do this evaluation before send the statement to the server?).
I'm creating result paging based on first letter of certain nvarchar column and not the usual one, that usually pages on number of results.
And I'm not faced with a challenge whether to filter results using LIKE operator or equality (=) operator.
select *
from table
where name like #firstletter + '%'
vs.
select *
from table
where left(name, 1) = #firstletter
I've tried searching the net for speed comparison between the two, but it's hard to find any results, since most search results are related to LEFT JOINs and not LEFT function.
"Left" vs "Like" -- one should always use "Like" when possible where indexes are implemented because "Like" is not a function and therefore can utilize any indexes you may have on the data.
"Left", on the other hand, is function, and therefore cannot make use of indexes. This web page describes the usage differences with some examples. What this means is SQL server has to evaluate the function for every record that's returned.
"Substring" and other similar functions are also culprits.
Your best bet would be to measure the performance on real production data rather than trying to guess (or ask us). That's because performance can sometimes depend on the data you're processing, although in this case it seems unlikely (but I don't know that, hence why you should check).
If this is a query you will be doing a lot, you should consider another (indexed) column which contains the lowercased first letter of name and have it set by an insert/update trigger.
This will, at the cost of a minimal storage increase, make this query blindingly fast:
select * from table where name_first_char_lower = #firstletter
That's because most database are read far more often than written, and this will amortise the cost of the calculation (done only for writes) across all reads.
It introduces redundant data but it's okay to do that for performance as long as you understand (and mitigate, as in this suggestion) the consequences and need the extra performance.
I had a similar question, and ran tests on both. Here is my code.
where (VOUCHER like 'PCNSF%'
or voucher like 'PCLTF%'
or VOUCHER like 'PCACH%'
or VOUCHER like 'PCWP%'
or voucher like 'PCINT%')
Returned 1434 rows in 1 min 51 seconds.
vs
where (LEFT(VOUCHER,5) = 'PCNSF'
or LEFT(VOUCHER,5)='PCLTF'
or LEFT(VOUCHER,5) = 'PCACH'
or LEFT(VOUCHER,4)='PCWP'
or LEFT (VOUCHER,5) ='PCINT')
Returned 1434 rows in 1 min 27 seconds
My data is faster with the left 5. As an aside my overall query does hit some indexes.
I would always suggest to use like operator when the search column contains index. I tested the above query in my production environment with select count(column_name) from table_name where left(column_name,3)='AAA' OR left(column_name,3)= 'ABA' OR ... up to 9 OR clauses. My count displays 7301477 records with 4 secs in left and 1 second in like i.e where column_name like 'AAA%' OR Column_Name like 'ABA%' or ... up to 9 like clauses.
Calling a function in where clause is not a best practice. Refer http://blog.sqlauthority.com/2013/03/12/sql-server-avoid-using-function-in-where-clause-scan-to-seek/
Entity Framework Core users
You can use EF.Functions.Like(columnName, searchString + "%") instead of columnName.startsWith(...) and you'll get just a LIKE function in the generated SQL instead of all this 'LEFT' craziness!
Depending upon your needs you will probably need to preprocess searchString.
See also https://github.com/aspnet/EntityFrameworkCore/issues/7429
This function isn't present in Entity Framework (non core) EntityFunctions so I'm not sure how to do it for EF6.
Is it possible to use the name of a table as a parameter in t-sql?
I want to insert data into a table, but I want one method in C# which has a parameter for the table.
Is this a good approach? I think if I have one form and I am choosing the table and fields to insert data into, I am essentially looking to write my own dynamic sql query built on the fly. This is another thing altogether which I am sure has its catches?
Thanks
Not directly. The only way to do this is through dynamic SQL - either EXEC or sp_ExecuteSQL. The latter has the advantage of query cache/re-use, and avoiding injection via parameters for the values - but you will have to concatenate the table-name itself into the query (you can't parameterise it), so be sure to white-list it against a list of known-good table names.