Dapper query times greatly increase with concurrent operations - entity-framework

Question

I've built a .NET Core 3.1 API. The API does a lot of work, and the bulk of it is executing SQL Queries. Most of the API uses EF Core for its data access. However, for one particularly critical query, we've discovered that Dapper offered a significant performance advantage.
One test scenario we're testing involves calling the API with many requests per second. When tested In a linear fashion, the requests take a little less than a second. However, if we bombard the API with 10, 15, (up to 120) calls within a second, the query performance drops dramatically. As an example, linear queries take around 500ms. At 15 rapid calls (within a second or so), an average query time is 3600ms.
It's true that with "Wide" scenarios effective throughput increases. However, I can't see the database being the bottleneck here, and I wonder why requests take so much longer when many go on in a small period of time.
Another critical point to note is that if I alternate calls to our API hosted in an environment and my local box for instance, performance and throughput increases by around 30%. This lends credence to the theory that the DB itself isn't the primary issue. I've also found only this question being something close to what I'm asking, and it's focused on EF.
Dapper's used in an unobtrusive way. The EF Context has a typical lifetime (scoped) though I've experimented with singleton and transient. Here's what it looks like:
using (var conn = new SqlConnection(_connectionString)) //from EF properties
{
using (var tx = await conn.BeginTransactionAsync(System.Data.IsolationLevel.ReadUncommitted))
{
var details = conn.QueryAsync<OurObject>(
#"SELECT columns
FROM table1
INNER JOIN table2
WHERE (someConditional = #paramValue)"
, new
{
paramValue
},tx);
var res1 = Success((await details).ToList());
return res1;
}
}
It should be noted that we need to use this isolation level and transactions for a specific reason. We're reading from tables and then writing to them for each request. Dirty Reads are much less of a concern than locks.
We have detailed logging throughout the request lifecycle showing effectively that reads (and to a lesser extent, writes) take longer. The processing of data does not change and is very fast throughout each request.
Finally, I know this is a bigger question than many on S.O. I appreciate any information or things to try you can provide!

Answer 1

As an example, linear queries take around 500ms. At 15 rapid calls (within a second or so), an average query time is 3600ms.
...
I can't see the database being the bottleneck here,
I can.
Assuming that 500ms is database server CPU time, 15 requests would require 7,500ms of CPU time. If the database server has 2 CPU cores it can process 4 queries/second. 15 queries would require 4.5sec. But the server is quite capable of accepting all 15 requests at the same time (after 15 separate client sessions have been esatablished), and time-slicing the CPUs among the running requests. So it may well be that all of the requests take an average of 3600ms.
For perspective 5ms is a "cheap" database query. EG a single-row lookup by key will take well under 1ms of CPU time. For a commonly-run query 500ms is expensive-enough to spend some time on analyzing the query plan and optimizing the execution.