How to build a select using Zend with a DISTINCT specific column? - postgresql

I'm using Zend Framework for my website and I'd like to retrieve some data from my PostgreSQL database.
I have a request like :
SELECT DISTINCT ON(e.id) e.*, f.*, g.* FROM e, f, g
WHERE e.id = f.id_e AND f.id = g.id_f
This request works well but I don't know how to convert the DISTINCT ON(e.id) with Zend.
It seems that I can get DISTINCT rows but no distinct columns.
$select->distinct()->from("e")->join("f", "e.id = f.id_e")
->join("g", "f.id = g.id_f");
Any idea on how to make a select with distinct column ?
Thanks for help

You probably can't do this with Zend Framework since distinct on is not part of the SQL standard (end of page in Postgres documentation). Although Postgres supports it, I would assume its not part of Zend Framework because you could in theory configure another database connection which does not offer support.
If you know in advance that you're developing for a specific database (Postgres in this case), you could use manually written statements instead. You'll gain more flexibility within the queries and better performance at the cost of no longer being able to switch databases.
You would then instantiate a Zend_Db_Apdapter for Postgres. There a various methods available to get results for SQL queries which are described in the frameworks documentation starting at section Reading Query Results. If you choose to go this route I'd recommend to create an own subclass of the Zend_Db_Adapter_Pgsql class. This is to be able to convert data types and throw exceptions in case of errors instead of returning ambiguous null values and hiding error causes.

Related

Converting SQL query with FORMAT command to use in entity framework core

I have an SQL query:
SELECT
FORMAT(datetime_scrapped, 'MMMM-yy') [date],
count(FORMAT(datetime_scrapped, 'MMMM-yy')) as quantity
FROM scrap_log
GROUP BY FORMAT(datetime_scrapped, 'MMMM-yy')
It basically summarises all the entries in the scrap_log table by month/year and counts how many entries are in each month/year. Returns two columns (date and quantity). But I need to execute this in an ASP.NET core API using Entity Framework core. I tried using .fromSqlRaw(), but this expects all columns to be returned and so doesn't work.
I can find plenty of info on EF to implement group by and count etc... But I cannot find anything for the FORMAT(datetime, "MMMM-yy") part. Please could somebody explain to me how to do this?
EDIT: Seems already I appear to be going about this the wrong way in terms of efficiency. I will look into alternative solutions based on comments already made. Thanks for the fast response.

How do SQL dialects actually work internally in frameworks like hibernate and JOOQ

As we have seen that after you get a data source. We need to configure SQL dialects based on the database we use. After we select a particular dialect, How would that be used to make SQL queries specific to DB. Do frameworks like hibernate and JOOQ construct SQL queries in string based on the selected dialect ? If so which would be the most optimal way to support this in a framework of our own ?
Do frameworks like hibernate and JOOQ construct SQL queries in string based on the selected dialect
Yes. In jOOQ, there's an internal StringBuilder that collects SQL fragments from your expression tree, which are generated for your target SQL dialect specifically. You can see how that works in action on this website: https://www.jooq.org/translate. Try translating for example this input: SELECT * FROM t LIMIT 1 (which could correspond to your jOOQ API usage ctx.selectFrom(T).limit(1). It translates to:
-- Oracle 12c and more
SELECT * FROM t FETCH NEXT 1 ROWS ONLY
-- Oracle 11g and less
SELECT *
FROM (
SELECT x.*, rownum rn
FROM (SELECT * FROM t) x
WHERE rownum <= 1
)
WHERE rn > 0
If so which would be the most optimal way to support this in a framework of our own ?
You need:
An expression tree representation of your SQL query.
Optionally, you can parse a string to build this expression tree, like jOOQ's parser if you want to support actual SQL, or you can have your own language abstraction like Hibernate did with HQL / JPQL
Traverse that expression tree using something like a visitor to collect the SQL strings and bind variables.
But!
Do not build your own when you have off the shelf products like jOOQ or to some lesser extent Hibernate that can do the same. Building such a generic SQL abstraction is really difficult, and unless you want to actually sell such a product (you probably don't given your question), investing this time into building this product is not worth it at all.
The above LIMIT emulation is one of the more simple examples from jOOQ. Here's a lot more to help you decide against rolling your own, and that answer is still just scratching the surface of what jOOQ does behind the scenes.

Faster/efficient alternative to IN clause in custom/native queries in spring data jpa

I have a custom query along these lines. I get the list of orderIds from outside. I have the entire order object list with me, so I can change the query in any way, if needed.
#Query("SELECT p FROM Person p INNER JOIN p.orders o WHERE o.orderId in :orderIds)")
public List<Person> findByOrderIds(#Param("orderIds") List<String> orderIds);
This query works fine, but sometimes it may have anywhere between 50-1000 entries in the orderIds list sent from outside function. So it becomes very slow, taking as much as 5-6 seconds which is not fast enough. My question is, is there a better, faster way to do this? When I googled, and on this site, I see we can use ANY, EXISTS: Postgresql: alternative to WHERE IN respective WHERE NOT IN or create a temporary table: https://dba.stackexchange.com/questions/12607/ways-to-speed-up-in-queries-under-postgresql or join this to VALUES clause: Alternative when IN clause is inputed A LOT of values (postgreSQL). All these answers are tailored towards direct SQL calls, nothing based on JPA. ANY keyword is not supported by spring-data. Not sure about creating temporary tables in custom queries. I think I can do it with native queries, but have not tried it. I am using spring-data + OpenJPA + PostgresSQL.
Can you please suggest a solution or give pointers? I apologize if I missed anything.
thanks,
Alice
You can use WHERE EXISTS instead of IN Clause in a native SQL Query as well as in HQL in JPA which results in a lot of performance benefits. Please see sample below
Sample JPA Query:
SELECT emp FROM Employee emp JOIN emp.projects p where NOT EXISTS (SELECT project from Project project where p = project AND project.status <> 'Active')

Is it possible to run a SQL query with EntityFramework that joins three tables between two databases?

So I've got a SQL query that is called from an API that I'm trying to write an integration test for. I have the method that prepares the data totally working, but I realized that I don't know how to actually execute the query to check that data (and run the test). Here is what the query looks like (slightly redacted to protect confidental data):
SELECT HeaderQuery.[headerid],
kaq.[applicationname],
HeaderQuery.[usersession],
HeaderQuery.[username],
HeaderQuery.[referringurl],
HeaderQuery.[route],
HeaderQuery.[method],
HeaderQuery.[logdate],
HeaderQuery.[logtype],
HeaderQuery.[statuscode],
HeaderQuery.[statusdescription],
DetailQuery.[detailid],
DetailQuery.[name],
DetailQuery.[value]
FROM [DATABASE1].[dbo].[apilogheader] HeaderQuery
LEFT JOIN [DATABASE1].[dbo].[apilogdetails] DetailQuery
ON HeaderQuery.[headerid] = DetailQuery.[headerid]
INNER JOIN [DATABASE2].[dbo].[apps] kaq
ON HeaderQuery.[applicationid] = kaq.[applicationid]
WHERE HeaderQuery.[applicationid] = #applicationid1
AND HeaderQuery.[logdate] >= #logdate2
AND HeaderQuery.[logdate] <= #logdate3
For the sake of the test, and considering I already have the SQL script, I was hoping to be able to just execute that script above (providing the where clause programmatically) using context.Database.SqlQuery<string>(QUERY) but since I have two different contexts, I'm not sure how to do that.
The short answer is no, EF doesn’t support cross database queries. However there are a few things you can try.
You can use two different database contexts (one for each database).
Run your respective queries and then merge / massage the data after
the query returns.
Create a database view and query the view through EF.
Using a SYNONYM
https://rachel53461.wordpress.com/2011/05/22/tricking-ef-to-span-multiple-databases/
If the databases are on the same server, you can try using a
DbCommandInterceptor
I’ve had this requirement before and personally like the view option.

How to optimize generic SQL to retrieve DDL information

I have a generic code that is used to retrieve DDL information from a Firebird database (FB2.1). It generates SQL code like
SELECT * FROM MyTable where 'c' <> 'c'
I cannot change this code. Actually, if that matters, it is inside Report Builder 10.
The fact is that some tables from my database are becoming a litle too populated (>1M records) and that query is starting to take too long to execute.
If I try to execute
SELECT * FROM MyTable where SomeIndexedField = SomeImpossibleValue
it will obviously use that index and run very quickly.
Well, it wouldn´t be that hard to the database find out that that is an impossible matcher and make some sort of optimization and avoid testing it against each row.
Is there any way to make my firebird database to optimize that search?
As the filter condition is a negative proposition (and also doesn't refer a column to search, but only a value to compare to another value), Firebird need to do a full table scan (without use any index) to confirm that aren't any record that meet your criteria.
If you can't change you need to wait for the upcoming 3.0 version, that will implement the Boolean data type, and therefore should start to evaluate "constant" fake comparisons in advance (maybe the client library will do this evaluation before send the statement to the server?).