I'm running the Google Cloud SQL Proxy to connect to a remote PostgreSQL (v9.6) instance on Google Cloud (GCP).
I'm having a very weird issue with the number of returned rows from these 2 queries:
select "id", "original_image"
from "tableA"
where "image_id" is NULL
order by "id" asc
limit 581
returns 0 rows
select "id", "original_image"
from "tableA"
where "image_id" is NULL
order by "id" asc
limit 580
returns 73 rows
If I remove the WHERE clause (where "image_id" is NULL) both return the limit amount (581 and 580, respectively).
Any ideas of what could be causing this?
Related
I have data in a Postgres table with roughly this form:
CREATE TABLE jobs
(
id BIGINT PRIMARY KEY,
started_at TIMESTAMPTZ,
duration NUMERIC,
project_id BIGINT
)
I also came up with a query that is kinda what I want:
SELECT
$__timeGroupAlias(started_at,$__interval),
avg(duration) AS "durations"
FROM jobs
WHERE
project_id = 720
GROUP BY 1
ORDER BY 1
This query filters for one exact project_id. What I actually want is one line in the chart for each project that has an entry in the table, not for just one.
I fail to find a way to do that. I tried all different flavors of group by clauses I could think of, and also tried the examples I found online but none of them worked.
Try this Grafana PostgreSQL query:
SELECT
$__timeGroupAlias(started_at, $__interval),
project_id::text AS "metric",
AVG(duration) AS "durations"
FROM jobs
WHERE $__timeFilter(started_at)
GROUP BY 1,2
ORDER BY 1
I have 3 tables:
table1:{id, uid}
table2:{id, uid}
table1_table2:{table1_id, table2_id}
I need to execute the following queries:
SELECT 1 FROM table1_table2
LEFT JOIN table1 ON table1.id = table1_table2.table1_id
LEFT JOIN table2 ON table2.id = table1_table2.table2_id
WHERE table1.uid = ? and table2.uid = ?
I have unique indices on UUID columns, so I expected the search to be fast. When I have an almost empty database, select takes 0 ms, when there are 50,000 records in table 1, 100 records in table 2 and 110,000 records in table1_table2, select takes 10 ms, which is a lot, because I have to make 400,000 queries. Can I have O(1) on select?
Now I'm using hibernate(spring data) and postgres.
You have unique indices but have you updated statistics with ANALYZE as well?
What type is used for UID column and what type are you feeding it with from Java?
Is there any difference, when you run it from Hibernate/Java and from Postgres console?
Run the query with "EXPLAIN", get the execution plan - from Java as well as from Postgres console, and observe any differences. See How to get query plan information from Postgres into JDBC
I have an AWS RDS PostgreSQL 12.3 (t3.small, 2CPU 2GB RAM). I have this table:
CREATE TABLE public.phones_infos
(
phone_id integer NOT NULL DEFAULT nextval('phones_infos_phone_id_seq'::regclass),
phone character varying(50) COLLATE pg_catalog."default" NOT NULL,
company_id integer,
phone_tested boolean DEFAULT false,
imported_at timestamp with time zone NOT NULL,
CONSTRAINT phones_infos_pkey PRIMARY KEY (phone_id),
CONSTRAINT fk_phones_infos FOREIGN KEY (company_id)
REFERENCES public.companies_infos (id) MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE CASCADE
)
There are exactly 137468 records in this table, using:
SELECT count(1) FROM phones_infos;
The ERROR: out of memory for query result occurs with this simple query when I use pgAdmin 4.6:
SELECT * FROM phones_infos;
I have tables with 5M+ records and never had this problem before.
EXPLAIN SELECT * FROM phones_infos;
Seq Scan on phones_infos (cost=0.00..2546.68 rows=137468 width=33)
I read this article to see if I could find answers, but unfortunately as we can see on metrics: there are no old pending connections that could eat memory.
As suggested, the shared_buffers seems to be correctly sized:
SHOW shared_buffers;
449920kB
What should I try?
The problem must be on the client side. A sequential scan does not require much memory in PostgreSQL.
pgAdmin will cache the complete result set in RAM, which probably explains the out-of-memory condition.
I see two options:
Limit the number of result rows in pgAdmin:
SELECT * FROM phones_infos LIMIT 1000;
Use a different client, for example psql. There you can avoid the problem by setting
\set FETCH_COUNT 1000
so that the result set is fetched in batches.
I have a table called streams which has several rows, I like to change the order of the physical rows in the database according to the created_on column . I have a total of 179 rows, row (id) 179 is supposed to have the latest created_on date, then row (id) 178 should have the second newest created_on date . As you can see from the small snippet below things are not in order for example the date on id 172 is newer than the one on id 179 . I could get the correct data by doing this
select * from streams order by created_on desc
however they want it to search by id such as this
select * from streams order by id desc
The following query above produced the results below . We had everything working correctly but we did a migration from Postgres 10 to Postgres 11 and messed up on the order of things.
You can re-generate the id column values with an UPDATE such as the following:
UPDATE streams SET id = tmp.row_id
FROM (SELECT id, row_number() OVER (ORDER BY created_on, id) AS row_id FROM streams) AS tmp
WHERE streams.id = tmp.id;
After that, you might want to re-order the rows physically on disk using CLUSTER:
CLUSTER streams USING <index-for-id-column>;
I'm having an issue with limiting the SQL query. I'm using SQL 2000 so I can't use any of the functions like ROW_NUMBER(),CTE OR OFFSET_ROW FETCH.
I have tried the Select TOP limit * FROM approach and excluded the already shown results but this way the query is so slow because sometimes my result query fetches more than 10000 records.
Also I have tried the following approach:
SELECT * FROM (
SELECT DISTINCT TOP 100 PERCENT i.name, i.location, i.image ,
( SELECT count(DISTINCT i.id) FROM image WHERE i.id<= im.id ) AS recordnum
FROM images AS im
order by im.location asc, im.name asc) as tmp
WHERE recordnum between 5 AND 15
same problem here plus issue because I couldn't add ORDER option in sub query from record um. I have placed both solution in stored procedure but still the query execution is still so slow.
So my question is:
IS there an efficient way to limit the query to pull 20 records per page in SQL 2000 for large amounts of data i.e more than 10000?
Thanks.
Now the subquery is only run once
where im2.id is null will skip the first 40 rows
SELECT top 25 im1.*
FROM images im1
left join ( select top 40 id from images order by id ) im2
on im1.id = im2.id
where im2.id is null
order by im1.id
Query-wise, there is no great performing way. If performance is critical and the data will always be grouped/ordered the same, you could add a int column and set the value by trigger based on the grouping/ordering. Index it and it should be extremely fast for reads; writes will be a bit slower.
Also, make sure you have indexes on the Id columns on image and images.