I am using psql 8.1.18 on Glassfishserver. I have a query like this:
select ip,round((select sum(t1.size) from table t1))
from table
where date > '2011.07.29'
and date < '2011.07.30'
and ip = '255.255.255.255'
group by ip;
When I run this query I got this error:
ERROR: could not read block 4707 of relation 1663/16384/16564: Success
However this query works fine:
select ip,round(sum(size)/175)
from table
where date > '2011.07.29'
and l_date < '2011.07.30'
and ip = '255.255.255.255'
group by ip;
I think it might be a database error and I need to restore the table from the backup, maybe. But first I need to learn where this corrupted data exist. Does anyone know how to find 1663/16384/16564 relation? Or 4707 block?
EDIT:
I tried this code:
select relname , relfilenode from pg_class where relname in ('1663','16384','16564');
but it returns:
relname | relfilenode
---------+-------------
(0 rows)
It looks like there are bad blocks in a table or an index.
To find the bad data, Maybe you can query pg_class views ;
select oid,relname from pg_class where oid =1663 or oid=16564;
just see what's the result!
IF the result is an index, just recreate the corrupted index;
IF the result is a table , than it means that there are some data of the table is damaged,
you can set the parameter "zero_damaged_pages" to on to by pass those corrupted data or
restore the table from your recently backup set !
more information about the parameter "zero_damaged_pages"
http://www.postgresql.org/docs/9.0/static/runtime-config-developer.html
Related
Trying to get a list of COPY commands run on a particular date and the tables that were updated for each COPY command.
Working with this query:
select
slc.query as query_id,
trim(slc.filename) as file,
slc.curtime as updated,
slc.lines_scanned as rows,
sq.querytxt as querytxt
from stl_load_commits slc
join stl_query sq on sq.query = slc.query
where trunc(slc.curtime) = '2020-05-07';
How can we get the table that was updated for each COPY command? Maybe using a Redshift RegEx function on querytxt? Or joining to another system table to find the table id or name?
this regex will select the table or schema.table from stl_query.querytxt
select
slc.query as query_id,
trim(slc.filename) as file,
slc.curtime as updated,
slc.lines_scanned as rows,
sq.querytxt as querytxt,
REGEXP_REPLACE(LOWER(sq.querytxt), '^copy (analyze )?(\\S+).*$', '$2') AS t
from stl_load_commits slc
join stl_query sq on sq.query = slc.query
where trunc(updated) = '2020-05-07';
I have the following table in my PostgreSQL database:
CREATE TABLE values
(
dt timestamp,
series_id integer,
value real
);
CREATE INDEX idx_values_date ON public."values" USING btree (dt);
ALTER TABLE ONLY public."values" ADD CONSTRAINT values_series_id_fkey FOREIGN KEY (series_id) REFERENCES public.series(id) ON DELETE CASCADE;
I'm parsing some CSV files and extracting floats which I add to this table together with a timestamp and series_id which is a foreign key to another table.
The directory containing my raw data files amounts to about 28MB on my drive.
After feeding the data into my table I do a
SELECT pg_size_pretty( pg_total_relation_size('values') );
And find that the table now has ~871503 rows and is now 98MB in size. Is this normal? I was expecting my table to be way less in
size than actual text files containing raw data.
I'd like to mention that the PostgreSQL instance also has PostGIS installed but I'm not using it
in this particular schema. Furthermore, I'm running PostgreSQL from a docker container.
Later edit ...
After doing some more research and running the following query:
SELECT *, pg_size_pretty(total_bytes) AS total
, pg_size_pretty(index_bytes) AS INDEX
, pg_size_pretty(toast_bytes) AS toast
, pg_size_pretty(table_bytes) AS TABLE
FROM (
SELECT *, total_bytes-index_bytes-COALESCE(toast_bytes,0) AS table_bytes FROM (
SELECT c.oid,nspname AS table_schema, relname AS TABLE_NAME
, c.reltuples AS row_estimate
, pg_total_relation_size(c.oid) AS total_bytes
, pg_indexes_size(c.oid) AS index_bytes
, pg_total_relation_size(reltoastrelid) AS toast_bytes
FROM pg_class c
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE relkind = 'r'
) a
) a WHERE a.table_name = 'values';
I came up with the following results:
Index: 61MB
Table: 38MB
Can I somehow optimize the index? Maybe it's using some defaults that make it take up so much space?
When I populate a table with your structure with that number of rows, I get:
table: 37 MB
index: 24 MB
So either your index is bloated (you can drop and recreate it, or use REINDEX) or you have more indexes than you are admitting to.
But perhaps the better answer is "Yes, relational databases have a lot of overhead, get used to it." If you try to investigate every difference between a database and a flat file, you will drive yourself crazy and accomplish very little from it.
I'm working on a postgreSQL database with 22 table. I need a query which returns the columns with null values. May be a static sql statement that I can launch to each table.
I would be pleased to get some help.
Best.
Assuming that you run VACUUM ANALYZE periodically, pg_stats.null_frac can help you to get that:
--Get columns "filled" entirely with null values
SELECT
schemaname,
tablename,
attname,
null_frac
FROM
pg_stats
WHERE
null_frac = 1.0
AND schemaname = 'yourschema'
Could you tell me how to check what indexes are created for some table in postgresql ?
The view pg_indexes provides access to useful information about each index in the database, eg.
select *
from pg_indexes
where tablename not like 'pg%';
if you're in psql, then:
\d tablename
show Indexes, Foreign Keys and references...
You can use this query:
select tablename,indexname,tablespace,indexdef from pg_indexes where tablename = 'your_table_name';
where has tablename is a field in pg_indexes ,you an get an accurate indices by matching user defined table at 'your_table_name' at WHERE clause . This will give you the desired details.
You can find all the index related information inside the pg_indexes view. Sometimes, a table may be part of some schema ("owner") or might have had a different name in the past (see: PostgreSQL Rename Table).
So first find out what is the schema ("owner") of the table:
SELECT schemaname, tablename FROM pg_tables WHERE tablename='table_name';
and then query indexes on the table with either of these queries:
SELECT tablename, indexname FROM pg_indexes WHERE tablename='table_name';
-- or
SELECT * FROM pg_indexes WHERE tablename='schema_name.table_name';
As an alternative to all the above, you can also use \d:
\d table_name;
The command
\di
will list all indexes for the current schema.
Does anyone know how to find the OID of a table in Postgres 9.1?
I am writing an update script that needs to test for the existence of a column in a table before it tries to add the column. This is to prevent errors when running the script repeatedly.
To get a table OID, cast to the object identifier type regclass (while connected to the same DB):
SELECT 'mytbl'::regclass::oid;
This finds the first table (or view, etc.) with the given name along the search_path or raises an exception if not found.
Schema-qualify the table name to remove the dependency on the search path:
SELECT 'myschema.mytbl'::regclass::oid;
In Postgres 9.4 or later you can also use to_regclass('myschema.mytbl'), which doesn't raise an exception if the table is not found:
How to check if a table exists in a given schema
Then you only need to query the catalog table pg_attribute for the existence of the column:
SELECT TRUE AS col_exists
FROM pg_attribute
WHERE attrelid = 'myschema.mytbl'::regclass
AND attname = 'mycol'
AND NOT attisdropped -- no dropped (dead) columns
-- AND attnum > 0 -- no system columns (you may or may not want this)
;
The postgres catalog table pg_class is what you should look at. There should be one row per table, with the table name in the column relname, and the oid in the hidden column oid.
You may also be interested in the pg_attribute catalog table, which includes one row per table column.
See: http://www.postgresql.org/docs/current/static/catalog-pg-class.html and http://www.postgresql.org/docs/current/static/catalog-pg-attribute.html
SELECT oid FROM pg_class WHERE relname = 'tbl_name' AND relkind = 'r';
Just to complete the possibilities I'd like to add that there exists a syntax for dropping columns in order to no error out:
ALTER TABLE mytbl
DROP COLUMN IF EXISTS mycol
See http://www.postgresql.org/docs/9.0/static/sql-altertable.html
Then you can safely add your column.