Is it possible to search every column of every table for a particular value in PostgreSQL?
A similar question is available here for Oracle.
How about dumping the contents of the database, then using grep?
$ pg_dump --data-only --inserts -U postgres your-db-name > a.tmp
$ grep United a.tmp
INSERT INTO countries VALUES ('US', 'United States');
INSERT INTO countries VALUES ('GB', 'United Kingdom');
The same utility, pg_dump, can include column names in the output. Just change --inserts to --column-inserts. That way you can search for specific column names, too. But if I were looking for column names, I'd probably dump the schema instead of the data.
$ pg_dump --data-only --column-inserts -U postgres your-db-name > a.tmp
$ grep country_code a.tmp
INSERT INTO countries (iso_country_code, iso_country_name) VALUES ('US', 'United States');
INSERT INTO countries (iso_country_code, iso_country_name) VALUES ('GB', 'United Kingdom');
Here's a pl/pgsql function that locates records where any column contains a specific value.
It takes as arguments the value to search in text format, an array of table names to search into (defaults to all tables) and an array of schema names (defaults all schema names).
It returns a table structure with schema, name of table, name of column and pseudo-column ctid (non-durable physical location of the row in the table, see System Columns)
CREATE OR REPLACE FUNCTION search_columns(
needle text,
haystack_tables name[] default '{}',
haystack_schema name[] default '{}'
)
RETURNS table(schemaname text, tablename text, columnname text, rowctid text)
AS $$
begin
FOR schemaname,tablename,columnname IN
SELECT c.table_schema,c.table_name,c.column_name
FROM information_schema.columns c
JOIN information_schema.tables t ON
(t.table_name=c.table_name AND t.table_schema=c.table_schema)
JOIN information_schema.table_privileges p ON
(t.table_name=p.table_name AND t.table_schema=p.table_schema
AND p.privilege_type='SELECT')
JOIN information_schema.schemata s ON
(s.schema_name=t.table_schema)
WHERE (c.table_name=ANY(haystack_tables) OR haystack_tables='{}')
AND (c.table_schema=ANY(haystack_schema) OR haystack_schema='{}')
AND t.table_type='BASE TABLE'
LOOP
FOR rowctid IN
EXECUTE format('SELECT ctid FROM %I.%I WHERE cast(%I as text)=%L',
schemaname,
tablename,
columnname,
needle
)
LOOP
-- uncomment next line to get some progress report
-- RAISE NOTICE 'hit in %.%', schemaname, tablename;
RETURN NEXT;
END LOOP;
END LOOP;
END;
$$ language plpgsql;
See also the version on github based on the same principle but adding some speed and reporting improvements.
Examples of use in a test database:
Search in all tables within public schema:
select * from search_columns('foobar');
schemaname | tablename | columnname | rowctid
------------+-----------+------------+---------
public | s3 | usename | (0,11)
public | s2 | relname | (7,29)
public | w | body | (0,2)
(3 rows)
Search in a specific table:
select * from search_columns('foobar','{w}');
schemaname | tablename | columnname | rowctid
------------+-----------+------------+---------
public | w | body | (0,2)
(1 row)
Search in a subset of tables obtained from a select:
select * from search_columns('foobar', array(select table_name::name from information_schema.tables where table_name like 's%'), array['public']);
schemaname | tablename | columnname | rowctid
------------+-----------+------------+---------
public | s2 | relname | (7,29)
public | s3 | usename | (0,11)
(2 rows)
Get a result row with the corresponding base table and and ctid:
select * from public.w where ctid='(0,2)';
title | body | tsv
-------+--------+---------------------
toto | foobar | 'foobar':2 'toto':1
Variants
To test against a regular expression instead of strict equality, like grep, this part of the query:
SELECT ctid FROM %I.%I WHERE cast(%I as text)=%L
may be changed to:
SELECT ctid FROM %I.%I WHERE cast(%I as text) ~ %L
For case insensitive comparisons, you could write:
SELECT ctid FROM %I.%I WHERE lower(cast(%I as text)) = lower(%L)
to search every column of every table for a particular value
This does not define how to match exactly.
Nor does it define what to return exactly.
Assuming:
Find any row with any column containing the given value in its text representation - as opposed to equaling the given value.
Return the table name (regclass) and the tuple ID (ctid), because that's simplest.
Here is a dead simple, fast and slightly dirty way:
CREATE OR REPLACE FUNCTION search_whole_db(_like_pattern text)
RETURNS TABLE(_tbl regclass, _ctid tid) AS
$func$
BEGIN
FOR _tbl IN
SELECT c.oid::regclass
FROM pg_class c
JOIN pg_namespace n ON n.oid = relnamespace
WHERE c.relkind = 'r' -- only tables
AND n.nspname !~ '^(pg_|information_schema)' -- exclude system schemas
ORDER BY n.nspname, c.relname
LOOP
RETURN QUERY EXECUTE format(
'SELECT $1, ctid FROM %s t WHERE t::text ~~ %L'
, _tbl, '%' || _like_pattern || '%')
USING _tbl;
END LOOP;
END
$func$ LANGUAGE plpgsql;
Call:
SELECT * FROM search_whole_db('mypattern');
Provide the search pattern without enclosing %.
Why slightly dirty?
If separators and decorators for the row in text representation can be part of the search pattern, there can be false positives:
column separator: , by default
whole row is enclosed in parentheses:()
some values are enclosed in double quotes "
\ may be added as escape char
And the text representation of some columns may depend on local settings - but that ambiguity is inherent to the question, not to my solution.
Each qualifying row is returned once only, even when it matches multiple times (as opposed to other answers here).
This searches the whole DB except for system catalogs. Will typically take a long time to finish. You might want to restrict to certain schemas / tables (or even columns) like demonstrated in other answers. Or add notices and a progress indicator, also demonstrated in another answer.
The regclass object identifier type is represented as table name, schema-qualified where necessary to disambiguate according to the current search_path:
Find the referenced table name using table, field and schema name
What is the ctid?
How do I decompose ctid into page and row numbers?
You might want to escape characters with special meaning in the search pattern. See:
Escape function for regular expression or LIKE patterns
There is a way to achieve this without creating a function or using an external tool. By using Postgres' query_to_xml() function that can dynamically run a query inside another query, it's possible to search a text across many tables. This is based on my answer to retrieve the rowcount for all tables:
To search for the string foo across all tables in a schema, the following can be used:
with found_rows as (
select format('%I.%I', table_schema, table_name) as table_name,
query_to_xml(format('select to_jsonb(t) as table_row
from %I.%I as t
where t::text like ''%%foo%%'' ', table_schema, table_name),
true, false, '') as table_rows
from information_schema.tables
where table_schema = 'public'
)
select table_name, x.table_row
from found_rows f
left join xmltable('//table/row'
passing table_rows
columns
table_row text path 'table_row') as x on true
Note that the use of xmltable requires Postgres 10 or newer. For older Postgres version, this can be also done using xpath().
with found_rows as (
select format('%I.%I', table_schema, table_name) as table_name,
query_to_xml(format('select to_jsonb(t) as table_row
from %I.%I as t
where t::text like ''%%foo%%'' ', table_schema, table_name),
true, false, '') as table_rows
from information_schema.tables
where table_schema = 'public'
)
select table_name, x.table_row
from found_rows f
cross join unnest(xpath('/table/row/table_row/text()', table_rows)) as r(data)
The common table expression (WITH ...) is only used for convenience. It loops through all tables in the public schema. For each table the following query is run through the query_to_xml() function:
select to_jsonb(t)
from some_table t
where t::text like '%foo%';
The where clause is used to make sure the expensive generation of XML content is only done for rows that contain the search string. This might return something like this:
<table xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<row>
<table_row>{"id": 42, "some_column": "foobar"}</table_row>
</row>
</table>
The conversion of the complete row to jsonb is done, so that in the result one could see which value belongs to which column.
The above might return something like this:
table_name | table_row
-------------+----------------------------------------
public.foo | {"id": 1, "some_column": "foobar"}
public.bar | {"id": 42, "another_column": "barfoo"}
Online example for Postgres 10+
Online example for older Postgres versions
Without storing a new procedure you can use a code block and execute to obtain a table of occurences. You can filter results by schema, table or column name.
DO $$
DECLARE
value int := 0;
sql text := 'The constructed select statement';
rec1 record;
rec2 record;
BEGIN
DROP TABLE IF EXISTS _x;
CREATE TEMPORARY TABLE _x (
schema_name text,
table_name text,
column_name text,
found text
);
FOR rec1 IN
SELECT table_schema, table_name, column_name
FROM information_schema.columns
WHERE table_name <> '_x'
AND UPPER(column_name) LIKE UPPER('%%')
AND table_schema <> 'pg_catalog'
AND table_schema <> 'information_schema'
AND data_type IN ('character varying', 'text', 'character', 'char', 'varchar')
LOOP
sql := concat('SELECT ', rec1."column_name", ' AS "found" FROM ',rec1."table_schema" , '.',rec1."table_name" , ' WHERE UPPER(',rec1."column_name" , ') LIKE UPPER(''','%my_substring_to_find_goes_here%' , ''')');
RAISE NOTICE '%', sql;
BEGIN
FOR rec2 IN EXECUTE sql LOOP
RAISE NOTICE '%', sql;
INSERT INTO _x VALUES (rec1."table_schema", rec1."table_name", rec1."column_name", rec2."found");
END LOOP;
EXCEPTION WHEN OTHERS THEN
END;
END LOOP;
END; $$;
SELECT * FROM _x;
If you're using IntelliJ add your DB to Database view then right click on databases and select full text search, it will list all tables and all fields for your specific text.
And if someone think it could help. Here is #Daniel Vérité's function, with another param that accept names of columns that can be used in search. This way it decrease the time of processing. At least in my test it reduced a lot.
CREATE OR REPLACE FUNCTION search_columns(
needle text,
haystack_columns name[] default '{}',
haystack_tables name[] default '{}',
haystack_schema name[] default '{public}'
)
RETURNS table(schemaname text, tablename text, columnname text, rowctid text)
AS $$
begin
FOR schemaname,tablename,columnname IN
SELECT c.table_schema,c.table_name,c.column_name
FROM information_schema.columns c
JOIN information_schema.tables t ON
(t.table_name=c.table_name AND t.table_schema=c.table_schema)
WHERE (c.table_name=ANY(haystack_tables) OR haystack_tables='{}')
AND c.table_schema=ANY(haystack_schema)
AND (c.column_name=ANY(haystack_columns) OR haystack_columns='{}')
AND t.table_type='BASE TABLE'
LOOP
EXECUTE format('SELECT ctid FROM %I.%I WHERE cast(%I as text)=%L',
schemaname,
tablename,
columnname,
needle
) INTO rowctid;
IF rowctid is not null THEN
RETURN NEXT;
END IF;
END LOOP;
END;
$$ language plpgsql;
Bellow is an example of usage of the search_function created above.
SELECT * FROM search_columns('86192700'
, array(SELECT DISTINCT a.column_name::name FROM information_schema.columns AS a
INNER JOIN information_schema.tables as b ON (b.table_catalog = a.table_catalog AND b.table_schema = a.table_schema AND b.table_name = a.table_name)
WHERE
a.column_name iLIKE '%cep%'
AND b.table_type = 'BASE TABLE'
AND b.table_schema = 'public'
)
, array(SELECT b.table_name::name FROM information_schema.columns AS a
INNER JOIN information_schema.tables as b ON (b.table_catalog = a.table_catalog AND b.table_schema = a.table_schema AND b.table_name = a.table_name)
WHERE
a.column_name iLIKE '%cep%'
AND b.table_type = 'BASE TABLE'
AND b.table_schema = 'public')
);
Here's #Daniel Vérité's function with progress reporting functionality.
It reports progress in three ways:
by RAISE NOTICE;
by decreasing value of supplied {progress_seq} sequence from
{total number of colums to search in} down to 0;
by writing the progress along with found tables into text file,
located in c:\windows\temp\{progress_seq}.txt.
_
CREATE OR REPLACE FUNCTION search_columns(
needle text,
haystack_tables name[] default '{}',
haystack_schema name[] default '{public}',
progress_seq text default NULL
)
RETURNS table(schemaname text, tablename text, columnname text, rowctid text)
AS $$
DECLARE
currenttable text;
columnscount integer;
foundintables text[];
foundincolumns text[];
begin
currenttable='';
columnscount = (SELECT count(1)
FROM information_schema.columns c
JOIN information_schema.tables t ON
(t.table_name=c.table_name AND t.table_schema=c.table_schema)
WHERE (c.table_name=ANY(haystack_tables) OR haystack_tables='{}')
AND c.table_schema=ANY(haystack_schema)
AND t.table_type='BASE TABLE')::integer;
PERFORM setval(progress_seq::regclass, columnscount);
FOR schemaname,tablename,columnname IN
SELECT c.table_schema,c.table_name,c.column_name
FROM information_schema.columns c
JOIN information_schema.tables t ON
(t.table_name=c.table_name AND t.table_schema=c.table_schema)
WHERE (c.table_name=ANY(haystack_tables) OR haystack_tables='{}')
AND c.table_schema=ANY(haystack_schema)
AND t.table_type='BASE TABLE'
LOOP
EXECUTE format('SELECT ctid FROM %I.%I WHERE cast(%I as text)=%L',
schemaname,
tablename,
columnname,
needle
) INTO rowctid;
IF rowctid is not null THEN
RETURN NEXT;
foundintables = foundintables || tablename;
foundincolumns = foundincolumns || columnname;
RAISE NOTICE 'FOUND! %, %, %, %', schemaname,tablename,columnname, rowctid;
END IF;
IF (progress_seq IS NOT NULL) THEN
PERFORM nextval(progress_seq::regclass);
END IF;
IF(currenttable<>tablename) THEN
currenttable=tablename;
IF (progress_seq IS NOT NULL) THEN
RAISE NOTICE 'Columns left to look in: %; looking in table: %', currval(progress_seq::regclass), tablename;
EXECUTE 'COPY (SELECT unnest(string_to_array(''Current table (column ' || columnscount-currval(progress_seq::regclass) || ' of ' || columnscount || '): ' || tablename || '\n\nFound in tables/columns:\n' || COALESCE(
(SELECT string_agg(c1 || '/' || c2, '\n') FROM (SELECT unnest(foundintables) AS c1,unnest(foundincolumns) AS c2) AS t1)
, '') || ''',''\n''))) TO ''c:\WINDOWS\temp\' || progress_seq || '.txt''';
END IF;
END IF;
END LOOP;
END;
$$ language plpgsql;
-- Below function will list all the tables which contain a specific string in the database
select TablesCount(‘StringToSearch’);
--Iterates through all the tables in the database
CREATE OR REPLACE FUNCTION **TablesCount**(_searchText TEXT)
RETURNS text AS
$$ -- here start procedural part
DECLARE _tname text;
DECLARE cnt int;
BEGIN
FOR _tname IN SELECT table_name FROM information_schema.tables where table_schema='public' and table_type='BASE TABLE' LOOP
cnt= getMatchingCount(_tname,Columnames(_tname,_searchText));
RAISE NOTICE 'Count% ', CONCAT(' ',cnt,' Table name: ', _tname);
END LOOP;
RETURN _tname;
END;
$$ -- here finish procedural part
LANGUAGE plpgsql; -- language specification
-- Returns the count of tables for which the condition is met.
-- For example, if the intended text exists in any of the fields of the table,
-- then the count will be greater than 0. We can find the notifications
-- in the Messages section of the result viewer in postgres database.
CREATE OR REPLACE FUNCTION **getMatchingCount**(_tname TEXT, _clause TEXT)
RETURNS int AS
$$
Declare outpt text;
BEGIN
EXECUTE 'Select Count(*) from '||_tname||' where '|| _clause
INTO outpt;
RETURN outpt;
END;
$$ LANGUAGE plpgsql;
--Get the fields of each table. Builds the where clause with all columns of a table.
CREATE OR REPLACE FUNCTION **Columnames**(_tname text,st text)
RETURNS text AS
$$ -- here start procedural part
DECLARE
_name text;
_helper text;
BEGIN
FOR _name IN SELECT column_name FROM information_schema.Columns WHERE table_name =_tname LOOP
_name=CONCAT('CAST(',_name,' as VarChar)',' like ','''%',st,'%''', ' OR ');
_helper= CONCAT(_helper,_name,' ');
END LOOP;
RETURN CONCAT(_helper, ' 1=2');
END;
$$ -- here finish procedural part
LANGUAGE plpgsql; -- language specification
I need obtain table names from schema, except some tables
CREATE OR REPLACE FUNCTION func(unnecessary_tables TEXT)
returns void
as $$
begin
EXECUTE 'SELECT table_name FROM information_schema.tables
WHERE
table_schema=''public''
AND
table_name NOT IN( $1 )
' USING unnecessary_tables
--here execute retrieved result, etc ...
end;
$$language plpgsql
Then call function
select func('table1'',''table2');
This not works and returns in result table1 and table2 also.
Question is: How to pass text parameter to stored function, for IN operator ?
Pass a text array in instead of text:
create or replace function func(unnecessary_tables text[])
returns void as $$
begin
select table_name
from information_schema.tables
where
table_schema = 'public'
and
not(table_name = any($1))
;
end;
$$language plpgsql
Call it like:
select func(array['t1','t2']::text[]);
BTW the code above can be plain SQL in instead of PL/pgSQL
To answer you exact question (How to pass to function text for IN operator) You need:
SELECT func( '''table1'',''table2''');
The reason is that table names must by string, so they need to by inside quotes.
To make it works there is one change in code needed which I did't see at first:
CREATE OR REPLACE FUNCTION func(unnecessary_tables TEXT)
returns void
as $$
begin
EXECUTE 'SELECT table_name FROM information_schema.tables
WHERE
table_schema=''public''
AND
table_name NOT IN(' || unnecessary_tables || ')';
--here execute retrieved result, etc ...
end;
$$language plpgsql
It's needed because USINGis aware of types and don't just "paste" parameter in place of $1.
I don't think none of above answers are correct.
select pg_typeof(table_name),table_name::text
from information_schema.tables
where table_schema = 'public';
It will return:
pg_typeof | table_name
-----------------------------------+--------------------------
information_schema.sql_identifier | parent_tree
from Which means at least the table_name should be cast to text.
Here is my solution:
create or replace function n_fnd_tbl(_other_tables text[])
returns table(__table_name text) as
$$
begin
return query EXECUTE format('
select table_name::text
from information_schema.tables
where table_schema = ''public''
and table_name <> ''%s''',_other_tables );
end
$$language plpgsql;
Then call it:
select * from n_fnd_tbl(array['tableb','tablea']::text[]);
I'm trying to figure out how to resolve this ERROR: null values cannot be formatted as an SQL identifier when trying to select my function:
select * from store_keys();
ERROR: null values cannot be formatted as an SQL identifier
CONTEXT: SQL statement "SELECT string_agg(
format('SELECT %1$I, count(email_store_key), email_store_key, form_created_datetime
FROM %1$I where email_store_key=0 GROUP BY 3, 4', tbl_name),
' UNION ') FROM information_schema.tables
WHERE table_schema = 'abc_dev_sch_1234'
AND table_name LIKE 'fact_%'"
The base query it uses, does not produce null values. So where is this coming from??
select count(email_store_key), email_store_key, form_created_datetime
FROM <table_name> where email_store_key=0 GROUP BY email_store_key, form_created_datetime;
Here's my create statement:
DROP FUNCTION store_keys();
CREATE OR REPLACE FUNCTION store_keys()
RETURNS TABLE (tbl_name varchar, count_keys bigint, email_store_key integer, form_created_datetime timestamp)
AS $$
DECLARE
qry text;
BEGIN
SELECT string_agg(
format('SELECT %1$I, count(email_store_key), email_store_key, form_created_datetime
FROM %1$I where email_store_key=0 GROUP BY 3, 4', tbl_name),
' UNION ') INTO qry
FROM information_schema.tables
WHERE table_schema = 'abc_dev_sch_1234'
AND table_name LIKE 'fact_%';
RETURN QUERY EXECUTE qry;
END
$$ LANGUAGE plpgsql;
Can any one of you tell me how to approach this:
CREATE OR REPLACE FUNCTION name()
RETURNS ????? AS
$func$
BEGIN
SELECT * FROM tbl_a a;
SELECT * FROM tbl_b b;
END
$func$ LANGUAGE plpgsql;
Both tables have different structures.
You can use cursors but I can hardly imagine why you need such a function.
CREATE OR REPLACE FUNCTION my_multiselect(refcursor, refcursor) RETURNS VOID AS
$func$
BEGIN
OPEN $1 FOR SELECT * FROM information_schema.routines;
OPEN $2 FOR SELECT * FROM information_schema.sequences;
END
$func$ LANGUAGE plpgsql;
BEGIN;
SELECT my_multiselect('first_cursor_to_routines', 'second_cursor_to_sequences');
FETCH ALL IN first_cursor_to_routines;
FETCH ALL IN second_cursor_to_sequences;
COMMIT;
I'm not really sure what you're doing with this, but it sounds like you just want to return a union of these distinct result sets. You can do this with a dynamic query. I'm using Postgres 9.4.
CREATE OR REPLACE FUNCTION make_query(IN p_tables text[])
RETURNS void AS
$BODY$
DECLARE
v_qry text;
v_cols text;
v_types text;
v_as text;
BEGIN
EXECUTE format('
WITH sub AS (
SELECT
table_name,
column_name,
data_type
FROM
information_schema.columns
WHERE
table_name = ANY(%L)
ORDER BY
table_name,
ordinal_position)
,sub2 AS(
SELECT
DISTINCT ON (column_name, data_type)
column_name || '' '' || data_type AS def
FROM
sub
)
SELECT
string_agg(def, '','')
FROM
sub2;
',
p_tables
) INTO v_types;
v_qry := '
CREATE OR REPLACE FUNCTION name()
RETURNS TABLE(' || v_types || ') AS
$func$';
FOR i IN 1..array_upper(p_tables, 1)
LOOP
v_as := 'tbl' || i;
EXECUTE format('
WITH sub AS (
SELECT
table_name,
column_name,
data_type
FROM
information_schema.columns
WHERE
table_name = ANY(%L)
ORDER BY
table_name,
ordinal_position)
,sub2 AS(
SELECT
DISTINCT ON (column_name, data_type)
CASE WHEN table_name = ''%I''
THEN %L || ''.'' || column_name
ELSE ''NULL::'' || data_type
END AS cols
FROM
sub
)
SELECT
string_agg(cols, '','')
FROM
sub2;
',
p_tables,
p_tables[i],
v_as
) INTO v_cols;
IF i > 1 THEN
v_qry := v_qry || '
UNION ALL';
END IF;
v_qry := v_qry || '
SELECT ' || v_cols || ' FROM ' || p_tables[i] || ' AS ' || v_as;
IF i = array_upper(p_tables, 1) THEN
v_qry := v_qry || ';';
END IF;
END LOOP;
v_qry := v_qry || '
$func$ LANGUAGE sql;
';
EXECUTE v_qry;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
Sorry it looks ugly here, but this formatting helps the final product look nicer. If you're shy about executing a dynamic query like this off the bat, just replace EXECUTE v_qry; with RAISE INFO 'v_qry: %', v_qry; and it will simply print the dynamic query out in a message without executing it, so you can review what it will do once executed.
Then execute make_query() with a list of tables you want to display like this:
SELECT make_query(ARRAY['tbl_a', 'tbl_b']);
The result is that you will now have a function called name() which you can call in order to see the results of both tables at the same time, with all the union details already sorted out:
SELECT * FROM name();