Related
I have one array with column values as
{james=UC/james,adam=C/james,chris=UC/james,john=U/james}
The above column values are not json. They are in string in the following form:
{ username=privilegestring/grantor }
How to convert above column into multiple rows
Edit #3:
Updated the query to specifically target pg_catalog.pg_namespace for acl permissions grants, via CTE pg_catalog. Currently this CTE is filtered in the where clause to select a single namespace name ('avengers'); if you want to select from multiple namespace names, you should be able to add them into the WHERE clause of this CTE directly, or in the case of wanting all namespace names, remove the clause altogether.
It's worth noting as well, that you will need to expand the case statements in access_privilege_types to handle all permissions cases: 'r', 'w', 'a', 'd', and 'x', for the operations: SELECT, UPDATE, INSERT, DELETE, REFERENCE, respectively.
Edit #2:
The final posted version of the query below should get you the data you want in the format that you want it in. I don't know how many possible values there are for the permissions types; if you have more than the two specified currently, you will need to expand the case statements in the CTE* access_privilege_types*. Obviously you'll also need to replace your table name within the query, etc.. Let me know if you run into any trouble and I'll help as necessary.
Edit #1:
Was able to validate that this query works in Redshift. Updated the query to break out separate rows by grantee and owner. The current version doesn't break out individual permissions by row yet -- Will take a look later tonight to see if I can get that working as well.
Original:
I don't have access to my Redshift cluster to test this at the moment, but I will when I get home. The general idea behind the following method, is to create a numbered index table to cross join against that will expand the data in the permissions field into a row-based representation.
I had inquired about the size limit, because this will currently only handle 10,000 possible delimited values, however you can adjust the CTEs to scale up to larger amounts if needed for your specific application:
Revision 3 Query:
WITH
pg_namespace AS (
SELECT
nspname
, nspowner
, rtrim(ltrim(array_to_string(nspacl, ','), '{'), '}') as nspacl
FROM pg_catalog.pg_namespace
WHERE nspname = 'public'
),
-- Generating a table with the numbers 1 - 10 in a single column.
ten_numbers AS (
SELECT
1 AS num
UNION SELECT 2
UNION SELECT 3
UNION SELECT 4
UNION SELECT 5
UNION SELECT 6
UNION SELECT 7
UNION SELECT 8
UNION SELECT 9
UNION SELECT 0
),
-- Expands the values in ten_numbers to create a single column with the values 1 - 10,000.
depivot_index AS (
SELECT
(1000 * t1.num) + (100 * t2.num) + (10 * t3.num) + t4.num AS gen_num
FROM ten_numbers AS t1
JOIN ten_numbers AS t2 ON 1 = 1
JOIN ten_numbers AS t3 ON 1 = 1
JOIN ten_numbers AS t4 ON 1 = 1
),
-- Filters down generated_numbers to house only the numbers up to the maximum times that the delimiter appears.
splitter AS (
SELECT
*
FROM depivot_index
WHERE gen_num BETWEEN 1 AND (
SELECT max(REGEXP_COUNT(nspacl, '\\,') + 1)
FROM pg_namespace
)
),
-- Cross joins permissions_groups and splitter to populate all requests, delimited on ','.
expanded_input AS (
SELECT
pg.nspname
, pg.nspacl
, trim(split_part(pg.nspacl, ',', s.gen_num)) AS raw_permissions_string
FROM pg_namespace AS pg
JOIN splitter AS s ON 1 = 1
WHERE split_part(nspacl, ',', s.gen_num) <> ''
),
-- Breaks out the owner and grantee fields into their own columns respectively.
users_with_raw_permissions_data AS (
SELECT
e.raw_permissions_string
, e.nspname
, trim(split_part(e.raw_permissions_string, '=', 1)) AS grantee
, trim(split_part(trim(split_part(e.raw_permissions_string, '=', 2)), '/', 2)) AS owner
, trim(split_part(trim(split_part(e.raw_permissions_string, '=', 2)), '/', 1)) AS raw_permissions_data
FROM
expanded_input e
),
-- Mines privilege types from raw string data.
access_privilege_types AS (
SELECT
u.nspname
, u.owner
, u.grantee
,CASE
WHEN position('C*' IN u.raw_permissions_data) > 0 THEN 'C*'
WHEN position('U*' IN u.raw_permissions_data) > 0 THEN 'U*'
WHEN position('C' IN u.raw_permissions_data) > 0 THEN 'C'
WHEN position('U' IN u.raw_permissions_data) > 0 THEN 'U'
ELSE u.raw_permissions_data
END AS first_access_privilege
, CASE
WHEN position('U*' IN u.raw_permissions_data) > 0 THEN 'U*'
WHEN position('C*' IN u.raw_permissions_data) > 0 THEN 'C*'
WHEN position('U' IN u.raw_permissions_data) > 0 THEN 'U'
WHEN position('C' IN u.raw_permissions_data) > 0 THEN 'C'
ELSE u.raw_permissions_data
END AS second_access_privilege
, first_access_privilege || ',' || second_access_privilege AS merged_access_privileges
FROM users_with_raw_permissions_data u
),
-- Cross joins access_privilge_types and splitter to populate all privilege_types, delimited on ','.
expanded_access_privilege_types AS (
SELECT
a.nspname
, a.owner
, a.grantee
, trim(split_part(a.merged_access_privileges, ',', s.gen_num)) AS access_privileges
FROM access_privilege_types AS a
JOIN splitter AS s ON 1 = 1
WHERE split_part(a.merged_access_privileges, ',', s.gen_num) <> ''
GROUP BY 1, 2, 3, 4
)
SELECT
ea.nspname
, ea.owner
, ea.grantee
, LEFT(ea.access_privileges, 1) AS access_privilege
, CASE
WHEN POSITION('*' IN ea.access_privileges) > 0 THEN 'YES'
ELSE 'NO'
END AS is_grantable
FROM expanded_access_privilege_types ea
ORDER BY 1, 2, 3, 4, 5
Edit #4:
Adding some clarification on how the ten_numbers, depivot_index, and splitter tables work to break apart the pg_catalog.pg_namespace.nspacl field. The general overview, is that ten_numbers and depivot_index are created purely to return tables with numbered rows to use as an index when joining in thesplit_partvalues ofnspacl`.
ten_numbers generates a table with a single column, containing the numbers 0-9:
-------
| num |
-------
| 0 |
-------
| 1 |
-------
| etc |
-------
| 9 |
-------
This table is then expanded to house the range 0-9999 during the CTE depivot_index:
-----------
| gen_num |
-----------
| 0 |
-----------
| 1 |
-----------
| 2 |
-----------
| etc |
-----------
| 9998 |
-----------
| 9999 |
-----------
splitter then narrows down the table to house only the numbers up to the maximum count of the specified delimiter within the nspacl field:
-------
| num |
-------
| 0 |
-------
| 1 |
-------
| etc |
-------
| 6 |
-------
The table returned by splitter is then used as the target of a CROSS JOIN via the join on 1 = 1 in CTE expanded_input. This ensures that each member returned by split_part will have its own row:
---------------------------------------------------------------------------
| nspname | nspacl | raw_permissions_string |
---------------------------------------------------------------------------
| avengers | "{james=UC/james,adam=C/james}" | "james=UC/james" |
---------------------------------------------------------------------------
| avengers | "{james=UC/james,adam=C/james}" | "adam=C/james" |
---------------------------------------------------------------------------
| avengers | etc. | etc. |
---------------------------------------------------------------------------
the title may not be very clear so let's consider this example (this is not my code, just taking this example to model my request)
I have a table that references itself (like a filesystem)
id | parent | name
----+----------+-------
1 | null | /
2 | 1 | home
3 | 2 | user
4 | 3 | bin
5 | 1 | usr
6 | 5 | local
Is it possible to make a sql request so if I choose :
1 I will get a table containing 2,3,4,5,6 (because this is the root) so matching :
/home
/home/user
/home/user/bin
/usr
etc...
2 I will get a table containing 3,4 so matching :
/home/user
/home/user/bin
and so on
Use recursive common table expression. Always starting from the root, use an array of ids to get paths for a given id in the WHERE clause.
For id = 1:
with recursive cte(id, parent, name, ids) as (
select id, parent, name, array[id]
from my_table
where parent is null
union all
select t.id, t.parent, concat(c.name, t.name, '/'), ids || t.id
from cte c
join my_table t on c.id = t.parent
)
select id, name
from cte
where 1 = any(ids) and id <> 1
id | name
----+-----------------------
2 | /home/
5 | /usr/
6 | /usr/local/
3 | /home/user/
4 | /home/user/bin/
(5 rows)
For id = 2:
with recursive cte(id, parent, name, ids) as (
select id, parent, name, array[id]
from my_table
where parent is null
union all
select t.id, t.parent, concat(c.name, t.name, '/'), ids || t.id
from cte c
join my_table t on c.id = t.parent
)
select id, name
from cte
where 2 = any(ids) and id <> 2
id | name
----+-----------------------
3 | /home/user/
4 | /home/user/bin/
(2 rows)
Bidirectional query
The question is really interesting. The above query works well but is inefficient as it parses all tree nodes even when we're asking for a leaf. The more powerful solution is a bidirectional recursive query. The inner query walks from a given node to top, while the outer one goes from the node to bottom.
with recursive outer_query(id, parent, name) as (
with recursive inner_query(qid, id, parent, name) as (
select id, id, parent, name
from my_table
where id = 2 -- parameter
union all
select qid, t.id, t.parent, concat(t.name, '/', q.name)
from inner_query q
join my_table t on q.parent = t.id
)
select qid, null::int, right(name, -1)
from inner_query
where parent is null
union all
select t.id, t.parent, concat(q.name, '/', t.name)
from outer_query q
join my_table t on q.id = t.parent
)
select id, name
from outer_query
where id <> 2; -- parameter
I want to select certain elements from an array column. I know you can do it by position, but I want to filter on content. Here's my data:
table_name | column_names
---------------------+---------------------------------------------------------------
attribute_definition | {attribute_type_concept_id}
cohort_definition | {definition_type_concept_id,subject_concept_id}
condition_occurrence | {condition_concept_id,condition_source_concept_id,condition_type_concept_id}
death | {cause_concept_id,cause_source_concept_id,death_impute_concept_id,death_type_concept_id}
device_exposure | {device_concept_id,device_source_concept_id,device_type_concept_id}
drug_exposure | {dose_unit_concept_id,drug_concept_id,drug_source_concept_id,drug_type_concept_id,route_concept_id}
What I would like to say is something like:
SELECT table_name,
array_agg(SELECT colname FROM column_names WHERE colname LIKE '%type%') AS type_cols,
array_agg(SELECT colname FROM column_names WHERE colname NOT LIKE '%type%') AS other_cols
FROM mytable
GROUP BY table_name
And the result I would like would be:
table_name | type_cols | other_cols
----------------------+--------------------------------------------------------------------------------------------------------------
attribute_definition | {attribute_type_concept_id} | {}
cohort_definition | {definition_type_concept_id} | {subject_concept_id}
condition_occurrence | {condition_type_concept_id} | {condition_concept_id,condition_source_concept_id}
death | {death_type_concept_id} | {cause_concept_id,cause_source_concept_id,death_impute_concept_id}
device_exposure | {device_type_concept_id} | {device_concept_id,device_source_concept_id}
drug_exposure | {drug_type_concept_id} | {dose_unit_concept_id,drug_concept_id,drug_source_concept_id,route_concept_id}
So, I want to end up with the same number of rows but different columns. There's gotta be a simple way to do this. Why can't I find it?
unnest is your friend. As in:
SELECT table_name,
array(SELECT colname FROM unnest(column_names) AS colname WHERE colname LIKE '%type%') AS type_cols,
array(SELECT colname FROM unnest(column_names) AS colname WHERE colname NOT LIKE '%type%') AS other_cols
FROM mytable
GROUP BY table_name, column_names
Here is Dan Getz's answer again but in a self-contained statement so it's easily runnable without copying my data.
with grps as
(
with numlist as
(
select '1 - 10' as grp, generate_series(1,10) num
union
select '11 - 20', generate_series(11,20) order by 1,2
)
select grp, array_agg(num) as nums
from numlist
group by 1
)
select grp,
(select array_agg(evens) from unnest(nums) as evens where evens % 2 = 0) as evens,
(select array_agg(odds) from unnest(nums) as odds where odds % 2 != 0) as odds
from grps
group by grp, nums;
grp | evens | odds
---------+------------------+------------------
11 - 20 | {12,14,16,18,20} | {11,13,15,17,19}
1 - 10 | {2,4,6,8,10} | {1,3,5,7,9}
I have a single table laid out as such:
id | name | count
1 | John |
2 | Jim |
3 | John |
4 | Tim |
I need to fill out the count column such that the result is the number of times the specific name shows up in the column name.
The result should be:
id | name | count
1 | John | 2
2 | Jim | 1
3 | John | 2
4 | Tim | 1
I can get the count of occurrences of unique names easily using:
SELECT COUNT(name)
FROM table
GROUP BY name
But that doesn't fit into an UPDATE statement due to it returning multiple rows.
I can also get it narrowed down to a single row by doing this:
SELECT COUNT(name)
FROM table
WHERE name = 'John'
GROUP BY name
But that doesn't allow me to fill out the entire column, just the 'John' rows.
you can do that with a common table expression:
with counted as (
select name, count(*) as name_count
from the_table
group by name
)
update the_table
set "count" = c.name_count
from counted c
where c.name = the_table.name;
Another (slower) option would be to use a co-related sub-query:
update the_table
set "count" = (select count(*)
from the_table t2
where t2.name = the_table.name);
But in general it is a bad idea to store values that can easily be calculated on the fly:
select id,
name,
count(*) over (partition by name) as name_count
from the_table;
Another method : Using a derived table
UPDATE tb
SET count = t.count
FROM (
SELECT count(NAME)
,NAME
FROM tb
GROUP BY 2
) t
WHERE t.NAME = tb.NAME
Say I have this table
id | data | value
-----------------
1 | a | A
2 | a | A
3 | a | A
4 | a | B
5 | b | C
6 | c | A
7 | c | C
8 | c | C
I want to remove those rows with duplicated value for each data while keeping the one with the min id, e.g. the result will be
id | data | value
-----------------
1 | a | A
4 | a | B
5 | b | C
6 | c | A
7 | c | C
I know a way to do it is to do a union like:
SELECT 1 [id], 'a' [data], 'A' [value] INTO #test UNION SELECT 2, 'a', 'A'
UNION SELECT 3, 'a', 'A' UNION SELECT 4, 'a', 'B'
UNION SELECT 5, 'b', 'C' UNION SELECT 6, 'c', 'A'
UNION SELECT 7, 'c', 'C' UNION SELECT 8, 'c', 'C'
SELECT * FROM #test WHERE id NOT IN (
SELECT MIN(id) FROM #test
GROUP BY [data], [value]
HAVING COUNT(1) > 1
UNION
SELECT MIN(id) FROM #test
GROUP BY [data], [value]
HAVING COUNT(1) <= 1
)
but this solution has to repeat the same group by twice (consider the real case is a massive group by with > 20 columns)
I would prefer a simpler answer with less code as oppose to complex ones. Is there any more concise way to code this?
Thank you
You can use one of the methods below:
Using WITH CTE:
WITH CTE AS
(SELECT *,RN=ROW_NUMBER() OVER(PARTITION BY data,value ORDER BY id)
FROM TableName)
DELETE FROM CTE WHERE RN>1
Explanation:
This query will select the contents of the table along with a row number RN. And then delete the records with RN >1 (which would be the duplicates).
This Fiddle shows the records which are going to be deleted using this method.
Using NOT IN:
DELETE FROM TableName
WHERE id NOT IN
(SELECT MIN(id) as id
FROM TableName
GROUP BY data,value)
Explanation:
With the given example, inner query will return ids (1,6,4,5,7). The outer query will delete records from table whose id NOT IN (1,6,4,5,7).
This fiddle shows the records which are going to be deleted using this method.
Suggestion: Use the first method since it is faster than the latter. Also, it manages to keep only one record if id field is also duplicated for the same data and value.
I want to add MYSQL solution for this query
Suggestion 1 : MySQL prior to version 8.0 doesn't support the WITH clause
Suggestion 2 : throw this error (you can't specify table TableName for update in FROM clause
So the solution will be
DELETE FROM TableName WHERE id NOT IN
(SELECT MIN(id) as id
FROM (select * from TableName) as t1
GROUP BY data,value) as t2;