I am trying to work out the best schema structure to represent a BoM in Postgres. Assuming a part can have multiple of the same child part, I could add a quantity column, but those parts may also have multiple children.
If I wanted to know the total usage of each part does postgres have a way of using the quantity column in a hierarchical query?
BOM means Bill Of Material.
As far as I understand your question, then yes, you can include the quantity when using a hierarchical BOM. The way I understand your question is, that if one BOM entry has an amount of e.g. 10, then the amount for its children needs to be multiplied with 10 (because you have 10 times that "child" item).
With the following table and sample data:
create table bom_entry
(
entry_id integer primary key,
product text, -- should be a foreign key to the product table
amount integer not null,
parent_id integer references bom_entry
);
insert into bom_entry
values
(1, 'Box', 1, null),
(2, 'Screw', 10, 1),
(3, 'Nut', 2, 2),
(4, 'Shim', 2, 2),
(5, 'Lock', 2, 1),
(6, 'Key', 2, 5);
So our box needs 10 screws and every screw needs 2 nuts and 2 shims, so we need a total of 20 nuts and 20 shims. We also have two locks and each lock has two keys, so we have a total of 4 keys.
You can use a recursive CTE to go through the tree and calculate the amount for each item.
with recursive bom as (
select *, amount as amt, 1 as level
from bom_entry
where parent_id is null
union all
select c.*, p.amt * c.amount as amt, p.level + 1
from bom_entry c
join bom p on c.parent_id = p.entry_id
)
select rpad(' ', (level - 1)*2, ' ')||product as product, amount as entry_amount, amt as total_amount
from bom
order by entry_id;
The rpad/level is used to do the indention to visualize the hierarchy. The above query returns the following:
product | entry_amount | total_amount
---------+--------------+-------------
Box | 1 | 1
Screw | 10 | 10
Nut | 2 | 20
Shim | 2 | 20
Lock | 2 | 2
Key | 2 | 4
Related
We have a third party BI tool on the project that can only add a where clause with specified filters on a table/view select. We use a set of 4 source tables, they have indexes for columns that can be filtered using BI's UI. We have a view for each table that do grouping by indexed columns and add 1 additional column. Then we have another view that joins all the data from those 4 views using index columns, that is a view that is queried from our BI's UI, BI adds where clause to queries.
The problem is indexes on source tables are not utilized, filters are not pushed down on the level of tables, instead they are applied at the very end. We can't use Set Returning Function, all our BI tool can do is just select from table\view and add a where clause.
We thought about intercepting a select's where condition in Pg but I'm not sure is it possible. Or maybe it's possible to hint a optimizer that filters need to be pushed down. We can query source tables directly without using views but it will multiply a number of data sources\elements on UI which is not desirable. Is there any other ways we can solve it in PostgreSQL?
Update 1
Bellow examples of schemas/queries we use for our tables and views
CREATE TABLE source_table_1
(
dim1 VARCHAR(255) NOT NULL,
dim2 VARCHAR(255) NOT NULL,
dim3 VARCHAR(255) NOT NULL,
meausre1 Bigint NOT NULL,
meausre2 Bigint NOT NULL,
meausre3 Bigint NOT NULL
);
CREATE INDEX ON uc13_failures_by_cell (dim1, dim2, dim3);
... another 3 tables
CREATE OR REPLACE VIEW view1 AS
SELECT
"type1" as type,
dim1,
dim2,
dim3,
sum(meausre1) AS meausre1,
sum(meausre2) AS meausre2,
sum(meausre3) AS meausre3
FROM source_table_1
GROUP BY 1, 2, 3, 4;
... another 3 views
CREATE OR REPLACE VIEW view_uinion AS
SELECT
coalesce(view1.dim1, view2.dim1, view3.dim1, view4.dim1) AS dim1,
... two other dims
view1.meausre1 AS meausre1_1,
view2.meausre1 AS meausre2_1,
view3.meausre1 AS meausre3_1,
view4.meausre1 AS meausre4_1,
... two meausres
FROM view1
FULL JOIN view2 ON
view1.dim1 = view2.dim1 AND
view1.dim2 = view2.dim2 AND
view1.dim3 = view2.dim3 AND
FULL JOIN view3 ON ...
FULL JOIN view4 ON ...
WHERE -- this is were filters on dims are inserted
;
You cannot push a WHERE condition into a full outer join.
See this example:
CREATE TABLE a(id integer NOT NULL, a1 integer NOT NULL);
INSERT INTO a VALUES (1, 20), (2, 20);
CREATE TABLE b(id integer NOT NULL, b1 integer NOT NULL);
INSERT INTO b VALUES (2, 30), (3, 30);
SELECT *
FROM a
FULL JOIN b USING (id)
WHERE b1 = 30;
id | a1 | b1
----+----+----
2 | 20 | 30
3 | | 30
(2 rows)
SELECT *
FROM a
FULL JOIN (SELECT *
FROM b
WHERE b1 = 30) AS b_red
USING (id);
id | a1 | b1
----+----+----
1 | 20 |
2 | 20 | 30
3 | | 30
(3 rows)
So you would have to modify the underlying queries/views.
If you used inner joins, it would not be a problem.
I got a table with linestrings that I want to divide into chunks that have a list of id not higher than provided number for each and store only lines that are within certain distance.
For example, I got a table with 14 rows
create table lines ( id integer primary key, geom geometry(linestring) );
insert into lines (id, geom) values ( 1, 'LINESTRING(0 0, 0 1)');
insert into lines (id, geom) values ( 2, 'LINESTRING(0 1, 1 1)');
insert into lines (id, geom) values ( 3, 'LINESTRING(1 1, 1 2)');
insert into lines (id, geom) values ( 4, 'LINESTRING(1 2, 2 2)');
insert into lines (id, geom) values ( 11, 'LINESTRING(2 2, 2 3)');
insert into lines (id, geom) values ( 12, 'LINESTRING(2 3, 3 3)');
insert into lines (id, geom) values ( 13, 'LINESTRING(3 3, 3 4)');
insert into lines (id, geom) values ( 14, 'LINESTRING(3 4, 4 4)');
create index lines_gix on lines using gist(geom);
I want to split it into chunks with 3 ids for each chunk with lines that are within 2 meters from each other or the first one.
The result I am trying to get from this example is:
| Chunk No.| Id chunk list |
|----------|----------------|
| 1 | 1, 2, 3 |
| 2 | 4, 5, 6 |
| 3 | 7, 8, 9 |
| 4 | 10, 11, 12 |
| 5 | 13, 14 |
I tried to use st_clusterwithin but when lines are close to each other it will return all of them not split into chunks.
I also tried to use some with recursive magic like the one from the answer provided by Paul Ramsey here. But I don't know how to modify the query to return limited grouped id list.
I am not sure if it is the best possible answer so if anyone has a better method or know how to improve provided answer feel free to update it. With a little modification of Paul answer, I've managed to create following queries that are doing what I asked for.
-- Create function for easier interaction
CREATE OR REPLACE FUNCTION find_connected(integer, double precision, integer, integer[])
returns integer[] AS
$$
WITH RECURSIVE lines_r AS -- Recursive allow to use the same query on the output - is like continues append to result and use it inside a query
(SELECT ARRAY[id] AS idlist,
geom, id
FROM lines
WHERE id = $1
UNION ALL
SELECT array_append(lines_r.idlist, lines.id) AS idlist, -- append id list to array
lines.geom AS geom, -- keep geometry
lines.id AS id -- keep source table id
FROM (SELECT * FROM lines WHERE NOT $4 #> array[id]) lines, lines_r -- from source table and recursive table
WHERE ST_DWITHIN(lines.geom, lines_r.geom, $2) -- where lines are within 2 meters
AND NOT lines_r.idlist #> ARRAY[lines.id] -- recursive id list array not contain lines array
AND array_length(idlist, 1) <= $3
)
SELECT idlist
FROM lines_r WHERE array_length(idlist, 1) <= $3 ORDER BY array_length(idlist, 1) DESC LIMIT 1;
$$
LANGUAGE 'sql';
-- Create id chunks
WITH RECURSIVE groups_r AS (
(SELECT find_connected(id, 2, 3, ARRAY[id]) AS idlist, find_connected(id, 2, 3, ARRAY[id]) AS grouplist, id
FROM lines WHERE id = 1)
UNION ALL
(SELECT array_cat(groups_r.idlist, find_connected(lines.id, 2, 3, groups_r.idlist)) AS idlist,
find_connected(lines.id, 2, 3, groups_r.idlist) AS grouplist,
lines.id
FROM lines,
groups_r
WHERE NOT groups_r.idlist #> ARRAY[lines.id]
LIMIT 1))
SELECT
-- (SELECT array_agg(DISTINCT x) FROM unnest(idlist) t (x)) idlist, -- left for better understanding what is happening
row_number() OVER () chunk_id,
(SELECT array_agg(DISTINCT x) FROM unnest(grouplist) t (x)) grouplist,
id input_line_id
FROM groups_r;
The only problem is that performance is quite pure when the number of ids in the chunk increase. For a table with 300 rows and 20 ids per chunk, execution time is around 15 min, even with indexes on geometry and id columns.
Suppose I have a table with two columns id and val. I wan't to find all the distinct ids where there exist a pair of equal and opposite vals. For example suppose you have the following table
id | val
------+------
1 | 3
2 | 5
2 | -5
1 | 4
2 | 6
3 | 9
2 | -6
3 | -9
I want the result to be
result
2
3
2 in the result set because there are values 5, -5 and 6, -6. 3 is in the result set because of 9, -9.
I can do this by using where exists. Something like
select distinct tab1.id from tab tab1
where exists (
select * from tab tab2
where tab1.id = tab2.id
and tab1.val = -tab2.val
);
However I worry that a query like this has time complexity O(n^2) because it is computed like nested loops (?). However it is possible to compute this in O(n) time by scanning the table (and keeping track of previously seen results in a data structure with O(1) lookup time). What is the optimal way to write such a query?
We should have an explain of your request and how you sets indexes.
May be it could be done like this too :
WITH pos AS (
SELECT id, val FROM tab WHERE val > 0),
neg AS (
SELECT id, val FROM tab WHERE val < 0)
SELECT DISTINCT id
FROM pos JOIN neg USING (id)
WHERE pos.val = neg.val;
With right indexation, this could be quick. Depend also of the volume of data.
I have a table like this:
id product amount
1 A 6
1 A 8
1 A
1 B 1
1 B
2 C 2
2 C
2 C 4
2 C
2 C
and I need to make it like this:
id product amount
1 A 6
1 A 8
1 A 8
1 B 1
1 B 1
2 C 2
2 C 2
2 C 4
2 C 4
2 C 4
Copy amount by previous non-missing value.
I tried to use lag() function. however, aggregation function lag() is not allowed in UPDATE.
update tableA set amount = lag(amount);
What can I do using PostgreSQL?
You can SELECT what you want to UPDATE, but there is no (easy) way to actually do the UPDATE, because the table fox does not have a primary key (yet).
CREATE TABLE fox (
id integer NOT NULL,
product text NOT NULL,
amount integer
);
To populate the fox with some data.
INSERT INTO fox VALUES
(1, 'A', 6),
(1, 'A', 8),
(1, 'A', NULL),
(1, 'B', 1),
(1, 'B', NULL),
(2, 'C', 2),
(2, 'C', NULL),
(2, 'C', 4),
(2, 'C', NULL),
(2, 'C', NULL),
(3, 'What does the fox say?', 5);
The query.
WITH ranks (rank, id, product, amount) AS (
SELECT ROW_NUMBER() OVER (), id, product, amount FROM foo
)
SELECT r.id, r.product,
(SELECT amount FROM ranks
WHERE id = r.id AND product = r.product
AND rank < r.rank AND amount IS NOT NULL
ORDER BY amount DESC LIMIT 1
)
FROM ranks r WHERE r.amount IS NULL ORDER BY 1, 2, 3;
Yields the rows which previously had a NULL and now have the appropriate amount.
id | product | amount
----+---------+--------
1 | A | 8
1 | B | 1
2 | C | 2
2 | C | 4
2 | C | 4
But you cannot use this data to update, because rows are still not uniquely identified by (id, product) - which means you cannot write a WHERE condition identifying your rows uniquely. How would the WHERE clause know whether to change the amount to 2 or 4 in the UPDATE? The multiple rows with (id, product) = (2, 'C') are indistinguishable in the WHERE of the UPDATE.
Let's give the fox a primary key.
ALTER TABLE fox ADD COLUMN IF NOT EXISTS pkey serial ;
ALTER TABLE fox ADD PRIMARY KEY (pkey) ;
Now we can identify the rows by the PRIMARY KEY pkey.
WITH nulls AS (
SELECT pkey, id, product
FROM fox
WHERE amount IS NULL
)
SELECT pkey,
id, product, -- you can leave these out in your UPDATE: pkey is UNIQUE
(SELECT amount FROM fox
WHERE id = n.id AND product = n.product
AND n.pkey > pkey AND amount IS NOT NULL
ORDER BY pkey DESC LIMIT 1)
FROM nulls n ORDER BY 1, 2, 3, 4;
to display the changes to be made
pkey | id | product | amount
------+----+---------+--------
3 | 1 | A | 8
5 | 1 | B | 1
7 | 2 | C | 2
9 | 2 | C | 4
10 | 2 | C | 4
And we can use pkey in the UPDATE.
BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE ;
WITH nulls AS (
SELECT pkey, id, product
FROM fox
WHERE amount IS NULL
), changes AS (
SELECT pkey,
(SELECT amount FROM fox
WHERE id = n.id AND product = n.product
AND n.pkey > pkey AND amount IS NOT NULL
ORDER BY pkey DESC LIMIT 1)
FROM nulls n
) UPDATE fox f SET amount = c.amount FROM changes c WHERE f.pkey = c.pkey ;
Check the result is okay:
SELECT * FROM fox ORDER BY 1, 2, 3, 4;
And accept using COMMIT or ROLLBACK accordingly.
Alternative to adding a PRIMARY KEY
Every table should always have a primary key.
If you insist not to have one, then you could also compute the rows with their then-not-NULL amount and instead of UPDATEing them, you could INSERT them into your table and then DELETE FROM fox WHERE amount IS NULL remove the rows which had no amount. This way you get around adding a primary key, which is unique. Of course the UPDATE and DELETE are packaged into a TRANSACTION such as not to interfere with other Transactions running concurrently. For example another Transaction adding rows with NULL amount AFTER you have calculated the data to be INSERTed using SELECT and before you DELETE all NULL amounts. You'd miss the concurrently added row with NULL amount in this case (data loss due to concurrency; think ACID).
But a missing primary key will probably bite you later on, anyway.
Without knowing what defines "previous rows" all is a guess. But you can use a anonymous block to do what your want, just make your changes:
CREATE TEMPORARY TABLE test_lag AS
SELECT column1 AS id, column2 AS product, column3 AS amount FROM (
VALUES (1, 'A', 6),
(1, 'A', 8),
(1, 'A', NULL),
(1, 'B', 1),
(1, 'B', NULL),
(2, 'C', 2),
(2, 'C', NULL),
(2, 'C', 4),
(2, 'C', NULL),
(2, 'C', NULL)) AS tmp;
DO $$
BEGIN
--Loop until update all null amounts
--Why we need this? It's because PostgreSQL don't supports IGNORE NULLS clause on lag()
LOOP
WITH tmp AS (
SELECT ctid, lag(amount) OVER() AS last_amount FROM test_lag ORDER BY id, product -- You MUST change this ORDER to right columns (What's previous row?)
)
UPDATE test_lag SET amount = tmp.last_amount FROM tmp WHERE test_lag.ctid = tmp.ctid AND amount IS NULL;
IF NOT FOUND THEN
EXIT;
END IF;
END LOOP;
END $$;
SELECT * FROM test_lag ORDER BY id, product, amount;
I've got three tables: users, courses, and grades, the latter of which joins users and courses with some metadata like the user's score for the course. I've created a SQLFiddle, though the site doesn't appear to be working at the moment. The schema looks like this:
CREATE TABLE users(
id INT,
name VARCHAR,
PRIMARY KEY (ID)
);
INSERT INTO users VALUES
(1, 'Beth'),
(2, 'Alice'),
(3, 'Charles'),
(4, 'Dave');
CREATE TABLE courses(
id INT,
title VARCHAR,
PRIMARY KEY (ID)
);
INSERT INTO courses VALUES
(1, 'Biology'),
(2, 'Algebra'),
(3, 'Chemistry'),
(4, 'Data Science');
CREATE TABLE grades(
id INT,
user_id INT,
course_id INT,
score INT,
PRIMARY KEY (ID)
);
INSERT INTO grades VALUES
(1, 2, 2, 89),
(2, 2, 1, 92),
(3, 1, 1, 93),
(4, 1, 3, 88);
I'd like to know how (if possible) to construct a query which specifies some users.id values (1, 2, 3) and courses.id values (1, 2, 3) and returns those users' grades.score values for those courses
| name | Algebra | Biology | Chemistry |
|---------|---------|---------|-----------|
| Alice | 89 | 92 | |
| Beth | | 93 | 88 |
| Charles | | | |
In my application logic, I'll be receiving an array of user_ids and course_ids, so the query needs to select those users and courses dynamically by primary key. (The actual data set contains millions of users and tens of thousands of courses—the examples above are just a sample to work with.)
Ideally, the query would:
use the course titles as dynamic attributes/column headers for the users' score data
sort the row and column headers alphabetically
include empty/NULL cells if the user-course pair has no grades relationship
I suspect I may need some combination of JOINs and Postgresql's crosstab, but I can't quite wrap my head around it.
Update: learning that the terminology for this is "dynamic pivot", I found this SO answer which appears to be trying to solve a related problem in Postgres with crosstab()
I think a simple pivot query should work here, since you only have 4 courses in your data set to pivot.
SELECT t1.name,
MAX(CASE WHEN t3.title = 'Biology' THEN t2.score ELSE NULL END) AS Biology,
MAX(CASE WHEN t3.title = 'Algebra' THEN t2.score ELSE NULL END) AS Algebra,
MAX(CASE WHEN t3.title = 'Chemistry' THEN t2.score ELSE NULL END) AS Chemistry,
MAX(CASE WHEN t3.title = 'Data Science' THEN t2.score ELSE NULL END) AS Data_Science
FROM users t1
LEFT JOIN grades t2
ON t1.id = t2.user_id
LEFT JOIN courses t3
ON t2.course_id = t3.id
GROUP BY t1.name
Follow the link below for a running demo. I used MySQL because, as you have noticed, SQLFiddle seems to be perpetually busted the other databases.
SQLFiddle