I'm using sqlalchemy 1.3.0 with postgres 11. I'm trying to use an INSERT with ON CONFLICT DO UPDATE ... RETURNING * in order to create an instance of my model.
class Model(Base):
__tablename__ = 'mytable'
pk = Column(String(64), primary_key=True)
col2 = Column(String(64))
#classmethod
def upsert(cls, pk, col2, session):
return session.query(cls).from_statement(
text(
"""
INSERT INTO {} (pk, col2) VALUES (:pk, :col2)
ON CONFLICT (pk) DO UPDATE SET col2=EXCLUDED.col2 RETURNING *;
""".format(cls.__tablename__)
)
).params(pk=pk, col2=col2).one()
obj1 = Model.upsert(1, 'one', session)
obj2 = Model.upsert(1, 'two', session)
print(obj2.col2) ----> outputs "one"
session.commit()
print(obj2.col2) ----> outputs "two"
The second upsert does issue the correct command to the database, but printing the col2 attribute of the object returned shows the value of the column that was inserted during the first upsert. Then, if I do a session.commit(), the object magically gets updated to show the new value. What am I missing? I want the object returned from the function to reflect the values that the row was updated to, without having to do a commit, because I want this along with several other things to happen within a transaction.
It turns out that the session.commit() was essentially expiring the object so that the following print(obj2.col2) would requery the database to populate its attributes.
In order to have obj2.col2 be the correct value without committing (and requerying the database), I just needed to do session.expunge(obj1) before the second upsert call to create obj2.
Related
I need to do the sequential steps:
to lock a table or specific rows (with a condition) for writing;
to check some rows with specific conditions;
if there are no rows with the conditions, add a new row
unlock
How can I do that in SQLAlchemy, Flask, PostgreSQL? What ways are correct?
I'm using Flask-Sqlalchemy, so try to make adjustments.
You can use with_for_update() here. docs
An example would be.
class TestTable(db.Model):
__tablename__ = "test_table"
id = Column(db.Integer(), primary_key=True)
filename = Column(db.String(1024))
is_complete = Column(db.Boolean(), default=False, server_default="f")
You can lock them by doing:
for table in TestTable.query.with_for_update().filter_by(is_complete=False).all():
# Example Check condition
if table.filename != 'bar' and table.id >= 10:
# Add a new row
newtable = TestTable(filename='foo')
db.session.add(newtable)
# Save and release lock
db.session.commit()
If you want that to be safe from race conditions, you need to use a transaction with SERIALIZABLE isolation level. Perform the whole operation in that transaction.
I am trying to single insert data into table with assigned PK. Manually assiging PK.
XML file
<insert id = "insertStd" parameterType = "com.org.springboot.dao.StudentEntity" useGeneratedKeys = "false" keyProperty = "insertStd.id", keyColumn = "id">
INSERT INTO STUDENT (ID, NAME, BRANCH, PERCENTAGE, PHONE, EMAIL )
VALUES (ID=#{insertStd.id}, NAME=#{insertStd.name}, BRANCH=#{insertStd.branch}, PERCENTAGE=#{insertStd.percentage}, PHONE=#{insertStd.phone}, EMAIL =#{insertStd.email});
</insert>
Service call method
public boolean saveStudent(Student student){
LOGGER.info("Student object save");
int savedId= studentMapper.insertStd(student);
}
Log file
org.springframework.jdbc.badsqlgrammarexception
### Error updating database Causes: cause org.postgresql.util.psqlexception error column id does not exist
HINT: There is a column named "id" in the table "student" but it can't be referenced from this part of the query.
Position 200
### Error may exist in file [c:\.....\StudentMapper.xml]
### Error may involve in com.org.springboot.dao.StudentMapper.insertStd-InLine
### The error occurred while setting parameters
### SQL INSERT INTO STUDENT (ID, NAME, BRANCH, PERCENTAGE, PHONE, EMAIL )
VALUES (ID=?, NAME=?,BRANCH=?, PERCENTAGE=?, PHONE=?, EMAIL=?);
### cause org.postgresql.util.psqlexception ERROR column "id" doesn't exist. //It did worked with JPA id assigned manually.
### There is a column named "ID" in the table "STUDENT", Bbut it cannot be referenced from the part of the query.
The INSERT statement of malformed. The VALUES clause should not include the column names.
Also, since there's no primary auto-generation, you can remove all the other attributes. Just leave the mapper id.
Note: if you want to manually assign the PK value, you need to make sure the table does not have a GENERATED ALWAYS clause for the column. If this is the case, the table will ignore the value you are providing and will use its own rules to generate the PK.
Use:
<insert id="insertStd">
INSERT INTO STUDENT (ID, NAME, BRANCH, PERCENTAGE, PHONE, EMAIL)
VALUES (
#{insertStd.id}, #{insertStd.name}, #{insertStd.branch},
#{insertStd.percentage}, #{insertStd.phone}, #{insertStd.email}
);
</insert>
Your error is easily reproduceable:
create table t (a int, b varchar(10));
insert into t (a, b) values (123, 'ABC'); -- succeeds
insert into t (a, b) values (a=123, b='ABC'); -- fails!
error: column "a" does not exist
See the Fiddle.
I have a few rows stored in a source table (as defined as $schema.$sourceTable in the UPDATE query below). This table has 3 columns: TABLE_NAME, PERMISSION_TAG_COL, PT_DEPLOYED
I have an update statement stored in a string like:
var update_PT_Deploy = s"UPDATE $schema.$sourceTable SET PT_DEPLOYED = 'Y' WHERE TABLE_NAME = '$tableName';"
My source table does have rows with TABLE_NAME as $tableName (parameter) as I inserted rows into this table using another function of my program. The default value of PT_DEPLOYED when I inserted the rows was specified as NULL.
I'm trying to execute update using JDBC in the following manner:
println(update_PT_Deploy)
val preparedStatement: PreparedStatement = connection.prepareStatement(update_PT_Deploy)
val row = preparedStatement.execute()
println(row)
println("row updated in table successfully")
preparedStatement.close()
The above piece of code does not throw any exception, but when I query my table in a tool like DBeaver, the NULL value of PT_DEPLOYED does not get updated to Y.
If I execute the same query as mentioned in update_PT_Deploy inside DBeaver, the query works and the table updates. I am sure I am following the correct steps..
I have the following table:
CREATE TABLE scoped_data(
owner_id text,
scope text
key text,
data json,
PRIMARY KEY (owner_id, scope, key)
);
As part of each transaction we will potentially be inserting data for multiple scopes. Given this table has the potential to grow very quickly I would like not to store data if it is NULL or an empty JSON object.
An upsert felt like the idiomatic approach to this. The following is within the context of a PL/pgSQL function:
WITH upserts AS (
INSERT INTO scoped_data (owner_id, scope, key, data)
VALUES
(p_owner_id, 'broad', p_broad_key, p_broad_data),
(p_owner_id, 'narrow', p_narrow_key, p_narrow_data),
-- etc.
ON CONFLICT (owner_id, scope, key)
DO UPDATE SET data = scoped_data.data || COALESCE(EXCLUDED.data, '{}')
RETURNING scope, data
)
SELECT json_object_agg(u.scope, u.data)
FROM upserts u
INTO v_all_scoped_data;
I include the RETURNING as I would like the up-to-date version of each scope's data included in a variable for subsequent use, therefore I need the RETURNING to return something even if logically no data has been updated.
For example (all for key = 1 and scope = 'narrow'):
data = '{}' => v_scoped_data = {}, no data for key = 1 in scoped_data.
data = '{"some":"data"}' => v_scoped_data = { "narrow": { "some": "data" } }, data present in scoped_data.
data = '{}' => v_scoped_data = { "narrow": { "some": "data" }, data from 2. remains unaffected.
data = '{"more":"stuff"}' => v_scoped_data = { "narrow": { "some": "data", "more": "stuff" }. Updated data stored in table.
I initially added a trigger BEFORE INSERT ON scoped_data which did the following:
IF NULLIF(NEW.data, '{}') IS NULL THEN
RETURN NULL;
END IF;
RETURN NEW;
This worked fine for preventing the insertion of new records but the issue was that this trigger also prevented subsequent inserts to existing rows thereby no INSERT happened therefore there was no ON CONFLICT therefore nothing returned in the RETURNING.
A couple of approaches I've considered, both of which feel inelegant or like they should be unnecessary:
Add a CHECK constraint to scoped_data.data: CHECK(NULLIF(data, '{}') IS NOT NULL), allow the insert and catch the exception in the PL/pgSQL code.
DELETE in an AFTER INSERT trigger if the data field was NULL or empty.
Am I going about this in the right way? Am I trying to coerce this logic into an upsert when there is a better way? Might explicit INSERTs and UPDATEs be a more logical fit?
I am using Postgres 9.6.
I would go with the BEFORE trigger ON INSERT to prevent unnecessary inserts and updates.
To return the values even in the case that the operation is not performed, you can UNION ALL your query with a query on scoped_data that returns the original row, ORDER the results so that any new row is ordered first (introduce an artifical column to both queries) and use LIMIT 1 to get the correct result.
update Claim
set first_name = random_name(7),
Last_name = random_name(6),
t2.first_name=random_name(7),
t2.last_name=random_name(6)
from Claim t1
inner join tbl_ecpremit t2
on t1.first_name = t2.first_name
I am getting below error
column "t2" of relation "claim" does not exist
You can do this with a so-called data-modifying CTE:
WITH c AS (
UPDATE claim SET first_name = random_name(7), last_name = random_name(6)
WHERE <put your condition here>
RETURNING *
)
UPDATE tbl_ecpremit SET last_name = c.last_name
FROM c
WHERE first_name = c.first_name;
This assumes that random_name() is a function you define, it is not part of PostgreSQL afaik.
The nifty trick here is that the UPDATE in the WITH query returns the updated record in the first table using the RETURNING clause. You can then use that record in the main UPDATE statement to have exactly the same data in the second table.
This is all very precarious though, because you are both linking on and modifying the "first_name" column with some random value. In real life this will only work well if you have some more logic regarding the names and conditions.