We are facing org.springframework.dao.DataIntegrityViolationException while trying to insert the data into “BATCH_JOB_INSTANCE” table using spring batch2.0.
This issue does not occurs frequently on our system .It occurs randomly in our test environment. We are using SQL server database server.
Below error message is writing to logs :-
[PreparedStatementCallback; SQL [INSERT into BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION) values (?, ?, ?, ?)]; Violation of PRIMARY KEY constraint 'PK__BATCH_JO__4848154AB1CB6AD0'. Cannot insert duplicate key in object 'dbo.BATCH_JOB_INSTANCE'. The duplicate key value is (402).; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Violation of PRIMARY KEY constraint 'PK__BATCH_JO__4848154AB1CB6AD0'. Cannot insert duplicate key in object 'dbo.BATCH_JOB_INSTANCE'. The duplicate key value is (402).; caused by Violation of PRIMARY KEY constraint 'PK__BATCH_JO__4848154AB1CB6AD0'. Cannot insert duplicate key in object 'dbo.BATCH_JOB_INSTANCE'. The duplicate key value is (402).] and cannot be retried automatically. The job must be manually recovered
org.springframework.dao.DataIntegrityViolationException: PreparedStatementCallback; SQL [INSERT into BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION) values (?, ?, ?, ?)]; Violation of PRIMARY KEY constraint 'PK__BATCH_JO__4848154AB1CB6AD0'. Cannot insert duplicate key in object 'dbo.BATCH_JOB_INSTANCE'. The duplicate key value is (402).; nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Violation of PRIMARY KEY constraint 'PK__BATCH_JO__4848154AB1CB6AD0'. Cannot insert duplicate key in object 'dbo.BATCH_JOB_INSTANCE'. The duplicate key value is (402).
at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:228)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:607)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:792)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:850)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:854)
at org.springframework.batch.core.repository.dao.JdbcJobInstanceDao.createJobInstance(
We have also implemented the solution which is mention in below link.
https://github.com/spring-projects/spring-batch/issues/1232?jql=project%2520%253D%2520BATCH%2520AND%2520text%2520~%2520%2522key%2522
We thought it to be a database issue and asked team to recreate the database. Issue is still appearing after database recreate as well.
We found some details from the spring jira and found that you have worked on similar issues earlier. If you could provide us any insight we will be thankful.
We have done some changes mentioned earlier and none is helping us. If you could provide us with some pointers that will be helpful.
Related
[Issue resolved. See Answer below.]
I have just encountered a series of “duplicate key value violates unique constraint” errors with a system that has been working well for months. And I cannot determine why they occurred.
Here is the error:
org.springframework.dao.DuplicateKeyException: PreparedStatementCallback;
SQL [
INSERT INTO transaction_item
(transaction_group_id, transaction_type, start_time, end_time) VALUES
(?, ?::transaction_type_enum, ?, ?)
];
ERROR: duplicate key value violates unique constraint "transaction_item_pkey"
Detail: Key (transaction_id)=(67109) already exists.;
Here is the definition of the relevant SEQUENCE and TABLE:
CREATE SEQUENCE transaction_id_seq AS bigint;
CREATE TABLE transaction_item (
transaction_id bigint PRIMARY KEY DEFAULT NEXTVAL('transaction_id_seq'),
transaction_group_id bigint NOT NULL,
transaction_type transaction_type_enum NOT NULL,
start_time timestamp NOT NULL,
end_time timestamp NOT NULL
);
And here is the only SQL statement used for inserting to that table:
INSERT INTO transaction_item
(transaction_group_id, transaction_type, start_time, end_time) VALUES
(:transaction_group_id, :transaction_type::transaction_type_enum, :start_time, :end_time)
As you can see, I’m not explicitly trying to set the value of transaction_id. I've defined a default value for the column definition, and using that to fetch a value formthe SEQUENCE.
I have been under the impression that the above approach is safe, even for use in high-concurrency situation. A SEQUENCE should never return the same value twice, right?
I’d really appreciate some help to understand why this has occurred, and how to fix it. Thank you!
I found the cause of this issue.
A few months ago (during development of this system) an issue was discovered that meant it was necessary to purge any existing test data from the database. I did this using DELETE FROM statements for all TABLES and ALTER ... RESTART statements for all SEQUENCES. These statements were added to the Liquibase configuration to be executing during startup for the new code. From inspecting the logs at the time, it appears that an instance of the system was still running at the time of the migration. And this happened: The new instance of the system deleted all data from the TRANSACTION_ITEM table, the still-running instance then added more data to that table, and then the new instance restarted the SEQUENCE use for inserting those records. So yesterday, when I received the duplicate key violations, it was because the SEQUENCE finally reached the ID values corresponding to the TRANSACTION_ITEM records that were added by still-running instance back when DB purge and migration occurred.
Long story, but it all makes sense now. Thanks to those who commented on this issue.
Was solved
We have 2 schemas in the db: public and umler.
After that, the dba team has generated the spring batch schema to both schemas above.
And the foreign key was set:
umler.job_inst_exec_fk (job_instance_id reference to public.batch_job_instance(job_instance_id) ) =)))
----------------Problem-------------------
I was using spring batch in springboot,in which I had a postgres datasource for the job. But when I ran the job, I've got the error:
Detail: Key (job_instance_id)=(72) is not present in table "batch_job_instance".; nested exception is org.postgresql.util.PSQLException: ERROR: insert or update on table "batch_job_execution" violates foreign key constraint "job_inst_exec_fk"
Detail: Key (job_instance_id)=(72) is not present in table "batch_job_instance".] with root cause
org.postgresql.util.PSQLException: ERROR: insert or update on table "batch_job_execution" violates foreign key constraint "job_inst_exec_fk"
Detail: Key (job_instance_id)=(72) is not present in table "batch_job_instance".
The step that I can see are:
Find the job by name
if not existing, will create a new record into job_instance (with id = 72)
Inserting new record into the job_execution. At this step we would meet the error.
Was solved
We have 2 schemas in the db: public and umler.
After that, the dba team has generated the spring batch schema to both schemas above.
And the foreign key was set:
umler.job_inst_exec_fk (job_instance_id reference to public.batch_job_instance(job_instance_id) ) =)))
I imported all tables from MySQL to PostgreSQL but now I have problems with id's.
The way I converted my MySQL DB was simple exported DB and copied all "INSERTS" with edited syntax, import was successful because I can see all
the data correct.
SQLSTATE[23505]: Unique violation: 7 ERROR: duplicate key value violates unique constraint "elements_pkey"
DETAIL: Key (id)=(1) already exists.
Is there any way to fix issues with id's?
It works after reset the sequence.
SELECT setval('my_sequence_name', (SELECT max(id) FROM my_table));
In postgres I have created a table by name twitter_tweets. In this table I have assigned constraint for tweet_text column by using the command
ALTER TABLE ONLY twitter_tweets
ADD CONSTRAINT twitter_tweets_pkey PRIMARY KEY (tweet_text);
The constraint has applied by getting message i.e., alter table
but while parsing the data it showing runtime exception i.e.,
java.lang.RuntimeException: Failed to execute insert query insert into twitter_tweets (tweet_created_at, tweet_id, tweet_id_str, tweet_text, tweet_source, tweet_truncated, tweet_in_reply_to_status_id, tweet_in_reply_to_status_id_str, tweet_in_reply_to_user_id, tweet_in_reply_to_user_id_str, tweet_in_reply_to_screen_name, tweet_geo,tweet_coordinates, tweet_at_reply, tweet_is_quote_status, tweet_retweet_count, tweet_favorite_count, tweet_favorited, tweet_retweeted, tweet_lang, tweet_possibly_sensitive, tweet_filter_level, tweet_scopes_S)values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) at Demo.JdbcClient.executeInsertQuery(JdbcClient.java:62) at Demo.PsqlBolt.execute(PsqlBolt.java:91) at backtype.storm.daemon.executor$fn__5694$tuple_action_fn__5696.invoke(executor.clj:690) at backtype.storm.daemon.executor$mk_task_receiver$fn__5615.invoke(executor.clj:436) at backtype.storm.disruptor$clojure_handler$reify__5189.onEvent(disruptor.clj:58) at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132) at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106) at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) at backtype.storm.daemon.executor$fn__5694$fn__5707$fn__5758.invoke(executor.clj:819) at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "twitter_tweets_pkey" Detail: Key (tweet_text)=() already exists. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892) at com.zaxxer.hikari.proxy.StatementProxy.executeBatch(StatementProxy.java:116) at com.zaxxer.hikari.proxy.PreparedStatementJavassistProxy.executeBatch(PreparedStatementJavassistProxy.java) at Demo.JdbcClient.executeInsertQuery(JdbcClient.java:50) ... 11 more
The below image1 is the table to which i have used constraint
This is my output after keeping constraints
Your problem is described here:
ERROR: duplicate key value violates unique constraint "twitter_tweets_pkey" Detail: Key (tweet_text)=() already exists. at
You set tweet_text to be your PRIMARY KEY (PK), and as PK it cant get duplicated data.
At some point you already insert the data that you are trying to insert now into this column (tweet_text).
Now, why not create an Integer column, AUTO INCREMENTED, something like ID? The way as it now, you are telling me that no one should post a same text that was posted by other user.
Ex. If User A post a tweet with content (tweet_text) : "Hello World", no other user can post the same content.
Unique Constraint Violation
You asked for a primary key. A primary key in Postgres automatically creates an index and a UNIQUE constraint.
Then you inserted rows of data. At least two of those rows had the same value in that primary key field. The duplicate data violated the UNIQUE constraint. Postgres then did its duty in refusing to store the offending data. That refusal is reported back to you, the Java programmer, as an Exception.
At least that is my guess based on this excerpt from the middle of your error text:
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "twitter_tweets_pkey" Detail: Key (tweet_text)=() already exists.
I have a table with two columns with foreign key constraints, for example:
CREATE TABLE example
(
id integer PRIMARY KEY,
f1 integer REFERENCES example(id),
f2 integer REFERENCES example(id)
);
If I then perform the insert:
insert into example (id, f1, f2) values (1, 2, 2);
I will obviously get an error, but only for the first failed constraint:
ERROR: insert or update on table "example" violates foreign key constraint "example_f1_fkey"
DETAIL: Key (f1)=(2) is not present in table "example".
My question is: Is it possible to configure postgres so it returns an error with both of the failed key constraints?
Thanks very much,
Ben
Is it possible to configure postgres so it returns an error with both of the failed key constraints?
No, it isn't. The first FK failure aborts the transaction so no further checks are run.
It would be interesting to be able to capture all violations but there's no way to do that in current versions (true in 9.3, at least).
To do it you'd need to be able to selectively change ERROR level reports for CHECK constraints, foreign key constraint checks, etc into WARNINGs that also set a flag that'd force the transaction to abort at the end of the current statement. That might not be too hard to do technically, but it's certainly going to involve a chunk of work on the PostgreSQL source code.