My replication suddenly stopped, data are not replicated to the subscriber. I don't really know the cause may be because of ID conflict.
I try a couple of things, but none work :
ALTER SUBSCRIPTION test_mysub REFRESH publication;
ALTER SUBSCRIPTION test_mysub ENABLE;
(Note: ALTER SUBSCRIPTION test_mysub REFRESH publication works when there are columns difference between publisher and subscriber.)
The subscriber has 2000 less records than the publisher. Seems that there are some WALs not sent to the subscriber. Then I deleted all records in the subscriber, and automatically that 2000 records are inserted and the replication works again.
Addendum:
I know what causing the problem : ID conflict.
Here is what I do to replicate the problem :
Insert a record with an ID 1000 on the Subscriber
Insert a record with an ID 1000 on the Publisher
Check the Publisher : select * from pg_stat_replication;. The result is NULL. Its is an indication that the subscription stopped.
Insert a record with an ID 1001 on the Publisher
After that I check the Subscriber, ID 1001 does not present.
How I recover :
Delete the record with ID 1000 on the Subscriber
Check the Publisher : select * from pg_stat_replication;. There is a record, with State column = streaming.
Check the Subscriber. There are 2 new record : ID 1000 and 1001, both from publisher.
It seems that the replication works again if I remove the causing record. After that, the replication works again automatically without having to be restarted.
Questions :
How can I entry the records that are not sent (the 2000 records) from the publisher without deleting the duplicating ID existing on the subscriber ?
I am wondering If I can see the records on the failed WAL files and be able to delete the causing ID in the WAL. Is it possible ? Or is there any better solution ?
Thanks
Related
I'm Beginner of Debezium concept,
Just Before i created Maven project for debezium integration. i created 2 java classes one is Configuration part and another one is OPERATIONS Part like CREATE,DELETE,UPDATE.
Step 1--: My Configuration part i added few my database related properties like host name port, dbname etc like as uploaded image(1.1).
enter image description here
So i'm inserting one record that (id is 1000) to my table debezium read that record. I take that records inserting some like .. ABC table.
Now i'm Down application(debezium Shut Down) then i'm inserting one Records previously not inserted but after adding snapshot.mode="never" its work fine data inserted (id is 1001).
Here is my issue---> : data inserted along with previous records like when (debezium Shut Down) then restart 1001 and 1000 id also inserted into ABC Table next time Again Down my application (debezium Shut Down) now i'm not inserting any record then Restart my application then again 1001 and 1000 id(s) insert every time with out any operation 2 records inserted.
I Don't want previous related records just i need how many records inserted in (debezium down time that much records i need)
Could you please tell what type of properties i need to add for Resolving this issue.
I want to track the states of subscriptions between a contact and a list. The states are...
pending
subscribed
resubscribed (a subset of subscribed)
unsubscribed
I also want to know when each step happened. Rather than using a single field I'm using three timestamps.
created_at
subscribed_at
unsubscribed_at
The states correspond to:
pending: subscribed_at is null
subscribed: subscribed_at > coalesce(unsubscribed_at, '-infinity')
resubscribed: ... and unsubscribed_at is not null
unsubscribed: coalesce(unsubscribed_at, '-infinity') > subscribed_at
The major operations are...
Find pending subscriptions for a contact and list.
Find active subscriptions for a contact, a list, or both.
Find unsubscriptions for a contact, a list, or both.
Delete old pending subscriptions.
Delete old unsubscriptions.
This table can potentially get quite large. We have in the order of 10,000,000 contacts and 100 lists. A contact has about 10 active subscriptions. A list can have in the order of 100,000 active subscriptions. Old pending and unsubscriptions will be purged periodically.
I'm anticipating most queries will use subscribed_at and unsubscribed_at which will complicate indexing. I'd like to partition the table into pending, subscribed, and unsubscribed, but I can't figure out how.
This is using PostgreSQL 12.
It's possible to get all the new or update records from one table in postgresql by
specified date?
something like this:
Select NEW OR UPDATED FROM anyTable WHERE dt_insert or dt_update = '2015-01-01'
tks
You can only do this if you added a trigger-maintained field that keeps track of the last change time.
There is no internal row timestamp in PostgreSQL, so in the absence of a trigger-maintained timestamp for the row, there's no way to find rows changed/added after a certain time.
PostgreSQL does have internal information on the transaction ID that wrote a row, stored in the xmin hidden column. There's no record of what transaction ID committed when, though, until PostgreSQL 9.5 which keeps track of this if and only if the new track_commit_timestamps setting is turned on. Additionally, PostgreSQL eventually clears the creator transaction ID information from a tuple because it re-uses transaction IDs, so it only works for fairly recent transactions.
In other words: it's kind-of possible in a rough way, if you understand the innards of the database, but should really only be used for forensic and recovery purposes.
I am using EF code first. Recently I had to replace the following code:
User user = userRepository.GetByEmail("some#email.com");
if (user == null)
{
user = New User { Email = email, CreatedAt = DateTime.Now };
userRepository.Add(user);
unitOfWork.Commit();
}
with
Context.ExecuteSqlCommand("IF NOT EXISTS(SELECT 1 FROM Users WHERE Email = '{0}')
INSERT INTO Users(Email, CreatedAt)
VALUES ('email', GETDATE())");
The reason behind this is that it took EF a very long time to run the first piece of code when trying to add thousands of rows. By changing it to a ExecuteSqlCommand, the time to handle that many rows decreased by a multitude.
The problem I am seeing now (only occurred twice so far) is the following message from the database: Transaction (Process ID 52) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
How would I go about resolving this? Most of my data access is done through EF with a few exceptions like the the one above. I have never seen a deadlock in my logs before so I assume this ha something to do with the query.
My questions are:
Is there a way to write the query using No LOCK? How would that
query look?
Is there a way to tell EF to use NO LOCK for certain queries?
What you actually want is to lock the table earlier, not to prevent locking. Locking is necessary to ensure that between the two statements in your command, some other process hasn't come along and inserted the same user. (Locking is always necessary when inserting data because the physical storage is being modified.)
Assuming that this is actually the command that is causing deadlock, the following should resolve it, because it only asks for an exclusive lock :
Context.ExecuteSqlCommand("IF NOT EXISTS(SELECT 1 FROM Users WHERE Email = '{0}')
INSERT INTO Users(Email, CreatedAt)
VALUES ('email', GETDATE())");
We have a table called Contracts. These contract records are created by users on an external site and must be approved or rejected by staff on an internal site. When a contract is rejected, it's simply deleted from the db. When it's accepted, however, a new record is generated called Contract Acceptance which is written to its own table and is derived from data that exists on the contract.
The problem is that two internal staff members may each end up opening the same contract. The first user accepts and a contract acceptance record is generated. Then, with the same contract record still open on the page, the second user accepts the contract again, creating a duplicate acceptance record.
The quick and dirty way to get past this is to retrieve the contract from the db just before it's accepted, check the status, and produce an error message saying that it's already been accepted. This would probably work for most circumstances, but the users could still click the Accept button at the exact same time and sneak by this validation code.
I've also considered a thread lock deep in the data layer that prevents two threads from entering the same region of code at the same time, but the app exists on two load-balanced servers, so the users could be on separate servers which would render this approach useless.
The only method I can think of would have to exist at the database. Conceptually, I would like to somehow lock the stored procedure or table so that it can't be updated twice at the same time, but perhaps I don't understand Oracle enough here. How do updates work? Are update requests somehow queued up so that they do not occur at the exact same time? If this is so, I could check the status of the record in th SQL and return a value in an out parameter stating it has already been accepted. But if update requests aren't queued then two people could still get into the update sql at the exact same time.
Looking for good suggestions on how to go about this.
First, if there can only be one Contract Acceptance per Contract, then Contract Acceptance should have the Contract ID as its own primary (or unique) key: that will make duplicates impossible.
Second, to prevent the second user from trying to accept the contract while the first user is accepting it, you can make the acceptance process lock the Contract row:
select ...
from Contract
where contract_id = :the_contract
for update nowait;
insert into Contract_Acceptance ...
The second user's attempt to accept will then fail with an exception :
ORA-00054: resource busy and acquire with nowait specified
In general, there are two approaches to the problem
Option 1: Pessimistic Locking
In this scenario, you're pessimistic so you lock the row in the table when you select it. When a user queries the Contracts table, they'd do something like
SELECT *
FROM contracts
WHERE contract_id = <<some contract ID>>
FOR UPDATE NOWAIT;
Whoever selects the record first will lock it. Whoever selects the record second will get an ORA-00054 error that the application will then catch and let them know that another user has already locked the record. When the first user completes their work, they issue their INSERT into the Contract_Acceptance table and commit their transaction. This releases the lock on the row in the Contracts table.
Option 2: Optimistic Locking
In this scenario, you're being optimistic that the two users won't conflict so you don't lock the record initially. Instead, you select the data you need along with a Last_Updated_Timestamp column that you add to the table if it doesn't already exist. Something like
SELECT <<list of columns>>, Last_Updated_Timestamp
FROM Contracts
WHERE contract_id = <<some contract ID>>
When a user accepts the contract, before doing the INSERT into Contract_Acceptance, they issue an UPDATE on Contracts
UPDATE Contracts
SET last_updated_timestamp = systimestamp
WHERE contract_id = <<some contract ID>>
AND last_update_timestamp = <<timestamp from the initial SELECT>>;
The first person to do this update will succeed (the statement will update 1 row). The second person to do this will update 0 rows. The application detects the fact that the update didn't modify any rows and tells the second user that someone else has already processed the row.
In Either Case
In either case, you probably want to add a UNIQUE constraint to the Contract_Acceptance table. This will ensure that there is only one row in the Contract_Acceptance table for any given Contract_ID.
ALTER TABLE Contract_Acceptance
ADD CONSTRAINT unique_contract_id UNIQUE (Contract_ID)
This is a second line of defense that should never be needed but protects you in case the application doesn't implement its logic correctly.