Get tenant_id from a delete CDC operation in Debezium - postgresql

I am trying to develop a SpringBoot application to track CDC changes of a Postgres database.
In DebeziumConfig I have
private io.debezium.config.Configuration config() {
return io.debezium.config.Configuration.create()
.with("name","dbz-postgres-poc")
.with("connector.class", "io.debezium.connector.postgresql.PostgresConnector")
.with("offset.storage", "org.apache.kafka.connect.storage.FileOffsetBackingStore")
.with("offset.storage.file.filename", "/tmp/debezium/offsets.dat")
.with("offset.flush.interval.ms", 60000)
.with("database.hostname", "localhost")
.with("database.port", "5432")
.with("database.user", "postgres")
.with("database.password", "password")
.with("database.dbname", "postgres")
.with("database.server.name", "dbserver")
.with("database.history", "io.debezium.relational.history.FileDatabaseHistory")
.with("database.history.file.filename", "/tmp/debezium/dbhistory.dat")
.with("table.whitelist", "public.customers,public.orders")
.with("value.converter.schemas.enable", "false")
.with("key.converter", "org.apache.kafka.connect.json.JsonConverter")
.with("value.converter", "org.apache.kafka.connect.json.JsonConverter")
.with("transforms", "route, extractTenantId")
.with("transforms.route.type", "org.apache.kafka.connect.transforms.RegexRouter")
.with("transforms.route.regex", "([^.]+)\\.([^.]+)\\.([^.]+)")
.with("transforms.route.replacement", "$3.tenant")
.with("transforms.extractTenantId.type", "com.example.postgres.service.ExtractTenantId")
.build();
}
And in ExtractTenantId, I have
public class ExtractTenantId implements Transformation {
#Override
public ConnectRecord apply(ConnectRecord connectRecord) {
Struct value = (Struct) connectRecord.value();
if (value!=null) {
Struct beforeValue = (Struct) value.get("before");
Struct afterValue = (Struct) value.get("after");
Struct newValue = afterValue!=null?afterValue:beforeValue;
String tenantId = (String)newValue.get("tenant_id");
return connectRecord.newRecord(connectRecord.topic().replace("tenant",tenantId), connectRecord.kafkaPartition(), connectRecord.keySchema(), connectRecord.key(),
newValue.schema(), newValue, connectRecord.timestamp());
}
return connectRecord;
}
I can get the tenant_id out from insert and update events but not delete event, for example
insert into customers (id, tenant_id, name, address)
values (6,'tenant1','May Doe','111 main st');
SourceRecord{sourcePartition={server=dbserver}, sourceOffset={transaction_id=null, lsn_proc=24060344, lsn_commit=24060288, lsn=24060344, txId=586, ts_usec=1674863096056242}} ConnectRecord{topic='customers.tenant', kafkaPartition=null, key=Struct{id=6}, keySchema=Schema{dbserver.public.customers.Key:STRUCT}, value=Struct{after=Struct{id=6,tenant_id=tenant1,name=May Doe,address=111 main st},source=Struct{version=1.9.7.Final,connector=postgresql,name=dbserver,ts_ms=1674863096056,db=postgres,sequence=["24060288","24060344"],schema=public,table=customers,txId=586,lsn=24060344},op=c,ts_ms=1674863096145}, valueSchema=Schema{dbserver.public.customers.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}
update customers set name='May may' where id=6
SourceRecord{sourcePartition={server=dbserver}, sourceOffset={transaction_id=null, lsn_proc=24060856, lsn_commit=24060800, lsn=24060856, txId=587, ts_usec=1674863186386044}} ConnectRecord{topic='customers.tenant', kafkaPartition=null, key=Struct{id=6}, keySchema=Schema{dbserver.public.customers.Key:STRUCT}, value=Struct{after=Struct{id=6,tenant_id=tenant1,name=May may,address=111 main st},source=Struct{version=1.9.7.Final,connector=postgresql,name=dbserver,ts_ms=1674863186386,db=postgres,sequence=["24060800","24060856"],schema=public,table=customers,txId=587,lsn=24060856},op=u,ts_ms=1674863186678}, valueSchema=Schema{dbserver.public.customers.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}
delete from customers where id=6
SourceRecord{sourcePartition={server=dbserver}, sourceOffset={transaction_id=null, lsn_proc=24060176, lsn_commit=24058760, lsn=24060176, txId=585, ts_usec=1674863000249380}} ConnectRecord{topic='customers.tenant', kafkaPartition=null, key=Struct{id=6}, keySchema=Schema{dbserver.public.customers.Key:STRUCT}, value=Struct{before=Struct{id=6,tenant_id=,name=,address=},source=Struct{version=1.9.7.Final,connector=postgresql,name=dbserver,ts_ms=1674863000249,db=postgres,sequence=["24058760","24060176"],schema=public,table=customers,txId=585,lsn=24060176},op=d,ts_ms=1674863000777}, valueSchema=Schema{dbserver.public.customers.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}
Is there a way to get the tenant id of the deleted records? I don't want create records to store the delete keys with tenant_id. And here is just one example of customer table, but I will want a generic CDC applications for all tables so adding caching will be too cumbersome

Related

Creating view for update table via foregien key access in django

I am really new to django and stuck in using foreign key in django queries.Can anyone help me.
Situation:I created a sign up page with username and email field.So when a new user registered, its username and email address saved to auth_user table.
Q: I want to create a model instance in class UserDetails each time a new user register. or you can say entry of user as a foregien key in table userdetails. I am using postgresql database.
Myapp/models.py
class Pincode(models.Model):
pincode = models.CharField("PinCode",null=False)
geom = GeopositionField("Location")
objects = models.GeoManager()
def __unicode__(self):
return u'%s' %(self.pincode)
class UserDetails(models.Model):
user = models.OneToOneField(User, related_name='User_Details',unique=True)
pin= models.ForeignKey(Pincode,related_name='pin', null= True, blank=True)
rating= models.CharField(max_length=12, null=True, blank=True)
def __unicode__(self):
return u'%s, %s, %s' % (self.user, self.pin, self.rating)
views.py
def index(request):
update_user_details()
return render_to_response("larb/index.html",
RequestContext(request))
def update_user_details(request): ## **Stuck here**
user_details = UserDetails.objects.all()
new_entry= User.objects.exclude() # Don't know how to do
currently i am thinking of creating a function in the views that update the table.
first it checks for new entries means entries in auth_user which are not in userdetails table. If found update userdetails table.

I hope i solved it and it's seems fine now:
Here is views.py code
def index(request):
update_user_details()
return render_to_response("larb/index.html",
RequestContext(request))
def update_user_details():
id_past_entries = UserDetails.objects.all().values_list('user_id', flat = True)
new_entries = User.objects.exclude(id__in=id_past_entries)
if new_entries:
for new_id in new_entries:
new = User.objects.get(id=new_id)
UserDetails.objects.create(user=new)
UserDetails.save()

How to enable Seperate Audits Table in Entity Framework

I have a Entity Framework based database with a few entities/models/table. For e.g. Documents Model, I am want to track all changes to each record in that table, in a seperate table called DocumentChanges Model/Table.
Could you please guide me on how to enable/tell EF to track/audit all changes to the table in a separate table?, not just a date time stamp, but save the full record for every change in a separate table.

The library Audit.EntityFramework can help you to do what you want.
You'll need to implement your own DataProvider to store the data formatted as you wish.
For example:
void StartUp()
{
//Setup to use your own provider to store the data
Audit.Core.Configuration.Setup()
.UseCustomProvider(new YourDataProvider());
//Setup to audit EF operations only for the table Documents
//(Your DbContext must inherit from AuditDbContext)
Audit.EntityFramework.Configuration.Setup()
.ForAnyContext(x => x.IncludeEntityObjects())
.UseOptIn()
.Include<Documents>();
}
class YourDataProvider : AuditDataProvider
{
public override object InsertEvent(AuditEvent auditEvent)
{
//Get some enviroment info:
var userName = auditEvent.Environment.UserName
//Get the complete log for the EF operation:
var efEvent = auditEvent.GetEntityFrameworkEvent();
foreach(var entry in efEvent.Entries)
{
// each entry is a modified entity (updated, deleted or inserted)
if (entry.Action == "Update")
{
//You can access the column values
var value = entry.ColumnValues["ID"];
//...or the columns changes
var changes = entry.Changes.Select(ch => ch.ColumnName + ": " +
ch.OriginalValue + " -> " + ch.NewValue);
}
//... insert into DocumentChanges table
}
return id;
}
}

Test class for trigger

I just wrote this trigger and it seems to be working fine in dev, I need to move it into production, however, the test class I wrote passes the test but does not cover the trigger. Any help would be greatly appreciated. I am a bit green here. I know I should be inserting a contact (the account is a req field) then updating the contact field I just have no earthly clue how to do taht. Thank you
trigger PropOwned on Contact (after update) {
for (Contact c : Trigger.new) {
McLabs2__Ownership__c ownNew = new McLabs2__Ownership__c();
Contact oldContact = Trigger.oldMap.get(c.id);
if (c.One_Prop_Owned__c != oldContact.One_Prop_Owned__c && c.One_Prop_Owned__c != null) {
ownNew.McLabs2__Contact__c = c.id;
ownNew.McLabs2__Property__c = c.One_Prop_Owned__c;
insert ownNew;
}
}
}
This is the test class I wrote.
#isTest
public class TestOwnership {
static testMethod void createOwnership() {
McLabs2__Ownership__c ownNew = new McLabs2__Ownership__c();
ownNew.McLabs2__Contact__c = 'Michael Webb';
ownNew.McLabs2__Property__c = '131 West 33rd Street';
insert ownNew;
}
}

Your test class just creates a McLabs2__Ownership__c object and inserts this object in database. As a result of this trigger on McLabs2__Ownership__c(if exist) will be invoked, but you have to test a trigger on Contact object. Thus you need to insert an account and after that update it because your contact trigger works in after update mode.
So, you need something like that
#isTest
private class TestOwnership {
static testMethod void whenContactUpdatedNewOwnershipIsInserted() {
// create contact, you have to replace 'value1' with appropriate data type
Contact contact = new Contact(name = 'Test Contact', One_Prop_Owned__c = 'value1');
insert contact;
contact.One_Prop_Owned__c = 'value2'; // you have to replace value2 with appropriate data type
update contact;
// in this place you should has only one record of McLabs2__Ownership__c in database, because in test context real data isn't visible
List<McLabs2__Ownership__c> ownerships = [SELECT Id, McLabs2__Contact__c, McLabs2__Property__c FROM McLabs2__Ownership__c];
System.assertEquals(1, ownerships.size());
System.assertEquals(contact.Id, ownerships[0].McLabs2__Contact__c);
System.assertEquals(contact.One_Prop_Owned__c, ownerships[0].McLabs2__Property__c);
}
}
Read the following articles which might be pretty useful for you:
Apex Trigger best practice
SOQL in loop

get primary key of last inserted record with JPA

I've been using JPA to insert entities into a database but I've run up against a problem where I need to do an insert and get the primary key of the record last inserted.
Using PostgreSQL I would use an INSERT RETURNING statement which would return the record id, but with an entity manager doing all this, the only way I know is to use SELECT CURRVAL.
So the problem becomes, I have several data sources sending data into a message driven bean (usually 10-100 messages at once from each source) via OpenMQ and inside this MDB I persists this to PostgreSQL via the entity manager. It's at this point I think there will be a "race condition like" effect of having so many inserts that I won't necessarily get the last record id using SELECT CURRVAL.
My MDB persists 3 entity beans via an entity manager like below.
Any help on how to better do this much appreciated.
public void onMessage(Message msg) {
Integer agPK = 0;
Integer scanPK = 0;
Integer lookPK = 0;
Iterator iter = null;
List<Ag> agKeys = null;
List<Scan> scanKeys = null;
try {
iag = (IAgBean) (new InitialContext()).lookup(
"java:comp/env/ejb/AgBean");
TextMessage tmsg = (TextMessage) msg;
// insert this into table only if doesn't exists
Ag ag = new Ag(msg.getStringProperty("name"));
agKeys = (List) (iag.getPKs(ag));
iter = agKeys.iterator();
if (iter.hasNext()) {
agPK = ((Ag) iter.next()).getId();
}
else {
// no PK found so not in dbase, insert new
iag.addAg(ag);
agKeys = (List) (iag.getPKs(ag));
iter = agKeys.iterator();
if (iter.hasNext()) {
agPK = ((Ag) iter.next()).getId();
}
}
// insert this into table always
iscan = (IScanBean) (new InitialContext()).lookup(
"java:comp/env/ejb/ScanBean");
Scan scan = new Scan();
scan.setName(msg.getStringProperty("name"));
scan.setCode(msg.getIntProperty("code"));
iscan.addScan(scan);
scanKeys = (List) iscan.getPKs(scan);
iter = scanKeys.iterator();
if (iter.hasNext()) {
scanPK = ((Scan) iter.next()).getId();
}
// insert into this table the two primary keys above
ilook = (ILookBean) (new InitialContext()).lookup(
"java:comp/env/ejb/LookBean");
Look look = new Look();
if (agPK.intValue() != 0 && scanPK.intValue() != 0) {
look.setAgId(agPK);
look.setScanId(scanPK);
ilook.addLook(look);
}
// ...

The JPA spec requires that after persist, the entity be populated with a valid ID if an ID generation strategy is being used. You don't have to do anything.

updating table in entity framework 4 /mvc 3!

could you help with this? I bet this isn't any tough one..but am new to EF and facing a weekend deadline. I want to update a table with values.. but the primary key is identity column. So my task is like this.. if it exists, update.. if it doesn't add to the
table.. this is my code..and am stuck in this else part..!
Table structure is like this
Primary Key table - System: SystemId, SystemName
Foreign Key table - SystemConfiguration: SystemConfigurationId, SystemId, SystemRAM, SystemHard-Disk
public void SaveSystemConfigurations(SystemConfiguration systemConfig)
{
var config = (from s in Context.SystemConfiguration
where s.SystemId == systemConfig.SystemId
select s).FirstOrDefault();
if (config == null)
{
Context.SystemConfigurations.AddObject(systemConfig);
Context.SaveChanges();
}
else
{
// EntityKey systemConfigKey= new EntityKey("systemConfig", "systemConfigId", config.SystemConfigurationId);
Context.SystemConfigurations.Attach(systemConfig);
Context.SaveChanges();
}
}

Try this:
public void SaveSystemConfigurations(SystemConfiguration systemConfig)
{
var config = (from s in Context.SystemConfiguration
where s.SystemId == systemConfig.SystemId
select s).FirstOrDefault();
if (config == null)
{
Context.SystemConfigurations.AddObject(systemConfig);
}
else
{
config.Attribute = value; // Do your update here
}
Context.SaveChanges();
}
Edit. It should be config not systemConfig.

The ApplyCurrentValues method will apply scalar attributes to an entity that matches the same key. My assumption is that you are modifying a real entity (an object that has a valid entity key).
This would work:
var eSet = config.EntityKey.EntitySetName;
Context.ApplyCurrentValues(eSet, systemConfig);
Context.SaveChanges();

public void SaveSystemConfigurations(SystemConfiguration systemConfig)
{
var context = new EntitiesModel();
//here is the name of the partial class created on the Context area of the edmx designer.cs
var config = (from s in context.SystemConfiguration
where s.SystemId == systemConfig.SystemId
select s).FirstOrDefault();
context.ApplyCurrentValues(config.EntityKey.EntitySetName, systemConfig);
// systemConfig comes from the view with values set by the user
// you dont have to manually need to map one value at the time
context.SaveChanges();
}