I need to implement a cassandra trigger in java that will take a different action for INSERT vs UPDATE operations.
I've seen how it is possible to identify a DELETE operation in the question Cassandra sample trigger to get the deleted row and column values but I can't see any methods in the ColumnFamily object that would allow allow the code to differentiate between INSERT vs UPDATE, is there a way to achieve this?
There isn't conceptually any difference between INSERT and UPDATE. Indeed an INSERT and an UPDATE are just mutation. Cassandra gives them different name so that people coming from the relational DB have familiar concepts
As #doanduyhai mentioned and according to Casssandra sources only INSERT operation updates row timestamp (LivenessInfo).
For other operations this timestamp is not updated and equals Long.MIN_VALUE.
Custom Trigger which checks for INSERT operation
public class CustomTrigger implements ITrigger {
#Override
public Collection<Mutation> augment(Partition partition) {
checkQueryType(partition));
return Collections.emptyList();
}
private void checkQueryType(Partition partition) {
UnfilteredRowIterator it = partition.unfilteredIterator();
while (it.hasNext()) {
Unfiltered unfiltered = it.next();
Row row = (Row) unfiltered;
if (isInsert(row)) {
// Implement insert related logic
}
}
}
private boolean isInsert(Row row) {
return row.primaryKeyLivenessInfo().timestamp() != Long.MIN_VALUE;
}
}
Related
This is the mysql table.
create table Customer
(
id int auto_increment primary key,
birth date null,
createdTime time null,
updateTime datetime(6) null
);
This my java code
#Before
public void init() {
this.entityManagerFactory=Persistence.createEntityManagerFactory("jpaLearn");
this.entityManager=this.entityManagerFactory.createEntityManager();
this.entityTransaction=this.entityManager.getTransaction();
this.entityTransaction.begin();
}
#Test
public void persistentTest() {
this.entityManager.setFlushMode(FlushModeType.COMMIT); //don't work.
for (int i = 0; i < 1000; i++) {
Customer customer = new Customer();
customer.setBirth(new Date());
customer.setCreatedTime(new Date());
customer.setUpdateTime(new Date());
this.entityManager.persist(customer);
}
}
#After
public void destroy(){
this.entityTransaction.commit();
this.entityManager.close();
this.entityManagerFactory.close();
}
When I reading the wikibooks of JPA, it said "This means that when you call persist, merge, or remove the database DML INSERT, UPDATE, DELETE is not executed, until commit, or until a flush is triggered."
But at same time my code runing, I read the mysql log, I find each time the persist execution, mysql will execute the sql. And I also read the wireShark, each time will cause the request to Database.
I remember jpa saveAll method can send SQL statements to the database in batches? If I wanna to insert 10000 records, how to improve the efficiency?
My answer below supposes that you use Hibernate as jpa implementation. Hibernate doesn't enable batching by default. This means that it'll send a separate SQL statement for each insert/update operation.
You should set hibernate.jdbc.batch_size property to a number bigger than 0.
It is better to set this property in your persistence.xml file where you have your jpa configuration but since you have not posted it in the question, below it is set directly on the EntityManagerFactory.
#Before
public void init() {
Properties properties = new Properties();
properties.put("hibernate.jdbc.batch_size", "5");
this.entityManagerFactory = Persistence.createEntityManagerFactory("jpaLearn", properties);
this.entityManager = this.entityManagerFactory.createEntityManager();
this.entityTransaction = this.entityManager.getTransaction();
this.entityTransaction.begin();
}
Then by observing your logs you should see that the Customer records are persisted in the database in batches of 5.
For further reading please check: https://www.baeldung.com/jpa-hibernate-batch-insert-update
you should enable batch for hibernate
spring.jpa.properties.hibernate.order_inserts=true
spring.jpa.properties.hibernate.order_updates=true
spring.jpa.properties.hibernate.jdbc.batch_size=20
and use
reWriteBatchedInserts&reWriteBatchedStatements=true
end of your connectionString
I am trying my hands on spring-data-r2dbc. I am try this on Postgresql. I have tried spring-data-mongodb-reactive before. I couldn't help but to compare both.
I see that Query Derivation is not yet supported. But I was wondering if there is an equivalent for #Tailable. This way I would be notified of the database changes in real time. Ca anyone share any code samples with respect to this.
I understand that the underlying database should support this. I believe Postgresql does support this kinda thing using Logical Decoding(Correct me if I am wrong here).
Is there a #Tailable equivalent in spring-data-r2dbc ?
I was on the same issue not sure if you found a solution or not but I was able to accomplish something similar by doing the following. First, I added trigger to my table
CREATE TRIGGER trigger_name
AFTER INSERT OR DELETE OR UPDATE
ON table_name
FOR EACH ROW
EXECUTE PROCEDURE trigger_function_name;
This will set a trigger on the table whenever a row, is updated, deleted, or inserted. Then it will call the trigger function I have set up which looked something like this:
CREATE FUNCTION trigger_function_name
RETURNS trigger
LANGUAGE 'plpgsql'
COST 100
VOLATILE NOT LEAKPROOF
AS
$BODY$
DECLARE
payload JSON;
BEGIN
payload = row_to_json(NEW);
PERFORM pg_notify('notification_name', payload::text);
RETURN NULL;
END;
$BODY$;
This will allow me to 'listen' to the any of these updates from my spring boot project and it will send the entire row as a payload.
Next, in my spring boot project I configured a connection to my db.
#Configuration
#EnableR2dbcRepositories("com.(point to wherever repository is)")
public class R2DBCConfig extends AbstractR2dbcConfiguration {
#Override
#Bean
public ConnectionFactory connectionFactory() {
return new PostgresqlConnectionFactory(PostgresqlConnectionConfiguration.builder()
.host("host")
.database("db")
.port(port)
.username("username")
.password("password")
.schema("schema")
.connectTimeout(Duration.ofMinutes(2))
.build());
}
}
With that I Autowire (dependency injection) it into the constructor in my service class and cast it to a r2dbc PostgressqlConnection class like so:
this.postgresqlConnection = Mono.from(connectionFactory.create()).cast(PostgresqlConnection.class).block();
Now we want to 'listen' to our table and get the notified when perform some update to our table. To do that we set up an initialization method that is performed after dependency injection by using the #PostContruct annotation
#PostConstruct
private void postConstruct() {
postgresqlConnection.createStatement("LISTEN notification_name").execute()
.flatMap(PostgresqlResult::getRowsUpdated).subscribe();
}
Notice that we listen to whatever name we put inside the pg_notify method. Also we want to set up a method to close the the connection when the bean is about to be tossed away, like so:
#PreDestroy
private void preDestroy() {
postgresqlConnection.close().subscribe();
}
Now I simply create a method that returns a Flux of whatever is currently in my table, and I also merge it with my notifications, as I said before the notifications come in as a json, so I had to deserialize it and I decided to use ObjectMapper. So, it will look something like this:
private Flux<YourClass> getUpdatedRows() {
return postgresqlConnection.getNotifications().map(notification -> {
try {
//deserialize json
return objectMapper.readValue(notification.getParameter(), YourClass.class);
} catch (IOException e) {
//handle exception
}
});
}
public Flux<YourClass> getDocuments() {
return documentRepository.findAll().share().concatWith(getUpdatedRows());
}
Hope this helps.
Cheers!
To audit an entity conditionally, there is the option of using the Integrators and extends Envers event liteners.
But can we audit a field or a property conditionally ?
In our case , we have a blob column, and to avoid the increase of the volume of the audit tables , we want to set the value of this column only when a condition is valid. Is there any way to do it ?
Thanks
As indicated, the current and only way to do Conditional Auditing is to extend the Envers listeners as you described, register your custom ones via an Integrator and make sure to configure Envers not skip its event listener registration step.
As an example, lets assume you have extended EnversPostUpdateEventListenerImpl:
public class CustomPostUpdateEventListener extends EnversPostUpdateEventListenerImpl {
#Override
public void onPostUpdate(PostUpdateEvent event) {
if ( event.getEntity() instanceof YourCustomEntityType ) {
if ( !isSpecialConditionSet( event ) ) {
return;
}
}
super.onPostUpdate( event );
}
private boolean isSpecialConditionSet(PostUpdateEvent event) {
final Object conditionValue = event.getPersister()
.getPropertyValue( event.getEntity(), "nameOfPropertyCondition" );
/* check your condition and return true if you should not audit the entity */
}
}
One thing to consider when doing Conditional Auditing is when a PostInsertEvent for an entity is fired, it should not be considered conditional if you use the ValidityAuditStrategy to avoid downstream issues when you update that entity in a future transaction.
I am trying to set the default value SQL for a computed column called 'Duration' in a table 'dbo.Table1', in code-first Entity Framework 6 migration through SqlServerMigrationSqlGenerator class.
I tried setting this in Generate methods for AddColumnOperation as well as for CreateTableOperation. While the code under Generate method for column never fires, but the code under Generate table fires and throws an error saying the following. (the column EndTime is a column in table dbo.Table1 and so is StartTime)
The name "EndTime" is not permitted in this context. Valid expressions
are constants, constant expressions, and (in some contexts) variables.
Column names are not permitted.
Question: How could I make this work in either of the Generate methods in code below?
internal class CustomImplForSqlServerMigration: SqlServerMigrationSqlGenerator {
protected override void Generate(AlterColumnOperation alterColumnOperation) {
base.Generate(alterColumnOperation);
}
protected override void Generate(AddColumnOperation addColumnOperation) {
if (addColumnOperation.Table == "dbo.Table1" && addColumnOperation.Column.Name == "Duration") {
addColumnOperation.Column.DefaultValueSql = "(CAST(CAST(EndTime AS DATETIME) - CAST(StartTime AS DATETIME) AS TIME))";
}
base.Generate(addColumnOperation);
}
protected override void Generate(CreateTableOperation createTableOperation) {
if (createTableOperation.Name == "dbo.Table1") {
foreach(ColumnModel cm in createTableOperation.Columns) {
if (cm.Name == "Duration") {
cm.DefaultValueSql = "(CAST(CAST(EndTime AS DATETIME) - CAST(StartTime AS DATETIME) AS TIME))";
}
}
}
base.Generate(createTableOperation);
}
}
UPDATE 1:
I used another simple approach to add my custom logic for modifying database objects using ExecuteSqlCommand. Just follow the steps below to use this in your situation.
come up with the custom script for modifying or creating a database object
execute a command in Seed method for every custom script
make sure that the ExecuteSqlCommand statement is placed at end of Seed method and also context.SaveChanges( ) method is called before the code for custom scripts in case there is a dependency on seed data
protected override void Seed(EfCodeFirst.ShiftsDb context)
{
//Write your seed data statements
//call SaveChanges in case your custom script depends
//on some seed data
context.SaveChanges();
//include your custom scripts like ALTER TABLE
//or CREATE PROCEDURE or anything else
//use a ExecuteSqlCommand for every custom script
context.Database.ExecuteSqlCommand(#"ALTER TABLE ShiftTypes DROP COLUMN Duration;
ALTER TABLE TABLE1 ADD Duration AS (CAST(CAST(EndTime AS DATETIME) -
CAST(StartTime AS DATETIME) AS TIME)); ");
}
This is not a limitation of EF but of the database itself - you cannot refer to other columns within Default value specification. What I would recommend you instead is to write stored procedure for inserting new Table1 entities and then map with fluent api according to article.
I am reading from source table (using JpaPagingItemReader) and passing to ItemProcessor.
My requirement is if Item is processed successfully then it should write to TABLE_A and if processing failed then write to TABLE_B.
I got it working, but I dont feel it as nice way.
My current implementation is
// my processor
public class MyItemProcessor implements ItemProcessor<SourceEntity, BaseOutputEntity>{
#Override
public BaseOutputEntity process(SourceEntity input) {
// NOTE: EntityA, EntityB both extend BaseOutputEntity
try {
EntityA a = callMyBusiness.method(input);
return a;
} catch (MyBusinessException e) {
EntityB b = createMyFailureObj(input)
return b;
}
}
}
// my itemwriter
public class MyItemWriter extends JpaItemWriter<MyBaseOutputEntity> {
// donthing as JpaItemWriter methods will take care
}
It is doing functionally what exactly I want.
One drawback of above is when I see job execution / step execution history, I can't know how many are successful or how many are failure, as it shows e.g. if 100 reads then 100 writes.
Can anyone suggest better approach. Are conditional steps useful here?
You can throw an exception on your processor, and declare this Exception as Skipable (if not, chunck will be broken).
If you implements an ItemProcessListener you can catch the invalid item on the onProcessError(Entry item, Exception t) function and write it on the table B.
(Read the documentation carefully: Some listeners functions are on transactions, others not)
At the end of the batch, writedItemsCount is the number of valids item, skippedItemCount is the number of invalid items.
Other way to write in different tables is use the ClassifierCompositeItemWriter with the BackToBackPatternClassifier but you loose the count of invalid items.