Does R2dbc support instant updates if some proces inserts, updates, deletes data directly from SQL databases?
Or does it only support publishing of such events when data modification actions are done via R2dbc?
When I tried running queries and expose as WebFlux stream the stream was completed after successfully streaming all query rrsults
Is there a cold / hot Flux that will get either whole result set streamed on new subscription or only modified row(s) for existing / new subscriptions ?
Related
I've a mongoDB instance with a collection holding calendar events. This is fed using a Kafka application.
These events need to feed into other downstream systems, using Kafka Streams, but what I'd like to invesitgate is whether is would be possible to only trigger an event to a downstream system when the event has just happened (rather then passing future events downstream).
So if an event is received and written to mongo for a date in the future, the downstream system will only know about it as that date is reached and not before.
I've looked at the normal connectors (mongoDB -> Kafka https://www.mongodb.com/kafka-connector) and that functionaility isn't provided.
One of the ways I thought about doing this would be to write a custom application which queries the mongo DB collection on a schedule between the "last run" and "now" to get all the events which occur within these times and create a downstream event into Kafka. (setting indexes on the query elements in the mongo document).
Is there any other way?
Many thanks for reading.
Jill
Instead of query the mongodb I would suggest to create a consumer group to the original kafka topic, which the mongodb data is ingested from and do if you recognize that the date is in the future -> create a rundeck / airflow scheduled task configured to that date, so your consumer logic will be simple.
Another solution you can try is to do some changes to the source connector that you found and try to merge it.
Good luck! Im here if you have any questions
I have a question about mongodb and outbox pattern (and i'm quite newbie in mongo).
I am working on application, that uses mongodb as primary database.
I have some use cases, in which i need to save document to the database and then publish some event to message broker (RabbitMq).
Saved document must be consistent with published event - this means that if I save document i MUST also send message (solution must be resilient to for example server shutdown between document save and message send) so I decided to use outbox pattern.
In relational (sql) database this problem is trivial: I just start new transaction, then persist/change new object, then persist some kind of database scheduler record (that sends message to RabbitMq after transaction is committed), and then commit.
How do I achieve this in mongodb (once again with emphasis on no data loss and no 'phantom' message send). Should I use mongodb transactions (transactions are strongly disadvised in mongo community) or there are other, better solutions?
you can try mongodb change streams at a collection level. It is similar to triggers on the RDBMS world. You can then listen on that stream for events of your choice, in your case - an insert event, post which you can send that event to downstream systems.
I have a Google Cloud Dataflow pipeline (written with the Apache Beam SDK) that, in its normal mode of operation, handles event data published to Cloud Pub/Sub.
In order to bring the pipeline state up to date, and to create the correct outputs, there is a significant amount of historical event data which must be processed first. This historical data is available via JDBC. In testing, I am able to use the JdbcIO.Read PTransform to read and handle all historical state, but I'd like to initialize my production pipeline using this JDBC event data, and then cleanly transition to reading events from Pub/Sub. This same process may happen again in the future if the pipeline logic is ever altered in a backward incompatible way.
Note that while this historical read is happening, new events are continuing to arrive into Pub/Sub (and these end up in the database also), so there should be a clean cutover from only historical events read from JDBC, and only newer events read from Pub/Sub.
Some approaches I have considered:
Have a pipeline that reads from both inputs, but filters data from JDBC before a certain timestamp, and from pub/sub after a certain timestamp. Once the pipeline is caught up deploy an update removing the JDBC input.
I don't think this will work because removal of an I/O transform is not backward compatible. Alternately, the JDBC part of the pipeline must stay there forever, burning CPU cycles for no good reason.
Write a one-time job that populates pub/sub with the entirety of the historical data, and then starts the main pipeline reading only from pub/sub.
This seems to use more pub/sub resources than necessary, AND I think newer data interleaved in the pipeline with much older data will cause watermarks to be advanced too early.
Variation of option #2 -- stop creating new events until the historical data is handled, to avoid messing up watermarks.
This requires downtime.
It seems like it would be a common requirement to backfill historical data into a pipeline, but I haven't been able to find a good approach to this.
Your first option, reading from a Bounded source (filtered to timestamp <= cutoff) and PubSub (filtered to timestamp > cutoff) should work well.
Because JDBC.Read() is a bounded source, it will be read all the data and then "finish" i.e. never produce any more data, advance its watermark to +infinity, and not be invoked again (so there's no concern about it consuming CPU cycles, even if it's present in your graph).
I have a spark job which will be periodically submitted to do some job for every hour, so now I need to store that job complete status to some database whenever some action(for example filtering and writing is complete) is completed .
What is best way to get job progress(from spark stage) and store its progress, completion or error if any ?
I thought of using Hbase or some other no SQL but for this simple information Hbase or other DB will be overhead, also looks like there is no support of SQLite with spark, so What is the best way to store this information
I need below information to be put it db
Current JOB running, its progress, its in/output paths etc
Job completion , Pending Failure status etc
We are using Debezium + PostgreSQL.
Notice that we get 4 types of events for create, read, update and delete - c, r, u and d.
The read type of event is unused for our application. Actually, I could not think of an use case for the 'r' events unless we are doing auditing or mirroring the activities of a transaction.
We are facing difficulties scaling & I suspect it is because of network getting hogged by read type of events.
How do we filter out those events in postgreSQL itself?
I got a clue from one of the contributors to use snapshot.mode. I guess something that has to be done when Debezium creates a snapshot. I am unable to figure out how to do that.
It is likely that your database has existed for some time and contains data and changes that have been purged from the logical decoding logs. If you then start using the Debezium PostgreSQL connector to start capturing changes into Kafka, the question becomes what a consumer of the events in Kafka should be able to see.
One scenario is that a consumer should be able to see events for all rows in the database, even those that existed prior to the start of CDC. For example, this allows a consumer to completely reproduce/replicate all of the existing data and keep that data in sync over time. To accomplish this, the Debezium PostgreSQL connector starts up can begin by creating a snapshot of the database contents before it starts capturing the changes. This is done atomically, so that even if the snapshot process takes a while to run, the connector will still see all of the events that occurred since the snapshot process was started. These events are represented as "read" events, since in effect the connector is simply reading the existing rows. However, they are identical to "insert" events, so any application could treat reads and inserts in the same way.
On the other hand, if consumers of the events in Kafka do not need to see events for all existing rows, then the connector can be configured to avoid the snapshot and to instead begin by capturing the changes. This may be useful in some scenarios where the entire database state need not be found in Kafka, but instead the goal is to simply capture the changes that are occurring.
The Debezium PostgreSQL connector will work either way, so you should use the approach that works for how you're consuming the events.