I set a cascaded crossSectionEngine and set the triggeringPattern to ‘perBatch’. Does the data in the following engine refresh after I insert one batch data into the first engine?
Yes, the parameter perBatch is to update the engine once a batch of data is injected.
For detailed parameter introduction, please refer to the introduction of createcrosssectionengine function
https://www.dolphindb.com/help/FunctionsandCommands/FunctionReferences/c/createCrossSectionalEngine.html
Related
In my project I am reading the data for DB table using StoredProcedure reader and calling an API to process and then saving the output using writer. I need to maintain the processing status as Processed or Error for each record that I am reading. As of now I am using the writer to update the input table column STATUS to P (Processed) or E(Error) and add logs the in case of any error to LOGS column.
Can you please suggest if this is the efficient way to maintain the processing status of each record. Does Spring batch provides any default implementation for same?
Thanks
No, Spring Batch does not provide a "default implementation" for such a requirement.
That said, a flag on each item as you did is a reasonable way to address your requirement in my opinion.
I have a materialized view that is updated from many streams. Every one enrich it partially. Order doesn't matter. Updates comes in not specified time. Is following algorithm is a good approach:
Update comes and I check what is stored in materialized view via get(), that this is an initial one so enrich and save.
Second comes and get() shows that partial update exist - add next information
... and I continue with same style
If there is a query/join, object that is stored has a method that shows that the update is not complete isValid() that could be used in KafkaStreams#filter().
Could you share please is this a good plan? Is there any pattern in Kafka streams world that handle this case?
Please advice.
Your plan looks good , you have the general idea, but you'll have to use the lower Kafka Stream API : Processor API.
There is a .transform operator that allow you to access a KeyValueStatestore, inside this operation implementation you are free to decide if you current aggregated value is valid or not.
Therefore send it downstream or returning null waiting for more information.
Hi so i understand firestore write triggers run out of order with respect to time. Is is possible to get timestamp information on when a write occured within the trigger functions execution context?
If you're using the Firebase CLI to deploy, every background function is delivered an EventContext object as its second parameter. You can use its timestamp property. Or, you can have the client write it into the document.
I assume something similar is available for the context object provided to code deployed by gcloud.
After working for awhile with the C driver , reading the tutorials and the API .
I little confused ,
According to this tutorial : http://api.mongodb.org/c/current/executing-command.html
i can execute DB and Collections commands which include also the CRUD commands.
And i can even get the Document cursor if i don't use "_simple" in the command API
so why do i need to use for example the mongoc_collection_insert() API command ?
What are the differences ? what is recommended ?
Thanks
This question is probably similar to what's the difference between using insert command or db.collection.insert() via the mongo shell.
mongoc_collection_insert() is specific function written to insert a document into a collection while mongoc_collection_command() is for executing any valid database commands on a collection.
I would recommend to use the API function (mongoc_collection_insert) whenever possible. For the following reasons:
The API functions had been written as an abstraction layer with a specific purpose so that you don't have to deal with other details related to the command.
For example, mongoc_collection_insert exposes the right parameters for inserting i.e. mongoc_write_concern_t and mongoc_insert_flags_t with the respective default value. On the other hand, mongoc_collection_command has broad range of parameters such as mongoc_read_prefs_t, skip, or limit which may not be relevant for inserting a document.
Any future changes for mongoc_collection_insert will more likely be considered with the correct context for insert.
Especially for CRUD, try to avoid using command because the MongoDB wire protocol specifies different request opcodes for command (OP_MSG: 1000) and insert (OP_INSERT: 2002).
I have following requirement.I am generating unique id from ItemProcessor and writing the same to database using JdbcItemWriter.
I wanted to pass this unique id as a query param in next JdbcItemReader,so that i can select all the records from database based on this unique id.
currently i am using max(uniqueid) from database.I have tried using {jobParameters['unqueid']} but it didn't worked.
Please let me know how to pass value from ItemProcessor to DataBaseItemReader.
I think using step execution context might work for you here. There is the option for setting some transient data on the step execution context and having that be available to other components in the same step.
There is a previous answer here that elaborates a bit more on this and a quick google search on "spring batch step execution context" also provides quite a few q/a on the subject.