I have created a custom Item Reader just like
public class AggregateItemReader<T> implements ItemReader<List<T>> {
private static final Log LOG = LogFactory.getLog(AggregateItemReader.class);
private ItemReader<AggregateItem<T>> itemReader;
}
When I call the reader from step, I pass the chunk size but that chunk size is not being passed to ItemReader inside AggregateItemReader and it reads all the records mentioned in the query. Could you please let me know how to pass chunk size to ItemReader inside another Reader
An item reader is not aware of the chunk size. It is the step that calls the reader and decides when the chunk is complete. So in your case, neither the AggregateItemReader nor the delegate can get this info, it is up to you to pass this info at configuration time to both of these components.
Related
I'm trying to batch the records constantly emitted from a streaming source (Kafka) and call my service in a batch of 100.
What I get as the input is a single record. I'm trying what's the best way to achieve it in the Reactive way using Spring Reactor without having to have a mutation and locking outside the pipeline.
Here is my naive attempt which simply reflects my sequential way of thinking:
Mono.just(input)
.subscribe(i -> {
batches.add(input);
if(batches.size() >= 100) {
// Invoke another reactive pipeline.
// Clear the batch (requires locking in order to be thread safe).
}
});
What's the best way to achieve batching on a streaming source using reactor.
.buffer(100) or bufferTimeout(100, Duration.ofSeconds(xxx) comes to the rescue
Using Flux.buffer or Flux.bufferTimeout you will be capable of gathering the fixed amount of elements into the List
StepVerifier.create(
Flux.range(0, 1000)
.buffer(100)
)
.expectNextCount(10)
.expectComplete()
.verify()
Update for the use case
In case, when the input is a single value, suppose like an invocation of the method with parameter:
public void invokeMe(String element);
You may adopt UnicastProcessor technique and transfer all data to that processor so then it will take care of batching
class Batcher {
final UnicastProcessor processor = UnicastProcessor.create();
public void invokeMe(String element) {
processor.sink().next(element);
// or Mono.just(element).subscribe(processor);
}
public Flux<List<String>> listen() {
return processor.bufferTimeout(100, Duration.ofSeconds(5));
}
}
Batcher batcher = new Batcher();
StepVerifier.create(
batcher.listen()
)
.then(() -> Flux.range(0, 1000)
.subscribe(i -> batcher.invokeMe("" + i)))
.expectNextCount(10)
.thenCancel()
.verify()
From that example, we might learn how to provide a single point of receiving events and then listen to results of the batching process.
Please note that UnicastPorcessor allows only one subscriber, so it will be useful for the model when there is one interested party in batching results and many data producers. In a case when you have subscribers as many as producers you may want to use one of the next processors -> DirectProcessor, TopicProcessor, WorkerQueueProcessor. To learn more about Reactor Processors follow the link
I am writing a small CEP program using Siddhi. I can add a callback whenever a given filter outputs a data like this
executionPlanRuntime.addCallback("query1", new QueryCallback() {
#Override
public void receive(long timeStamp, Event[] inEvents, Event[] removeEvents) {
EventPrinter.print(inEvents);
System.out.println("data received after processing");
}
});
but is there is a way to know that the filter has finished processing and it won't give any more of the above callback. Something like didFinish. I think that would be the ideal place for shutting down SiddhiManager and ExecutionPlanRuntime instances.
No. There in no such functionality and can't be supported in the future also. Rationale behind that is, in real time stream processing queries will process the incoming stream and emit an output stream. There is no concept as 'finished processing'. Query will rather process event as long as there is input.
Since your requirement is to shutdown SiddhiManager and ExecutionPlanRuntime, recommended way is to do this inside some cleaning method of your program. Or else you can write some java code inside callback to count responses or time wait and call shutdown. Hope this helps!!
I'm using ProtoBuf-Net for serialize and deserialize TCP_Messages.
I've tried all the suggestions I've found here, so I really don't know where the mistake is.
The serialize is made server side, and the deserialize is made on an application client-side.
Serialize code:
public void MssGetCardPersonalInfo(out RCPersonalInfoRecord ssPersonalInfoObject, out bool ssResult) {
ssPersonalInfoObject = new RCPersonalInfoRecord(null);
TCP_Message msg = new TCP_Message(MessageTypes.GetCardPersonalInfo);
MemoryStream ms = new MemoryStream();
ProtoBuf.Serializer.Serialize(ms, msg);
_tcp_Client.Send(ms.ToArray());
_waitToReadCard.Start();
_stopWaitHandle.WaitOne();
And the deserialize:
private void tpcServer_OnDataReceived(Object sender, byte[] data, TCPServer.StateObject clientState)
{
TCP_Message message = new TCP_Message();
MemoryStream ms = new MemoryStream(data);
try
{
//ms.ToArray();
//ms.GetBuffer();
//ms.Position = 0;
ms.Seek(0, SeekOrigin.Begin);
message = Serializer.Deserialize<TCP_Message>(ms);
} catch (Exception ex)
{
EventLog.WriteEntry(_logSource, "Error deserializing: " + ex.Message, EventLogEntryType.Error, 103);
}
As you can see, I've tried a bunch of different approache, now comented.
I have also tried to deserialize using the DeserializeWithLengthPrefix but it didn't work either.
I'm a bit noob on this, so if you could help me I would really appreciate it.
Thank's
The first thing to look at here is: is the data you receive the data you send. Until you can answer "yes" to that, all other questions are moot. It is very easy to confuse network code and end up reading partial frames, etc. As a crude debugger test:
Debug.WriteLine(Convert.ToBase64String(ms.GetBuffer(), 0, (int)ms.Length));
should work. If the two base-64 strings are not identical, then you aren't working with the same data. This can be because of a range of reasons, including packet splitting and combining. You need to keep in mind that in a stream, what you send is not what you get - at least, not down to the fragment level. You might "send" data in the way of:
one bundle of 20 bytes
one bundle of 10 bytes
but at the receiving end, it would be entirely legitimate to read:
1 byte
22 bytes
7 bytes
All that TCP guarantees is the order and accuracy of the bytes. It says nothing about their breakdown in terms of chunks. When writing network code, there are basically 2 approaches:
have one thread that synchronously reads from a stream and local buffer (doesn't scale well)
async code (very scalable), but accept that you're going to have to do a lot of "do I have a complete frame? if not, append to an input buffer; if so, process any available frame data (could be multiple), then shuffle any incomplete data to the start of the buffer"
I've a register space of 16 registers.
These are accessible through serial bus (single as well as burst).
I've UVM reg model defined for these registers.
However none of the reg model method supports burst transaction on bus.
As a workaround
I can declare memory model for same space and whenever I need burst access I use memory model but it seems redundant to declare 2 separate classes for same thing and this approach won't mirror register values correctly.
create a function which loops for number of bytes iterations and access registers one by one however this method doesn't create burst transaction on bus.
So I would like to know if there is a way to use burst_read and burst_write methods with register model. It would be nice if burst_read and burst_write support mirroring (current implementation doesn't support this) but if not I can use .predict and .set so its not big concern.
Or can I implement a method for register model easily to support burst operation.
I found this to help get you started:
http://forums.accellera.org/topic/716-uvm-register-model-burst-access/
The guy mentions using the optional 'extension' argument that read/write take. You could store the length of the burst length inside a container object (think int vs. Integer in Java) and then pass that as an argument when calling write() on the first register.
A rough sketch (not tested):
// inside your register sequence
uvm_queue #(int) container = new("container");
container.push_front(4);
start_reg.write(status, data, .extension(container));
// inside your adapter
function uvm_sequence_item reg2bus(const ref uvm_reg_bus_op rw);
int burst_len = 1;
uvm_reg_item reg_item = get_item();
uvm_queue #(int) extension;
if ($cast(extension, reg_item.extension))
burst_len = extension.pop_front();
// do the stuff here based on the burst length
// ...
endfunction
I've used uvm_queue because there isn't any trivial container object in UVM.
After combining opinions provided by Tudor and links in the discussion, here is what works for adding burst operation to reg model.
This implementation doesn't show all the code but only required part for adding burst operation, I've tested it for write and read operation with serial protocols (SPI / I2C). Register model values are updated correctly as well as RTL registers are updated.
Create a class to hold data and burst length:
class burst_class extends uvm_object;
`uvm_object_utils (....);
int burst_length;
byte data [$];
function new (string name);
super.new(name);
endfunction
endclass
Inside register sequence (for read don't initialize data)
burst_class obj;
obj = new ("burstInfo");
obj.burst_length = 4; // replace with actual length
obj.data.push_back (data1);
obj.data.push_back (data2);
obj.data.push_back (data3);
obj.data.push_back (data4);
start_reg.read (status,...., .extension(obj));
start_reg.write (status, ...., .extension (obj));
After successful operation data values should be written or collected in obj object
In adapter class (reg2bus is updated for write and bus2reg is updated for read)
All the information about transaction is available in reg2bus except data in case of read.
adapter class
uvm_reg_item start_reg;
int burst_length;
burst_class adapter_obj;
reg2bus implementation
start_reg = this.get_item;
adapter_obj = new ("adapter_obj");
if($cast (adapter_obj, start_reg.extension)) begin
if (adapter_obj != null) begin
burst_length = adapter_obj.burst_length;
end
else
burst_length = 1; /// so that current implementation of adapter still works
end
Update the size of transaction over here according to burst_length and assign data correctly.
As for read bus2reg needs to be updated
bus2reg implementation (Already has all control information since reg2bus is always executed before bus2reg, use the values captured in reg2bus)
According to burst_length only assign data to object passed though extension in this case adapter_obj
I'm looking for some sample code to show me how to add metadata to the wav files we create.
Anyone?
One option is to add your own chunk with a unique id. Most WAV players will ignore it.
Another idea would to be use a labl chunk, associated with a que set at the beginning or end of the file. You'd also need a que chunk. See here for a reference
How to write the data is simple
Write "RIFF".
save the file position.
Write 4 bytes of 0's
Write all the existing chunks. Keep count of bytes written.
Add your chunk. Be sure to get the chunksize right. Keep
count of bytes written.
rewind to the saved position. Write the new size (as a 32-bit
number).
Close the file.
It's slightly more complicated if you are adding things to an existing list chunk, but the same principle applies.
Maybe the nist file format will give you what you want:
NIST
Here is a lib that could help, but im afraid it looks old. NIST Lib
Cant find more useful information right now how exactly to use it, and im afraid the information papers from my company must stay there. :L/
Try code below
private void WaveTag()
{
string fileName = "in.wav";
WaveReadWriter wrw = new WaveReadWriter(File.Open(fileName, FileMode.Open, FileAccess.ReadWrite));
//removes INFO tags from audio stream
wrw.WriteInfoTag(null);
//writes INFO tags into audio stream
Dictionary<WaveInfo, string> tag = new Dictionary<WaveInfo, string>();
tag[WaveInfo.Comments] = "Comments...";
wrw.WriteInfoTag(tag);
wrw.Close();
//reads INFO tags from audio stream
WaveReader wr = new WaveReader(File.OpenRead(fileName));
Dictionary<WaveInfo, string> dir = wr.ReadInfoTag();
wr.Close();
if (dir.Count > 0)
{
foreach (string val in dir.Values)
{
Console.WriteLine(val);
}
}
}
from http://alvas.net/alvas.audio,articles.aspx#id3-tags-for-wave-files
If you examine the wave file spec you'll see that there does not seem to be room for annotations of any kind. An option would be to wrap the wave file with your own format that includes custom information but you would in effect be creating a whole new format that would not be readable by users who do not have your app. But you might be ok with that.