I am trying to write test code for tumbling window for kafka in scala but I have no idea how I can write the window timing for testing.
Expected :List("[45,55#1650555264000/1650556225000],1", "[45,55#1650555264000/1650556225000],2", "[45,55#1650555264000/1650556225000],3", "[45,55#1650555264000/1650556225000],4", "[45,55#1650555264000/1650556225000],5", "[45,55#1650555264000/1650556225000],6", ...
This is the test code I got above. How can I write test code for "#1650555264000/1650556225000"?
Related
Good morning,
I have created a callback in Dash that makes the job of a scheduler.
Every 10 minutes (with the help of an interval component), my callback is running to fetch the data from a server and to update the csv file that I use in my app.
The problem is that my callback is called only when I have the webpage opened. As soon as I close the page, the scheduler stops and runs again when I open the page again.
As the data process of updating data can be long sometimes, I want the scheduler to always run and fetch the data every 10 minutes.
I assume that a callback is a client side process right? So how can I make it run in server side?
Thank you,
Dash is probably not the right solution for this. I think it would make more sense to set up the Python code you need for this job in a simple .py script file, and set a cron job to run that script every 10 min.
Thank you #coralvanda for the help.
I finally did a python script in my container A that calls the container B every 10 minutes. The container B is fetching the data.
It makes the job.
import schedule
import time
import docker
def worker_restart():
client = docker.from_env()
container = client.containers.get('container_worker')
container.restart()
schedule.every(10).minutes.do(worker_restart)
while True:
schedule.run_pending()
time.sleep(1)
I’m trying to submit a JCL through an SQR Program using Call System Command on MVS z/os. The JCL resides in specific Dataset.
What I’m trying do is something like this:
let $jclcmd= 'SUBMIT PSLIBDSN.O92.CUST7.JCLSRC(UTILI)'
call system using $jclcmd #rtnstat
Up to this point, I have not been able to submit the JCL. What I get from the mainframe is this error:
**** WARNING **** ERRNO = ESYS
Generated in SYSTEM called from line 389 of SYS(UCALL) , offset 000118
Program SUBMIT was abnormally terminated with a system code of 66D.SYS(UCALL) , offset 000118
I also tried let $jclcmd= 'TSO SUBMIT PSLIBDSN.O92.CUST7.JCLSRC(UTILI)' but gets this:
Program TSO was abnormally terminated with a system code of 806.
SYSTEM COMPLETION CODE=806 REASON CODE=00000004
Up to this point I have thought that the call system function does not allow operating system commands to be executed for reasons of incompatibility with MVS. The reality is that the SQR documentation does not mention that it is not, but always mentions Windows and UNIX as an example. I have made a thousand attempts to execute a REXX program, submit a JCL and others but looks like the function is not right assembling the command.
Any idea will be welcome.
I have flink stream and I am calucating few things on some time window say 30 seconds.
here what happens it is giving me result my aggregating previous windows as well.
say for first 30 seconds I get result 10.
next thiry seconds I want fresh result, instead I get last window result + new
and so on.
so my question is how I get fresh result for each window.
You need to use a purging trigger. What you want is FIRE_AND_PURGE (emit and remove window content), what the default flink trigger does is FIRE (emit and keep window content).
input
.keyBy(...)
.timeWindow(Time.seconds(30))
// The important part: Replace the default non-purging ProcessingTimeTrigger
.trigger(new PurgingTrigger[..., TimeWindow](ProcessingTimeTrigger))
.reduce(...)
For a more in depth explanation have a look into Triggers and FIRE vs FIRE_AND_PURGE.
A Trigger determines when a window (as formed by the window assigner) is ready to be processed by the window function. Each WindowAssigner comes with a default Trigger. If the default trigger does not fit your needs, you can specify a custom trigger using trigger(...).
When a trigger fires, it can either FIRE or FIRE_AND_PURGE. While FIRE keeps the contents of the window, FIRE_AND_PURGE removes its content. By default, the pre-implemented triggers simply FIRE without purging the window state.
The functionality you describe can be found in Tumbling Windows: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/windows.html#tumbling-windows
A bit more detail and/or code would help :)
I'm little late into this question but I encountered the same issue with OP's. What I found out later was a bug in my own code. FYI my mistake could be good reference for your problem.
// Old code (modified to be an example):
val tenSecondGrouping: DataStream[MyCustomGrouping] = userIdsStream
.keyBy(_.somePartitionedKey)
.window(TumblingProcessingTimeWindows.of(Time.of(10, TimeUnit.SECONDS)))
.trigger(ProcessingTimeTrigger.create())
.aggregate(new MyCustomAggregateFunc(new MyCustomGrouping()))
Bug happened at new MyCustomGrouping: I unintentionally created a singleton MyCustomGrouping object and reusing it in MyCustomAggregateFunc. As more tumbling windows created, the later aggregation results grow crazy! The fix was to create new MyCustomGrouping each time MyCustomAggregateFunc is triggered. So:
// New code, problem solved
...
.aggregate(new MyCustomAggregateFunc(() => new MyCustomGrouping()))
// passing in a func to create new object per trigger
I have a kafka simulation test using Gatling and currently we are able to run with the constantUsersPerSec for a given duration. But we wanted to run this simulation at constantUsersPerSec (say some 1000/sec) and this test should be running forever until somebody kills the test/process.
Currently we have adjusted to use days in place of duration. But is there an option to run a test forever that uses a feeder. Here is the code snippet:
val scn = scenario("Kafka Test Demo")
.feed(ssv(conf.getString("filename"), true).circular)
.exec(kafka("request")
// message to send
.send[String]("${t}"))
setUp(
scn.inject(constantUsersPerSec(conf.getInt("constantUsersPerSec")) during (conf.getInt("run_duration_in_mins") hours)))
.protocols(kafkaConf)
Any suggestions/directions please?
many thanks,
mpha
Is code bellow going to work?
val scn = scenario("Kafka Test Demo")
.feed(ssv(conf.getString("filename"), true)).circular.forever() {
exec(kafka("request")
// message to send
.send[String]("${t}"))
}
I have a few questions regarding quartz.net.
What is it that keeps track of if there has been a missfire situation i Quartz.net?
What happens in the following scenarios:
If a job is run but cannot finnish due to some bug, does that count as a missfire or not?
What happens if i republish the solution, is the tracking reset?
Is there a way to receive information on what the scheduler has done and not been able to do?
I have the following code in my Run method:
IJobDetail dailyUserMailJob = new JobDetailImpl("DailyUserMailJob", null, typeof(Jobs.TestJob));
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.WithCronSchedule("0 0 4 1 * ?", x => x.WithMisfireHandlingInstructionFireAndProceed())
.Build();
this.Scheduler.ScheduleJob(dailyUserMailJob, trigger);
this.Scheduler.Start();
The job is supposed to run the first every month on 4 am.
When testing I have set the system clock so that the jobb is missed for one month. According to the documentation when using WithMisfireHandlingInstructionFireAndProceed the job should be run the first thing that happens, but it dosent. Is there something wrong with the code or could it be some other reason the job is not run when using WithMisfireHandlingInstructionFireAndProceed() ?
If a job is missed, there is logic to bring it back. However, there is a "window" on how far back to go.
<add key="quartz.jobStore.misfireThreshold" value="60000"/>
You can increase this value.
If you have an ADOStore, misfires are persisted. Thus "if the power goes out", when restarting...you can recover from misfires.
If you have a RamStore...if "the power goes out", everything was in memory to begin with..so you won't get mis-fire handling, because everything was "in memory" and the memory is lost.
..
If you use Sql Server (AdoStore) and put a Profiler/Trace on it, you'll see the engine "poll" for misfires.......with a "go back this far in time" based on the misfireThreshold.
See this link:
http://nurkiewicz.blogspot.com/2012/04/quartz-scheduler-misfire-instructions.html
for more detailed info. Which has a "withMisfireHandlingInstructionFireAndProceed" note.