My Goal : Prevent batch Job from lunching while project startup.
I want to run a batch job from spring boot project once in every one hour using #scheduled annotation. However the job is starting as soon as I run the project....
spring.batch.job.enabled=false
#Scheduled( cron = "* * */1 * * *")
public void jobScheduled(){
logger.info("Job triggered");
getAllFileNames().stream().forEach( fileName-> {
jobRunner.runFlatFileBatchJob(fileName);
});
}
I have declared my class like below
#Configuration
#EnableScheduling
public class JobScheduler {
#Autowired
public JobScheduler(JobRunner jobRunner){
logger.info("job scheduler created... ");
this.jobRunner = jobRunner;
}
The Job is starting as soon as Run the application. I want it to wait till the project loads completely and other integration objects prepare themselves.
Thanks in advance.
Santrupta Dash
Related
Is there a way to add custom shutdown hook to kafka connect which I can put in the classpath using the plugins.path property?
Use Case:
The Kafka Connect cluster tries to connect to Kafka Cluster.
If it fails it logs and shutsdown immediately
The error logs does not reach to the remote log server like Splunk
I need to delay the shutdown so that the log collector agent can push the logs to the target log server.
I am using Confluent Platform v 6.1
The best way to accomplish what you are looking for is write the log file to a location that outlives the container. This could be a persistent volume like #OneCricketeer mentioned previously, or a network based logging service.
If for some reason you can't do that, you can create a JVM shutdown hook using a Java Agent. This hook could delay the JVM's shutdown long enough (risky) or it could force a flush of the logging library, or any other cleanup. Since the agent is configured as a JVM command line argument, it should work for your kafka-connect workers. You just need to modify the command line that runs the workers.
There are good instructions for creating a Java Agent here and an example for setting up a shutdown hook here: Java shutdown hook
Here is a super simple example class that has both the applications main() method and the Agent's premain() in the same class:
public class App
{
public static void main( String[] args ) throws InterruptedException {
System.out.println( System.currentTimeMillis() + ": Main Started!" );
Thread.sleep(1000);
System.out.println( System.currentTimeMillis() + ": Main Ended!" );
}
public static void premain(String agentArgs, Instrumentation inst) {
System.out.println(System.currentTimeMillis() + ": Agent Started");
Runtime.getRuntime()
.addShutdownHook(
new Thread() {
#Override
public void run() {
System.out.println(System.currentTimeMillis() + ": Shutdown Hook is running!");
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// do nothing
}
System.out.println(System.currentTimeMillis() + ": Shutdown hook is completed");
}
});
}
}
Note that in your case, you only need the premain method, as the main method is implemented by the connect worker.
Running the above class with the following command line:
java -javaagent:<MY_AGENT_FILE>.jar -classpath <MY_APP_FILE>.jar
org.example.App
generates the following output:
1630187996652: Agent Started
1630187996672: Main Started!
1630187997672: Main Ended!
1630187997673: Shutdown Hook is running!
1630188002675: Shutdown hook is completed
So you have your delay.
I'm developing system/integration tests based on Testcontainers solution. It is required for me to use our own database PostgreSQL image with already applied database schema.
For this reason I'm using Testcontainers GenericContainer.
private static final GenericContainer postgresDb = new GenericContainer(POSTGRES_IMAGE).withExposedPorts(5432);
I'm developing tests with Spring Boot, so for this reason I've created abstract class which will hold this config for all tests
#ActiveProfiles("test")
#SpringBootTest
public abstract class AbstractTests {
private static final DockerImageName POSTGRES_IMAGE = DockerImageName.parse("docker-name:latest");
private static final GenericContainer postgresDb;
static {
postgresDb = new GenericContainer(POSTGRES_IMAGE)
.withExposedPorts(5432);
postgresDb.withStartupTimeout(Duration.ofSeconds(30))
.start();
}
#DynamicPropertySource
static void properties(DynamicPropertyRegistry registry) throws InterruptedException {
final String s = "jdbc:postgresql://"+ postgresDb.getHost() +":"+ postgresDb.getMappedPort(5432) + "/test";
registry.add("spring.datasource.url", () ->s);
}
}
However, the problem is that when when tests are running, container still is starting. This withStartupTimeout(Duration.ofSeconds(30)) doesn't for work some reason.
When I stop in debug at properties method and give a couple of seconds to start a container all tests run fine.
When tests fail I see next log:
org.postgresql.util.PSQLException: FATAL: the database system is starting up
If I put Thread.sleep(..) it also works, it is not preferable solution.
What is the right solution to wait or right strategy to know that container is ready?
I think the answer is in their documentation, particularly this part:
Log output Wait Strategy
In some situations a container's log output is a simple way to determine if it is ready or not. For example, we can wait for a `Ready' message in the container's logs as follows:
public GenericContainer containerWithLogWait = new GenericContainer(DockerImageName.parse("redis:5.0.3"))
.withExposedPorts(6379)
.waitingFor(
Wait.forLogMessage(".*Ready to accept connections.*\\n", 1)
);
Note: you will want to change the message to something like:
".*database system is ready to accept connections.*"
I'm just getting started with using Spring Batch. I'm working on a command line launcher using AWS Batch with a Docker image. Trying to just sort the job instance naming.
Would it be acceptable to use a #Value for the below literal string in the jobBuilder? Essentially I'm passing in S3 File keys which will be unique values already as I have a task that's grabbing the file before my FlatFileReader runs. The goal being to facilitate retries against the job when required due to a failure.
#Bean
public Job jobParametersJob() {
return jobBuilderFactory.get("PassedInValue")
.start(step1())
.next(step2())
.next(step3())
.build();
}
I ended up solving by using a job incrementer that implements JobParametersIncrementer. Note in my case, I'm not currently passing in any job parameters so this is setting them here as my parameters are currently just passed in via environment variables to the docker container.
public class JobIncrementer implements JobParametersIncrementer {
#Value("${s3filekey}")
String s3filekey;
#Override
public JobParameters getNext(JobParameters jobParameters) {
return new JobParametersBuilder().addString("s3filekey",s3filekey).toJobParameters();
}...//then in job configuration...
#Bean
public Job jobParametersJob() {
return jobBuilderFactory.get("jobParametersJob")
.incrementer(jobIncrementer)
.start(step1())
.next(step2())
.next(step3())
.build();
}
I have service which runs some init scripts after application startup (implemented with ApplicationListener<ApplicationReadyEvent>). In this scripts I need to call another services with RestTemplate which is #LoadBalanced. When the call to service is invoked there's no information about instances of remote service because discovery server was not contacted at that time (I guess).
java.lang.IllegalStateException: No instances available for api-service
at org.springframework.cloud.netflix.ribbon.RibbonLoadBalancerClient.execute(RibbonLoadBalancerClient.java:79)
So is there way how to get list of available services from discovery server at application startup, before my init script will execute?
Thanks
edit:
The problem is more related to fact, that in current environment (dev) all services are tied together in one service (api-service). So from within api-service I'm trying to call #LoadBalanced client api-service which doesn't know about self? Can I register some listener or something similar to know when api-service (self) will be available?
here are the sample applications. I'm mainly interested how to have working this method
edit2:
Now there could be the solution to create EurekaListener
public static class InitializerListener implements EurekaEventListener {
private EurekaClient eurekaClient;
private RestOperations restTemplate;
public InitializerListener(EurekaClient eurekaClient, RestOperations restTemplate) {
this.eurekaClient = eurekaClient;
this.restTemplate = restTemplate;
}
#Override
public void onEvent(EurekaEvent event) {
if (event instanceof StatusChangeEvent) {
if (((StatusChangeEvent) event).getStatus().equals(InstanceInfo.InstanceStatus.UP)) {
ResponseEntity<String> helloResponse = restTemplate.getForEntity("http://api-service/hello-controller/{name}", String.class, "my friend");
logger.debug("Response from controller is {}", helloResponse.getBody());
eurekaClient.unregisterEventListener(this);
}
}
}
}
and then register it like this:
EurekaEventListener initializerListener = new InitializerListener(discoveryClient, restTemplate);
discoveryClient.registerEventListener(initializerListener);
However this is only executed only when application is registered to discovery service first time. Next time when I stop the api-service and run it again, event is not published. Is there any other event which can I catch?
Currently, in Camden and earlier, applications are required to be registered in Eureka before they can query for other applications. Your call is likely too early in the registration lifecycle. There is an InstanceRegisteredEvent that may help. There are plans to work on this in the Dalston release train.
I have a scheduler (extension "scheduler" 6.2.0) task class:
namespace MyVendor\MyExt\Scheduler;
class MultiStepProcessTask extends \TYPO3\CMS\Scheduler\Task\AbstractTask {
public function execute() {
$objectManager = \TYPO3\CMS\Core\Utility\GeneralUtility::makeInstance('TYPO3\\CMS\\Extbase\\Object\\ObjectManager');
$businessLogic = $objectManager->get(\MyVendor\MyExt\Scheduler\MultiStepProcessTaskBusinessLogic::class);
return $businessLogic->run();
}
}
And a class implementing business logic for the scheduler task:
namespace MyVendor\MyExt\Scheduler;
class MultiStepProcessTaskBusinessLogic {
public function run() {
...
}
}
I created a scheduler task in BE. If I run it manually - it's OK. If it is started automatically - following error message comes:
[scheduler]: Task failed to execute successfully. Class:
MyVendor\MyExt\Scheduler\MultiStepProcessTask, UID: 8. Could not
analyse class:
MyVendor\MyExt\Scheduler\MultiStepProcessTaskBusinessLogic maybe not
loaded or no autoloader? (msg#4.0.255)
The best is - after being once run manually, the task runs automatically without errors until the cache is cleared. After that it needs one manual run again.
One more strange thing: in the main scheduler task class (MultiStepProcessTask) no injection is possible. That's why I had to create business logic object via objectManager
The best solution would be to write a command controller based on extbase. There you can use the ObjectManager and you can run it directly from CLI and of course also call it via scheduler!