I have two Java Spring Boot web service apps on the same server calling each other via REST. Service A calls Service B and the latter successfully acts upon the notfication.
THE PROBLEM is that Service A never receives the acknowlegement from Service B, so it thinks it has failed, and in accordance with its looping recovery logic, it tries again…and again…and again. Service B ends up doing 3 times the work for no added benefit.
The relevant code (stripped down and falsified to protect the guilty) is as follows:
Service A:
public void giveOrderToServiceB(#RequestBody CustomClass message) {
...
org.springframework.web.client.RestTemplate template = new RestTemplate(clientHttpRequestFactory());
com.mycompany.CustomReply reply = template.postForObject(serviceBUrl, message, CustomReply.class);
Service B REST Controller:
#PostMapping(value="ExecuteTheWork", produces=org.springframework.http.MediaType.APPLICATION_JSON_VALUE, consumes=MediaType.APPLICATION_JSON_VALUE)
public #ResponseBody CustomReply executeTheWork(#RequestBody CustomClass thing) {
// do something with the thing...
CustomReply reply = new CustomReply();
reply.setReply("Successfully executed the work.");
return reply;
}
The actual exception caught by Service A after calling RestTemplate.postForObject() is
java.net.SocketTimeoutException: Read timed out
Please advise.
OK, I think I got it. I don't send the response back from Service B until after the method has completed all of its work, which can take several seconds to several minutes.
If I immediately answer (and skip the processing), it works consistently.
Need to spin off the actual work to a separate thread.
Cheeers
When you are registering the bean of rest template in your application it must then configure it with a timeout. Following is the Spring application config file
package com.temp.project.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.ClientHttpRequestFactory;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;
#Configuration
public class TempProjectConfig {
/**
* Set Timeout for HTTP requests
* #return
*/
#Bean
public ClientHttpRequestFactory getClientHttpRequestFactory() {
int timeout = 1200000; // here is the timeout property set for rest template
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory
= new HttpComponentsClientHttpRequestFactory();
clientHttpRequestFactory.setConnectTimeout(timeout);
return clientHttpRequestFactory;
}
/**
* RestTemplate to call REST endpoints
* #param clientHttpRequestFactory
* #return
*/
#Bean
public RestTemplate getRestTemplate(ClientHttpRequestFactory clientHttpRequestFactory) {
return new RestTemplate(clientHttpRequestFactory);
}
}
Related
I have a config file that has:
#Configuration
#ComponentScan(basePackages = { "xxxxxxxx", "xxxxxxxx" })
#EnableBatchProcessing
#Import(DataSourceConfig.class)
#EnableRetry
public class BatchConfiguration extends DefaultBatchConfigurer {
and another method in a separate class that has:
#Override
#Retryable(ValidationException.class)
public FlowExecutionStatus decide(JobExecution jobExec, StepExecution stepExec) {
boolean passed = validationStatus.getValidationStatus();
if(!passed) {
LOG.info("******Batch job validation FAILED******");
throw new ValidationException("Batch job validation FAILED");
}
else return FlowExecutionStatus.COMPLETED;
}
At the very least it should retry and print this 3 times instead of once, since it does not pass. All the annotations are imported successfully. Its just not doing what I would expect.:
******Batch job validation FAILED******
All I get is the stacktrace for the one ValidationException
Caused by: org.springframework.batch.item.validator.ValidationException: Batch job validation FAILED
at com.xxxxx.xxxxx.decision.ValidationFlowDecider.decide(ValidationFlowDecider.java:31)
The decide method is part of the JobExecutionDecider contract and will be driven by the flow you defined for your job. So if you want to "retry" some logic in your job flow, you should define it in the flow definition itself, not using an annotated method (with Spring Retry or any other library).
A typical usage of a decider is shown with a code example in the reference documentation here: Programmatic Flow Decisions.
My goal is to create a pipeline that invokes a back-end (Cloud hosted) service a maximum number of times per second ... how can I achieve that?
Back story: Imagine a back-end service that is invoked with a single input and returns a single output. This service has quotas associated with it that permit a maximum number of requests per second (let's say 10 requests per second). Now imagine an unbounded source PCollection where I wish to transform the elements in the input by passing them through my back-end service. I can envisage a ParDo invoking the back-end service once for each element in the input PCollection. However, this doesn't perform any kind of flow control against the back-end.
I could imagine my DoFn logic testing the response from the back-end response and retrying till it succeeds but this doesn't feel right. If I have 100 workers, then I seem to be burning a lot of resources and putting a load on the back-end. What I think I want to do is throttle the calls to the back-end from the pipeline.
Good Day, kolban. In addition to Bruno Volpato's helpful RampupThrottlingFn example, I've seen a combination of the following. Please do not hesitate at all to let me know how I can update the example with more clarity.
PeriodicImpulse - emits an Instant at a fixed specified interval.
Fix the number of workers with the maxNumWorkers and numWorkers (Please see Dataflow Pipeline Options), if using the Dataflow runner.
Beam Metrics API to monitor the actual resource request count over time and set alerts. When using Dataflow, the Beam Metrics API automatically connects to Cloud Monitoring as Custom metrics
The following shows abbreviated code starting from the whole pipeline followed by some details as needed to provide clarity. It assumes a target of 10 workers, using Dataflow with the arguments --maxNumWorkers=10 and --numWorkers=10 and a goal to limit the resource requests among all workers to 10 requests per second. This translates to 1 request per second per worker.
PeriodicImpulse limits the Request creation to 1 per second
public class MyPipeline {
public static void main(String[] args) {
Pipeline pipeline = Pipeline.create(/* Usually with options */);
PCollection<Response> responses = pipeline.apply(
"PeriodicImpulse",
PeriodicImpulse
.create()
.withInterval(Duration.standardSeconds(1L))
).apply(
"Build Requests",
ParDo.of(new RequestFn())
)
.apply(ResourceTransform.create());
}
}
RequestFn DoFn emits Requests per Instant emitted from PeriodicImpulse
class RequestFn extends DoFn<Instant, Request> {
#ProcessElement
public void process(#Element Instant instant, OutputReceiver<Request> receiver) {
receiver.output(
Request.builder().build()
);
}
}
ResourceTransform transforms Requests to Responses, incrementing a Counter
class ResourceTransform extends PTransform<PCollection<Request>, PCollection<Response>> {
static ResourceTransform create() {
return new ResourceTransform();
}
public PCollection<Response> expand(PCollection<Request> input) {
return ParDo.of("Consume Resource", new ResourceFn());
}
}
class ResourceFn extends DoFn<Request, Response> {
private Counter counter = Metrics.counter(ResourceFn.class, "some:resource");
private transient ResourceClient client = null;
#Setup
public void setup() {
client = new ResourceClient();
}
#ProcessElement
public void process(#Element Request request, OutputReceiver<> receiver)
{
counter.inc(); // Increment the counter.
// not showing error handling
Response response = client.execute(request);
receiver.output(response);
}
}
Request and Response classes
(Aside: consider creating a Schema for the request input and response output classes. Example below uses AutoValue and AutoValueSchema)
#DefaultSchema(AutoValueSchema.class)
#AutoValue
abstract class Request {
/* abstract Getters. */
abstract String getId();
#AutoValue.Builder
static abstract class Builder {
/* abstract Setters. */
abstract Builder setId(String value);
abstract Request build();
}
}
#DefaultSchema(AutoValueSchema.class)
#AutoValue
abstract class Response {
/* abstract Getters. */
abstract String getId();
#AutoValue.Builder
static abstract class Builder {
/* abstract Setters. */
abstract Builder setId(String value);
abstract Response build();
}
}
Using Spring Cloud Dalston, we have built a proxy service that, of course, uses Zuul. I am now trying to add Spring Cloud Contract tests to verify our proxy service works as expected from a contract adherence perspective. What is strange is that I can send a request and receive a 200 status code back, but the expected response body, content type header, etc. is empty, and my tests fail as a result.
Are there additional configurations not specified in the Spring Cloud Contract documentation necessary for testing services that use Zuul capabilities?
Here you have an example https://github.com/spring-cloud/spring-cloud-contract/issues/450
/**
* Abstraction of configuration to test contracts with external microservices which are using a zuul-gateway in between.
*
* When a provider of data only allows communication through a Zuul-Gateway, a special way to ensure valid contracts is needed.
* The Goal is to emulate the Zuul-Interception to be able to use the contracts stored within the providing service.
*
* F.e.: Consumer-Service consumes the Provider-Service, but the provider can not be called directly.
* The Consumer-Service calls an URL with a prefix consisting of the name of the gateway ("gateway") and name of the
* service (in this example "Provider-Service"). For example: http://gateway/provider/path/to/resource.
* The contract provided by Provider-Service holds only a part of the provided URL (here: "/path/to/resource").
* The resolution of "gateway" and "provider" is done within this class.
*/
#RunWith(SpringRunner.class)
#SpringBootTest(classes = EmbeddedZuulProxy.class,
properties = {"server.port: 8080",
"zuul.routes.provider.path: /provider/**",
"zuul.routes.provider.service-id: provider",
"zuul.routes.provider.url: http://localhost:9090/" //the url of the provider-stub is at port 9090
},
webEnvironment = WebEnvironment.DEFINED_PORT) //defined port is important! Ribbon points to zuul, which then points to the provider-stub
#AutoConfigureMockMvc
#AutoConfigureJsonTesters
//the stub runs on fixed port 9090, so that zuul can point to it
#AutoConfigureStubRunner(ids = "<group-id>:<artifact-id>:+:stubs:9090")
#DirtiesContext
public abstract class ZuulContractBase {
}
/**
* Configuration and Setup of an embedded Zuul-Gateway.
* This way it is possible to use contracts, stored in providing service
*/
#Configuration
#ComponentScan(basePackages = "<path.to.feign.client>") //autowiring feign client
#EnableAutoConfiguration
#EnableZuulProxy
#EnableFeignClients
#RibbonClient(name = "gateway", configuration = SimpleRibbonClientConfiguration.class)
class EmbeddedZuulProxy {
#Bean
RouteLocator routeLocator(DiscoveryClient discoveryClient,
ZuulProperties zuulProperties) {
return new DiscoveryClientRouteLocator("/", discoveryClient, zuulProperties);
}
}
/**
* Ribbon Load balancer with fixed server list for "simple" pointing to localhost,
* which holds the mocked zuul-gateway
*
* f.e. a call with feign would normally look like:
* http://gateway/provider/rest/path/to/your/{url}
* which is mapped to:
* http://localhost:{zuulport}/provider/rest/path/to/your/{url}
*/
#Configuration
class SimpleRibbonClientConfiguration {
#Value("${server.port}")
private Integer zuulPort;
#Bean
public ServerList<Server> ribbonServerList() {
return new StaticServerList<>(new Server("localhost", zuulPort));
}
}
I have implementated a Rest web service (the function is not relevant) using JAX-RS. Now I want to generate its documentation using Swagger. I have followed these steps:
1) In build.gradle I get all the dependencies I need:
compile 'org.glassfish.jersey.media:jersey-media-moxy:2.13'
2) I documentate my code with Swagger annotations
3) I hook up Swagger in my Application subclass:
public class ApplicationConfig extends ResourceConfig {
/**
* Main constructor
* #param addressBook a provided address book
*/
public ApplicationConfig(final AddressBook addressBook) {
register(AddressBookService.class);
register(MOXyJsonProvider.class);
register(new AbstractBinder() {
#Override
protected void configure() {
bind(addressBook).to(AddressBook.class);
}
});
register(io.swagger.jaxrs.listing.ApiListingResource.class);
register(io.swagger.jaxrs.listing.SwaggerSerializers.class);
BeanConfig beanConfig = new BeanConfig();
beanConfig.setVersion("1.0.2");
beanConfig.setSchemes(new String[]{"http"});
beanConfig.setHost("localhost:8282");
beanConfig.setBasePath("/");
beanConfig.setResourcePackage("rest.addressbook");
beanConfig.setScan(true);
}
}
However, when going to my service in http://localhost:8282/swagger.json, I get this output.
You can check my public repo here.
It's times like this (when there is no real explanation for the problem) that I throw in an ExceptionMapper<Throwable>. Often with server related exceptions, there are no mappers to handle the exception, so it bubbles up to the container and we get a useless 500 status code and maybe some useless message from the server (as you are seeing from Grizzly).
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Response;
import javax.ws.rs.ext.ExceptionMapper;
public class DebugMapper implements ExceptionMapper<Throwable> {
#Override
public Response toResponse(Throwable exception) {
exception.printStackTrace();
if (exception instanceof WebApplicationException) {
return ((WebApplicationException)exception).getResponse();
}
return Response.serverError().entity(exception.getMessage()).build();
}
}
Then just register with the application
public ApplicationConfig(final AddressBook addressBook) {
...
register(DebugMapper.class);
}
When you run the application again and try to hit the endpoint, you will now see a stacktrace with the cause of the exception
java.lang.NullPointerException
at io.swagger.jaxrs.listing.ApiListingResource.getListingJson(ApiListingResource.java:90)
If you look at the source code for ApiListingResource.java:90, you will see
Swagger swagger = (Swagger) context.getAttribute("swagger");
The only thing here that could cause the NPE is the context, which scrolling up will show you it's the ServletContext. Now here's the reason it's null. In order for there to even be a ServletContext, the app needs to be run in a Servlet environment. But look at your set up:
HttpServer server = GrizzlyHttpServerFactory
.createHttpServer(uri, new ApplicationConfig(ab));
This does not create a Servlet container. It only creates an HTTP server. You have the dependency required to create the Servlet container (jersey-container-grizzly2-servlet), but you just need to make use of it. So instead of the previous configuration, you should do
ServletContainer sc = new ServletContainer(new ApplicationConfig(ab));
HttpServer server = GrizzlyWebContainerFactory.create(uri, sc, null, null);
// you will need to catch IOException or add a throws clause
See the API for GrizzlyWebContainerFactory for other configuration options.
Now if you run it and hit the endpoint again, you will see the Swagger JSON. Do note that the response from the endpoint is only the JSON, it is not the documentation interface. For that you need to use the Swagger UI that can interpret the JSON.
Thanks for the MCVE project BTW.
Swagger fixed this issue in 1.5.7. It was Issue 1103, but the fix was rolled in last February. peeskillet's answer will still work, but so will OP's now.
Error Description
Hey all,
I'm having trouble getting a response from my manually added controllers in a JHipster-based project. I scaffolded up the original project, and then hand-wrote my own services and controllers.
When I execute the call, the error result I get from SoapUI (which I am using for initial validation) is at the following url: http://imgur.com/04FpmEZ,Havk1EL#0
And if I look at my Eclipse console error, I see the following: http://imgur.com/04FpmEZ,Havk1EL#1
Controller
/**
* GET /courses/json -> get all the courses.
*/
#RequestMapping(value = "/json",
method = RequestMethod.GET,
produces = "application/json")
#Timed
public List<Course> getAll() {
log.debug("REST request to get all Courses");
return courseService.findAllCourses();
}
Service
package com.testapp.myapp.service;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import com.testapp.myapp.domain.Course;
import com.testapp.myapp.repository.CourseRepository;
#Service
#Transactional
public class CourseServiceImpl implements CourseService {
#Autowired
CourseRepository courseRepository;
public long countAllCourses() {
return courseRepository.count();
}
public void deleteCourse(Course course) {
courseRepository.delete(course);
}
public Course findCourse(Integer id) {
return courseRepository.findOne(id);
}
public List<Course> findAllCourses() {
return courseRepository.findAll();
}
public List<Course> findCourseEntries(int firstResult, int maxResults) {
return courseRepository.findAll(new org.springframework.data.domain.PageRequest(firstResult / maxResults, maxResults)).getContent();
}
public void saveCourse(Course course) {
courseRepository.save(course);
}
public Course updateCourse(Course course) {
return courseRepository.save(course);
}
}
What is confusing about this is that I ran the query provided by hibernate directly against my DB, and it returns the record set just fine. Is it possible that the service is being blocked due to some security or authentication constraint auto-loaded by JHipster?
A few issues existed, all related to migrating from Roo into JHipster:
I had built my new Controller class with org.sprinframework.stereotype.Controller's #Controller annotation, rather than #RestController... The original controller annotation was scaffolded up by Spring Roo (which is highly effective at generating services from an existing DB using their DBRE addon, I might add).
After switching over to #RestController, I ran into the second hurdle, which I had originally expected as a JHipster implementation : the service was being blocked due to authentication constraints.
This was fixed by going into com.[projectname].config and updating the SecurityConfiguration.java file, exposing specifically the APIs that I wanted.
Then, I had to make sure Hibernate was getting the full collection of the objects being requested (I had a lot of complex relational entities being built by Roo)... failed to lazily initialize a collection of role...
In the Domain entity, change your #OneToMany annotation as follows:
#OneToMany(fetch = FetchType.EAGER, mappedBy = "courseId", cascade = CascadeType.REMOVE)
Source of answer: Solve "failed to lazily initialize a collection of role" exception
Voila! Functioning, secure-able JSON-based APIs, fully reverse engineered from an existing Postgresql DB, loaded into a prescaffolded Angular front-end.