I'm learning the Spring 4 stuff by converting an existing Spring 3 project. In that project I have a custom query. That query fetches data in a straightforward way, after which some heavy editing is done to the query results. Now the data is sent to the caller.
I plan on extending CrudRepository for most of my simple query needs. The data will be output in HATEOAS format.
For this custom query I think I should be adding custom behavior (spring.io, "Working with Spring Data Repositories", Section 1.3.1, "Adding custom behavior to single repositories").
As an example:
#Transactional(readOnly = true)
public List<Offer> getFiltered(List<Org> orgs, OfferSearch criteria) {
List<Offer> filteredOffers = getDateTypeFiltered(criteria);
filteredOffers = applyOrgInfo(orgs, filteredOffers);
filteredOffers = applyFilterMatches(filteredOffers, criteria);
return sortByFilterMatches(filteredOffers);
}
(The code merely illustrates that I don't have a simple value fetch going on.)
If I could use the raw results of getDateTypeFiltered(criteria) then I could put that into a CrudRepository interface and the output would be massaged into HATEOAS by the Spring libraries. But I must do my massaging in an actual Java object, and I don't know how to tell Spring to take my output and emit it in my desired output format.
Is there an easy way to get there from here? Or must I try things like do my filtering in the browser?
Thanks,
Jerome.
To properly get HAL formatted results, your query controllers must return some form of Spring HATEOAS Resource type.
#RequestMapping(method = RequestMethod.GET, value = "/documents/search/findAll")
public ResponseEntity<?> findAll() {
List<Resource<Document>> docs = new ArrayList<>();
docs.add(new Resource<Document>(new Document("doc1"), new Link("localhost")));
docs.add(new Resource<Document>(new Document("doc2"), new Link("localhost")));
Resources<Resource<Document>> resources = new Resources<Resource<Document>>(docs);
resources.add(linkTo(methodOn(ApplicationController.class).findAll()).withSelfRel());
resources.add(entityLinks.linkToCollectionResource(Document.class).withRel("documents"));
return ResponseEntity.ok(resources);
}
I have submitted a pull request to Spring Data REST to update its reference docs to specify this in http://docs.spring.io/spring-data/rest/docs/2.4.0.RELEASE/reference/html/#customizing-sdr.overriding-sdr-response-handlers
I am not sure I perfectly got your question. If I did this should be the answer: http://docs.spring.io/spring-data/jpa/docs/1.9.0.RELEASE/reference/html/#repositories.custom-implementations
Related
Please look at the below code from controller(Added comments) which uses RestTemplate:
#GetMapping("/{courseid}")
public Course getCourseDetails(#PathVariable Long courseid) {
// Get Course info (ID, Name, Description) from pre-populated Array List
CourseInfo courseInfo = getCourseInfo(courseid);
// Get Price info of a course from another microservice using RESTTemplate
Price price = restTemplate.getForObject("http://localhost:8002/price/"+courseid, Price.class);
// Get enrollment info of a course from another microservice using RESTTemplate
Enrollment enrollment = restTemplate.getForObject("http://localhost:8003/enrollment/"+courseid, Enrollment.class);
//Consolidate everything in to Course object and send it as response
return new Course(courseInfo.getCourseID(), courseInfo.getCourseName(), courseInfo.getCourseDesc(), price.getDiscountedPrice(),
enrollment.getEnrollmentOpen());
}
Now I am trying to achieve the same using Reactive programming. I now use WebClient and Mono from Web-Flux. But, I am so confused as to how to combine the results? Take a look at the below code (Just using Mono Everywhere. Rest of the code remained same)
#GetMapping("/{courseid}")
public Mono<Course> getCourseDetails(#PathVariable Long courseid) {
// Get Course info (ID, Name, Description) from pre-populated Array List
CourseInfo courseInfo = getCourseInfo(courseid);
// Get Price info of a course from another microservice using RESTTemplate
Mono<Price> price = webClient.get().uri("http://localhost:8002/price/{courseid}/",courseid).retrieve().bodyToMono(Price.class);
// Get enrollment info of a course from another microservice using RESTTemplate
Mono<Enrollment> inventory = webClient.get().uri("http://localhost:8003/enrollment/{courseid}/",courseid).retrieve().bodyToMono(Enrollment.class);
//Question : How do we Consolidate everything and form a Mono<Course> object and send it as response?
}
Question 1 : How do we Consolidate everything and form a Mono object and send it as response?
Question 2 : Does the statement "CourseInfo courseInfo = getCourseInfo(courseid);" cause blocking operation?
Thanks!
Answering to:
Question 1 : How do we Consolidate everything and form a Mono object and send it as response?
Mono.zip(..) is what you need to combine the two results. This diagram is from the doc :
Note that, zip will result in an empty Mono if one of A or 1 is empty! Use switchIfEmpty/defaultIfEmpty to protect against that case.
Thus the code looks like:
#GetMapping("/{courseid}")
public Mono<Course> getCourseDetails(#PathVariable Long courseid) {
CourseInfo courseInfo = getCourseInfo(courseid);
Mono<Price> priceMono = webClient.get().uri("http://localhost:8002/price/{courseid}/",courseid).retrieve().bodyToMono(Price.class);
Mono<Enrollment> enrollmentMono = webClient.get().uri("http://localhost:8003/enrollment/{courseid}/",courseid).retrieve().bodyToMono(Enrollment.class);
return Mono.zip(priceMono, enrollmentMono).map(t -> new Course(courseInfo.getCourseID(), courseInfo.getCourseName(), courseInfo.getCourseDesc(), t.getT1().getDiscountedPrice(),
t.getT2().getEnrollmentOpen()));
}
Now answering to:
Question 2 : Does the statement "CourseInfo courseInfo = getCourseInfo(courseid);" cause blocking operation?
Since you mentioned that Get Course info (ID, Name, Description) from pre-populated Array List, if it is just an in-memory Array containing the course information, then it's not blocking.
But (as #mslowiak also mentioned), if getCourseInfo contains logic which involves querying a database, ensure that you are not using a blocking JDBC driver. If so, then there is no point of using Webflux and Reactor. Use Spring R2DBC if that is the case.
restTemplate.getForObject returns simple object - in your case Price or Enrollment. To convert them to Mono you can simply Mono.just(object), however better solution would be to switch to Webclient which is default HTTP client for Spring Reactive
getCourseInfo it depends what is the logic behind this method. For sure if there is a JDBC connection behind that method it is blocking.
To make a final response with Mono<Course> you should think about zip operator which will help you with that.
For ex:
Mono<Course> courseMono = Mono.zip(price, enrollment)
.map(tuple -> new Course(courseInfo, tuple.getT1(), tuple.getT2()));
In OData V3, I can select just fields from parent/ancestor entities like this:
http://services.odata.org/V3/Northwind/Northwind.svc/Order_Details(OrderID=10248,ProductID=11)?&$select=Product/Category/CategoryName&$expand=Product/Category
That query returns only CategoryName, it does not include any fields from Order_Details or Product. This behavior is very important to our application for performance reasons. Selecting all fields when we don't need them can have a significant impact on query performance.
There does not seem to be a way to accomplish the same in OData V4. The equivalent query returns all fields from Order_Details and Product
http://services.odata.org/V4/Northwind/Northwind.svc/Order_Details(OrderID=10248,ProductID=11)?$expand=Product($expand=Category($select=CategoryName))
The closest I can get is to just select one field from each level, introduces a lot of complexity into our code, and it has been difficult to ensure that all queries (future and existing) adhere to this rule.
The closest I can get is to just select one field from each level,
introduces a lot of complexity into our code, and it has been
difficult to ensure that all queries (future and existing) adhere to
this rule.
Looks something like this:
http://services.odata.org/V4/Northwind/Northwind.svc/Order_Details(OrderID=10248,ProductID=11)?$expand=Product($select=Category;$expand=Category($select=CategoryName))&$select=Product
There is certainly a bit of added complexity here, but this was acceptable in my case.
The syntax for this is:
https://services.odata.org/V4/Northwind/Northwind.svc/Order_Details(OrderID=10248,ProductID=11)/Product/Category?$select=CategoryName
results in:
{
"#odata.context": "https://services.odata.org/V4/Northwind/Northwind.svc/$metadata#Categories(CategoryName)/$entity",
"CategoryName": "Dairy Products"
}
The key OData v4 concept here is that the path, that is everything before the ? defines the resource that is being served, and by that the shape of the resulting graph. The output of $select and $expand (projections) is constrained to match the requested resource.
So in v3 with $select you could return a more arbitrary structure but in v4 the $select and $expand can only mask the graph by returning what is essentially a subset of $select=*&$expand=*.
To get around this but still allow similar query scenarios in v4 we can compose an entity path expression to *any resource within the parent path.
So we move the resource selector path from the v3 $select Product/Cateogry' and append it the path of our resource ~/Order_Details(OrderID=10248,ProductID=11)`
NOTE: There is a strong caveat to this, whilst the OData specification describes this behaviour, not all implementations support deep resource selections like this. The specification is a guidance document on the standard protocol, not all implemenations are 100% compliant.
A simplification of this is to try selecting just the Product from the same query, notice here we do not use any query parameters at all:
~/Order_Details(OrderID=10248,ProductID=11)/Product
{
"#odata.context": "https://services.odata.org/V4/Northwind/Northwind.svc/$metadata#Products/$entity",
"ProductID": 11,
"ProductName": "Queso Cabrales",
"SupplierID": 5,
"CategoryID": 4,
"QuantityPerUnit": "1 kg pkg.",
"UnitPrice": 21.0000,
"UnitsInStock": 22,
"UnitsOnOrder": 30,
"ReorderLevel": 30,
"Discontinued": false
}
You can see in this response that the context or the resource that is being returned is a $metadata#Products/$entity and not an Order_Details/$entity
Once your resource is selected, normal v4 $select and $expand logic is evaluated. This is documented in the specification under 4.3 Addressing Entities
These rules are recursive, so it is possible to address a single entity via another single entity, a collection via a single entity and even a collection via a collection; examples include, but are not limited to:
By following a navigation from a single entity to another related entity (see rule: entityNavigationProperty)
Example 14:
http://host/service/Products(1)/Supplier
Update:
I've substantially edited this post from my original answer, at the time i misinterpreted OP's request and the structure they were expecting, but this is still relevant information in 2022 and none of the answers directly produces the desired behaviour.
The simplest solutions would be to create View with required schema on your db server and try to fetch data from this datasource with filters and column name(s) instead.
Especially when facing issues with performance.
The best way would be to register this class to your IoC as singleton
public class InternalODataEdmModelBuilder
{
private readonly ODataConventionModelBuilder _oDataConventionModelBuilder = new ODataConventionModelBuilder();
private IEdmModel _edmModel;
public InternalODataEdmModelBuilder()
{
ODataEntitySetsConfigInternal.Register(_oDataConventionModelBuilder);
}
// cache
public IEdmModel GetEdmModel()
{
return _edmModel ?? (_edmModel = _oDataConventionModelBuilder.GetEdmModel());
}
}
internal static class ODataEntitySetsConfigInternal
{
public static void Register(ODataConventionModelBuilder oDataModelBuilder)
{
if (oDataModelBuilder == null)
{
throw new Exception("'ODataConventionModelBuilderWebApi' cannot be null");
}
oDataModelBuilder.EntitySet<YourView>("YourView").EntityType.HasKey(x => x.YourKey);
}
}
And then inject this registered object in your API controller and build your query from URL like this:
ODataQueryContext queryContext = new ODataQueryContext(_oDataConventionModel, typeof(YourViewEFType), null);
var option = new ODataQueryOptions(queryContext, Request);
var data = option.ApplyTo(data, new ODataQuerySettings { EnsureStableOrdering = false });
And then map data into your POCO (API EDM model shown to the public world).
Hope this helps.
I have a Spring Clound Feign Client mapping defined as following
#RequestMapping(method = RequestMethod.GET, value = "/search/findByIdIn")
Resources<MyClass> get(#RequestParam("ids") List<Long> ids);
when I call
feignClient.get(Arrays.asList(1L,2L,3L))
according to what I can see in the debugger, the feign-core library forms the following request:
/search/findByIdIn?ids=1&ids=2&ids=3
instead of expected
/search/findByIdIn?ids=1,2,3
which would be correct for the server Spring Data REST endpoint declared in the same way as my Feign client method.
Thus, because of this issue, the request always returns empty set.
I have seen similar question, but it looks like the Feign client was working as I expect back in 2015.
I am using:
spring-cloud-starter-feign version 1.2.4.RELEASE
feign-httpclient version 9.4.0
feign-core version 9.4.0
Is there a way to correct the behaviour and "marry" the Spring Cloud Feign Client with the Spring Data REST defined endpoints?
I had the same issue with multiple occurence of the parametre instead of the expected comma separated sequence of items. The solution was really simple:
In my feign client I used arrays
feignClient.get(new Long[]{1L,2L,3L})
instead of collection/list:
feignClient.get(Arrays.asList(1L,2L,3L))
In Feign you can annotate your controller with the following
#CollectionFormat(feign.CollectionFormat.CSV) and it will process collections in
the CSV format findByIdIn?ids=1&ids=2&ids=3
Thanks #prola for your answer.
Just to add an explicit example, #CollectionFormat(feign.CollectionFormat.CSV) annotation targets a method; you can't apply globally to your Feign Client interface.
So each method will be similar to:
#RequestMapping(value = ["/objects"], method = [RequestMethod.GET])
#CollectionFormat(feign.CollectionFormat.CSV)
fun findById(
#RequestParam(value = "object.id", required = true) id: String,
#RequestParam(value = "object.fields", required = false) objectFields: List<String> = DEFAULT_FIELDS_LIST,
#RequestParam(value = "format") format: String = FORMAT,
): ResponseEntity<ObjectsDTO>
The result will be
/objects?object.fields=size,weight,location
instead of
/objects?object.fields=size&object.fields=weight&object.fields=location
You can also refer to:
1.16.Feign CollectionFormat support
OpenFeign #542: Support Multiple Collection Formats
I've just battled with this today, and the solution for me was surprisingly simple.
If you use brackets [] for denoting query array:
Resources<MyClass> get(#RequestParam("ids[]") List<Long> ids);
it will create a request that looks like this
/search/findByIdIn?ids[]=1&ids[]=2&ids[]=3
Most server side frameworks will interpret this as an array.
If your server is also in spring then you can pick this up like this
#GetMapping("/search/findByIdIn")
public ResponseEntity findByIdIn(#RequestParam("ids[]") List<Long> ids) { ... }
Just keep in mind that the query has to be encoded, [] gets encoded to %5B%5D.
I am building a REST API and facing this issue: How can REST API pass very large JSON?
Basically, I want to connect to Database and return the training data. The problem is in Database I have 400,000 data. If I wrap them into a JSON file and pass through GET method, the server would throw Heap overflow exception.
What methods we can use to solve this problem?
DBTraining trainingdata = new DBTraining();
#GET
#Produces("application/json")
#Path("/{cat_id}")
public Response getAllDataById(#PathParam("cat_id") String cat_id) {
List<TrainingData> list = new ArrayList<TrainingData>();
try {
list = trainingdata.getAllDataById(cat_id);
Gson gson = new Gson();
Type dataListType = new TypeToken<List<TrainingData>>() {
}.getType();
String jsonString = gson.toJson(list, dataListType);
return Response.ok().entity(jsonString).header("Access-Control-Allow-Origin", "*").header("Access-Control-Allow-Methods", "GET").build();
} catch (SQLException e) {
logger.warn(e.getMessage());
}
return null;
}
The RESTful way of doing this is to create a paginated API. First, add query parameters to set page size, page number, and maximum number of items per page. Use sensible defaults if any of these are not provided or unrealistic values are provided. Second, modify the database query to retrieve only a subset of the data. Convert that to JSON and use that as the payload of your response. Finally, in following HATEOAS principles, provide links to the next page (provided you're not on the last page) and previous page (provided you're not on the first page). For bonus points, provide links to the first page and last page as well.
By designing your endpoint this way, you get very consistent performance characteristics and can handle data sets that continue to grow.
The GitHub API provides a good example of this.
My suggestion is no to pass the data as a JSON but as a file using multipart/form-data. In your file, each line could be a JSON representing a data record. Then, it would be easy to use a FileOutputStream to receive te file. Then, you can process the file line by line to avoid memory problems.
A Grails example:
if(params.myFile){
if(params.myFile instanceof org.springframework.web.multipart.commons.CommonsMultipartFile){
def fileName = "/tmp/myReceivedFile.txt"
new FileOutputStream(fileName).leftShift(params.myFile.getInputStream())
}
else
//print or signal error
}
You can use curl to pass your file:
curl -F "myFile=#/mySendigFile.txt" http://acme.com/my-service
More details on a similar solution on https://stackoverflow.com/a/13076550/2476435
HTTP has the notion of chunked encoding that allows you send a HTTP response body in smaller pieces to prevent the server from having to hold the entire response in memory. You need to find out how your server framework supports chunked encoding.
I have an ASP.NET MVC 2 application which in part allows a user to filter data and view that data in a JQGrid.
Currently this consists of a controller which initialises my filter model and configures how I wish my grid to be displayed. This information is used by a view and a partial view to display the filter and the grid shell. I use an editor template to display my filter. The JQGrid makes use of a JsonResult controller action (GET) to retrieve the results of the filter (with the addition of the paging offered by the grid - only a single page of data is returned by the GET request. The Uri used by the grid to request data contains the filter model as a RouteValue - and currently contains a string representation of the current state of the filter. A custom IModelBinder is used to convert this representation back into an instance of the filter model class.
The user can change the filter and press a submit button to get different results - this is then picked up by an (HttpPost) ViewResult action which takes the filter model - reconstituted by a further model binder and causes the grid shell to be updated.
So I have:
FilterModel
Represents the user's desired filtering characteristics
FilterModelEditorTemplateSubmissionBinder : DefaultModelBinder - used to convert the request information supplied from a user changing their filtering characteristics into the appropriate FilterModel instance.
FilterModelStringRepresentationBinder : IModelBinder - used to convert the encoded filter from the JQGrid GET request for data so the correct request is made of the service which is ultimately performing the query and returning the relevant data.
ViewResult Index() - constructs a default filter, configures the grid specification and returns the view to render the filter's editor template, and the grid shell.
[HttpPost]ViewResult Filter(FilterModel filter) - takes the new filter characteristics and returns the same view as Index(). Uses FilterModelEditorTemplateSubmissionBinder to bind the filter model.
JsonResult GetData(FilterModel filter, string sidx, string sord, int page, int rows) - called from the JQGrid in order to retrieve the data. Uses FilterModelStringRepresentationBinder to bind the filter model.
As a complication, my filter model contains a option to select a single value from a collection of items. This collection is retrieved from a service request and I don't want to keep querying for this data everytime I show the filter, currently I get it if the property is null, and then include the options hidden in the editor template and encoding in the string representation. These options are then reconstituted by the relevant model binder.
Although this approach works I can't help but feel that I am having to basically reinvent viewstate in order to maintain my filter and the included options. As I am new to ASP.NET MVC but am very happy with classic ASP and ASP.NET Web Forms I thought I'd throw this out there for comment and guidance as to find a way which more closely fits with the MVC pattern.
It seems to me that the best way in to divide some actions which provide pure data for the jqGrid from other controller action. Such jqGrid-oriented actions can have prototype like:
JsonResult GetData(string filter, string sidx, string sord, int page, int rows)
I personally prefer to implement this part as WCF service and to have this WCF service as a part of the same ASP.NET site. In general it's much more the matter of taste and depends on your other project requirements.
This part of you ASP.NET site could implement users authentication which you need and can be tested with unit tests exactly like other actions of your controllers.
The views of the ASP.NET MVC site can have empty data for jqGrids, and have only correct URLs and probably generate the HTML code depends on the users permission in the site. Every page will fill the data of jqGrids with respect of the corresponds requests to the server (request to the corresponding GetData action).
You can use HTTP GET for the data for the best data caching. The caching of data is the subject of a separate discussion. If you do this, you should use prmNames: { nd:null } in the definition of jqGrid to remove unique nd parameter with the timestamp added per default to every GET request. To have full control of the data caching on the server side you can for example add in HTTP headers of the server responses both "Cache-Control" set to "max-age=0" and "ETag" header with the value calculated based of the data returned in the response. You should test whether the request from the client has "If-None-Match" HTTP header with the value of "ETag" coresponds the data cached on the client. Then you should verify whether the current data on the server (in the database) are changed and, if there are not changed, generate a response with an empty body (set SuppressEntityBody to true) and return "304 Not Modified" status code (HttpStatusCode.NotModified) instead of default "200 OK". A more detail explanation is much more longer.
If you don't want optimize you site for caching of HTTP GET data for jqGrids you can either use HTTP POST or don't use prmNames: { nd:null } parameter.
The code inside of JsonResult GetData(string filter, string sidx, string sord, int page, int rows) is not very short of cause. You should deserialise JSON data from the filter string and then construct the request to the data model depends on the method of the data access which you use (LINQ to SQL, Entity Model or SqlCommand with SqlDataReader). Because you have this part already implemented it has no sense to discuss this part.
Probably the main part of my suggestion is the usage of clear separation of controller actions which provide the data for all your jqGrids and the usage of MVC views with empty data (having only <table id="list"></table><div id="pager"></div>). You should also has no doubt with having a relative long code for analyzing of filters which come from the Advance Searching feature of the jqGrid and generating or the corresponding requests to your data model. Just implement it one time. In my implementation the code in also relatively complex, but it is already written one time, it works and it can be used for all new jqGrids.
I made this once, very simple.
pseudo code:
Controller
[HttpGet]
public ActionResult getList(int? id){
return PartialView("Index", new ListViewModel(id??0))
}
ViewModel
public class ListViewModel{
//ObjectAmountPerPage is the amount of object you want per page, you can modify this as //parameter so the user
//can choose the amount
public int ObjectAmountPerPage = 20 //you can make this into a variable of any sort, db/configfile/parameter
public List<YourObjectName> ObjectList;
public int CurrentPage;
public ListViewModel(id){
Currentpage = id;
using (MyDataContext db = new MyDataContext()){
ObjectList = db.YourObjectName.OrderBy(object=>object.somefield).getListFromStartIndexToEndIndex(id*ObjectAmountPerPage ,(id*ObjectAmountPerPage) +20).toList();
}
}
}
Now Create A RenderPartial:
PartialView
<#page inherit="IEnumerable<ListViewMode>">
<%foreach(YourObjectName object in Model.ObjectList){%>
Create a table with your fields
<%}%>
And create a view that implements your Jquery, other components+your partialView
View
<javascript>
$(function(){
$("#nextpage").click(function(){
(/controller/getlist/$("#nextpage").val(),function(data){$("#yourlist").html = data});
});
});
</javascript>
<div id="yourlist">
<%=Html.RenderPartial("YourPartialView", new ListViewModel())%>
</div>
<something id="nextpage" value"<%=Model.CurentPage+1%>">next page</something>
I hope this helps, this is according to the MVC- mv-mv-c principle ;)
Model-View -(modelview) - control