We are trying to store data from XML in Apache ignite cache, which is set up in a Kubernates cluster in Azure. We have a spring boot application, added ignite dependency, parsing the XML using jaxb to pojos and storing the data. But the ignite cache put operation is taking a lot of time. ( Because we are dealing with 500K - 1M lines of XML
As an alternative, we trying to use this REST API calls provide by ignite itself in our application. (https://apacheignite.readme.io/docs/rest-api)
My questions:
Does anybody have working example to do this in spring boot? Can it be done using Ignite SDK? (We know we can send get and put request using rest template in spring boot.)
Is it possible to send json response to this rest API? If so, How? Examples please.
Thanks for your response. Will help us a lot.
If your underlying problem is loading 1mm records is too slow, using the REST interface is not the solution. Instead, you should consider using the DataStreamer API, which is designed to ingest large amounts of data. You probably also don't need to convert your XML objects into POJO, but that's an optimisation you might not need right now.
Related
I have just started learning Kafka and I am trying to build a prototype to have a producer which is a REST API and send the data to Kafka consumer. I went through quite a lot of documentation to figure out some particular procedure.
I wasn't able to understand if there is a single connector that I could just use just like the fileconnector or the JDBC connectors provided for Apache Kafka. Should I be writing a custom connector for this ?
I am pretty much confused on where to start. I am particularly looking for some structured documentation or idea on how to get this done.
It sounds like you're talking about the functionality that already exists in the REST Proxy. This provides a REST API for producing data into Kafka, or consuming data from Kafka.
Edit: From your comment I understand your question to be different. If you want to pull data from a REST endpoint into Kafka you can use Kafka Connect and the kafka-connect-rest plugin. There's an example of it in use here.
There's no need to write a connector (besides, it's not possible with python). One already exists for HTTP.
https://github.com/llofberg/kafka-connect-rest
I have a set of simple views in my database that I want to expose via a REST API. I'd rather not write specific code to do this. Is there a framework or toolset that could allow me to generate the REST API either at build time or run-time? Java eco-system preferred.
From the Java world: Spring Data REST does that: https://spring.io/guides/gs/accessing-data-rest/
I've got a Python Django Backend running, and want to design a microservice. This microservices has a rest POST endpoint open. Now I want to transfer multiple binary files and some meta data (as json?) in a single POST request from Django to the microservice. What would be the besteht way to accomplish this.
I thought about transfering the data as multipart, but I think it's Mord for HTML forms. Also thought about protobuf. Would appreciate if you could help me, what's the most common way for such problems? What's the most efficient way?
It shouldn't be important, that I am using Django or Python for the answer.
The simplest solution is to use multipart/form-data. This is supported (as in builin) by all web application servers and almost web clients. It supports multiple arbitrary binary files and other data in the same request.
So my answer is KISS.
P.S. It is not only for forms.
Could anyone please help me to use the best caching techniques for REST API's using spring boot. Reason looking for caching is, when database goes down we should be able to retrieve the details from cache for few API's.
Given that one of the main benefits of Spring Data and the related REST repositories is that most of the time the developer doesn't have to worry about the underlying implementations, is there an out-of-the-box way to leverage the Spring Cloud Netflix libraries, specifically the Hystrix annotations in this case, without extending every call in the provided Repository interfaces or creating my own implementation?
Currently you need to wrap calls in another service whose methods are annotated with #HystrixCommand. Because of the way both Spring Data and the Hystrix Aspect work (they both create proxies), there would need to be specific integration in Spring Data for #HystrixCommand. #ccit-spence is right, you really want to put #HystrixCommand on the services calling into a Spring Data REST repository.