I've created an API in my controller to return Flux:
#GetMapping(value = "/orderStreams", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<OrderResponseDTO> getAllOrders() {
return orderService.getAll()
.map(order -> modelMapper.map(order, OrderResponseDTO.class));
}
Now there can be up to 100k order documents that this orderService can return as server set events.
My question is whether Flux takes up heap memory to hold all 100k documents in this process?
Related
I want to load only 100000 records which are in NOT_STARTED status in mongodb and want to process those records and update status to STARTED. I want to repeat this process until all the records which are in NOT_STARTED status processed.
Currently i am using Pagerequest as shown in the below code and it seems working. But is there a way i can do this without pagerequest having my repository extends spring MongoRepository. Because Pagerequest seems for pagination. But i am not doing any pagination only loading 100000 records each time and processing them
Sort sort = new Sort(Sort.Direction.ASC, "_id");
int count = (int) PaymentReportRepository.count();
for(int i = 0; i < count; i += reportProperties.getPageSize()) {
List<PaymentReport> paymentReportList =
MongoTraceability.capture(() ->
PaymentReportRepository.findByStatusAndDateLessThan("NOT_STARTED",
LocalDateTime.now().minusSeconds(reportProperties.getTimeInterval()),
,PageRequest.of(0, reportProperties.getPageSize(), sort)));
if (paymentReportList != null && !paymentReportList.isEmpty()) {
for (PaymentReport paymentReport : paymentReportList) {
messageService.processMessage(paymentReport);
}
}
}
It appears that you're processing each record synchronously. Do you have any desire/ability to process asynchronously?
Will this solution be run off a single JVM?
From your question I'm assuming synchronous processing and a single JVM.
I would use Spring's MongoTemplate class. Example tutorials/examples here: https://www.baeldung.com/queries-in-spring-data-mongodb
MongoTemplate will allow you to write your query along the lines of query("NOT_STARTED").limit(100000) to return the results you want. Assuming your messageService.processMessage(paymentReport); is doing an update() to the document after it is done processing and updates its status, then your next query will retrieve the next 100000 messages with your desired status.
You can try to rename findByStatusAndDateLessThan to findFirst100000ByStatusAndDateLessThan
I'm playing around with Springboot 2 with the webflux stack.
Within the application I'm looking to make multiple HTTP requests in parallel to downstream services to reduce the overall response time back to the client. Is this possible without playing around with threads?
I'm currently using org.springframework.web.reactive.function.client.WebClient but open to other clients that would support this; or even RXJava.
I managed to achieve it by something like below. It's a naive example but the async/http requests are made in downstream.request1() and downstream.request2(). If is a more elegant way to achieve this I'd be interested.
#GetMapping("/sample")
public Mono<String> getMultipleRequests() {
Mono<String> monoResponse1 = downstream.request1();
Mono<String> monoResponse2 = downstream.request2();
return Mono.zip(monoResponse1, monoResponse2)
.flatMap(a -> myTransform(a));
}
private Mono<String> myTransform(Tuple2<String, String> tuple) {
String t1 = tuple.getT1();
String t2 = tuple.getT2();
return Mono.just(t1 + t2);
}
I'm trying to load 20 million rows into partitioned stateful service ReliableDictionary. I partitioned stateful service into 10 partitions. Based on MSDN documentation, I understood that I need to use some hashing algorithm to find the correct partition and send data to it to load into IReliabledictionary. So I used the Hydra to get the partition number based on the value. All I'm storing is a List<long> in the IReliableDictionary.
So I created a Stateless service as wrapper,
which will fetch the rows from the SQL Server (20 million),
get the partition number using Hydra for each row,
group them by partition number
call the Stateful service for each partition using ServiceRemoting. However, I get fabric message too large exception if I send 1 million rows of data per each request so I chunked it into 100000 per request.
This is taking 74 minutes for it to complete. This is too long. Below is the code for uploading -
Please advise.
foreach (var itemKvp in ItemsDictionary)
{
var ulnv2Uri = new Uri("fabric:/TestApp/dataservice");
//Insert to the correct shard based on the hash algorithm
var dataService = _serviceProxyFactory.CreateServiceProxy<IDataService>(
dataStoreUri,
new ServicePartitionKey(itemKvp.Key), TargetReplicaSelector.PrimaryReplica, "dataServiceRemotingListener");
var itemsShard = itemKvp.Value;
//if the total records count is greater then 100000 then send it in chunks
if (itemsShard.Count > 1_000_000)
{
//var tasks = new List<Task>();
var totalCount = itemsShard.Count;
var pageSize = 100000;
var page = 1;
var skip = 0;
while (skip < totalCount)
{
await dataService.InsertData(itemsShard.Skip(skip).Take(pageSize).ToList());
page++;
skip = pageSize * (page - 1);
}
}
else
{
//otherwise send all together
await dataService.InsertData(itemsShard);
}
}
You can likely save some time here, by uploading to all partitions in parallel.
So create 10 service proxies (one for each partition) and use them simultaneously.
Let's say, I have P processes running some business logic on N physical machines. These processes call some web service S, say. I want to ensure that not more than X calls are made to the service S per second by all the P processes combined.
How can such a solution be implemented?
Google Guava's Rate Limiter works well for processes running on single box, but not in distributed setup.
Are there any standard, ready to use, solutions available for JAVA? [may be based on zookeeper]
Thanks!
Bucket4j is java implementation of "token-bucket" rate limiting algorithm. It works both locally and distributed(on top of JCache). For distributed use case you are free to choose any JCache implementation like Hazelcast or Apache Ignite. See this example of using Bucket4j in cluster.
I have been working on an opensource solution for these kind of problems.
Limitd is a "server" for limits. The limits are implemented using the Token Bucket Algorithm.
Basically you define limits in the service configuration like this:
buckets:
"request to service a":
per_minute: 10
"request to service b":
per_minute: 5
The service is run as a daemon listening on a TCP/IP port.
Then your application does something along these lines:
var limitd = new Limitd('limitd://my-limitd-address');
limitd.take('request to service a', 'app1' 1, function (err, result) {
if (result.conformant) {
console.log('everything is okay - this should be allowed');
} else {
console.error('too many calls to this thing');
}
});
We are currently using this for rate-limiting and debouncing some application events.
The server is on:
https://github.com/auth0/limitd
We are planning to work on several SDKs but for now we only have node.js and partially implemented go:
https://github.com/limitd
https://github.com/jdwyah/ratelimit-java provides distributed rate limits that should do just this. You can configure your limit as S per second / minute etc and choose burst size / refill rate of the leaky bucket that is under the covers.
Simple rate limiting in java where you want to achieve a concurrency of 3 transactions every 3 seconds. If you want to centralize this then either store the tokens array in elasticache or any database. And in place of synchronized block you will have to implement a lock flag as well.
import java.util.Date;
public class RateLimiter implements Runnable {
private long[] tokens = new long[3];
public static void main(String[] args) {
// TODO Auto-generated method stub
RateLimiter rateLimiter = new RateLimiter();
for (int i=0; i<20; i++) {
Thread thread = new Thread(rateLimiter,"Thread-"+i );
thread.start();
}
}
#Override
public void run() {
// TODO Auto-generated method stub
long currentStartTime = System.currentTimeMillis();
while(true) {
if(System.currentTimeMillis() - currentStartTime > 100000 ) {
throw new RuntimeException("timed out");
}else {
if(getToken()) {
System.out.println(Thread.currentThread().getName() +
" at " +
new Date(System.currentTimeMillis()) + " says hello");
break;
}
}
}
}
synchronized private boolean getToken() {
// TODO Auto-generated method stub
for (int i = 0; i< 3; i++) {
if(tokens[i] == 0 || System.currentTimeMillis() - tokens[i] > 3000) {
tokens[i] = System.currentTimeMillis();
return true;
}
}
return false;
}
}
So with all distributed rate limiting architecture, you'll need a single backend store that acts as single source of true to track the number of requests. You can always use zookeeper as a in memory datastore for this out of convenience, although there are better choices such as Redis.
The application is a Windows Service that is retrieving a Generic List of my entity and then based on the data returned, I am running other tasks. This data is read-only. There will be no requirement to update the data in the backend store (Oracle 11g).
Why do I believe I have a memory leak?
Despite disposing the context via a using statement, it doesn't seem the GC is harvesting these objects. While profiling the application, I have noticed that the Heap memory gradually increases throughout the lifetime of the application. I also notice more than a few GEN 2 Collections, which of course means objects are existing for quite some time before being collected.
This doesn't occur when I just return a manually generated collection of objects. In that case I see very few GEN 2 collections (good) and the Heap memory doesn't increase gradually.
I am running Entity Framework 5.0/C# 4.5. I am accessing an Oracle database. I am using the database first approach.
Besides disposing the context immediately, what else can I do to ensure that the GC will collect the context?
public IEnumerable<EventServer> GetRegionalEventServers()
{
IEnumerable<EventServer> eventServers = null;
using (AssetEntities context = new AssetEntities())
{
context.Configuration.AutoDetectChangesEnabled = false;
context.Configuration.LazyLoadingEnabled = false;
context.Configuration.ProxyCreationEnabled = false;
try
{
eventServers = from p in context.EVENTSERVERS
select new EventServer
{
ServerName = p.SERVER_NAME,
Domain = p.DOMAIN,
ServerLocation = p.SERVER_LOCATION
};
return eventServers;
}
finally
{
eventServers = null;
}
}
}