I am trying to create a REST application over Apache geode. Application works well in case of limited data, but in cases when I need to get the complete data ( ~0.8M ), it fails with an OOM exception on server.
Exception :
HTTP GET Error: 500
REST OQL Response: {"cause":"Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space"}
I tried the same approach with cache client, it works seamlessly, but we need to use REST to integrate with our application.
Any ideas to go around with this?
I am thinking on the following approaches.
Can we break the data on server side and use something like "Range" with Apache Geode? I tried this quickly, but did not work well.
Can we start getting the data into smaller buffers at the client side and start reading buffer by buffer?
Is it possible to get data out from Geode as a data-stream?
Thanks,
Abhay
Would it be possible for your to share the stack trace for the OOM you are getting? Are you saying the results are 0.8 megabytes? That doesn't seem like it should cause on OOME.
You can get ranges of data using OQL queries, but if your data set is really that small it seems like something else strange is going on with the REST API.
Related
I am using GraphQLFederationModule in my nestjs project and now working on the exceptions handling part. I use formatError to catch all the exceptions thrown in resolvers.
Basically I have 4 kinds of source of exception:
Error in resolver/service logic that not related to other services.
Error from database, which is mongoDb (using nestjs/mongoose).
Error when calling api from other graphql servers using graphql-request.
Error when calling api from other restful api servers using axios.
For each kind of exceptions I need to handle it differently. So the question is, how can I determine which kind of exception it is from the error passed into formatError ?
Thank you.
I cannot find a "good" way to solve this so I just determine them by the content:
https://github.com/VictorCheng114668/nestjs-helper/blob/main/graphqlModule-formatError-helper/exceptionsHandler.ts
I have a confusing problem when I make several endpoints in spring boot to get data from mongodb
mongodb version 3.6.8 with 3 cluster
I have tried to request each endpoint, and that response is normal, like this:
myhost:123/getId
{"id":"123"}
myhost:123/getName
{"name":"myname"}
When trying to do lots of requests and simultaneous. My query is normal, but sometimes the response becomes strange. like this:
myhost:123/getId
{"name":"myname"}
or
myhost:123/getName
{"id":"123"}
Thats will throw an exception, because key not found. I don't know where the wrong is, because it doesn't return an error. Only returns different results.
I'm trying to use a custom DbExecutionStrategy that implements retries, but I'm getting the following error:
An exception of type 'System.InvalidOperationException' occurred in EntityFramework.dll but was not handled in user code
Additional information: Streaming queries are not supported by the configured execution strategy 'MyDbExecutionStrategy'. See http://go.microsoft.com/fwlink/?LinkId=309381 for additional information.
If you follow that link it will explain that EF6 doesn't use streaming by default but that you can turn it on by using AsStreaming().
By default, EF6 and later version will buffer query results rather than streaming them. If you want to have results streamed you can use the AsStreaming method to change a LINQ to Entities query to streaming.
However, the call that is being made does not use streaming and I don't have any code that calls AsStreaming().
dataModel.Set<DeploymentLog>().OrderByDescending(d => d.DeploymentTimestamp).FirstOrDefault()
I've copied the code and the strategy to a different command line app that I created to test it and it works fine there. But in my web application I'm getting that error.
Any ideas why? Is there some kind of setting to turn on streaming on all queries?
I am working on an application with ASP.Net/MVC/EF/Breeze/Angular and it is working in my development environment. I just deployed it to my test environment and I am getting an error when I try to query breeze entity data. I don't think it's a DB connection problem, because I can query the breeze metadata successfully.
However, when I try to query for entity data, for example:
http://server/path/breeze/data/Cities
I get the following error:
{"$id":"1","$type":"System.Web.Http.HttpError, System.Web.Http","Message":"An error has occurred."}
This shows up as a 500 error in the browser network history. Is there any way to get Breeze to return a more detailed error message? I am not getting this error in development so I can't debug it there. Or any suggestions as to what I should look for if I can get metadata but not data?
Well, it turns out it was a connection string problem! I have no idea how breeze was generating metadata without a DB connection. Maybe it caches it in a local file?
Anyways, it would be nice to be able to get more detailed messages in cases like this. Since breeze returns IQueryable<T> I couldn't see any obvious way to catch the exception and send back a message myself...
I have a questing regarding MSMQ...
I designed an async arhitecture like this:
CLient - > WCF Service (hosted in WinService) -> MSMQ
so basically the WCF service takes the requests, processes them, adds them to an INPUT queue and returns a GUID. The same WCF service (through a listener) takes first message from queue (does some stuff...) and then it puts it back into another queue (OUTPUT).
The problem is how can I retrieve the result from the OUTPUT queue when a client requests it... because MSMQ does not allow random access to it's messages and the only solution would be to iterate through all messages and push them back in until I find the exact one I need. I do not want to use DB for this OUTPUT queue, because of some limitations imposed by the client...
You can look in your Output-Queue for your message by using
var mq = new MessageQueue(outputQueueName);
mq.PeekById(yourId);
Receiving by Id:
mq.ReceiveById(yourId);
A queue is inherently a "first-in-first-out" kind of data structure, while what you want is a "random access" data structure. It's just not designed for what you're trying to achieve here, so there isn't any "clean" way of doing this. Even if there was a way, it would be a hack.
If you elaborate on the limitations imposed by the client perhaps there might be other alternatives. Why don't you want to use a DB? Can you use a local SQLite DB, perhaps, or even an in-memory one?
Edit: If you have a client dictating implementation details to their own detriment then there are really only three ways you can go:
Work around them. In this case, that could involve using a SQLite DB - it's just a file and the client probably wouldn't even think of it as a "database".
Probe deeper and find out just what the underlying issue is, ie. why don't they want to use a DB? What are their real concerns and underlying assumptions?
Accept a poor solution and explain to the client that this is due to their own restriction. This is never nice and never easy, so it's really a last resort.
You may could use CorrelationId and set it when you send the message. Then, to receive the same message you can pick the specific message with ReceiveByCorrelationId as follow:
message = queue.ReceiveByCorrelationId(correlationId);
Moreover, CorrelationId is a string with the following format:
Guid()\\Number