I'd like to forward all notifications from PostgreSQL into task queues in RabbitMQ named the same as the channel given in NOTIFY channel. Does PostgreSQL have something that would act like LISTEN *?
Inspecting the source for Skeeter it seems that PQnotifies might be of interest. PostgreSQL's documentation on libpq also mentions PQconsumeInput as a way to consume input from the server. From the documentation:
PQconsumeInput normally returns 1 indicating "no error", but returns 0 if there was some kind of trouble (in which case PQerrorMessage can be consulted). Note that the result does not say whether any input data was actually collected. After calling PQconsumeInput, the application can check PQisBusy and/or PQnotifies to see if their state has changed.
Am I on the right path? Since I'm using .NET I'd prefer not writing any C, so any suggestions are welcome.
I've tried pgsql-listen-exchange but either I'm doing something wrong or the plugin doesn't work for RabbitMQ 3.6 (there's only a 3.5 release). I created an issue.
Specific to RabbitMQ: As an alternative to listening for everything from PostgreSQL, I guess I could create an exchange and have something poll that for queues and just create a listener for each queue. Will be looking into this as well.
Related
I am struggling to read my Postgres NOTICE messages in my C++ API. I can only read EXCEPTIONmessages using the function PQresultErrorMessage(PGresult), but not lower level messages.
PQresultErrorField(res, PG_DIAG_SEVERITY) returns null pointer.
How do I read NOTICE and other low level messages?
(Using PostgreSQL 9.2)
Set up a notice receiver or notice processor using PQsetNoticeReceiver / PQsetNoticeProcessor. Both set up callbacks that are invoked when asynchronous notifications are received. Note that this may happen before, during, or after processing of query data.
It's safe to assume that after all query results are returned (PQexec or whatever has returned) and you've called PQconsumeInput to make sure there's nothing else waiting, then all notices for the last command are received. The PQconsumeInput shouldn't really be necessary, it's just to be cautious.
See the documentation for libpq.
Is there a command to know if the kdb server is busy running a query? Even better, knowing what is the percentage completion of the query being run?
So far I've been looking at the top screen on linux to know which server to use...
Unfortunately, not directly. The reason is due to the single threaded nature of a KDB process. In practice, this is easily worked around by adding some basic logging to your server. So whenever a query comes in just log to a file the time the query came in and when the result was returned to the user.
Take a look at the .z.pg and the .z.ps functions which are called to handle synchronous or asynchronous requests, respectively. By default they are just set to "value", which means evaluate the string and return the result. Just replace this with your own function to log events to a file or a log server.
Besides above solution, a more simple way is: keep checking the port.
Normally all queries will be running against port, and kdb server can launched multiple ports for different purpose.
Details:
Use below code to query again port, if the port is busy, null res will return. And you can further kill the port and restart it or whatever the requirement is.
The code will send out 1 to the port and calculate.
.server.testQuery:{[inPort]
res:#[{hopen(x;3000)};`$":",":" sv string `,inPort;0N];
if[not null res;hclose res];
:res
};
I'm looking for a data store that has a "get or block" operation. This operation would return the value associated with a key/query if that value exists or block until that value is created.
It's like a pub/sub message queue but with a memory to handle the case when the subscriber connects after the publisher has published the result.
This operation allows unrelated processes to rendezvous with each other, and it seems that it would be a very useful architectural building block to have - especially in a web environment - i.e. a web request comes in which kicks off a backend server process to do some work and the web client can get the results via a future AJAX call.
Here is an blog post I found on how to accomplish this sort of operation with mongodb:
http://blog.mongodb.org/post/29495793738/pub-sub-with-mongodb
What other solutions are in use today? Can I do the same thing with redis or rabbitmq? I've looked at the docs for both, but it's unclear exactly how it would work. Should I roll my own server with 0MQ? Is there something out there specifically tailored for this problem?
Your are correct both Redis[1] and rabbitmq[2] have pub/sub capabilities.
[1] http://redis.io/topics/pubsub
[2] http://www.rabbitmq.com/tutorials/tutorial-three-python.html
I just installed memcache in my machine.Actually,i have to create regions in there.
For eg there should be 3 regions created ,each storing a set of data.Am not sure how can i do that in memcache.Can anyone please help/give eg as it is "urgent".
Thanks in advance
Memcached itself is just a service that runs on your server. It can be connected to and commands sent to it over a text based protocol. Once the service is running, you can use a tool such as telnet or netcat to interact with it.
As for accessing from Java, you'll probably want a library to do most of the work for you. There were a few listed on this question: Java Memcached Client
Now as for your regions: memchached is basically a key/value table. To set a region you'd do something like memcached.set("key", yourData) and to get it back you'd do something like yourData = memcached.get("key")
Note that these functions will vary depending which library you are using.
I have a questing regarding MSMQ...
I designed an async arhitecture like this:
CLient - > WCF Service (hosted in WinService) -> MSMQ
so basically the WCF service takes the requests, processes them, adds them to an INPUT queue and returns a GUID. The same WCF service (through a listener) takes first message from queue (does some stuff...) and then it puts it back into another queue (OUTPUT).
The problem is how can I retrieve the result from the OUTPUT queue when a client requests it... because MSMQ does not allow random access to it's messages and the only solution would be to iterate through all messages and push them back in until I find the exact one I need. I do not want to use DB for this OUTPUT queue, because of some limitations imposed by the client...
You can look in your Output-Queue for your message by using
var mq = new MessageQueue(outputQueueName);
mq.PeekById(yourId);
Receiving by Id:
mq.ReceiveById(yourId);
A queue is inherently a "first-in-first-out" kind of data structure, while what you want is a "random access" data structure. It's just not designed for what you're trying to achieve here, so there isn't any "clean" way of doing this. Even if there was a way, it would be a hack.
If you elaborate on the limitations imposed by the client perhaps there might be other alternatives. Why don't you want to use a DB? Can you use a local SQLite DB, perhaps, or even an in-memory one?
Edit: If you have a client dictating implementation details to their own detriment then there are really only three ways you can go:
Work around them. In this case, that could involve using a SQLite DB - it's just a file and the client probably wouldn't even think of it as a "database".
Probe deeper and find out just what the underlying issue is, ie. why don't they want to use a DB? What are their real concerns and underlying assumptions?
Accept a poor solution and explain to the client that this is due to their own restriction. This is never nice and never easy, so it's really a last resort.
You may could use CorrelationId and set it when you send the message. Then, to receive the same message you can pick the specific message with ReceiveByCorrelationId as follow:
message = queue.ReceiveByCorrelationId(correlationId);
Moreover, CorrelationId is a string with the following format:
Guid()\\Number