How can I tell if a kdb server is busy? - kdb

Is there a command to know if the kdb server is busy running a query? Even better, knowing what is the percentage completion of the query being run?
So far I've been looking at the top screen on linux to know which server to use...

Unfortunately, not directly. The reason is due to the single threaded nature of a KDB process. In practice, this is easily worked around by adding some basic logging to your server. So whenever a query comes in just log to a file the time the query came in and when the result was returned to the user.
Take a look at the .z.pg and the .z.ps functions which are called to handle synchronous or asynchronous requests, respectively. By default they are just set to "value", which means evaluate the string and return the result. Just replace this with your own function to log events to a file or a log server.

Besides above solution, a more simple way is: keep checking the port.
Normally all queries will be running against port, and kdb server can launched multiple ports for different purpose.
Details:
Use below code to query again port, if the port is busy, null res will return. And you can further kill the port and restart it or whatever the requirement is.
The code will send out 1 to the port and calculate.
.server.testQuery:{[inPort]
res:#[{hopen(x;3000)};`$":",":" sv string `,inPort;0N];
if[not null res;hclose res];
:res
};

Related

How to perform an "I can reach my database" healthcheck?

I have a classic spray+slick http server which is my database access layer, and I'd like to be able to have an healthcheck route to ensure my server is still able to reach my DB.
I could do it by doing a generic sql query, but I was wondering if there was a better way to just check the connection is alive and usable without actually adding load on the database (or at least the minimum possible load).
So pretty much :
val db = Database.forConfig("app.mydb")
[...]
db.???? // Do the check here
Why do you want to avoid executing a query against the database?
I think the best health check is to actually use the database as your application would (actually connecting and running a query). With that in mind, you can perform a SELECT 1 against your DB, and verify that it responds accordingly.

LISTEN on all channels in PostgreSQL

I'd like to forward all notifications from PostgreSQL into task queues in RabbitMQ named the same as the channel given in NOTIFY channel. Does PostgreSQL have something that would act like LISTEN *?
Inspecting the source for Skeeter it seems that PQnotifies might be of interest. PostgreSQL's documentation on libpq also mentions PQconsumeInput as a way to consume input from the server. From the documentation:
PQconsumeInput normally returns 1 indicating "no error", but returns 0 if there was some kind of trouble (in which case PQerrorMessage can be consulted). Note that the result does not say whether any input data was actually collected. After calling PQconsumeInput, the application can check PQisBusy and/or PQnotifies to see if their state has changed.
Am I on the right path? Since I'm using .NET I'd prefer not writing any C, so any suggestions are welcome.
I've tried pgsql-listen-exchange but either I'm doing something wrong or the plugin doesn't work for RabbitMQ 3.6 (there's only a 3.5 release). I created an issue.
Specific to RabbitMQ: As an alternative to listening for everything from PostgreSQL, I guess I could create an exchange and have something poll that for queues and just create a listener for each queue. Will be looking into this as well.

Query URL without redirect in Go

I am writing a benchmark test for a redirect script.
I wisg my program to query certain URL that redirects to AppStore. But I do not wish to download AppStore page. I just wish to log redirect URL or error.
How do I tell Go to query URL without second redirect query?
UPDATE
Both answers are correct BUT:
I tried both solutions. I am doing benchmarking.
I run 1 or many go processes with 10 - 500 go routines. They query URL in a loop.
My server is also written in go. It reports number of requests every second.
First solution: http.DefaultTransport.RoundTrip - works slow, gives errors.
First 4 seconds works fine. Making 300-500 queries then performance drops to 80 query per second.
Then drops to 0-5 query per second and queryies script start getting errors like
dial tcp IP:80: A connection attempt failed because the connected
party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond.
I guess it re-use connection that is closed.
Second solution: CheckRedirect field works with constant performance. I am not sure if it re-uses connections or it opens a new connection for every request. I create client for every request in a loop. It is how it will behave in a real life (every request is a new connection). Is there way to ensure that connections are closed after each query and not re-used?
That is why I am going to mark second solution as such that answer my question. But for my research it is very important that each query was a new connection. How can I ensure with second solution?
You need to use an http.Transport instead of an http.Client. Transport is lower-level and does not follow redirects.
req, err := http.NewRequest("GET", "http://example.com/redirectToAppStore", nil)
// ...
resp, err := http.DefaultTransport.RoundTrip(req)
For completeness' sake, you can use an http.Client and not follow redirects. http.Client has a CheckRedirect field which is a function. It is called before following any redirection.
If this function returns an error, then httpClient.Do(...) will not follow the redirect (see doFollowingRedirects() function in Go's source code) and instead will return an error (its concrete type will be url.Error, and its URL field will be the redirect-to URL, aka the Location header value, see this code).
You can see my gocrawl library for a concrete example of this use.

'socketTimeoutMS' parameter in the connection string

I'm using Mongo server version 2.0.6
And the C# driver DLL version is: 1.3.1.4349
I'm using this connection string:
mongodb://a.b.c.d:27017,e.f.g.h:27017/abcd?connectTimeoutMS=30000;socketTimeoutMS=120000
The issue is that the flag socketTimeoutMS might not be honored, I think.
Because if I set it to 1ms, then most of my queries should fail, right?
I hope I've understood this parameter correctly.
Can anyone explain what might be going on?
The socket timeout parameter is used to time out sockets that are waiting to read or write data. If your server accepts writes and responds with data for reads within 1ms your query will not fail. Also it depends on the underlying OS if such a low timeout is actually honored. It might cap it.
Relevant code :
reads : https://github.com/mongodb/mongo-csharp-driver/blob/8e6850c91893743ebbbd53ebba84d3d4086cdecb/Driver/Internal/MongoConnection.cs#L322-L341
writes : https://github.com/mongodb/mongo-csharp-driver/blob/8e6850c91893743ebbbd53ebba84d3d4086cdecb/Driver/Internal/MongoConnection.cs#L374-L382

MSMQ querying for a specific message

I have a questing regarding MSMQ...
I designed an async arhitecture like this:
CLient - > WCF Service (hosted in WinService) -> MSMQ
so basically the WCF service takes the requests, processes them, adds them to an INPUT queue and returns a GUID. The same WCF service (through a listener) takes first message from queue (does some stuff...) and then it puts it back into another queue (OUTPUT).
The problem is how can I retrieve the result from the OUTPUT queue when a client requests it... because MSMQ does not allow random access to it's messages and the only solution would be to iterate through all messages and push them back in until I find the exact one I need. I do not want to use DB for this OUTPUT queue, because of some limitations imposed by the client...
You can look in your Output-Queue for your message by using
var mq = new MessageQueue(outputQueueName);
mq.PeekById(yourId);
Receiving by Id:
mq.ReceiveById(yourId);
A queue is inherently a "first-in-first-out" kind of data structure, while what you want is a "random access" data structure. It's just not designed for what you're trying to achieve here, so there isn't any "clean" way of doing this. Even if there was a way, it would be a hack.
If you elaborate on the limitations imposed by the client perhaps there might be other alternatives. Why don't you want to use a DB? Can you use a local SQLite DB, perhaps, or even an in-memory one?
Edit: If you have a client dictating implementation details to their own detriment then there are really only three ways you can go:
Work around them. In this case, that could involve using a SQLite DB - it's just a file and the client probably wouldn't even think of it as a "database".
Probe deeper and find out just what the underlying issue is, ie. why don't they want to use a DB? What are their real concerns and underlying assumptions?
Accept a poor solution and explain to the client that this is due to their own restriction. This is never nice and never easy, so it's really a last resort.
You may could use CorrelationId and set it when you send the message. Then, to receive the same message you can pick the specific message with ReceiveByCorrelationId as follow:
message = queue.ReceiveByCorrelationId(correlationId);
Moreover, CorrelationId is a string with the following format:
Guid()\\Number