I have Express server with TypeORM and I use Row Security Policies https://www.postgresql.org/docs/current/ddl-rowsecurity.html. I have issue with postgres configuration settings, because postgres uses connection pool under the hood and when I set configuration values on default connection I couldn't be sure it persist during life of all calls. I have considered few approaches on implement row security policies in TypeORM:
Avallone library https://github.com/Avallone-io/rls
Own implementation
EventSubscribers https://orkhan.gitbook.io/typeorm/docs/listeners-and-subscribers
Avallone has too many new dependencies.
EventSubscribers seems to be one of the approaches with less code except I really depend on using something like #BeforeLoad (as we have https://orkhan.gitbook.io/typeorm/docs/listeners-and-subscribers#afterload). And my question is how can we achieve this?
I've seen the similar discussion here Postgres SET runtime variables with TypeORM, how to persist variable during the life of the connection between calls
Related
I'd like to call a custom EVAL function from within postgres to a redis instance (so I'm not sure if redis_fdw will work in this case.)
The other option is to use plpython with a redis library
https://pypi.org/project/redis/.
import redis
r = redis.Redis(host='localhost', port=6379, db=0)
r.evalsha(<SHA>, <ARGS>)
I'd like to be able to create a redis pool so that each call to redis doesn't have to reestablish a connection. Is there any way to set some sort of global state within postgres itself so that a pool can be initiated on the first call and then subsequent calls just use the pool?
redis_fdw only mentions support for a handful of built-in data structures, so it doesn't look like much help for an EVAL.
A global connection pool will probably not be easy. Postgres runs each connection in its own server process, and you can't pass anything between them without allocating it in shared memory (which I don't see a PL/Python API for, so I think it'd need to be done in C).
But if you can afford to create one Redis connection per Postgres connection, you should be able to reuse it across PL/Python calls by putting it in a shared dictionary:
The global dictionary SD is available to store private data between repeated calls to the same function. The global dictionary GD is public data, that is available to all Python functions within a session; use with care.
So, something like
if 'redis' not in SD:
SD['redis'] = redis.Redis(host='localhost', port=6379, db=0)
SD['redis'].evalsha(<SHA>, <ARGS>)
There are already similar questions about this error and suggested solutions; e.g. increasing max_connections in postgresql.conf and / or adapting the max number of connections your app requests. However, my question is more specific to using jOOQ in a Spring Boot application.
I integrated jOOQ into my application as in the example on GitHub. Namely, I am using DataSourceConnectionProvider with TransactionAwareDataSourceProxy to handle database connections, and I inject the DSLContext in the classes that need it.
My application provides various web services to front-ends and I've never encountered that PSQLException on dev or test environments so far. I only started getting that error when running all integration tests (around 1000) locally. I don't expect some leak in handling the connection as Spring and jOOQ manage the resources; nevertheless that error got me worried if that would also happen on production.
Long story short, is there a better alternative to using DataSourceConnectionProvider to manage connections? Note that I already tried using DefaultConnectionProvider as well, and tried to make spring.datasource.max-active less than max_connections allowed by Postgres. Neither fixed my problem so far.
Since your question seems not to be about the generally best way to work with PostgreSQL connections / data sources, I'll answer the part about jOOQ and using its DataSourceConnectionProvider:
Using DataSourceConnectionProvider
There is no better alternative in general. In order to understand DataSourceConnectionProvider (the implementation), you have to understand ConnectionProvider (its specification). It is an SPI that jOOQ uses for two things:
to acquire() a connection prior to running a statement or a transaction
to release() a connection after running a statement (and possibly, fetching results) or a transaction
The DataSourceConnectionProvider does so by acquiring a connection from your DataSource through DataSource.getConnection() and by releasing it through Connection.close(). This is the most common way to interact with data sources, in order to let the DataSource implementation handle transaction and/or pooling semantics.
Whether this is a good idea in your case may depend on individual configurations that you have made. It generally is a good idea because you usually don't want to manually manage connection lifecycles.
Using DefaultConnectionProvider
This can certainly be done instead, in case of which jOOQ does not close() your connection for you, you'll do that yourself. I'm expecting this to have no effect in your particular case, as you'll implement the DataSourceConnectionProvider semantics manually using e.g.
try (Connection c = ds.getConnection()) {
// Implicitly using a DefaultConnectionProvider
DSL.using(c).select(...).fetch();
// Implicit call to c.close()
}
In other words: this is likely not a problem related to jOOQ, but to your data source.
Thanks in advance. We are trying to adopt drools as rules engine in our enterprise. After evaluating basic functionality in POC mode, we are exploring further. We have the following challenges and I am trying to validate some of the options we are considering. Any help is greatly appreciated.
Scenario-1: Say you get USA state (TX,CA,CO etc) in a fact's field. Now you want the rule to check if the 'state value on the fact' exists in a predetermined static list of state values(say the list contains three values TX,TN,MN).
Possible solution to Scenario-1: 'static list of state values' can be set as a global variable and the rule can access the global variable while performing the check.
Qustions on Scenario-1:
Is the 'possible solution to scenario-1' the standard practice? If so, is it possible to load the value of this global variable from a database during rule engine(KIE Server) startup? If yes, could you let me know the drools feature that enables us to load global variables from a database? Should a client application (a client application that calls kie-server) initialize global variables instead?
Scenario-2: We want to horizantally scale rule execution sever. Say we have one rule engine server(kie-server) exposing rest-api. Can we have multiple instances running behind a loadbalancer to have it scale horizontally? Is there any other way of achieving the scalability?
Q1: It depends. The usual solution for a small, rarely (if ever) changing set that is used just in a single rule is to put it into the rule, using the in operator. If you think you might have to change it or use it frequently, a global would be one way of achieving that but you must make sure that the global is initialized before any facts are inserted.
There is nothing out-of-the-box for accessing a DB.
Q2: A server running a Drools session is just another Java server program, so any load balancing applicable to this class of programs should apply to a Drools app as well. What are you fearing?
MongoDB 2.6 introduced the .maxTimeMS() cursor method, which allows you to specify a max running time for each query. This is awesome for ad-hoc queries, but I wondered if there was a way to set this value on a per-instance or per-database (or even per-collection) level, to try and prevent locking in general.
And if so, could that value then be OVERWRITTEN on a per-query basis? I would love to set an instance level timeout of 3000ms or thereabouts (since that would be a pretty extreme running time for queries issued by my application), but then be able to ignore it if I had a report to run.
Here's the documentation from mongodb.org, for reference: http://docs.mongodb.org/manual/reference/method/cursor.maxTimeMS/#behaviors
Jason,
Currently MongoDB does not support a global / universal "maxTimeMS". At the moment this option is applicable to the individual operations only. If you wish to have such a global setting available in MongoDB, I would suggest raising a SERVER ticket at https://jira.mongodb.org/browse/SERVER along with use-cases that can take advantage of such setting.
I know this is old but came here with a similar question and decided to post my findings, The timeout, as a global parameter, is supported by the drivers as part of the connection string, which makes sense because it's the driver the one that can control this global parameters, here you can find documentation about this: https://docs.mongodb.com/manual/reference/connection-string/, but each driver can handle this slightly different (c# for example uses Mongo Settings parameter for this, python has it as parameters in the init constructor, etc)
To test this you can start a test server using mtools like this:
mlaunch --replicaset --name testrepl --nodes 3 --port 27000
Then an example in python will be like:
from pymongo import MongoClient
c = MongoClient(host="mongodb://localhost:27000,localhost:27001,localhost:27002/?replicaSet=testrepl&wtimeoutMS=2000&w=3")
c.test_database.col.insert({ "name": "test" })
I'm using the URI method so this can be used in other drivers, but Python also supports the parameters w and wtimeout, in this example all the write operations will be defaulted to 2 segs and 3 nodes have to be confirmed before returning, if you restart the database and use the wtimeout of 1 (meaning 1 ms) you will see an exception because the replication will take a bit longer to initialize the first time you execute the python script.
Hope this helps others coming with the same question.
Is it possible to create namespaces in Redis?
From what I found, all the global commands (count, delete all) work on all the objects. Is there a way to create sub-spaces such that these commands will be limited in context?
I don't want to set up different Redis servers for this purpose.
I assume the answer is "No", and wonder why wasn't this implemented, as it seems to be a useful feature without too much overhead.
A Redis server can handle multiple databases... which are numbered. I think it provides 32 of them by default; you can access them using the -n option to the redis-cli shell scripting command and by similar options to the connection arguments or using the "select()" method on its connection objects. (In this case .select() is the method name for the Python Redis module ... I presume it's named similarly for other libraries and interfaces.
There's an option to control how many separate databases you want in the configuration file for the Redis server daemon as well. I don't know what the upper limit would be and there doesn't seem to be a way to dynamically change that (in other words it seems that you'd have to shutdown and restart the server to add additional DBs). Also, there doesn't seem to be an away to associate these DB numbers with any sort of name nor to impose separate ACLS, nor even different passwords, to them. Redis, of course, is schema-less as well.
If you are using Node, ioredis has transparent key prefixing, which works by having the client prepend a given string to each key in a command. It works in the same way that Ruby's redis-namespace does. This client-side approach still puts all your keys into the same database, but at least you add some structure, and you don't have to use multiple databases or servers.
var fooRedis = new Redis({ keyPrefix: 'foo:' });
fooRedis.set('bar', 'baz'); // Actually sends SET foo:bar baz
If you use Ruby you can look at these gems:
https://github.com/resque/redis-namespace
https://github.com/jodosha/redis-store