I am using golang in my application server and gorm as the ORM. I am using postgresql as the database in google cloud sql.
I created a 2 read replica's for postgres which are being used by the application server.
Previously, I used node.js and sequelize and there, I am able to define the read replicas as
read: [
{ host: '8.8.8.8', username: 'anotherusernamethanroot', password: 'lolcats!' },
{ host: 'localhost', username: 'root', password: null }
],
write: { host: 'localhost', username: 'root', password: null }
},
However for gorm, I dont see any way to do that(in the documentation).
So, is there a way that I can define read replicas and gorm takes care of it. If not, what is the best practice for this use case?
Now that Gorm V2 is out you can use the dbresolver plugin just for this use case. Replicating what you gave as an example would look like:
import (
"gorm.io/gorm"
"gorm.io/plugin/dbresolver"
"gorm.io/driver/postgres"
)
db, err := gorm.Open(postgres.Open("host=localhost user=root"), &gorm.Config{})
db.Use(dbresolver.Register(dbresolver.Config{
Replicas: []gorm.Dialector{
postgres.Open("host=8.8.8.8 user=anotherusernamethanroot password=lolcats!"),
postgres.Open("host=localhost user=root"),
},
Policy: dbresolver.RandomPolicy{},
})
Check out the documentation: https://gorm.io/docs/dbresolver.html
Related
I'm front dev and I need to test locally my front app with backend (nest js) and postgresql DB. Who can write me the right way How to run and connect to DB ? I get some errors on app start. I work on win 10 and there is my steps for start this app.
install postgresql
npm install for my nest js app
run pgAdmin4 and create DB for my app
npm start
There is my ormconfig
module.exports = {
"type": "postgres",
"host": process.env.POSTGRES_HOST || "localhost",
"port": process.env.POSTGRES_PORT || 5432,
"username": process.env.POSTGRES_USER || "", //<- Here I try to set all possible username
"password": process.env.POSTGRES_PASSWORD || "", //<- Here I try to set all possible password
"database": process.env.POSTGRES_DB || "my_database",
"entities": ["dist/**/*.entity{.ts,.js}"],
"synchronize": true,
"logging": true
}
There is error that I encountered
error
Also on other computer I try to do this and I get error like
[Nest] ERROR [TypeOrmModule] Unable to connect to the database.
FATAL: password authentication failed for user "postgres" (postgresql 14 with pgAdmin 4)
In your typeormconfig.ts , you should write this:
export class PostgresTypeormConfiguration implements TypeOrmOptionsFactory
{
createTypeOrmOptions(connectionName?: string): TypeOrmModuleOptions | Promise<TypeOrmModuleOptions> {
const TypeOrmOptions:TypeOrmModuleOptions=
{
type: "postgres",
host: process.env.POSTGRES_HOST ,
port: process.env.POSTGRES_PORT ,
username: process.env.POSTGRES_USER ,
password: process.env.POSTGRES_PASSWORD ,
database: process.env.POSTGRES_DB,
entities: ["dist/**/*.entity{.ts,.js}"],
synchronize: true,
logging: true
}
return TypeOrmOptions
}
}
and you should define this in your module like this:
#Module({
imports:[TypeOrmModule.forRootAsync({useClass:PostgresTypeormConfiguration})]
})
note: if you still got an error , you wrote one of the config option wrong in your .env file or you did not define .env file in your configModule
In GitHub actions, I am running a JavaScript file which connects to PostgreSQL and creates the table and extension for the database.
my script looks like this:
const { Client } = require('pg')
const pgclient = new Client({
host: process.env.POSTGRES_HOST,
port: process.env.POSTGRES_PORT,
user: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASSWORD,
database: process.env.POSTGRES_DB,
})
pgclient.connect()
const createDB = `
drop database mydb;
create database mydb;
\c mydb;
CREATE EXTENSION "pgcrypto";
`
pgclient.query(createDB, (err, res) => {
if (err) throw err
pgclient.end()
})
When I run the script, I get an error
error: syntax error at or near "c"
Which I am guessing is coming from \c flag.
How do I use PostgreSQL commands like this?
you can not use \c here because it is a psql meta-command, which I think you do not use here: See https://www.postgresql.org/docs/current/app-psql.html.
You need to reconnect to the new DB like so:
const pgclient_mydb = new Client({
host: process.env.POSTGRES_HOST,
port: process.env.POSTGRES_PORT,
user: process.env.POSTGRES_USER,
password: process.env.POSTGRES_PASSWORD,
database: 'mydb',
})
pgclient_mydb.connect()
See also https://stackoverflow.com/a/43670984/10743176
For known all databases (even with just users declared) I use with pymongo (pymongo==3.10.1 with mongodb 4.2)
db.command({'usersInfo': { 'forAllDBs': True },
'showCredentials': True})
this command need userAdminAnyDatabase builtin-role on client user.
But this role provide to much privileges. If someone crack the user password he can upgrade role to dbAdminAnyDatabases.
So I failed to create a any database role with authorization (role with only viewUser action on any databases) for userinfos previous command.
any can help me for this role definition?
You could try anyResource to get all the databases.
Thank's i create role :
with pymongo.MongoClient('mongodb://localhost:27017/',
username='SuperAdmin',
password='XXXXXXXXXX',
authSource='admin',
authMechanism='SCRAM-SHA-256') as client:
extendRole="showUsersAnyBase"
crole=client['admin'].command('rolesInfo',extendRole)
if reinit and extendRole in [x['role'] for x in crole['roles']]:
client['admin'].command('dropRole',extendRole)
if reinit or extendRole not in [x['role'] for x in crole['roles']]:
client['admin'].command({'createRole': extendRole,
'privileges': [{'resource': { 'anyResource': True },
'actions': [ "viewUser" ]}],
'roles':[ ]
})
create connector user:
usersInfos=client['admin'].command({'usersInfo': [{'user' : 'connector','db': 'admin'}]})
if reinitConnector and 'connector' in [x['user'] for x in usersInfos['users']]:
client['admin'].command('dropUser',"connector")
if reinitConnector or 'connector' not in [x['user'] for x in usersInfos['users']]:
client['admin'].command("createUser", "connector",
pwd="XXXXXXXXXX",
roles=[extendRole])
use the connector user with for userInfos command:
with pymongo.MongoClient('mongodb://localhost:27017/',
username='connector',
password='XXXXXXXXXX',
authSource='admin',
authMechanism='SCRAM-SHA-256') as client2:
usersInfos=client2['admin'].command({
'usersInfo': { 'forAllDBs': True },
'showCredentials': True
})
for user in usersInfos['users']:
print("user:",user['user'],
"db:",user['db'],
"roles:",[x['db']+'->'+x['role'] for x in user['roles']])
Works fine !!
and connector user can't grant role:
client2['admin'].command('grantRolesToUser','connector',
roles=['dbAdminAnyDatabase'])
throw exception :
.....
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: not authorized on admin to execute command { grantRolesToUser: "connector", .....
I have been given a knexfile like this:
require('dotenv').config()
module.exports = {
client: 'pg',
connection: process.env.DB_CONNECTION,
pool: {
min: 2,
max: 10
},
migrations: {
tableName: 'knex_migrations'
}
};
The connection string I supply is:
Host=localhost;Database=heypay;Username=postgres;Password=1234
However, Knex keeps issuing the error:
password authentication failed for user "user"
Apparently, the username I have given is not user. Moreover, I have tried to hardcore the connection string into the connection filed under module.exports. This still ended up in vain.
The trick is, the connection property can either be a string or an object. That's why you were able to supply an environment variable (it's a string).
The reason your original string was failing is not a Knex problem: Postgres connection strings have a slightly different format. You can use a similar approach as your first attempt, but pay attention to the key names:
host=localhost port=5432 dbname=mydb connect_timeout=10
Also note spaces, not semicolons. However in my experience most people use a Postgres URI:
postgresql://[user[:password]#][netloc][:port][,...][/dbname][?param1=value1&...]
So in your example, you'd use:
module.exports = {
client: 'pg',
connection: 'postgresql://your_database_user:password#localhost/myapp_test',
pool: {
min: 2,
max: 10
},
migrations: {
tableName: 'knex_migrations'
}
};
I was using a .NET style connection string, the correct one would be in the following format:
{
host : '127.0.0.1',
user : 'your_database_user',
password : 'your_database_password',
database : 'myapp_test'
}
I want to add monolog in mongodb with default handler(MongoDBHandler) in Symfony 4.
my monolog.yaml file in dev folder
monolog:
handlers:
mongo:
type: mongo
mongo:
id: monolog.logger.mongo
host: '%env(MONGODB_URL)%'
database: '%env(MONGODB_DB)%'
collection: logs
my services.yaml
services:
monolog.logger.mongo:
class: Monolog\Handler\MongoDBHandler
arguments: ['#doctrine_mongodb']
my doctrine_mongodb.yaml
doctrine_mongodb:
auto_generate_proxy_classes: '%kernel.debug%'
auto_generate_hydrator_classes: '%kernel.debug%'
connections:
default:
server: '%env(MONGODB_URL)%'
options:
db: '%env(MONGODB_DB)%'
log:
server: '%env(MONGODB_URL)%'
options:
db: '%env(MONGODB_DB)%'
connect: true
default_database: '%env(MONGODB_DB)%'
document_managers:
log:
auto_mapping: false
logging: false
But doesn't work.
one of the errors:
Cannot autowire service "monolog.logger.mongo": argument "$database"
of method "Monolog\Handler\MongoDBHandler::__construct()" is
type-hinted "string", you should configure its value explicitly.
While i use database option in monolog config.
Is there any document?
Another way to enable mongodb for monolog is:
monolog:
handlers:
mongo:
type: mongo
mongo:
host: '%env(MONGODB_URL)%'
user: myuser
pass: mypass
database: '%env(MONGODB_DB)%'
collection: logs
, So it mean you need to remove id field and add user and pass instead.
If you use doctrine mongodb already, it's possible to re-use it's connection, avoiding more ENV vars to separate the DSN:
monolog:
handlers:
mongo:
type: mongo
mongo:
id: "doctrine_mongodb.odm.default_connection"
database: "%env(MONGODB_DB)%"
collection: MyLogDocument # Keeping this the same, allows you to simply use a doctrine repository to access the documents in your app if needed
level: debug
I get the following error:
Attempted to load class "MongoClient" from the global namespace.
Did you forget a "use" statement?
protected function getMonolog_Handler_MongoService()
{
$this->privates['monolog.handler.mongo'] = $instance = new \Monolog\Handler\MongoDBHandler(new \MongoClient('mongodb://admin:pass#localhost:27017'), 'monolog', 'logs', 100, true);
$instance->pushProcessor(($this->privates['monolog.processor.psr_log_message'] ?? ($this->privates['monolog.processor.psr_log_message'] = new \Monolog\Processor\PsrLogMessageProcessor())));
return $instance;
}