I followed the instructions of https://github.com/karatelabs/karate/wiki/Docker to run the karate-chrome Docker image and worked it fine.
But when i try to connect karate with my local server of postgres i have the following error:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
I have the postgres server running with the host and port correct (localhost:5432). And I'm using the following configuration for the JDBC API:
Background:
* def config = { username: 'postgres', password: 'pass', url: 'jdbc:postgresql://localhost:5432/database_name_here', driverClassName: 'org.postgresql.Driver' }
* def DbUtils = Java.type('Testapi.DbUtils')
* def db = new DbUtils(config)
Anyone have any suggestions to solve this problem?. Thank you in advance.
Note: When i use a mysql online server everything runs fine (with its respective configuration).
I am trying to run my first flyway example using docker postgres image but getting the following error:
INFO: Flyway Community Edition 6.4.2 by Redgate
Exception in thread "main" org.flywaydb.core.internal.exception.FlywaySqlException:
Unable to obtain connection from database (jdbc:postgresql://localhost/flyway-service) for user 'flyway-service': FATAL: role "flyway-service" does not exist
-------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL State : 28000
Error Code : 0
Message : FATAL: role "flyway-service" does not exist
at org.flywaydb.core.internal.jdbc.JdbcUtils.openConnection(JdbcUtils.java:65)
at org.flywaydb.core.internal.jdbc.JdbcConnectionFactory.<init>(JdbcConnectionFactory.java:80)
I looked up into the docker container and can see that the user role flyway-service is created as part of the docker-compose execution:
$ docker exec -it flywayexample_postgres_1 bash
root#b2037e382112:/# psql -U flyway-service;
psql (12.2 (Debian 12.2-2.pgdg100+1))
Type "help" for help.
flyway-service=# \du;
List of roles
Role name | Attributes | Member of
----------------+------------------------------------------------------------+-----------
flyway-service | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
flyway-service=#
Main class is:
public static void main( String[] args ) {
var flyway = Flyway.configure().schemas("flyway_test_schema")
.dataSource("jdbc:postgresql://localhost/flyway-service", "flyway-service",
"password")
.load()
.migrate();
System.out.println( "Flyway example's hello world!" );
}
}
The migration called src/main/resources/db/migration/V1__Create_person_table.sql:
create table PERSON (
ID int not null,
NAME varchar(100) not null
);
Docker-compose yml file:
version: "3.8"
services:
postgres:
image: postgres:12.2
ports: ["5432:5432"]
environment:
- POSTGRES_PASSWORD=password
- POSTGRES_USER=flyway-service
I am running this code on MAC OSX. I assume, I am missing something obvious here, but not sure what! Any pointers would be appreciated.
Finally managed to figure out the issue with the help of a friend! The problem was not with the attached code but with a postgres daemon process running on the same port 5432 by an old Postgres installation.
I found the complete uninstallation procedure here. After removing the additional daemon process got only one port listening.
a$ lsof -n -i4TCP:5432
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
com.docke 654 root 50u IPv6 0x7ae1b5f8fbcf1cb 0t0 TCP *:postgresql (LISTEN)
Services
My service based on flask + postgresql + gunicorn + supervisor + nginx
When deploying by docker, after running the service, then accessing the api, sometimes it told the error message, and sometimes it workes well.
And the sqlachemy connect database add the parameters 'sslmode:disable'.
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
Return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
Distilled_params,
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
Util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
Reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
Cursor, statement, parameters, context
File "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
Cursor.execute(statement, parameters)
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Information
Docker for Mac: version: 2.0.0.3 (31259)
macOS: version 10.14.2
Python: version 2.7.15
Recurrence method
When view port information by command
lsof -i:5432
the port 5432 is postgresql database default port，if the outputconsole was
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 86469 user 4u IPv6 0xxddd 0t0 TCP *:postgresql (LISTEN)
postgres 86469 user 5u IPv4 0xxddr 0t0 TCP *:postgresql (LISTEN)
it would display the error message:
OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
but if the outputconsolelog show this:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
com.docke 62421 user 26u IPv4 0xe93 0t0 TCP 192.168.2.7:6435->192.168.2.7:postgresql (ESTABLISHED)
postgres 86460 user 4u IPv6 0xed3 0t0 TCP *:postgresql (LISTEN)
postgres 86460 user 5u IPv4 0xe513 0t0 TCP *:postgresql (LISTEN)
postgres 86856 user 11u IPv4 0xfe93 0t0 TCP 192.168.2.7:postgresql->192.168.2.7:6435 (ESTABLISHED)
the situation, the api would work well.
Becauce of Docker for mac?
Refer link https://github.com/docker/for-mac/issues/2442 , the issue can not solve my problem.
Notice a similar problem?
Refer link Python & Sqlalchemy - Connection pattern -> Disconnected from the remote server randomly
also this issue can not solve my problem.
Solution
flask_sqlachemy need the parameter pool_pre_ping
from flask_sqlalchemy import SQLAlchemy as _BaseSQLAlchemy
class SQLAlchemy(_BaseSQLAlchemy):
def apply_pool_defaults(self, app, options):
super(SQLAlchemy, self).apply_pool_defaults(self, app, options)
options["pool_pre_ping"] = True
db = SQLAlchemy()
Same logic for sqlalchemy.orm, ( on which flask_sqlalchemy is based btw )
engine = sqlalchemy.create_engine(connection_string, pool_pre_ping=True)
More protection strategies can be setup such as it is described in the doc: https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
For example, here is my engine instantiation:
engine = sqlalchemy.create_engine(connection_string,
pool_size=10,
max_overflow=2,
pool_recycle=300,
pool_pre_ping=True,
pool_use_lifo=True)
sqlalchemy.orm.sessionmaker(bind=engine, query_cls=RetryingQuery)
For RetryingQuery code, cf: Retry failed sqlalchemy queries
I'm posting my own answer to this, since none of the above addressed my particular setup (Postgres 12.2, SQLAlchemy 1.3).
To stop the OperationalErrors, I had to pass in some additional connect_args to create_engine:
create_engine(
connection_string,
pool_pre_ping=True,
connect_args={
"keepalives": 1,
"keepalives_idle": 30,
"keepalives_interval": 10,
"keepalives_count": 5,
}
)
Building on the Solution in the answer and the info from #MaxBlax360's answer. I think the proper way to set these config values in Flask-SQLAlchemy is by setting app.config['SQLALCHEMY_ENGINE_OPTIONS']:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
# pool_pre_ping should help handle DB connection drops
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {"pool_pre_ping": True}
app.config['SQLALCHEMY_DATABASE_URI'] = \
f'postgresql+psycopg2://{POSTGRES_USER}:{dbpass}#{POSTGRES_HOST}:{POSTGRES_PORT}/{POSTGRES_DBNAME}'
db = SQLAlchemy(app)
See also Flask-SQLAlchemy docs on Configuration Keys
My db configuration:
app = Flask(__name__)
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {"pool_pre_ping": True}
app.config['SQLALCHEMY_DATABASE_URI'] = os.environ['DATABASE_URL']
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
# Play with following options:
app.config['SQLALCHEMY_POOL_SIZE'] = 10
app.config['SQLALCHEMY_MAX_OVERFLOW'] = 20
app.config['SQLALCHEMY_POOL_RECYCLE'] = 1800
db = SQLAlchemy(app)
I searched for answers related to my question but not found what I'm looking for.
I have server with Linux CentOS 6.7. I need to connect to MS SQL Server 2005 on Windows Small Business Server 2003. In CentOS I installed FreeTDS and I managed to connect to SQL Server from terminal using this command:
# TDSVER=7.0 tsql -H ServerIPAdress -p 1433 -U username -P password
As far as I know this command bypass settings in freetds.conf.
Now I need to connect from PHP script. I'm using PDO extension and I tried this DSN string:
$db = new PDO("dblib:version=7.0;host=ServerIPAdress;dbname=Database;","username","password");
This leads me to first error:
Could not find driver
Ok, I understand that - my PHP 5.3.3 installation doesn't have installed PDO driver dblib.
My question: How can I install pdo_dblib driver in CentOS? Many tutorials and answers suggest to install through this command:
yum install php5-sybase
But that package doesn't exist:
No package php-sybase available.
If I check tsql configuration with command
tsql -C
I get these output:
Version: freetds v0.95
freetds.conf directory: /usr/local/etc
MS db-lib source compatibility: no
Sybase binary compatibility: no
Thread safety: yes
iconv library: yes
TDS version: 5.0
iODBC: no
unixodbc: yes
As I understand I need to configure FreeTDS:
./configure --enable-msdblib --with-pdo-dblib=/usr/local
But how can I configure FreeTDS after instalation? I tried to remove current version but I can't find way to do that. Also I tried to install another version but configuration doesn't change.
Any advice are appreciated.
When I run command # odbcinst -j I can see this output:
unixODBC 2.3.0
DRIVERS ...........: /usr/local/etc/odbcinst.ini
SYSTEM DATA SOURCES: /usr/local/etc/odbc.ini
FILE DATA SOURCES..: /usr/local/etc/ODBCDataSources
USER DATA SOURCES..: /root/.odbc.ini
SQLULEN Size.......: 8
SQLSETPOSIROW Size.: 8
Content of my freetds.conf:
# $Id: freetds.conf,v 1.12 2007/12/25 06:02:36 jklowden Exp $
#
# This file is installed by FreeTDS if no file by the same
# name is found in the installation directory.
#
# For information about the layout of this file and its settings,
# see the freetds.conf manpage "man freetds.conf".
# Global settings are overridden by those in a database
# server specific section
[global]
# TDS protocol version
tds version = 7.0
# Whether to write a TDSDUMP file for diagnostic purposes
# (setting this to /tmp is insecure on a multi-user system)
; dump file = /tmp/freetds.log
; debug flags = 0xffff
# Command and connection timeouts
; timeout = 10
; connect timeout = 10
# If you get out-of-memory errors, it may mean that your client
# is trying to allocate a huge buffer for a TEXT field.
# Try setting 'text size' to a more reasonable limit
text size = 64512
# A typical Sybase server
[egServer50]
host = symachine.domain.com
port = 5000
tds version = 7.0
# A typical Microsoft server
[MSServer]
host = 192.168.1.55
port = 1433
tds version = 7.0
odbcinst.ini:
[FreeTDS]
Description = FreeTDS v7.0
Driver = /usr/local/lib/libtdsodbc.so
UsageCount = 1
odbc.ini:
[MSSQL]
Driver = FreeTDS
Description = MSSQL Server Connection
TDS_Version = 7.0
Trace = No
Server = MSServer
Port = 1433
Database = MyDatabase
Username = DOMAIN\MyUsername
Password = MyPassword
I can make connection with isql command when I specify username and password:
# isql -v MSSQL "DOMAIN\MyUsername" MyPassword
But if I omit username and password isql generate error:
[S1000][unixODBC][FreeTDS][SQL Server]Unable to connect to data source
[01000][unixODBC][FreeTDS][SQL Server]Adaptive Server connection failed
I suspect that my username make problems because I have to use DOMAIN name followed by backslash before username when I connect to SQL Server 2005.
UPDATE:
I succesfully made connection in PHP and now I can run queries. I added this line on top of my script: putenv("FREETDSCONF=/usr/local/etc/freetds.conf")
Why is this necessary?
I am using mosquitto auth plugin here is my mosquitto.config file
listener 1883
#listener 9001 127.0.0.1
#protocol websockets
auth_opt_backends postgres
auth_plugin /etc/mosquitto/auth-plug.so
auth_opt_dbname mqtt
auth_opt_host localhost
auth_opt_port 5432
auth_opt_user postgres
auth_opt_pass postgres
auth_opt_userquery SELECT password FROM account WHERE username = $1 limit 1
auth_opt_superquery SELECT COALESCE(COUNT(*),0) FROM account WHERE username = $1 AND super = 1
auth_opt_aclquery SELECT topic FROM acls WHERE (username = $1) AND (rw >= $2)
all postgres setting are default postgres settings.
when I run
sudo /usr/local/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
command. I am getting this error:
1456228482: mosquitto version 1.4.8 (build date 2016-02-22 18:23:09+0530) starting
1456228482: Config loaded from /etc/mosquitto/mosquitto.conf.
1456228482: |-- *** auth-plug: startup
1456228482: |-- ** Configured order: postgres
1456228482: |-- }}}} POSTGRES
1456228482: |-- HERE: SELECT COALESCE(COUNT(*),0) FROM account WHERE username = $1 AND super = 1
1456228482: |-- HERE: SELECT topic FROM acls WHERE (username = $1) AND (rw >= $2)
|-- We were unable to connect to the database
|-- *** ABORT.
If I uncomment these two line
#listener 9001 127.0.0.1
#protocol websockets
i get this error:
Error: Websockets support not available.
Error found at /etc/mosquitto/mosquitto.conf:4.
Error: Unable to open configuration file.
I was following these tutorial
Tutorial1
Tutorial2
after changing password of postgres and chaging the default port 1883
1456232627: mosquitto version 1.4.8 (build date 2016-02-22 18:23:09+0530) starting
1456232627: Config loaded from /etc/mosquitto/mosquitto.conf.
1456232627: |-- *** auth-plug: startup
1456232627: |-- ** Configured order: postgres
1456232627: |-- }}}} POSTGRES
1456232627: |-- HERE: SELECT COALESCE(COUNT(*),0) FROM account WHERE username = $1 AND super = 1
1456232627: |-- HERE: SELECT topic FROM acls WHERE (username = $1) AND (rw >= $2)
1456232627: Opening ipv4 listen socket on port 1884.
1456232627: Opening ipv6 listen socket on port 1884.
1456232627: Error: Invalid user 'mosquitto'.
from where "mosquitto" user is comming ?
Taking the what has been given in the comment to form an actual answer:
The websockets problem is down to not building with Websockets enabled in config.mk by modifying the following line:
WITH_WEBSOCKETS:=no
And then rebuilding. You will need to ensure you have the libwebsocket development files installed.
The We were unable to connect to the database error was a database configuration issue
Error: Invalid user 'mosquitto'. is because you are trying to run mosquitto as root and mosquitto is trying to drop the root privileges in order to safely run at a lower level. By default mosquitto will try to run as a user called mosquitto, in this case there is no mosquitto user defined on the system hence the error. You can change the user that it will try and become by adding a the following to the mosquitto.conf
user foo
This would cause mosquitto to run as user foo. You really should not leave mosquitto running as root.