Set binlog for Mysql 5.6 & Ubuntu16.4 - ubuntu-16.04

in /etc/mysql/my.cnf:
[mysqld]
log_bin=mysql-bin
binlog_format=ROW
server-id=11
then on mysql client:
mysql> show variables like '%binlog%';
+-----------------------------------------+----------------------+
| Variable_name | Value |
+-----------------------------------------+----------------------+
| binlog_cache_size | 32768 |
| binlog_checksum | CRC32 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_format | STATEMENT |
| binlog_max_flush_queue_time | 0 |
| binlog_order_commits | ON |
| binlog_row_image | FULL |
| binlog_rows_query_log_events | OFF |
| binlog_stmt_cache_size | 32768 |
| innodb_api_enable_binlog | OFF |
| innodb_locks_unsafe_for_binlog | OFF |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| sync_binlog | 0 |
+-----------------------------------------+----------------------+
15 rows in set (0.00 sec)
I have googled "my.cnf dont take effect" and got "Changes to my.cnf don't take effect (Ubuntu 16.04, mysql 5.6)"
Changes to my.cnf don't take effect (Ubuntu 16.04, mysql 5.6)
I have tested the two answers. But when I start mysql with sudo service mysql start, there always be this error:
/etc/init.d/mysql[27197]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
/etc/init.d/mysql[27197]: [61B blob data]
/etc/init.d/mysql[27197]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
/etc/init.d/mysql[27197]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
/etc/init.d/mysql[27197]:
mysql[26527]: ...fail!
systemd[1]: mysql.service: Control process exited, code=exited status=1
systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
I have spent two days searching answers, and still not find nothing helpful. Could any one please help me out? Thank you!

Reinstall MySQL
Clean MySQL according to :
https://askubuntu.com/questions/172514/how-do-i-uninstall-mysql
sudo apt-get purge mysql-server mysql-client mysql-common mysql-server-core-* mysql-client-core-*
sudo rm -rf /etc/mysql /var/lib/mysql
sudo apt-get autoremove
sudo apt-get autoclean
as well as https://www.jianshu.com/p/c76b31df5d09
dpkg -l |grep ^rc|awk '{print $2}' |sudo xargs dpkg -P
Install by sudo apt-get install mysql-server-5.6
Make changes in my.cnf take effect
Second answer by Keeth(user:20588) on
Changes to my.cnf don't take effect (Ubuntu 16.04, mysql 5.6)
Configure binlog by editing /etc/mysql/my.cnf
[mysqld]
log_bin=mysql-bin
binlog_format=ROW
server-id=11

Related

Postgres Register Standby fails

I am trying to setup a Primary and a standby using repmgr. I think I have successfully setup master, but standby setup keeps failing.
On Standby node
/usr/pgsql-12/bin/repmgr -h master_ip standby clone
NOTICE: destination directory "/var/lib/pgsql/12/data" provided
INFO: connecting to source node
DETAIL: connection string is: host=master_ip
DETAIL: current installation size is 32 MB
ERROR: repmgr extension is available but not installed in database "(null)"
HINT: check that you are cloning from the database where "repmgr" is installed
On Master Node:
/usr/pgsql-12/bin/repmgr cluster show
ID | Name | Role | Status | Upstream | Location | Priority | Timeline | Connection string
----+-------------+---------+-----------+----------+----------+----------+----------+----------------------------------------------------------------
1 | hostname | primary | * running | | default | 100 | 1 | host=master_ip dbname=repmgr user=repmgr connect_timeout=2
postgres=# SELECT * FROM pg_available_extensions WHERE name='repmgr';
name | default_version | installed_version | comment
--------+-----------------+-------------------+------------------------------------
repmgr | 5.3 | | Replication manager for PostgreSQL
resolved after adding -U repmgr -d repmgr to the clone command.

pgpool/postgres - replication_delay is too high, how to reset?

in our setup the show pool_nodes shows a very high replication_delay and it keeps increasing, becuase of which any new queries are not replicated in the slave
following is the output of show pool_nodes command, is there a way to reset this, data loss if fine as this is not a live/production system.
[root#DB2 ~]# psql -h DB-HA-Hostname -U postgres -p 5432 -c 'show pool_nodes'
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | last_status_change
---------+--------------------------------------+------+--------+-----------+---------+------------+-------------------+-------------------+---------------------
0 | DB1-hostname | 5432 | up | 0.500000 | primary | 0 | true | 0 | 2021-01-11 19:32:00
1 | DB2-hostname | 5432 | up | 0.500000 | standby | 0 | false | 54986528 | 2021-01-11 19:32:00
(2 rows)
I have tried, restarting nodes, restarting pgpool, restarting postgresql , deleting database etc.. but no luck. As soon as the slave gets attached the replication_delay is high again..
You can run this command to check status of replication:
psql -h DB-HA-Hostname -U postgres -p 5432 -c "select * from pg_stat_replication" -x
if it shows:
if not, the configuration has failed.
You can show your configuration ?
Check the replication is running or not if it's not running re-configure the standby then attached the nodes
select * from pg_stat_replication;
after taking basebackup start the postgresql server then pcp-attach-node on pgpool

Error: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: Transport endpoint is not connected

I'm encountering an error when trying to do migrations bootstrap for kong, can anyone please help?
serene#ubuntu:~$ sudo kong migrations bootstrap -c /etc/kong/kong.conf --v
2020/10/19 02:21:04 [verbose] Kong: 2.1.4
2020/10/19 02:21:04 [verbose] reading config file at /etc/kong/kong.conf
2020/10/19 02:21:04 [verbose] prefix in use: /usr/local/kong
Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:95: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: Transport endpoint is not connected
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:95: in function 'cmd_exec'
/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88>
[C]: in function 'xpcall'
/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45>
/usr/local/bin/kong:9: in function 'file_gen'
init_worker_by_lua:50: in function <init_worker_by_lua:48>
[C]: in function 'xpcall'
init_worker_by_lua:57: in function <init_worker_by_lua:55>
I am using WSL, Ubuntu 18.04 LTS, I've installed kong v2.1.4, postgresql 12 on it
serene#ubuntu:~$ kong version
2.1.4
serene#ubuntu:~$ dpkg -l | grep postgres
ii pgdg-keyring 2018.2 all keyring for apt.postgresql.org
ii postgresql-12 12.4-1.pgdg18.04+1 amd64 object-relational SQL database, version 12 server
ii postgresql-client-12 12.4-1.pgdg18.04+1 amd64 front-end programs for PostgreSQL 12
ii postgresql-client-common 220.pgdg18.04+1 all manager for multiple PostgreSQL client versions
ii postgresql-common 220.pgdg18.04+1 all PostgreSQL database-cluster manager
I have started the postgresql service.
I have configured kong.conf as follows
database = postgres
pg_host = 127.0.0.1
pg_port = 5433
pg_timeout = 5000
pg_user = kong
pg_password = kong
pg_database = kong
pg_ssl = off
pg_ssl_verify = off
pg_max_concurrent_queries = 0
pg_semaphore_timeout = 60000
I have also configured a user in postgres called kong with password 'kong' as owner of database called 'kong.
serene#ubuntu:~$ sudo -u postgres psql
psql (12.4 (Ubuntu 12.4-1.pgdg18.04+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
kong | kong | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/kong +
| | | | | kong=CTc/kong
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres=#
Initially I was using WSL Ubuntu 20.04, Kong 2.1.4, Postgresql 13. Same error.
Did a fresh install of those. Same error.
Deleted Ubuntu 20.04 and installed Ubuntu 18.04 and reinstall Kong 2.1.4 and Postgresql 13. Same error.
Heard someone had troubles with Postgresql 13 when installing Kong, so I went and did a fresh install for Postgresql 12, still same error.
Is there anyone who knows what went wrong and how I can rectify this?

Upgrading Postgres from 10 to 12 -- problem with encodings

I'm trying to upgrade a postgres server from 10 12, and am having problems with the encodings. I'm following what I believe to be established recipes.
Behold:
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-------------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | postgres=CTc/postgres+
| | | | | =c/postgres
thingsboard | postgres | SQL_ASCII | C | C | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | nagios=c/postgres
(4 rows)
Notice the encodings on the 10 database. Time to create the database for 12.
postgres#serverbot:~$ sudo service postgresql stop
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/initdb -E SQL_ASCII --locale=C -D /var/lib/postgresql/12/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /var/lib/postgresql/12/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
Excellent! Let's fire up the new server.
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
And verify the encodings...
postgres#serverbot:~$ psql -l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Everything matches... time to upgrade!
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
--old-bindir=/usr/lib/postgresql/10/bin \
--new-bindir=/usr/lib/postgresql/12/bin \
--old-options '-c config_file=/etc/postgresql/10/main/postgresql.conf' \
--new-options '-c config_file=/etc/postgresql/12/main/postgresql.conf' \
--link --check
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
encodings for database "postgres" do not match: old "SQL_ASCII", new "UTF8"
Failure, exiting
postgres#serverbot:~$
Doh!
What is wrong here? I assert that the encodings do match, and I'm stuck.
Can anyone offer any advice?
What I see is:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_ctl -D /var/lib/postgresql/12/data -l logfile start
waiting for server to start.... done
server started
and then:
postgres#serverbot:~$ /usr/lib/postgresql/12/bin/pg_upgrade \
--old-datadir=/var/lib/postgresql/10/main \
--new-datadir=/var/lib/postgresql/12/main \
[...]
Note the $PGDATA directories. The database cluster you did the initdb for is different from the one you are doing the pg_upgrade to.
UPDATE. Since you seem to be using a Debian based OS and it's Postgres packaging, it might work better to stick with the packaging tools:
sudo pg_createcluster --locale=C 12 ascii
Creating new PostgreSQL cluster 12/ascii ...
/usr/lib/postgresql/12/bin/initdb -D /var/lib/postgresql/12/ascii --auth-local peer --auth-host md5 --locale C
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/12/ascii ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/Los_Angeles
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctlcluster 12 ascii start
Ver Cluster Port Status Owner Data directory Log file
12 ascii 5434 down postgres /var/lib/postgresql/12/ascii /var/log/postgresql/postgresql-12-ascii.log
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+-----------+---------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres +
| | | | | postgres=CTc/postgres
(3 rows)
Also I would use pg_upgradecluster, see here for more information. This keeps everything in the same system.

Can't connect the postgreSQL with psycopg2

It's the first time that I can't find the answer about some tech problems
Here's my problems:
>> conn=psycopg2.connect(database="mydb", user="postgres", password="123",port=5432)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
My postgreSQL is running
My listeningport is 5432 for sure
root#lanston-laptop:~# psql -l
Password:
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
---------------+----------+----------+------------+------------+-----------------------
checkdatabase | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
mydb | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
Thanks a lot!
Your libpq, which is used by psycopg2 expects Postgres socket to be in /var/run/postgresql/ but when you install Postgres from source it is by default it in /tmp/.
Check if there is a file /tmp/.s.PGSQL.5432 instead of /var/run/postgresql/.s.PGSQL.5432. Try:
conn=psycopg2.connect(
database="mydb",
user="postgres",
host="/tmp/",
password="123"
)
Only this solved my problem,
make a symbolic link to the /tmp/.s.PGSQL.5432:
sudo ln -s /tmp/.s.PGSQL.5432 /var/run/postgresql/.s.PGSQL.5432
Thanks to, Sukhjit Singh Sehra - s-postgresql-server-is-running
I originally intended to make it a comment to Tometzky's answer, but well, I have a lot to say here... Regarding the case where you don't call psycopg2.connect directly, but use third-party software.
tl;dr
Set unix_socket_directories in postgresql.conf to /var/run/postgresql, /tmp, and restart PostgreSQL.
intro
I tried PostgreSQL 9.2 (CentOS 7) and 9.5 (Ubuntu Xenial) from distro repos, PostgreSQL 9.3, 9.4, 9.5, 9.6, 10 on CentOS 7 from PostgreSQL repo, PostgreSQL 9.6, 10 on Ubuntu Xenial from PostgreSQL repo. Among them only 9.3 listens to only /tmp:
$ systemctl stop postgresql-9.4 && systemctl start postgresql-9.3
$ lsof -aUp $(ps --ppid 1 -o pid= -o comm= | awk '$2 == "postgres" || $2 == "postmaster" {print $1}')
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 25455 postgres 4u unix 0xffff9acb23bc5000 0t0 6813995 /tmp/.s.PGSQL.5432
$ systemctl stop postgresql-9.3 && systemctl start postgresql-9.4
$ lsof -aUp $(ps --ppid 1 -o pid= -o comm= | awk '$2 == "postgres" || $2 == "postmaster" {print $1}')
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 26663 postgres 4u unix 0xffff9ac8c5474c00 0t0 7086508 /var/run/postgresql/.s.PGSQL.5432
postgres 26663 postgres 5u unix 0xffff9ac8c5477c00 0t0 7086510 /tmp/.s.PGSQL.5432
python-psycopg2
That's not a big deal with psql, just a matter of running the matching binary. But if you, for instance, have python-psycopg2 installed from CentOS's base or update repo. It links dynamically to libpq that OS provides. With 9.3 and 9.4 installed OS provides 9.4's version:
$ alternatives --display pgsql-ld-conf
pgsql-ld-conf - status is auto.
link currently points to /usr/pgsql-10/share/postgresql-9.4-libs.conf
/usr/pgsql-9.3/share/postgresql-9.3-libs.conf - priority 930
/usr/pgsql-9.4/share/postgresql-9.4-libs.conf - priority 940
Current `best' version is /usr/pgsql-9.4/share/postgresql-9.4-libs.conf.
$ ls -l /etc/ld.so.conf.d
lrwxrwxrwx 1 root root 31 Feb 7 02:25 postgresql-pgdg-libs.conf -> /etc/alternatives/pgsql-ld-conf
$ ls -l /etc/alternatives/pgsql-ld-conf
lrwxrwxrwx 1 root root 43 Feb 7 02:25 /etc/alternatives/pgsql-ld-conf -> /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
$ cat /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
/usr/pgsql-9.4/lib/
But libpq that comes with PostgreSQL 9.4 looks for socket in /var/run/postgresql as opposed to 9.3:
$ strings /usr/pgsql-9.3/lib/libpq.so.5 | egrep '/(tmp|var)'
/tmp
$ strings /usr/pgsql-9.4/lib/libpq.so.5 | egrep '/(tmp|var)'
/var/run/postgresql
The solution comes from postinstall scripts of corresponding packages:
$ yum reinstall --downloadonly postgresql94-libs
$ rpm -qp /var/cache/yum/x86_64/7/pgdg94/packages/postgresql94-libs-9.4.15-1PGDG.rhel7.x86_64.rpm --scripts
postinstall scriptlet (using /bin/sh):
/usr/sbin/update-alternatives --install /etc/ld.so.conf.d/postgresql-pgdg-libs.conf pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf 940
/sbin/ldconfig
# Drop alternatives entries for common binaries and man files
postuninstall scriptlet (using /bin/sh):
if [ "$1" -eq 0 ]
then
/usr/sbin/update-alternatives --remove pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
/sbin/ldconfig
fi
Temporarily remove 9.4's alternative:
$ alternatives --remove pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf
$ ldconfig
When finished either reinstall postgresql94-libs, or add the alternative back:
$ alternatives --install /etc/ld.so.conf.d/postgresql-pgdg-libs.conf pgsql-ld-conf /usr/pgsql-9.4/share/postgresql-9.4-libs.conf 940
$ ldconfig
pip
If you install psycopg2 with pip on the other hand, it by default installs precompiled package which comes with its own libpq, which looks for socket in /var/run/postgresql:
$ python3.5 -m venv 1
$ . ./1/bin/activate
(1) $ pip install psycopg2
(1) $ python
>>> import psycopg2
>>>Ctrl-Z
[1]+ Stopped python
(1) $ pgrep python
26311
(1) $ grep libpq /proc/26311/maps | head -n 1
7f100b8cb000-7f100b90e000 r-xp 00000000 08:04 112980 /home/yuri/1/lib/python3.5/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10
(1) $ strings /home/yuri/1/lib/python3.5/site-packages/psycopg2/.libs/libpq-909a53d8.so.5.10 | egrep '/(tmp|var)'
/var/run
/var/run/postgresql
The solution is to ask pip to not install precompiled package, and make pg_config of the proper version of PostgreSQL available:
$ PATH=/usr/pgsql-9.3/lib:$PATH pip install --no-binary psycopg2 psycopg2
You can even add --no-binary switch to requirements.txt:
psycopg2==2.7.3.2 --no-binary psycopg2
unix_socket_directories
The easier option though is to make use of unix_socket_directories option:
Try change port to 5433 instead of 5432
a few years later, using the EnterpriseDB 'graphical' install on OSX 10.8, and pip install of psycopg2 (after linking the /Library/...dylib's as described here) i had this same issue.
for me the correct connect command was conn = psycopg2.connect('dbname=DBNAME user=postgres password=PWHERE host=/tmp/')
In my case with a conda installation had to: sudo ln -s /var/run/postgresql/.s.PGSQL.5432 /tmp/.s.PGSQL.5432
Having this happen to me after a brew upgrade, I googled for brew .s.PGSQL.5432.
Per the suggestion in this answer I ran the following:
postgres -D /usr/local/var/postgres
And got:
2019-10-29 17:43:30.860 IST [78091] FATAL: database files are incompatible with server
2019-10-29 17:43:30.860 IST [78091] DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.5.
I googled that FATAL error and per the suggestion in this answer I ran:
brew postgresql-upgrade-database
That solved it for me.
put vpc_access_connector:
name: project//locations/us-central1/connectors/
and host :'/cloudsql/::
It should work for private Ip postgresql on gcp
Try this once
cd /etc/postgresql/13/main
vi pg_hba.conf
Change the line after this
Database administrative login by Unix domain socket
local all postgres peer
To
local all postgres md5
Then execute following commands
sudo systemctl stop postgresql
sudo systemctl start postgresql
Then run the python program then it will work properly