I have successfully compiled pgagent from source on a CentOS 6.2 server.
When I try to launch pgagent with the following command :
/usr/bin/pgagent -l 2 hostaddr=serveur.com dbname=postgres user=postgres
I get the following error messages :
DEBUG: user : postgres
DEBUG: port : 0
DEBUG: host : server.com
DEBUG: dbname : postgres
DEBUG: password :
DEBUG: conn timeout : 0
DEBUG: Connection Information:
DEBUG: user : postgres
DEBUG: port : 0
DEBUG: host : server.com
DEBUG: dbname : postgres
DEBUG: password :
DEBUG: conn timeout : 0
DEBUG: Creating DB connection: user=postgres host=server.com dbname=postgres
DEBUG: Database sanity check
DEBUG: Clearing zombies
WARNING: Query error: ERROR: could not extend file "base/12870/12615": No space left on device
HINT: Check free disk space.
WARNING: Query error: ERROR: relation "pga_tmp_zombies" does not exist
LINE 1: INSERT INTO pga_tmp_zombies (jagpid) SELECT jagpid FROM pg...
^
WARNING: Query error: ERROR: table "pga_tmp_zombies" does not exist
WARNING: Query error: ERROR: could not extend file "base/12870/17167": No space left on device
HINT: Check free disk space.
WARNING: Couldn't create the primary connection (attempt 1): ERROR: could not extend file "base/12870/17167": No space left on device
HINT: Check free disk space.
DEBUG: Clearing all connections
DEBUG: Connection stats: total - 1, free - 0, deleted - 1
Any idea of the source of the problem ?
WARNING: Query error: ERROR: could not extend file "base/12870/12615": No space left on device
HINT: Check free disk space.
You're out of disk space, or (rather unlikely) a disk access quota limit has been hit for the database user. It's also possible you've run out of inodes on the file system for some platforms and file systems. Check df.
Related
We have an application pipeline and Postgres-12(TimescaleDB, managed through Patroni) on a separate server (VM with Ubuntu 18.04 LTS).
We are facing an issue with the DB, it suddenly stuck in the recovery mode, and also we can’t access it from the psql client and select queries also hung.
After an hour or late all got back to normal (As my current pipeline terminated) and able to run queries against the DB server.
Master DB error details:
2020-11-03 18:35:08.612 IST [9773] [unknown]#[unknown] LOG: connection received: host=x.x.x.x port=58780
2020-11-03 18:35:08.612 IST [9773] FATAL: the database system is in recovery mode
2020-11-03 18:35:08.596 IST [18276] LOG: could not send data to client: Broken pipe
Replica server error details:
2020-11-03 18:34:55 IST [18316]: [85649-1] user=postgres,db=postgres,app=[unknown],client=x.x.x.x LOG: duration: 10.228 ms statement: SELECT * FROM pg_stat_bgwriter;
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
2020-11-03 18:35:08 IST [18322]: [2-1] user=,db=,app=,client= FATAL: could not receive data from WAL stream: SSL SYSCALL error: EOF detected
2020-11-03 18:35:08 IST [20500]: [1-1] user=,db=,app=,client= FATAL: could not connect to the primary server: FATAL: the database system is in recovery mode
FATAL: the database system is in recovery mode
Pipeline error details:
Job aborted due to stage failure: Task 4 in stage 0.0 failed 3 times, most recent failure: Lost task 4.2 in stage 0.0 (TID 29, ip-x-x-x-x.ap-southeast-1.compute.internal, executor 19): org.postgresql.util.PSQLException: FATAL: the database system is in recovery mode at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:514) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:141) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) at org.postgresql.Driver.makeConnection(Driver.java:454) at org.postgresql.Driver.connect(Driver.java:256) at org.apache.spark.sql.execution.datasources.jdbc.DriverWrapper.connect(DriverWrapper.scala:45)
Please any advise on this issue?
What version of TimescaleDB are you running? In particular, there were some issues with 1.7.x if you try to query a read replica; we recommend upgrading to 1.7.4.
(Otherwise, there's not much information about to suggest what might have happened.)
https://github.com/timescale/timescaledb/releases/tag/1.7.4
I have a Greenplum deployment with several segments.
version: postgres (Greenplum Database) 8.2.15
A few days ago after an error as seen from pg_log (below)
FATAL 54000 out of on_shmem_exit slots
WARNING 1000 StartTransaction while in START state
PANIC XX000 Waiting on lock already held! (lwlock.c:557)
LOG 0 server process (PID 224596) was terminated by signal 6: Aborted
LOG 0 terminating any other active server processes
FATAL 57P01 terminating connection due to administrator command
LOG 0 sweeper process (PID 102635) exited with exit code 2
LOG 0 seqserver process (PID 102632) exited with exit code 2
FATAL 57P03 the database system is in recovery mode
LOG 0 ftsprobe process (PID 102633) exited with exit code 2
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
FATAL 57P03 the database system is in recovery mode
For 4 days now database remains in recovery mode, gpstart, gpstop all return with errors "the database system is in recovery mode" afterwards fails.
See ps response below:
[gpadmin#mdw1 ~]$ ps -ef | grep post
gpadmin 2979 189094 0 12:25 pts/0 00:00:00 grep post
postfix 3264 3251 0 2015 ? 00:34:40 qmgr -l -t fifo -u
gpadmin 102637 230099 0 May18 ? 00:01:47 postgres: port 5432, stats sender process
gpadmin 230099 1 0 Apr24 ? 02:47:53 /usr/local/greenplum-db-4.3.10.0/bin/postgres -D /data/master/gpseg-1 -p 5432 -b 1 -z 96 --silent-mode=true -i -M master -C -1 -x 194 -E
gpadmin 230100 230099 0 Apr24 ? 00:49:45 postgres: port 5432, master logger process
[gpadmin#mdw1 ~]$
I have searched a lot but am not able to find as a solution kindly assist with pointers on how to bring the database up.
I am running my application through JBoss with postgresql database but giving me max connection error but through PGAdmin I am able to connect database
17:14:44,917 ERROR [STDERR] [WARNING]
com.mchange.v2.resourcepool.BasicResourceP ool$AcquireTask#52900718 --
Acquisition Attempt Failed!!! Clearing pending acqui res. While trying
to acquire a needed new resource, we failed to succeed more th an the
maximum number of allowed acquisition attempts (2). Last acquisition
atte mpt exception: org.postgresql.util.PSQLException: FATAL:
remaining connection sl ots are reserved for non-replication superuser
connections
at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(Conn
Try to open file:
/etc/postgres/postgresql.conf
And increase number off connections like:
max_connections = 100
Web based application intermittently throwing '500 Internal Server Error'. This application is deployed on WAS 8.0 server with IHS and WAS servers hosted on different machines. Please find below the TRACE level plugin logs at IHS end.
DEBUG: ws_common: websphereExecute: Wrote the request; reading the response (timeout 900)
DETAIL: lib_htresponse: htresponseRead: Reading the response: 64040a70 TRACE: lib_rio: Blocking for read, waiting 900
DEBUG: lib_rio.c line 896 : Read failed, rc=104
DEBUG: lib_htresponse: htresponseSetError: Setting the error to: |READ_FAILED|(1, Line: 713)
DEBUG: ws_common: websphereExecute: Failed to read from an old stream; probably Keep-Alive timeout fired
DEBUG: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
DEBUG: lib_stream: destroyStream: Destroying the stream
DEBUG: lib_rio: rclose: socket 13 closed
STATS: ws_server: serverSetFailoverStatus: Server xxxxx : pendingRequests 0 failedRequests 7 affinityRequests 57 totalRequests 51.
ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'xxxxx' on host 'yyyyy'; will try another one
ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
ERROR: ESI: getResponse: failed to get response: rc = 10 DEBUG: ESI: esiHandleRequest: failed to get response
DEBUG: ESI: esiRequestUrlStackDestroy
DEBUG: ESI: cacheURL: '/site/index'
DETAIL: ESI: esiRequestPopUrl: '/site/index'
DEBUG: ESI: esiUrlDestroy: '/site/index'
ERROR: [://site/index] ws_common: websphereHandleRequest: Failed to handle request rc=10
DEBUG: ws_common: websphereCloseConnection
DETAIL: ws_common: websphereEndRequest: Ending the request
Please help me to get the issue resolved. Thanks in advance
rc=104 is the linux return code indicating that the socket was reset by the partner address so you need to look at the application server for errors.
I have installed mysql-5.6.14 from source pack
trying to start:
vaio1#vaio1-VPCEA3S1E:mysqld
2013-11-17 13:22:18 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
2013-11-17 13:22:18 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2013-11-17 13:22:18 1777 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
2013-11-17 13:22:18 1777 [Warning] One can only use the --user switch if running as root
2013-11-17 13:22:18 1777 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)
2013-11-17 13:22:18 1777 [Warning] Buffered warning: Changed limits: table_cache: 431 (requested 2000)
2013-11-17 13:22:18 1777 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
2013-11-17 13:22:18 1777 [Note] Plugin 'FEDERATED' is disabled.mysqld: Unknown error 1146
2013-11-17 13:22:18 1777 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2013-11-17 13:22:18 1777 [Note] InnoDB: The InnoDB memory heap is disabled
2013-11-17 13:22:18 1777 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-11-17 13:22:18 1777 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-11-17 13:22:18 1777 [Note] InnoDB: Using CPU crc32 instructions
2013-11-17 13:22:18 1777 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2013-11-17 13:22:18 1777 [Note] InnoDB: Completed initialization of buffer pool
2013-11-17 13:22:18 1777 [Note] InnoDB: Highest supported file format is Barracuda.
2013-11-17 13:22:19 1777 [Note] InnoDB: 128 rollback segment(s) are active.
2013-11-17 13:22:19 1777 [Note] InnoDB: Waiting for purge to start
2013-11-17 13:22:19 1777 [Note] InnoDB: 5.6.14 started; log sequence number 1600677
2013-11-17 13:22:19 1777 [ERROR] Aborting
Open the terminal and run this command for starting mysql service:
$ sudo mysqld_safe --skip-grant-tables
while mysql service is running, then open other terminal and run:
$ sudo mysql_upgrade -u root -p
mysqld reads this standard configuration file and can not read the 5.5 message file in
5.6, leading to this error.
Solutions :
1. pass your configuration file to mysql_install_db
./scripts/mysql_install_db --defaults-file=my.cnf
2. use --no-defaults, not to read other configuration files
./scripts/mysql_install_db --no-defaults --basedir=/home/praveen/mysql-5.6.10-linux-x86_65
--datadir=/home/praveen/data
Maybe, you got the MySQL 5.6 binary but my.cnf and data file is older version.
have you read MySQL upgrade Manual?
main reason MySQL has dead is :
Unknown error 1146 2013-11-17 13:22:18 1777 [ERROR] Can't open the mysql.plugin table.
Please run mysql_upgrade to create it
see Manual how to use mysql_upgrade. most of error message is related to deprecated options.
I have compiled from the sourceforge zip :mysql-5.6.14
Trying to mysql_upgrade:
vaio1#vaio1-VPCEA3S1E:~/Téléchargements/mysql-5.6.14$ mysql_upgrade
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
FATAL ERROR: Upgrade failed
vaio1#vaio1-VPCEA3S1E:~/Téléchargements/mysql-5.6.14$ mysqlcheck
Variables (--variable-name=value)
and boolean options {FALSE|TRUE} Value (after reading options)
--------------------------------- ----------------------------------------
all-databases FALSE
all-in-1 FALSE
auto-repair FALSE
bind-address (No default value)
character-sets-dir (No default value)
compress FALSE
databases FALSE
debug-check FALSE
debug-info FALSE
default-character-set (No default value)
default-auth (No default value)
fast FALSE
fix-db-names FALSE
fix-table-names FALSE
force FALSE
extended FALSE
host (No default value)
write-binlog TRUE
plugin-dir (No default value)
port 0
quick FALSE
silent FALSE
skip-database (No default value)
socket (No default value)
ssl FALSE
ssl-ca (No default value)
ssl-capath (No default value)
ssl-cert (No default value)
ssl-cipher (No default value)
ssl-key (No default value)
ssl-crl (No default value)
ssl-crlpath (No default value)
ssl-verify-server-cert FALSE
use-frm FALSE
user (No default value)
I don't understand why
user (No default value)
In my file: /etc/mysql/my.cnf
user = mysql
vaio1#vaio1-VPCEA3S1E:~/Téléchargements/mysql-5.6.14$ mysqld_safe --skip-grant-tables
131118 10:10:04 mysqld_safe Logging to '/var/log/mysql/error.log'.
touch: impossible de faire un touch «/var/log/mysql/error.log»: Permission non accordée
chmod: impossible d'accéder à «/var/log/mysql/error.log»: Aucun fichier ou dossier de ce type ## No files or directory
131118 10:10:04 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
/usr/local/mysql/bin/mysqld_safe: 129: /usr/local/mysql/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
/usr/local/mysql/bin/mysqld_safe: 1: eval: cannot create /var/log/mysql/error.log: Permission denied
touch: impossible de faire un touch «/var/log/mysql/error.log»: Permission non accordée
chown: impossible d'accéder à «/var/log/mysql/error.log»: Aucun fichier ou dossier de ce type ## No files or directory
chmod: impossible d'accéder à «/var/log/mysql/error.log»: Aucun fichier ou dossier de ce type
131118 10:10:04 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
/usr/local/mysql/bin/mysqld_safe: 129: /usr/local/mysql/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
...$ mysqlcheck --all-databases --check-upgrade --auto-repair
mysqlcheck: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying to connect
Trying this: sudo chown -R mysql /var/lib/mysql
also .. chown..var/log/mysql/
.. change nothing
YES! maybe problem from default user;
sudo cmod 777 var/log/mysql/ ## just test lol
Trying mysqld
vaio1#vaio1-VPCEA3S1E:~/Téléchargements/mysql-5.6.14$ mysqld
2013-11-18 11:36:36 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2013-11-18 11:36:36 24679 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
2013-11-18 11:36:36 24679 [Warning] One can only use the --user switch if running as root
2013-11-18 11:36:36 24679 [ERROR] Can't find messagefile '/usr/share/mysql/errmsg.sys'
Coping errmsg.sys
sudo cp /usr/local/mysql/share/english/errmsg.sys /usr/share/errmsg.sys
sudo chown mysql /usr/share/errmsg.sys
Resolve /usr/share/mysql/errmsg.sys
Whatever the list of MySQL "ini" files are present in : "MySQL\MySQL Server 5.2\" folder, replace them from your frined.
I wasted so much time and still dont know the root cause of the problem.
Do check ".err" file present in "data" folder for errors if any.
Now start the service & Done.
Its just makes entry somewhere & u try to debug for the problem soluton, instead this was faster approach.
Just ran into the same issue as mentioned in this question and tried lot of solutions, at last I figured out what was wrong. The issue came with the server upgrading. Files from mysql-libs-5.1.73-5.el6_6.x86_64 replaced those in MySQL-server-5.6.15-1.el6.x86_64, so after reinstalling, the server comes back. Here is what I did for a furture reference.
$ rpm -qf /usr/share/mysql/english/errmsg.sys
MySQL-server-5.6.15-1.el6.x86_64
mysql-libs-5.1.73-5.el6_6.x86_64
$ rpm -e mysql-libs-5.1.73-5.el6_6.x86_64
$ rpm -ivh --replacepkgs MySQL-*