When I saw this error message:
ERROR Shutdown broker because all log dirs in /tmp/kafka-logs have failed (kafka.log.LogManager)
The first thought is "well the /tmp directory probably got cleared out by the O/S (linux) - so I should update the kafka config to point to something permanent. However the directory is present and has not been wiped:
ll /tmp/kafka-logs/
total 20
drwxrwxr-x 2 ec2-user ec2-user 38 Apr 7 16:56 __consumer_offsets-0
drwxrwxr-x 2 ec2-user ec2-user 38 Apr 7 16:56 __consumer_offsets-7
drwxrwxr-x 2 ec2-user ec2-user 38 Apr 7 16:56 __consumer_offsets-42
..
drwxrwxr-x 2 ec2-user ec2-user 38 Apr 7 16:56 __consumer_offsets-32
drwxrwxr-x 2 ec2-user ec2-user 141 Apr 12 02:49 flights_raw-0
drwxrwxr-x 2 ec2-user ec2-user 178 Apr 12 08:25 air2008-0
drwxrwxr-x 2 ec2-user ec2-user 141 Apr 12 13:38 testtopic-0
-rw-rw-r-- 1 ec2-user ec2-user 1244 Apr 17 22:29 recovery-point-offset-checkpoint
-rw-rw-r-- 1 ec2-user ec2-user 4 Apr 17 22:29 log-start-offset-checkpoint
-rw-rw-r-- 1 ec2-user ec2-user 1248 Apr 17 22:30 replication-offset-checkpoint
So then what does this actually mean, why is it happening and what should be done to correct/avoid the error?
In related question best answer suggests to delete log dir both for Kafka /tmp/kafka-logs and Zookeper /tmp/zookeeper.
Probably it's because of Kafka issue which was resolved in August 2018.
Hope this will help.
Related
I have a DOCKER container built from a few different images using a .yml, Dockerfile(s), etc. Everything builds and runs fine so far, except for this one issue that I am seeing mentioned in the title:
index-db_1 | 2021-02-22 23:18:33.388 UTC [31] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
That Database Index is mapped to a Folder on the host in the root of the Docker Package, and everything else seems to work fine as far as the database is concerned. I am using a Mac, but if I list permission from CLI for the DB folder I get:
-rw-------# 1 sscotti staff 3 Feb 22 11:01 PG_VERSION
drwx------# 6 sscotti staff 192 Feb 22 11:54 base
drwx------# 60 sscotti staff 1920 Feb 22 16:00 global
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_commit_ts
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_dynshmem
-rw-------# 1 sscotti staff 4782 Feb 22 11:02 pg_hba.conf
-rw-------# 1 sscotti staff 1636 Feb 22 11:01 pg_ident.conf
drwx------# 5 sscotti staff 160 Feb 22 17:46 pg_logical
drwx------# 4 sscotti staff 128 Feb 22 11:01 pg_multixact
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_notify
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_replslot
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_serial
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_snapshots
drwx------# 2 sscotti staff 64 Feb 22 16:00 pg_stat
drwx------# 5 sscotti staff 160 Feb 22 17:50 pg_stat_tmp
drwx------# 3 sscotti staff 96 Feb 22 11:01 pg_subtrans
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_tblspc
drwx------# 2 sscotti staff 64 Feb 22 11:01 pg_twophase
drwx------# 4 sscotti staff 128 Feb 22 11:01 pg_wal
drwx------# 3 sscotti staff 96 Feb 22 11:01 pg_xact
-rw-------# 1 sscotti staff 88 Feb 22 11:01 postgresql.auto.conf
-rw-------# 1 sscotti staff 28073 Feb 22 11:01 postgresql.conf
-rw-------# 1 sscotti staff 36 Feb 22 16:00 postmaster.opts
-rw------- 1 sscotti staff 94 Feb 22 16:00 postmaster.pid
pg_stat folder is actually empty.
and pg_stat_temp has:
-rw------- 1 sscotti staff 1952 Feb 22 17:54 db_0.stat
-rw------- 1 sscotti staff 20360 Feb 22 17:54 db_13395.stat
-rw------- 1 sscotti staff 1151 Feb 22 17:54 global.stat
The .yml file has this:
index-db:
image: postgres
restart: unless-stopped
volumes:
- ./OrthancIndex:/var/lib/postgresql/data
Is that something that can just be ignored given that it is a Docker container.
Adding a comment about the same setup on UBUNTU.
Database Folder:
drwx------ 19 systemd-coredump root 4096 Jun 30 13:12 OrthancIndex
Database:
drwx------ 6 systemd-coredump systemd-coredump 4096 Jun 11 13:00 base
drwx------ 2 systemd-coredump systemd-coredump 4096 Jun 30 13:12 global
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_commit_ts
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_dynshmem
-rw------- 1 systemd-coredump systemd-coredump 4782 Mar 12 16:12 pg_hba.conf
-rw------- 1 systemd-coredump systemd-coredump 1636 Mar 12 16:12 pg_ident.conf
drwx------ 4 systemd-coredump systemd-coredump 4096 Jul 1 13:27 pg_logical
drwx------ 4 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_multixact
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_notify
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_replslot
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_serial
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_snapshots
drwx------ 2 systemd-coredump systemd-coredump 4096 Jun 30 13:12 pg_stat
drwx------ 2 systemd-coredump systemd-coredump 4096 Jul 1 13:29 pg_stat_tmp
drwx------ 2 systemd-coredump systemd-coredump 4096 Jun 24 21:04 pg_subtrans
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_tblspc
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_twophase
-rw------- 1 systemd-coredump systemd-coredump 3 Mar 12 16:12 PG_VERSION
drwx------ 3 systemd-coredump systemd-coredump 4096 Jul 1 12:37 pg_wal
drwx------ 2 systemd-coredump systemd-coredump 4096 Mar 12 16:12 pg_xact
-rw------- 1 systemd-coredump systemd-coredump 88 Mar 12 16:12 postgresql.auto.conf
-rw------- 1 systemd-coredump systemd-coredump 28073 Mar 12 16:12 postgresql.conf
-rw------- 1 systemd-coredump systemd-coredump 36 Jun 30 13:12 postmaster.opts
-rw------- 1 systemd-coredump systemd-coredump 94 Jun 30 13:12 postmaster.pid
pg_stat_temp
-rw------- 1 systemd-coredump systemd-coredump 2660 Jul 1 13:30 db_0.stat
-rw------- 1 systemd-coredump systemd-coredump 31157 Jul 1 13:30 db_13395.stat
-rw------- 1 systemd-coredump systemd-coredump 1151 Jul 1 13:30 global.stat
I actually get the same error on UBUNTU:
postgres_index-db_1 | 2021-07-01 18:06:45.140 UTC [266] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
postgres_index-db_1 | 2021-07-01 18:13:45.583 UTC [273] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
postgres_index-db2_1 | 2021-07-01 18:19:43.716 UTC [282] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
postgres_index-db2_1 | 2021-07-01 18:21:43.749 UTC [284] WARNING: could not open statistics file "pg_stat_tmp/global.stat": Operation not permitted
Although here the user and group are systemd-coredump.
Simple and dumb
I've tried another way: leaving the temporary file in container, where you need not deal with user permissions - and that worked. I've changed with inline command the Postgresql config in docker-compose.yml:
postgresdb:
image: 'postgres'
command: postgres -c stats_temp_directory=/tmp
.
.
.
This is quick and simple solution, but it does not set multiple postgres config parameters. Through the time I found another solution.
Advanced (added on 22/01/14 09:00PM CET)
The advanced solution consists in these steps:
Prepare your desired postgresql.conf - here you can set whichever postgres server parameters you need. The necessary one for solving problem in this thread is:
stats_temp_directory = '/tmp/stat_temporary'
Create bash file, that you will start using docker-entrypoint-initdb.d in docker-compose.yml with which you will copy your configuration to postgres config directory in docker and create your temporary folder.
If you setup the docker-compose.yml this way:
postgres-db:
image: postgres
container_name: 'dbcontainer'
volumes:
# point to your postgres init scripts (folder or file)
- $PWD/db_init:/docker-entrypoint-initdb.d
# if you need persistent data, point data to local folder
- $PWD/database:/var/lib/postgresql/data
# if you want some data restore, point to your backup
- $PWD/db_backup:/db_backup
# point docker to directory with your config
- $PWD/db_config:/db_config
Then database will init with scripts in db_init directory. (Or you can point directly to one sql or bash file. If you point to directory, init script will run all files in alphabetical order.
Your file with init bash script, that you will place in your db_init folder can look like this and will be started inside docker:
echo starting script...
# create temporary directory for postgres in docker
mkdir /tmp/stat_temporary
# copy your postgresql.conf to postgresql config location in docker
cp /db_config/postgresql.conf var/lib/postgresql/data/postgresql.conf
# alternatively you can run any pg script you need, here I restore backup to new database
pg_restore -U postgres -v -Fc -d analysis /db_backup/analysis_backup.bak
echo finished
After this script is run, the database service on docker is restarted and loads with new parameters you set up.
Writing this down as much for my future self as to answer the question: looking over at the documentation and taking #Fide's solution above, the following seems to work on MacOSX Monterey running a PostGIS image (which is just built on top of the Postgres images)...
Note that this is taken from a shell script to launch a Postgres+PostGIS image with a persistent local store on the host system:
WORK_DIR="$HOME/Documents/git/project"
DOCKER_NM="postgis"
DOCKER_IMG="postgis/postgis"
POSTGRES_PWD="XXX"
PORT_NO="5432"
docker run --name $DOCKER_NM \
-e POSTGRES_PASSWORD=$POSTGRES_PWD \
-e PGDATA=/var/lib/postgresql/data/pgdata \
-v "$WORK_DIR/data/postgres":/var/lib/postgresql/data \
-p "$PORT_NO:5432" -d $DOCKER_IMG \
postgres -c stats_temp_directory=/tmp
If you were running from the Terminal directly (i.e. not via zsh/bash) then you'd need to export each of the parameters instead of just specifying them.
With this the global.stat error seems to have gone away... though I should note that I've only just figured this out and have been running the container for less than 60 minutes. The alternative would be to follow the instructions on that page for creating and using a persistent custom conf file (extending #SScotti's comment above) as mounted from the local FS:
-v "$PWD/my-postgres.conf":/etc/postgresql/postgresql.conf
Mounting with a local volume requires user mapping within the postgres container. There is a section in the https://hub.docker.com/_/postgres README titled Arbitrary --user Notes which describes solutions to the problem.
One recommendation listed is to use a docker volume, initialize the contents of volume mounted, shutdown the container image, run a recursive ownership change and then restart the container normally.
$ docker volume create pgdata
$ docker run -it --rm -v pgdata:/var/lib/postgresql/data -e POSTGRES_PASSWORD=mysecretpassword postgres
The files belonging to this database system will be owned by user "postgres".
...
( once it's finished initializing successfully and is waiting for connections, stop it )
$ docker run -it --rm -v pgdata:/var/lib/postgresql/data bash chown -R 1000:1000 /var/lib/postgresql/data
$ docker run -it --rm --user 1000:1000 -v pgdata:/var/lib/postgresql/data postgres
Not an answer to the original question, but hence the lack of reputation points I can't add my two cents as a comment. So here is a working docker-compose.yml file using the tmpfs solution as an answer (for #SScotti and anyone else, who wants to try this without cluttering the default command or config):
Environment file (.env):
POSTGRES_VERSION=13.6
DATABASE_NAME=dbname
DATABASE_USER=dbuser
DATABASE_PASS=dbpass
Compose file (docker-compose.yml):
version: '3.6'
services:
postgres:
image: postgres:${POSTGRES_VERSION}-alpine
restart: always
container_name: postgres
environment:
POSTGRES_DB: ${DATABASE_NAME}
POSTGRES_USER: ${DATABASE_USER}
POSTGRES_PASSWORD: ${DATABASE_PASS}
volumes:
- /etc/timezone:/etc/timezone:ro
- .volumes/pgdata:/var/lib/postgresql/data
- type: tmpfs
target: /var/lib/postgresql/data/pg_stat_tmp
tmpfs:
# 256 Mb - beware the default, it's infinity(!)
size: 268435456
Edit: Note on first run
On first run, postgres tries to initialise the database and fails, because of non empty data directory (the existing/mounted tmpfs pg_stats_tmp in the data directory).
To avoid that, you have to run the service at first without the tmpfs section in the compose file, and when postgres finished initialising take the service down and enable the tmpfs section in the compose file before bringen up the service again.
I ran into this same problem - Looked into it in detail and finally found a simple solution.
First some basic background - your pg_stat_temp_directory location is a subdirectory under your data_directory.
show data_directory ;
If it returns the default value for Unix based systems like
/var/lib/postgresql/data
And your command to show stats_temp
show stats_temp_directory ;
Also returns the default value
pg_stat_tmp
Then your full path to stats_temp_directory is /var/lib/postgresql/data/pg_stat_tmp
I was using bind mapping to map the PG container data files to my host so that I can keep the data even after the container is stopped.
docker run -d --name gutlo-postgres -p 5434:5432 -v /data/postgres-docker/14:/var/lib/postgresql/data -e POSTGRES_PASSWORD=<PASSWORD> postgres:latest
Now everytime I start the container, it automatically creates files under stats_temp directory and once that it has a problem with is called global.stat
ls -la global.stat
-rw------- 1 ssengupta admin 3648 May 13 16:24 global.stat
ls -la . | head -2 | tail -1
drwxr-xr-x# 14 ssengupta admin 448 May 13 16:26 .
Now you see, where is the problem. The container is able to create the file global.stat ( its directory file "." has open rwx permissions. But the file itself is created without open permission hence my container cannot access its own created file.
From docker manual, there is a concept of temp mount. So simply create a temp mount point for the stats_temp directory and the docker will use tempfs / RAM - that eliminates permission problem.
--mount type=tmpfs,destination=/var/lib/postgresql/data/pg_stat_tmp
In addition it would improve performance too although you may not detect it.
My attempts...
Attach to the db container:
docker compose run --no-deps -it db bash
and do ls on the data directory:
root#dfc6399e2981:/var/lib/postgresql/data# ls -l
total 60
drwx------ 7 root root 224 Sep 8 23:38 base
drwx------ 60 root root 1920 Sep 8 23:38 global
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_commit_ts
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_dynshmem
-rw------- 1 postgres postgres 4821 Sep 8 23:35 pg_hba.conf
-rw------- 1 postgres postgres 1636 Sep 8 23:35 pg_ident.conf
drwx------ 5 root root 160 Sep 8 23:43 pg_logical
drwx------ 4 postgres postgres 128 Sep 8 23:35 pg_multixact
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_notify
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_replslot
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_serial
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_snapshots
drwx------ 2 root root 64 Sep 8 23:35 pg_stat
drwx------ 6 root root 192 Sep 9 00:20 pg_stat_tmp
drwx------ 3 postgres postgres 96 Sep 8 23:35 pg_subtrans
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_tblspc
drwx------ 2 postgres postgres 64 Sep 8 23:35 pg_twophase
-rw------- 1 postgres postgres 3 Sep 8 23:35 PG_VERSION
drwx------ 4 postgres postgres 128 Sep 8 23:35 pg_wal
drwx------ 3 postgres postgres 96 Sep 8 23:35 pg_xact
-rw------- 1 postgres postgres 88 Sep 8 23:35 postgresql.auto.conf
-rw------- 1 postgres postgres 28835 Sep 8 23:35 postgresql.conf
-rw------- 1 root root 36 Sep 8 23:35 postmaster.opts
-rw------- 1 postgres postgres 94 Sep 8 23:35 postmaster.pid
Obviously, note that some things are owned by root. Also, compare and contrast, from outside the container I see this:
% ls -l db/data
total 120
-rw-------# 1 pedz staff 3 Sep 8 18:35 PG_VERSION
drwx------# 7 pedz staff 224 Sep 8 18:38 base
drwx------# 60 pedz staff 1920 Sep 8 18:38 global
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_commit_ts
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_dynshmem
-rw-------# 1 pedz staff 4821 Sep 8 18:35 pg_hba.conf
-rw-------# 1 pedz staff 1636 Sep 8 18:35 pg_ident.conf
drwx------# 5 pedz staff 160 Sep 8 18:43 pg_logical
drwx------# 4 pedz staff 128 Sep 8 18:35 pg_multixact
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_notify
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_replslot
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_serial
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_snapshots
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_stat
drwx------# 6 pedz staff 192 Sep 8 19:27 pg_stat_tmp
drwx------# 3 pedz staff 96 Sep 8 18:35 pg_subtrans
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_tblspc
drwx------# 2 pedz staff 64 Sep 8 18:35 pg_twophase
drwx------# 4 pedz staff 128 Sep 8 18:35 pg_wal
drwx------# 3 pedz staff 96 Sep 8 18:35 pg_xact
-rw-------# 1 pedz staff 88 Sep 8 18:35 postgresql.auto.conf
-rw-------# 1 pedz staff 28835 Sep 8 18:35 postgresql.conf
-rw-------# 1 pedz staff 36 Sep 8 18:35 postmaster.opts
-rw-------# 1 pedz staff 94 Sep 8 18:35 postmaster.pid
So, while in the db container I did:
chown -R postgres:postgres /var/lib/postgresql/data
I can now see updates into pg_stat_tmp from outside of the container and it has been up for 30 minutes with no Warning messages so I think its "fixed" although I don't know enough about Docker's images to know for sure that the issue won't come back later. Perhaps someone with more Docker knowledge can chime in.
Update: this does NOT fix the issue. Rather than delete this answer I think I will leave it here. It is odd that some files are owned by root. And even after I did chown -R there are still some files owned by root:
root#ec24992481d1:/var/lib/postgresql# find . -user 0 -exec ls -ld {} +
-rw------- 1 root root 8 Sep 9 00:34 ./data/pg_logical/replorigin_checkpoint
-rw------- 1 root root 2225 Sep 9 01:12 ./data/pg_stat_tmp/db_0.stat
-rw------- 1 root root 6665 Sep 9 01:12 ./data/pg_stat_tmp/db_13757.stat
-rw------- 1 root root 7035 Sep 9 01:12 ./data/pg_stat_tmp/db_16384.stat
-rw------- 1 root root 94 Sep 9 00:29 ./data/postmaster.pid
root#ec24992481d1:/var/lib/postgresql#
This might be a clue for others more savvy with Docker.
I noticed that a lot of commands were missing when using VSCode terminal. So I tried a ls -l / on both my distro's and VSCode's terminals.
Linux Mint Xfce Terminal:
livy#linux-mint:~$ ls -l /
total 2097252
drwxr-xr-x 2 root root 4096 Aug 13 15:13 bin
drwxr-xr-x 4 root root 4096 Aug 13 15:15 boot
drwxr-xr-x 2 root root 4096 Aug 13 15:06 cdrom
drwxr-xr-x 20 root root 4260 Aug 15 08:40 dev
drwxr-xr-x 147 root root 12288 Aug 13 16:46 etc
drwxr-xr-x 4 root root 4096 Aug 13 15:06 home
lrwxrwxrwx 1 root root 33 Aug 13 15:07 initrd.img -> boot/initrd.img-4.15.0-54-generic
lrwxrwxrwx 1 root root 33 Aug 13 15:04 initrd.img.old -> boot/initrd.img-4.15.0-54-generic
drwxr-xr-x 23 root root 4096 Aug 13 15:07 lib
drwxr-xr-x 2 root root 4096 Jul 29 19:15 lib64
drwx------ 2 root root 16384 Aug 13 14:59 lost+found
drwxr-xr-x 2 root root 4096 Jul 29 19:14 media
drwxr-xr-x 2 root root 4096 Jul 29 19:14 mnt
drwxr-xr-x 3 root root 4096 Aug 13 15:18 opt
dr-xr-xr-x 214 root root 0 Aug 15 08:40 proc
drwx------ 4 root root 4096 Aug 13 15:07 root
drwxr-xr-x 32 root root 940 Aug 15 08:41 run
drwxr-xr-x 2 root root 12288 Aug 13 15:14 sbin
drwxr-xr-x 2 root root 4096 Jul 29 19:14 srv
-rw------- 1 root root 2147483648 Aug 15 08:40 swapfile
dr-xr-xr-x 13 root root 0 Aug 15 09:31 sys
drwxrwxrwt 15 root root 4096 Aug 15 08:53 tmp
drwxr-xr-x 11 root root 4096 Jul 29 19:14 usr
drwxr-xr-x 11 root root 4096 Jul 29 19:50 var
lrwxrwxrwx 1 root root 30 Aug 13 15:07 vmlinuz -> boot/vmlinuz-4.15.0-54-generic
And here is the result from VSCode terminal:
livy#linux-mint:~$ ls -l /
total 2097204
drwxr-xr-x 6 nfsnobody nfsnobody 4096 Aug 13 16:00 app
lrwxrwxrwx 1 livy livy 7 Aug 15 09:13 bin -> usr/bin
drwxr-xr-x 2 livy livy 80 Aug 15 09:13 boot
drwxr-xr-x 2 nfsnobody nfsnobody 4096 Aug 13 15:06 cdrom
drwxr-xr-x 5 livy livy 340 Aug 15 09:13 dev
drwxr-xr-x 15 livy livy 860 Aug 15 09:13 etc
drwxr-xr-x 4 nfsnobody nfsnobody 4096 Aug 13 15:06 home
lrwxrwxrwx 1 livy livy 33 Aug 15 09:13 initrd.img -> boot/initrd.img-4.15.0-54-generic
lrwxrwxrwx 1 livy livy 33 Aug 15 09:13 initrd.img.old -> boot/initrd.img-4.15.0-54-generic
lrwxrwxrwx 1 livy livy 7 Aug 15 09:13 lib -> usr/lib
lrwxrwxrwx 1 livy livy 9 Aug 15 09:13 lib64 -> usr/lib64
drwx------ 2 nfsnobody nfsnobody 16384 Aug 13 14:59 lost+found
drwxr-xr-x 2 nfsnobody nfsnobody 4096 Jul 29 19:14 media
drwxr-xr-x 2 nfsnobody nfsnobody 4096 Jul 29 19:14 mnt
drwxr-xr-x 3 nfsnobody nfsnobody 4096 Aug 13 15:18 opt
dr-xr-xr-x 209 nfsnobody nfsnobody 0 Aug 15 09:13 proc
drwxr-xr-x 6 livy livy 160 Aug 15 09:13 run
lrwxrwxrwx 1 livy livy 8 Aug 15 09:13 sbin -> usr/sbin
drwxr-xr-x 2 nfsnobody nfsnobody 4096 Jul 29 19:14 srv
-rw------- 1 nfsnobody nfsnobody 2147483648 Aug 15 08:40 swapfile
drwxr-xr-x 7 livy livy 140 Aug 15 09:13 sys
drwxr-xr-x 4 livy livy 80 Aug 15 09:13 tmp
drwxr-xr-x 12 nfsnobody nfsnobody 4096 Aug 13 16:00 usr
drwxr-xr-x 7 livy livy 160 Aug 15 09:13 var
lrwxrwxrwx 1 livy livy 30 Aug 15 09:13 vmlinuz -> boot/vmlinuz-4.15.0-54-generic
From the file owners, permissions, and modification date... it looks like they are working on different file systems. VSCode's /bin even points to /usr/bin, while it is not the case in Xfce terminal. The strange thing is that I can still use VSCode's terminal to navigate my home directory (/home/livy) and make changes to files. It even sources the content of my ~/.bashrc file.
What I am missing here?
I was having this exact same issue and discovered that the cause was that VSCode was installed as a Flatpak package. Uninstall it (be it a snap or a Flatpak package) and install the .deb file distributed by Microsoft.
I am trying to upgrade zookeeper from 3.4.8 to 3.4.13.
Before upgrade the content of /usr/lib/zookeeper
drwxr-xr-x 5 root root 4.0K Aug 23 08:39 .
drwxr-xr-x 77 root root 12K Aug 23 08:50 ..
drwxr-xr-x 2 root root 4.0K Aug 23 08:39 bin
lrwxrwxrwx 1 root root 19 May 24 11:25 conf -> /etc/zookeeper/conf
drwxr-xr-x 2 root root 4.0K Aug 23 08:39 lib
-rw-r--r-- 1 root root 12K May 24 11:25 LICENSE.txt
-rw-r--r-- 1 root root 170 May 24 11:25 NOTICE.txt
-rw-r--r-- 1 root root 1.3M Aug 23 08:39 zookeeper-3.4.8.jar
lrwxrwxrwx 1 root root 38 Aug 23 08:39 zookeeper.jar -> /usr/lib/zookeeper/zookeeper-3.4.8.jar
As mentioned in answer I have downloaded the zookeeper from this link and placed the zookeeper-3.4.13.jar in /usr/lib/zookeeper and pointed the symbolic link like below
lrwxrwxrwx 1 root root 39 Aug 30 03:19 zookeeper.jar -> /usr/lib/zookeeper/zookeeper-3.4.13.jar
But on checking the status after resarting zookeeper it is still pointing to 3.4.8
ubuntu#vrni-platform:/etc/zookeeper/conf$ telnet localhost 2181
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
Zookeeper version: 3.4.8--1, built on 02/06/2016 03:18 GMT
It appears this is because of the way the jars are loaded from /usr/lib/zookeeper/bin/zkEnv.sh
#release tarball format
for i in "$ZOOBINDIR"/../zookeeper-*.jar
do
CLASSPATH="$i:$CLASSPATH"
done
Can someone let me know is this some known issue is zkEnv.sh? Is this expected?
This has been answered in zookeeper mailing list. We should not have multiple zookeeper-<version>.jar in the CLASSPATH.
I am trying to install mongodb on Centos 6.6. I am following the steps here: https://www.liquidweb.com/kb/how-to-install-mongodb-on-centos-6/
I have added the following (where it states on the tutorial):
[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1
As you can see here:
However, when I try and install I get the following error
Where am I going wrong?
My file listing for /etc/yum.repos.d is;
4.0K drwxr-xr-x 2 root root 4.0K Jun 26 13:12 .
4.0K drwxr-xr-x 66 root root 4.0K Jun 24 14:19 ..
4.0K -rw-r--r-- 1 root root 2.0K Oct 23 2014 CentOS-Base.repo
4.0K -rw-r--r-- 1 root root 647 Oct 23 2014 CentOS-Debuginfo.repo
4.0K -rw-r--r-- 1 root root 289 Oct 23 2014 CentOS-fasttrack.repo
4.0K -rw-r--r-- 1 root root 630 Oct 23 2014 CentOS-Media.repo
8.0K -rw-r--r-- 1 root root 5.3K Oct 23 2014 CentOS-Vault.repo
4.0K -rw-r--r-- 1 root root 1.1K Jul 3 2015 city-fan.org.repo
4.0K -rw-r--r-- 1 root root 191 Jun 26 12:56 mongodb-org-3.2.repo
12K -rw------- 1 root root 12K Jun 26 12:53 .mongodb-org-3.2.repo.swp
4.0K -rw-r--r-- 1 root root 200 Jun 26 13:12 mongodb-org-3.4.repo
12K -rw------- 1 root root 12K Jun 26 12:55 .mongodb-org.repo.swp
12K -rw------- 1 root root 12K Jun 24 14:16 .mongodb.repo.swp
4.0K -rw-r--r-- 1 root root 472 Apr 26 2016 nodesource-el.repo
4.0K -rw-r--r-- 1 root root 219 Dec 18 2013 vz.repo
From your error message it would appear that at least one of your .repo files is corrupt.
The file mongodb-org-3.2.repo has an errant \n on line 2 But looking at your file listing it would appear you have several files, some of which are corrupted (hence the .swp files left behind)
You should do a full file listing with ls -lash /etc/yum.repos.d and delete all the files with 'mongo' in the name.
Then create a new file named mongodb.repo and add the following to it;
[mongodb-org-3.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc
Then run yum install mongodb-org
I think the param in [mongodb] is not enough and is a lack of header information. you should specify the version as well.
for example for MongoDB v2.0, il should look like:
[mongodb-org-2.0]
name=...ect
regards
This is on Ubuntu 14.04 LTS
EDIT:
WAL-E installed using pip, with secret keys managed by envdir, according to the instructions https://gist.github.com/elithrar/8682235 and https://github.com/wal-e/wal-e#dependencies.
I am trying to restore a database using WAL-E, and everything appears to go well initially, as I have Postgres installed and running, and can easily create or restore a database and access it locally and remotely via pgadmin. Where it goes bad is when I try to perform a restore from an S3 backup using wal-e fetch-backup. It appears to go well up until the point of starting the postgres.
There are a number of errors that come up, appearing to be either missing packages or permissions issues, like the following:
* Starting PostgreSQL 9.3 database server
* The PostgreSQL server failed to start. Please check the log output:
2015-07-11 00:41:11 EDT LOG: database system was interrupted; last known up at 2015-06-30 05:00:02 EDT
2015-07-11 00:41:11 EDT LOG: starting archive recovery
Traceback (most recent call last):
File "/usr/local/bin/wal-e", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3084, in <module>
#_call_aside
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3070, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3097, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 651, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 952, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 847, in resolve
new_requirements = dist.requires(req.extras)[::-1]
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2602, in requires
dm = self._dep_map
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2803, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2825, in _compute_dependencies
for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2794, in _parsed_pkg_info
metadata = self.get_metadata(self.PKG_INFO)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1617, in get_metadata
return self._get(self._fn(self.egg_info, name))
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1728, in _get
with open(path, 'rb') as stream:
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/six-1.9.0.dist-info/METADATA'
2015-07-11 00:41:11 EDT LOG: invalid checkpoint record
2015-07-11 00:41:11 EDT FATAL: could not locate required checkpoint record
2015-07-11 00:41:11 EDT HINT: If you are not restoring from a backup, try removing the file "/var/lib/postgresql/9.3/main/backup_label".
2015-07-11 00:41:11 EDT LOG: startup process (PID 1693) exited with exit code 1
2015-07-11 00:41:11 EDT LOG: aborting startup due to startup process failure
I had several of these and was able to resolve them by changing the group and modifying the permissions on the noted files to match others in the directory, but I suspect the problem has more to do with how and/or where these packages were installed. After resolving the above issue, postgres still fails to start up, returning the following:
* Starting PostgreSQL 9.3 database server
* The PostgreSQL server failed to start. Please check the log output:
2015-07-11 00:30:04 EDT LOG: database system was interrupted; last known up at 2015-06-30 05:00:02 EDT
2015-07-11 00:30:04 EDT LOG: starting archive recovery
Traceback (most recent call last):
File "/usr/local/bin/wal-e", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3084, in <module>
#_call_aside
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3070, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3097, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 651, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 952, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 839, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'wal-e==0.8.1' distribution was not found and is required by the application
2015-07-11 00:30:04 EDT LOG: invalid checkpoint record
2015-07-11 00:30:04 EDT FATAL: could not locate required checkpoint record
2015-07-11 00:30:04 EDT HINT: If you are not restoring from a backup, try removing the file "/var/lib/postgresql/9.3/main/backup_label".
2015-07-11 00:30:04 EDT LOG: startup process (PID 1495) exited with exit code 1
2015-07-11 00:30:04 EDT LOG: aborting startup due to startup process failure
It is complaining about the 'wal-e==0.8.1' distribution was not found..., but it is clearly installed and executable:
ls -l /usr/local/lib/python2.7/dist-packages
total 608
drwxr-sr-x 2 root staff 4096 Jul 10 20:14 argparse-1.3.0.dist-info
-rw-r--r-- 1 root staff 88400 Jul 10 20:14 argparse.py
-rw-r--r-- 1 root staff 65659 Jul 10 20:14 argparse.pyc
drwxr-sr-x 6 root staff 4096 Jul 9 22:36 azure
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 azure-0.11.1.egg-info
drwxr-sr-x 5 root staff 4096 Jul 9 22:36 babel
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 Babel-1.3.egg-info
drwxr-sr-x 57 root staff 4096 Jul 9 22:36 boto
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 boto-2.38.0.dist-info
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 concurrent
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 dateutil
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 debtcollector
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 debtcollector-0.5.0.dist-info
-rw-r--r-- 1 root staff 207 Jul 10 20:10 easy-install.pth
-rw-r--r-- 1 root staff 126 Jul 10 20:33 easy_install.py
-rw-r--r-- 1 root staff 315 Jul 10 20:33 easy_install.pyc
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 futures-3.0.3.dist-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 gevent
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 gevent-1.0.2.egg-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 greenlet-0.4.7.egg-info
-rwxr-xr-x 1 root staff 82869 Jul 9 22:36 greenlet.so
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 iso8601
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 iso8601-0.1.10.egg-info
drwxr-sr-x 15 root staff 4096 Jul 9 22:36 keystoneclient
drwxr-sr-x 2 root staff 4096 Jul 10 20:33 _markerlib
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 msgpack
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 msgpack_python-0.4.6.egg-info
drwxr-sr-x 5 root staff 4096 Jul 9 22:36 netaddr
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 netaddr-0.7.15.dist-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 netifaces-0.10.4.egg-info
-rwxr-xr-x 1 root staff 58386 Jul 9 22:36 netifaces.so
drwxr-sr-x 4 root staff 4096 Jul 9 22:36 oslo
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 oslo_config
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 oslo.config-1.14.0.dist-info
-rw-r--r-- 1 root root 299 Jul 9 22:35 oslo.config-1.14.0-py2.7-nspkg.pth
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 oslo_i18n
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 oslo.i18n-2.1.0.dist-info
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 oslo_serialization
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 oslo.serialization-1.7.0.dist-info
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 oslo_utils
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 oslo.utils-1.8.0.dist-info
-rw-r--r-- 1 root root 299 Jul 9 22:35 oslo.utils-1.8.0-py2.7-nspkg.pth
drwxr-sr-x 5 root staff 4096 Jul 10 20:14 pbr
drwxr-sr-x 2 root staff 4096 Jul 10 20:14 pbr-1.3.0.dist-info
drwxr-sr-x 4 root staff 4096 Jul 10 20:10 pip-7.1.0-py2.7.egg
drwxr-sr-x 3 root staff 4096 Jul 10 20:33 pkg_resources
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 python_dateutil-2.4.2.dist-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 python_keystoneclient-1.6.0.dist-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 python_swiftclient-2.4.0.dist-info
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 pytz
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 pytz-2015.4.dist-info
drwxr-sr-- 3 root staff 4096 Jul 9 23:25 requests
drwxr-sr-- 2 root staff 4096 Jul 9 23:25 requests-2.7.0.dist-info
drwxr-sr-x 3 root staff 4096 Jul 10 20:33 setuptools
drwxr-sr-x 2 root staff 4096 Jul 10 20:33 setuptools-18.0.1.dist-info
drwxr-sr-x 3 root staff 4096 Jul 9 22:36 simplejson
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 simplejson-3.7.3.egg-info
drwxr-sr-- 2 root staff 4096 Jul 9 23:26 six-1.9.0.dist-info
-rw-r--r-- 1 root root 29664 Jul 9 23:26 six.py
-rw-r--r-- 1 root root 29006 Jul 9 23:26 six.pyc
drwxr-sr-x 4 root staff 4096 Jul 9 22:36 stevedore
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 stevedore-1.6.0.dist-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 swiftclient
drwxr-sr-x 7 root staff 4096 Jul 9 22:35 wal_e
drwxr-sr-x 2 root staff 4096 Jul 9 22:35 wal_e-0.8.1.egg-info
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 wrapt
drwxr-sr-x 2 root staff 4096 Jul 9 22:36 wrapt-1.10.5.egg-info
I have done quite a bit of searching, but haven't found anything that helps. Any suggestions or points in the right direction are appreciated.
Additionally, it would be of great interest to solve this to ensure that my primary installation will behave in the event that a restoration is necessary. Backing up is pointless if I can't restore it.
I am not entirely sure what resolved the The 'wal-e==0.8.1' distribution was not found errro, but on rebooting I never saw that again.
Aside from that, fixing this was rather straightforward.
The executable bit on a number of python distributions was not set.
Chowning these fixed the error:
chmod o+x /usr/local/lib/python2.7/dist-packages/requests-2.7.0.dist-info/
chmod o+x /usr/local/lib/python2.7/dist-packages/requests
...
Reason backup_label is not created properly
Run the below command in your Master
su postgres
psql -c "select pg_start_backup('initial_backup');"
rsync -cva --inplace --exclude=pg_xlog
/var/lib/postgresql/9.1/main/
slave_IP_address:/var/lib/postgresql/9.1/main/
psql -c "select pg_stop_backup();"
cd /var/lib/postgresql/9.1/main
create file name recovery.conf add the below lines
standby_mode = 'on'
primary_conninfo = 'host=master_IP_address
port=5432 user=rep password=yourpassword' trigger_file =
'/tmp/postgresql.trigger.5432'
service postgresql restart