grafana postgres data source test returns "Metric request error" - postgresql

I have deployed a docker container using the TimescaleDB extension to Postgres as a persistent store for metrics collected by Prometheus. This includes deploying the Prometheus Postgres adapter in a separate container. And then I deployed grafana in a third container.
Inside the TimescaleDB container I can execute the query:
SELECT time_bucket('5 minutes', time) AS five_min_bucket, name, avg(value)
FROM metrics
WHERE (name='node_load5' OR name='node_memory_Active_bytes') AND
time > NOW() - interval '1 day'
GROUP BY five_min_bucket,name
ORDER BY five_min_bucket;
and it returns these rows from the metrics table:
five_min_bucket | name | avg
------------------------+--------------------------+-------------------
2019-07-13 16:15:00+00 | node_load5 | 1.84263157894737
2019-07-13 16:15:00+00 | node_memory_Active_bytes | 1055776552.42105
2019-07-13 16:20:00+00 | node_load5 | 1.62533333333333
2019-07-13 16:20:00+00 | node_memory_Active_bytes | 1070225817.6
2019-07-13 16:25:00+00 | node_load5 | 1.06166666666667
2019-07-13 16:25:00+00 | node_memory_Active_bytes | 1084650018.13333
2019-07-13 16:30:00+00 | node_load5 | 0.593333333333333
2019-07-13 16:30:00+00 | node_memory_Active_bytes | 1093145668.26667
2019-07-13 16:35:00+00 | node_load5 | 0.322333333333333
2019-07-13 16:35:00+00 | node_memory_Active_bytes | 1108816145.06667
2019-07-13 16:40:00+00 | node_load5 | 0.180666666666667
2019-07-13 16:40:00+00 | node_memory_Active_bytes | 1125213388.8
2019-07-13 16:45:00+00 | node_load5 | 0.126
2019-07-13 16:45:00+00 | node_memory_Active_bytes | 1134488098.13333
2019-07-13 16:50:00+00 | node_load5 | 0.133333333333333
2019-07-13 16:50:00+00 | node_memory_Active_bytes | 1162440704
(16 rows)
The grafana container is run as:
docker run \
-d \
-p 3000:3000 \
--name=grafana \
-v grafana-storage:/var/lib/grafana \
-e "GF_INSTALL_PLUGINS=grafana-kubernetes-app" \
grafana/grafana
When I launch a browser to http://104.XX.XX.XX:3000 I can log into the web-ui and define the Datasource to postgres, but I received the error 'Metric request error'.
I looked for the grafana logs in its container and the path /var/log/grafana is empty. This is the log location indicated when I do a docker inspect command.
Using Chrome, I am able to inspect the Console and it reports the server responded with an internal error of 500.
On the host, I can confirm the docker-proxy is listening on port 5432.
sudo netstat -antp | grep "5432"
tcp 0 0 172.17.0.1:49486 172.17.0.2:5432 ESTABLISHED 60631/docker-proxy
tcp6 0 0 :::5432 :::* LISTEN 60631/docker-proxy
tcp6 0 0 10.0.0.4:5432 172.18.0.2:38568 ESTABLISHED 60631/docker-proxy
How do I get grafana to fetch metrics stored in the postgres database?

Related

Issue in postgresql HA mode switching of Master node

I am new in postgresqlDB configuration. I am trying to configure postgresDB in HA mode with the help of pgpool and Elastic IP. Full setup is in AWS RHEL 8 servers.
pgpool version : 4.1.2
postgres version - 12
Below links I have followed during the configuration:
https://www.pgpool.net/docs/pgpool-II-4.1.2/en/html/example-cluster.html#EXAMPLE-CLUSTER-STRUCTURE
https://www.pgpool.net/docs/42/en/html/example-aws.html
https://www.enterprisedb.com/docs/pgpool/latest/03_configuring_connection_pooling/
Currently the postgres and pgpool services are up in all 3 component nodes. But if I am stopping master postgres service/server whole setup is going down and standby node is not taking the place of master. Please find the status of the pool nodes when master is down:
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
---------+--------------+------+--------+-----------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
0 | server1 | 5432 | down | 0.333333 | standby | 0 | false | 0 | | | 2022-10-12 12:10:13
1 | server2 | 5432 | up | 0.333333 | standby | 0 | true | 0 | | | 2022-10-13 09:16:07
2 | server3 | 5432 | up | 0.333333 | standby | 0 | false | 0 | | | 2022-10-13 09:16:07
Any help would be appreciated. Thanks in advance.

'another command is already in progress' returned when trying to rebalance shards

There are two Postgres instances running in a Citus cluster:
analysis=# select * from citus_check_cluster_node_health();
from_nodename | from_nodeport | to_nodename | to_nodeport | result
---------------+---------------+-------------+-------------+--------
myhost | 5400 | myhost | 5400 | t
myhost | 5400 | myhost | 5432 | t
myhost | 5432 | myhost | 5400 | t
myhost | 5432 | myhost | 5432 | t
(4 rows)
When trying to rebalance shards in a table:
SELECT * FROM rebalance_table_shards('mydist1');
There is returned an error:
NOTICE: Moving shard 102008 from myhost:5432 to myhost:5400 ...
ERROR: connection to the remote node myhost:5400 failed with the following error: another command is already in progress
CONTEXT: while executing command on localhost:5432
What is the reason of this error?

FIWARE orion with atlas mongodb

Hello I'm trying to launch the FIWARE orion context broker using an atlas mongodb
mongodb+srv://<user>:****************#<domainid>.mongodb.net/<aut_db>
I have tried multiple variations of the docker run command but I never get the connection to success
trace:
$ docker run --name orion -p 1027:1026 fiware/orion:latest -dbhost "<domainid>.mongodb.net/<aut_db>?ssl=true&retryWrites=true&w=majority" -dbuser <user> -dbpwd U9NjLafksdv9mavW -logLevel DEBUG
time=2021-12-01T14:38:42.188Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[1063]:main | msg=start command line </usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -disableFileLog -dbhost <domainid>.mongodb.net/<aut_db>?ssl=true&retryWrites=true&w=majority -dbuser <user> -dbpwd ****** -logLevel DEBUG>
time=2021-12-01T14:38:42.188Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[1137]:main | msg=Orion Context Broker is running
time=2021-12-01T14:40:23.011Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=mongoConnectionPool.cpp[220]:mongoConnect | msg=Database Startup Error (cannot connect to mongo - doing 100 retries with a 1000 millisecond interval)
time=2021-12-01T14:40:23.011Z | lvl=FATAL | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=MongoGlobal.cpp[142]:mongoInit | msg=Fatal Error (MongoDB error)
time=2021-12-01T14:40:23.012Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=contextBroker.cpp[591]:exitFunc | msg=Orion shutdown completed
Has someone know how I can configure orion context broker to connect to an mongo atlas?
EDIT: I have checked and I have access from that server to the mongo-db with the mongo shell
Orion currently supports only mongodb:// connection strings, built from the -db, -dbhost, -rplSet, -dbTimeout, -dbuser, -dbpwd, -dbAuthMech, -dbAuthDb, -dbSSL and -dbDisableRetryWrites CLI parameters(or default values, if omitted) as can be see in the source code.
There is an issue in the Orion repository with the aim of making this more flexible, allowing to provide the connection string directly overriding the above parameters.
While that issue comes (contributions welcome ;) you could hack the Orion source code cited above to build an specific version for you with support for mongo+srv:// connection strings.

Unable to create new database

When I create a new mysql db, slashdb's test connection fails.
Here is how I log into mysql:
$ mysql -u 7stud -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 15
Server version: 5.5.5-10.4.13-MariaDB Homebrew
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| chat |
| ectoing_repo |
| ejabberd |
| information_schema |
| mydb |
| mysql |
| performance_schema |
| test |
+--------------------+
8 rows in set (0.00 sec)
mysql> use mydb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------+
| Tables_in_mydb |
+----------------+
| cheetos |
| greetings |
| mody |
| people |
+----------------+
4 rows in set (0.00 sec)
mysql> select * from people;
+----+--------+------+
| id | name | info |
+----+--------+------+
| 1 | 7stud | abc |
| 2 | Beth | xxx |
| 3 | Diane | xyz |
| 4 | Kathy | xyz |
| 5 | Kathy | xyz |
| 6 | Dave | efg |
| 7 | Tom | zzz |
| 8 | David | abc |
| 9 | Eloise | abc |
| 10 | Jess | xyz |
| 11 | Jeffsy | 2.0 |
| 12 | XXX | xxx |
| 13 | XXX | xxx |
+----+--------+------+
13 rows in set (0.00 sec)
In the slashdb form for creating a new database, here is the info I entered:
Hostname: 127.0.0.1
Port: 80
Database Login: 7stud
Database Password: **
Database Name: mydb
Then I hit the "Test Connection" button, whereupon I get a spinning wheel, which disappears after a few minutes, but no "Connection Successful" message. What am I doing wrong?
Now, I'm using port 3306:
mysql> SHOW GLOBAL VARIABLES LIKE 'PORT';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| port | 3306 |
+---------------+-------+
1 row in set (0.00 sec)
but when slashdb tries to connect, I get the error:
Host localhost:3306 is not accessible
Your port is wrong in the database connection. You said that your MySQL is configured on port 3306, but you also posted your SlashDB config for database with port 80. Please change that to 3306.
Also, not sure how if you don't need to enable remote access to MySQL. Even if your SlashDB is running on the same machine as the MySQL database, it uses TCP/IP to connect.

pgpool-II 3.7.5 not caching PG connections

Shouldn't pgpool cache PG backend processes? After disconnecting and reconnecting pool_backendpidchanges.
Relevant parameters:
num_init_children = 1
max_pool = 1
child_life_time = 300
child_max_connections = 0
connection_life_time = 0
client_idle_limit = 0
connection_cache = on
Test:
postgres#node3:/etc/pgpool2$ psql -p 5433 -U postgres postgres
psql (9.6.10)
Type "help" for help.
postgres=# show pool_pools;
LOG: statement: show pool_pools;
pool_pid | start_time | pool_id | backend_id | database | username | create_time | majorversion | minorversion | pool_counter | pool_backendpid | pool_connected
----------+---------------------+---------+------------+----------+----------+---------------------+--------------+--------------+--------------+-----------------+----------------
3569 | 2018-09-13 20:18:22 | 0 | 0 | postgres | postgres | 2018-09-13 20:25:04 | 3 | 0 | 1 | 3631 | 1
(1 row)
postgres=# \q
postgres#node3:/etc/pgpool2$ psql -p 5433 -U postgres postgres
psql (9.6.10)
Type "help" for help.
postgres=# show pool_pools;
LOG: statement: show pool_pools;
pool_pid | start_time | pool_id | backend_id | database | username | create_time | majorversion | minorversion | pool_counter | pool_backendpid | pool_connected
----------+---------------------+---------+------------+----------+----------+---------------------+--------------+--------------+--------------+-----------------+----------------
3569 | 2018-09-13 20:18:22 | 0 | 0 | postgres | postgres | 2018-09-13 20:25:15 | 3 | 0 | 1 | 3640 | 1
(1 row)
Found out why:
connection_cache (boolean)
Caches connections to backends when set to on. Default is on. However,
connections to template0, template1, postgres and regression databases
are not cached even if connection_cache is on.
I was connecting to postgres database.