Where is the missing postgres-plpython-12 for Ubuntu 20.04, required for psycopg2? - postgresql

There is no available package for postgres-plpython-12
https://packages.ubuntu.com/search?keywords=postgresql-plpython
The psycopg2 library does not work with plpython3 (or does it)?
psycopg2.errors.UndefinedFile: could not access file "$libdir/plpython2": No such file or directory
So how does anyone get psycopg2 working on Ubuntu >= 20.04 ?
The libpq* libraries are installed.
libpq-dev is already the newest version (13.4-1.pgdg18.04+1).
libpq5 is already the newest version (13.4-1.pgdg18.04+1).
Maybe everything must be on postgres-13? Since there seems to be no explicit candidates for libpq* version 12.x
The error above is generated by pytest-postgresql from
postgresql = <connection object at 0x7fa79feaf548; dsn: 'user=postgres password=xxx dbname=app_test host=127.0.0.1 port=15988 options=''', closed: 0>
#pytest.fixture
def postgresql_config(postgresql) -> Dict:
# https://github.com/ClearcodeHQ/pytest-postgresql#how-to-use
# https://github.com/ClearcodeHQ/pytest-postgresql#configuration
# see pytest.ini for configuration options
# assert isinstance(postgresql, connection)
dsn = postgresql.get_dsn_parameters()
passwd_data = Path(dsn["passfile"]).read_text().strip()
passwd = passwd_data.split(":")[-1]
dsn_path = f"{dsn['user']}:{passwd}#{dsn['host']}:{dsn['port']}"
db_url = f"postgresql+psycopg2://{dsn_path}/{dsn['dbname']}"
admin_config = {
"timeout": 10,
"host": dsn["host"],
"port": dsn["port"],
"dbname": dsn["dbname"],
"username": dsn["user"],
"password": passwd,
"url": db_url,
}
# - init the test-db using the postgres admin user
# - for some tips on idempotent DDL, see:
# https://www.depesz.com/2008/06/18/conditional-ddl/
cursor = postgresql.cursor()
for statement in sqlparse.split(_setup_sql):
if statement:
> cursor.execute(statement)
E psycopg2.errors.UndefinedFile: could not access file "$libdir/plpython2": No such file or directory

Related

micronaut runs with the DB on dev/test but not with the build image

I'm trying to use postgres on a micronaut project. When running locally, or by executing the tests, everything works correctly. But when I try to generate a docker image and run it I got the following error:
Caused by: org.hibernate.HibernateException: HR000048: Could not determine Dialect from JDBC driver metadata (specify a connection URI with scheme 'postgresql:', 'mysql:', 'cockroachdb', or 'db2:')
According to the official documentation:
When you move to production, you will need to configure the properties
injected by Test Resources to point at your real production database.
This can be done via environment variables like so:
I should set some application items described onthe JDBC section of the guide
So here is my application.yaml:
micronaut:
application:
name: micronaut-guide
netty:
default:
allocator:
max-order: 3
#tag::jpa[]
jpa:
default:
entity-scan:
packages:
- 'example.micronaut.domain'
properties:
hibernate:
show-sql: true
hbm2ddl:
auto: update
connection:
db-type: postgres
reactive: true
#end::jpa[]
datasources:
default:
driverClassName: org.postgresql.Driver
db-type: postgresql
schema-generate: CREATE_DROP
dialect: POSTGRES
jpa.default.properties.hibernate.hbm2ddl.auto: update
Plus, I've tried to pass some environment variable when running it:
docker run micronaut-postgres -p 8080:8080 --rm -e DATASOURCES_DEFAULT_URL=$DATASOURCES_DEFAULT_URL -e DATASOURCES_DEFAULT_USERNAME=$DATASOURCES_DEFAULT_USERNAME -e DATASOURCES_DEFAULT_PASSWORD=$DATASOURCES_DEFAULT_PASSWORD
◼ ~/Code/Personal/micronaut-postgres $ echo $DATASOURCES_DEFAULT_URL 130
jdbc:postgresql://localhost:3306/db
to be sure there are all the pieces here is my build.gradle:
plugins {
id("org.jetbrains.kotlin.jvm") version "1.6.21"
id("org.jetbrains.kotlin.kapt") version "1.6.21"
id("org.jetbrains.kotlin.plugin.allopen") version "1.6.21"
id("org.jetbrains.kotlin.plugin.jpa") version "1.6.21"
id("com.github.johnrengelman.shadow") version "7.1.2"
id("io.micronaut.application") version "3.6.7"
id("io.micronaut.test-resources") version "3.6.7"
}
version = "0.1"
group = "example.micronaut"
repositories {
mavenCentral()
}
dependencies {
kapt("io.micronaut.data:micronaut-data-processor")
kapt("io.micronaut:micronaut-http-validation")
kapt("io.micronaut.serde:micronaut-serde-processor")
implementation("io.micronaut:micronaut-http-client")
implementation("io.micronaut.data:micronaut-data-hibernate-reactive")
implementation("io.micronaut.kotlin:micronaut-kotlin-runtime")
implementation("io.micronaut.reactor:micronaut-reactor")
implementation("io.micronaut.reactor:micronaut-reactor-http-client")
implementation("com.ongres.scram:client:2.1")
implementation("io.micronaut.serde:micronaut-serde-jackson")
// implementation("io.vertx:vertx-mysql-client")
implementation("io.vertx:vertx-pg-client")
implementation("io.micronaut.sql:micronaut-vertx-pg-client")
implementation("jakarta.annotation:jakarta.annotation-api")
implementation("org.jetbrains.kotlin:kotlin-reflect:${kotlinVersion}")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8:${kotlinVersion}")
runtimeOnly("ch.qos.logback:logback-classic")
// testResourcesService("mysql:mysql-connector-java")
testResourcesService("org.postgresql:postgresql")
compileOnly("org.graalvm.nativeimage:svm")
implementation("io.micronaut.sql:micronaut-vertx-pg-client")
implementation("io.micronaut:micronaut-validation")
runtimeOnly("com.fasterxml.jackson.module:jackson-module-kotlin")
}
application {
mainClass.set("example.micronaut.ApplicationKt")
}
java {
sourceCompatibility = JavaVersion.toVersion("11")
}
tasks {
compileKotlin {
kotlinOptions {
jvmTarget = "11"
}
}
compileTestKotlin {
kotlinOptions {
jvmTarget = "11"
}
}
}
graalvmNative.toolchainDetection = false
micronaut {
runtime("netty")
testRuntime("junit5")
processing {
incremental(true)
annotations("example.micronaut.*")
}
testResources {
additionalModules.add("hibernate-reactive-postgresql")
}
}
tasks.named("dockerfile") {
baseImage = "eclipse-temurin:17.0.5_8-jre-jammy"
}
Another more standard way to replicate the problem
I got into the same problem with the official example on the micronaut website available here.
I can generate the docker image, run it and the same problem arises:
Step 8/8 : ENTRYPOINT ["java", "-jar", "/home/app/application.jar"]
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in b4ac7195ad12
Removing intermediate container b4ac7195ad12
---> 98936716be8c
Successfully built 98936716be8c
Successfully tagged micronautguide:latest
Created image with ID '98936716be8c'.
...
# just checking the env var is there
$ echo $JDBC_URL
jdbc:mysql://production-server:3306/micronaut
$ docker run --rm -e DATASOURCES_DEFAULT_URL=$JDBC_URL -e DATASOURCES_DEFAULT_USERNAME=$JDBC_USER -e DATASOURCES_DEFAULT_PASSWORD=$JDBC_PASSWORD -it -p 8080:8080 98936716be8c 125
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
...
Caused by: org.hibernate.HibernateException: HR000048: Could not determine Dialect from JDBC driver metadata (specify a connection URI with scheme 'postgresql:', 'mysql:', 'cockroachdb', or 'db2:')

How can I start nest app with Postgress DB on Win 10?

I'm front dev and I need to test locally my front app with backend (nest js) and postgresql DB. Who can write me the right way How to run and connect to DB ? I get some errors on app start. I work on win 10 and there is my steps for start this app.
install postgresql
npm install for my nest js app
run pgAdmin4 and create DB for my app
npm start
There is my ormconfig
module.exports = {
"type": "postgres",
"host": process.env.POSTGRES_HOST || "localhost",
"port": process.env.POSTGRES_PORT || 5432,
"username": process.env.POSTGRES_USER || "", //<- Here I try to set all possible username
"password": process.env.POSTGRES_PASSWORD || "", //<- Here I try to set all possible password
"database": process.env.POSTGRES_DB || "my_database",
"entities": ["dist/**/*.entity{.ts,.js}"],
"synchronize": true,
"logging": true
}
There is error that I encountered
error
Also on other computer I try to do this and I get error like
[Nest] ERROR [TypeOrmModule] Unable to connect to the database.
FATAL: password authentication failed for user "postgres" (postgresql 14 with pgAdmin 4)
In your typeormconfig.ts , you should write this:
export class PostgresTypeormConfiguration implements TypeOrmOptionsFactory
{
createTypeOrmOptions(connectionName?: string): TypeOrmModuleOptions | Promise<TypeOrmModuleOptions> {
const TypeOrmOptions:TypeOrmModuleOptions=
{
type: "postgres",
host: process.env.POSTGRES_HOST ,
port: process.env.POSTGRES_PORT ,
username: process.env.POSTGRES_USER ,
password: process.env.POSTGRES_PASSWORD ,
database: process.env.POSTGRES_DB,
entities: ["dist/**/*.entity{.ts,.js}"],
synchronize: true,
logging: true
}
return TypeOrmOptions
}
}
and you should define this in your module like this:
#Module({
imports:[TypeOrmModule.forRootAsync({useClass:PostgresTypeormConfiguration})]
})
note: if you still got an error , you wrote one of the config option wrong in your .env file or you did not define .env file in your configModule

Wasm-rust and postgres

is there a way to get data from a database in postgres using wasm?. I'd tried to get it using a library in rust but I got some errore when I build the package using "wasm-pack building--target web". The idea is to build a function in lib.rs file that return data from a db. I have the below code inside lib.rs:
use postgres::{Client, Error, NoTls};
use wasm_bindgen::prelude::*;
...
struct Author {
_id: i32,
name: String,
}
#[wasm_bindgen]
pub fn select_name(name: &String) -> Result<(), Error> {
let mut client = Client::connect("postgresql://user:1234#localhost:5432/db", NoTls)?;
for row in client.query(
"SELECT id, name FROM author WHERE name = $1",
&[&name],
)? {
let author = Author {
_id: row.get(0),
name: row.get(1),
};
println!(
"Select_Name => Author {} :",
author.name
);
}
Ok(())
}
but I get some errors:
error[E0432]: unresolved import `crate::sys::IoSourceState`
error[E0432]: unresolved import `crate::sys`
...
It is not possible directly (as Njuguna Mureithi rightly wrote) but it can be circumvented.
We can use the project: https://github.com/PostgREST/postgrest
and expose the API to our sql server.
We download the latest version of postgrest
https://github.com/PostgREST/postgrest/releases/tag/v9.0.0
in case of Linux we unpack
tar -xf postgrest-v9.0.0-linux-static-x64.tar.xz
then run help
./postgrest -h
create a configuration file for ./postgrest
postgrest -e > cfg.psqlrest
change the user and password for the database in the configuration file.
e.g. with
db-uri = "postgres://user:pasword#localhost:5432/dbname"
to your dbname database access user:pasword configuration
db-uri = "postgres://postgres:zaqwsxc#localhost:5432/dbname"
and run the server which will issue the api to our postgres
./postgrest cfg.psqlrest
The address http://localhost:3000 will start accessing the database dbname, which must be created in the database server beforehand.
Here is a description of the libraries needed to call the query using the API.
https://rustwasm.github.io/wasm-bindgen/examples/fetch.html
examples of API
https://postgrest.org/en/stable/api.html

rust-lld: error: unable to find library -llibpq

Useful info
I am building a webapp and using as back-end rocket.rs, with diesel.rs to connect to the Postgres database. I wanto to dockerize this webapp: I'm on Windows 11, and I want to cross-compile for Linux alpine, so I can just add the executable in the container.
Toolchain: nightly-2021-12-05-x86_64-pc-windows-msvc
Target: x86_64-unknown-linux-musl
Cargo.toml file:
[package]
name = "project-api"
version = "1.0.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
rocket = { version = "0.4.10", default-features = false }
diesel = { version = "1.4.8", features = ["postgres"] }
serde = { version = "1.0.131", features = ["derive"]}
dotenv = "0.15.0"
[dependencies.rocket_contrib]
version = "0.4.10"
default-features = false
features = ["json", "serve", "diesel_postgres_pool"]
.cargo/config.toml file:
[target.x86_64-unknown-linux-musl]
linker = "rust-lld"
Compile command: cargo build --target x86_64-unknown-linux-musl --release
In my PATH I already added:
.../PostgreSQL/14/bin
.../PostgreSQL/14/lib
.../PostgreSQL/14/include
The problem
When I compile the project, I get the following error:
error: linking with `rust-lld` failed: exit code: 1
...
= note: rust-lld: error: unable to find library -llibpq
This is the complete error:
Compiling project-api v1.0.0 (D:\projects\project-management-system)
error: linking with `rust-lld` failed: exit code: 1
|
= note: "rust-lld" "-flavor" "gnu" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\rcrt1.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crti.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtbeginS.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.0.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.1.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.10.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.11.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.12.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.13.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.14.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.15.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.2.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.3.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.4.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.5.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.6.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.7.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.8.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.9.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.22qos1iwh0pu1l3f.rcgu.o" "--as-needed" "-L" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps" "-L" "D:\\projects\\project-management-system\\target\\release\\deps" "-L" "D:/programs/PostgreSQL/14/lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-Bstatic" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdotenv-17787ac5e36ba405.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_contrib-908e672c5a94e5a1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdiesel-b618b4aafb14a3d4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpq_sys-0deff7ab575185bc.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbitflags-cd7b464110d56932.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libr2d2-17991d433d6b8815.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscheduled_thread_pool-ed3fe870ef2578f4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot-084e1692fb5bda8c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot_core-57505ee8fbe2715a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblock_api-c4af51bd9cc3c613.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscopeguard-d10f930610969f1b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libinstant-5b170197aa126111.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde_json-f02c39ef48f71da7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libryu-62ce0cec239f97c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libitoa-8ec78a5baa91caac.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket-28b96e263956caad.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libatty-11406b7a761cb014.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-d9fe97a8863a409c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmemchr-1dd949379d427674.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtoml-6efc66767f0cacd7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde-4add0bb05fd83ce8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libyansi-077b82a357bfe296.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_http-9426f414126ee4c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhyper-5854b3142fe943a8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmime-41e7a782b7da09c1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-8290fbf24ebfc650.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-52629d6d7adb881b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcfg_if-b1347d601d516de4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblanguage_tags-1f120d4eb4c64568.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtypeable-37b90a66c0e53fd4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtraitobject-e42a82912129d07a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libnum_cpus-2c9ebf56e8506369.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhttparse-5fdd6843d2b90a53.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicase-1ff8862237f0ad7e.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liburl-cb2f1703c2051404.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libidna-2dccc330fa120c40.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_normalization-05c479f908735c18.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec-89542cb3fe5ec50d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec_macros-0d79dc173afd5963.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_bidi-25a8617db7cb03aa.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmatches-5a438e594be46ce5.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-1d678e0cfd664cb9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsafemem-7dc2f1cd7551f3b9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbyteorder-2481812d3e7c910d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_xid-d2b897ae1242d958.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libstate-d915de2c2db68eef.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libindexmap-3190e1dd9e2809b7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhashbrown-4794792eab7bd8e3.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcookie-b22759f25ba38a69.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtime-6373f03850cf300d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblibc-d520e3667288b8da.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-ee9e622516b87769.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsmallvec-2817ef9eba1ed532.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-dd42f0c66c80313d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpear-f2f7fd587bab036d.rlib" "--start-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd-84fa96c5a6567c2a.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libpanic_unwind-2694d6b5ee3337f3.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libminiz_oxide-58e7d510a302cb06.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libadler-3b97f36d6f391442.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libobject-7f2f74160dfc1203.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libmemchr-bc62538a6b3d16cf.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libaddr2line-531fc0db5fed2ff4.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libgimli-267be56b670e3626.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd_detect-5efe0f28e08ac5c2.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_demangle-5cb8086f7dea7d11.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libhashbrown-26229185d2048651.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_alloc-5e539cff88cca34d.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libunwind-a33879496a1412b2.rlib" "-lunwind" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcfg_if-cb8b44b450e6e086.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liblibc-9946a456dc1d9b49.rlib" "-lc" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liballoc-7fa17c9ac5f3bbe0.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_core-d23d1fdf2f9c2e82.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcore-4927c240c984c2ab.rlib" "--end-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcompiler_builtins-ff05cc303f3269e0.rlib" "-Bdynamic" "-llibpq" "--eh-frame-hdr" "-znoexecstack" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained" "-o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7" "--gc-sections" "-static" "-pie" "--no-dynamic-linker" "-z" "text" "-zrelro" "-znow" "-O1" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtendS.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtn.o"
= note: rust-lld: error: unable to find library -llibpq
error: could not compile `project-api` due to previous error
What I tried
I already tried adding the libpq.dll and libpq.lib files to the x86_64-unknown-linux-musl/lib folder, but I got the same error. I tried adding it to the default target too, but it didn't work. I found out it could be a problem of some build.rs file in a library... but I don't know which one. I also tried compiling with the regular nightly toolchain.

No grid/console views in buildbot, build view always empty, despite successful build

I have installed buildbot -- one docker image with a master, and another with a worker. Inter-container networking is allowed, and they share the same network; I have also a gitea instance, and installed the buildbot_gitea plugin.
So far I got a small project to run make on the worker after a push, and buildbot correctly reports success back to gitea (I can tell form the logs, and gitea also shows the green check image on the repo).
However,
the waterfall view is always empty; console and grid views do not load (they show the "loading" animation and never finish);
in the "Home" buildbot tab, sometimes the list of recent builds show up, sometimes it doesn't. (But the number of recent builds is always correct)
if I click on one of the builds (successful or not, doesn't matter), it shows a build page, but empty (no build steps, no build properties, nothing).
The only things that look strange on the master logs are periodic timeout messages, some connection drop messages:
2020-03-21 12:11:26+0000 [-] Timing out client: IPv4Address(type='TCP', host='172.27.0.1', port=56388)
2020-03-21 12:11:26+0000 [-] Timing out client: IPv4Address(type='TCP', host='172.27.0.1', port=56380)
2020-03-21 12:11:26+0000 [-] Timing out client: IPv4Address(type='TCP', host='172.27.0.1', port=56392)
2020-03-21 12:19:40+0000 [-] dropping connection to peer tcp4:172.27.0.1:56598 with abort=False: None
and this:
2020-03-21 12:10:49+0000 [-] Unhandled error in Deferred:
2020-03-21 12:10:49+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/_threads/_threadworker.py", line 46, in work
task()
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/_threads/_team.py", line 190, in doWork
task()
--- <exception caught here> ---
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/bbot/sandbox/lib/python3.8/site-packages/buildbot/buildbot_net_usage_data.py", line 204, in _sendBuildbotNetUsageData
res = _sendWithRequests(PHONE_HOME_URL, data)
File "/bbot/sandbox/lib/python3.8/site-packages/buildbot/buildbot_net_usage_data.py", line 197, in _sendWithRequests
r = requests.post(url, json=data)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='events.buildbot.net', port=443): Max retries exceeded with url: /events/phone_home (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7297704f0>: Failed to establish a new connection: [Errno 110] Operation timed out'))
which seems to happen only once (why is buildbot trying to phone home anyway? there is no mention of events.buildbot.net in any of my config files.
The docker containers have full network access, ipv6, routing and DNS are all fine (tested with the buildbot-master image).
This is my master.cfg:
import os
from twisted.application import service
from buildbot.master import BuildMaster
from buildbot.plugins import *
from buildbot_gitea.auth import GiteaAuth
from buildbot_gitea import *
basedir = '/bbot/bbot-master'
rotateLength = 10000000
maxRotatedFiles = 10
configfile = 'master.cfg'
umask = None
if basedir == '.':
basedir = os.path.abspath(os.path.dirname(__file__))
application = service.Application('buildmaster')
from twisted.python.logfile import LogFile
from twisted.python.log import ILogObserver, FileLogObserver
logfile = LogFile.fromFullPath(os.path.join(basedir, "twistd.log"), rotateLength=rotateLength,
maxRotatedFiles=maxRotatedFiles)
application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
m = BuildMaster(basedir, configfile, umask)
m.setServiceParent(application)
m.log_rotation.rotateLength = rotateLength
m.log_rotation.maxRotatedFiles = maxRotatedFiles# -*- python -*-
# ex: set filetype=python:
from buildbot.plugins import *
c = BuildmasterConfig = {}
####### WORKERS
c['workers'] = [worker.Worker("bbot-worker", "BUILDBOT_PASSWORD")]
c['protocols'] = {'pb': {'port': 9989}}
####### CHANGESOURCES
c['change_source'] = []
c['change_source'].append(changes.PBChangeSource())
####### SCHEDULERS
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
name="all",
change_filter=util.ChangeFilter(branch='master'),
treeStableTimer=None,
builderNames=["runtests"]))
c['schedulers'].append(schedulers.ForceScheduler(
name="force",
builderNames=["runtests"]))
####### BUILDERS
factory = util.BuildFactory()
factory.addStep(steps.Gitea(repourl='gitea#gitea.mydomain:myself/repo.git',
mode='incremental',
workdir="build",
branch="master",
progress=True,
logEnviron=False,
))
factory.addStep(steps.ShellCommand(command=["make"]))
c['builders'] = []
c['builders'].append(
util.BuilderConfig(name="runtests",
workernames=["bbot-worker"],
factory=factory))
####### BUILDBOT SERVICES
c['services'] = [
reporters.GiteaStatusPush(
baseURL="https://gitea.mydomain/",
token="GITEA_API_ACCESS_TOKEN",
verbose=True)
]
####### PROJECT IDENTITY
c['title'] = "My Domain!"
c['titleURL'] = "https://buildbot.mydomain"
c['buildbotURL'] = "https://buildbot.mydomain/"
c['www'] = dict(port=8010,
plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
c['www']['authz'] = util.Authz(
allowRules = [
util.AnyEndpointMatcher(role="admins")
],
roleMatchers = [
util.RolesFromUsername(roles=['admins'], usernames=['myself'])
]
)
c['www']['auth'] = GiteaAuth(
endpoint="https://gitea.mydomain/",
client_id="MY_CLIENT_ID_FROM_GITEA",
client_secret='MY_CLIENT_SECRET_FROM_GITEA')
c['www']['change_hook_dialects'] = {
'gitea': {
'secret': 'THE_GITEA_WEBHOOK_SECRET',
'onlyIncludePushCommit': True
}
}
####### DB URL
c['db'] = {
'db_url' : "postgresql://buildbot:MY_SECRET_DB_PASSWORD#172.25.0.2/buildbot",
}
The Dockerfile for the master is
FROM alpine:3.11.3
EXPOSE 9989
RUN apk update
RUN apk add python3 bash busybox-extras w3m gcc python3-dev libffi-dev openssl-dev musl-dev postgresql-dev
RUN mkdir /bbot
COPY entrypoint.sh /root/
RUN chmod a+x /root/entrypoint.sh
RUN mkdir /root/.ssh && chmod og-rwx /root/.ssh/
COPY bbot-gitea bbot-gitea.pub /root/.ssh/
RUN chmod og-w /root/.ssh/bbot-gitea*
RUN cd /bbot && \
python3 -m venv sandbox && \
source sandbox/bin/activate && \
pip3 install 'buildbot[bundle]' && \
pip3 install 'requests[security]' && \
pip3 -v install buildbot_gitea && \
pip3 install treq && \
pip3 install psycopg2
RUN apk del gcc python3-dev libffi-dev openssl-dev musl-dev
RUN ls -la /root
RUN cat /root/entrypoint.sh
ENTRYPOINT [ "/root/entrypoint.sh" ]
and the entrypoint does nothing special -- it is this,
#!/bin/bash
cd /bbot
echo " BBOT MASTER ENTRYPOINT"
source sandbox/bin/activate
buildbot upgrade-master bbot-master
# debug: check everything that was pip-installed:
echo "\n\n=====\n"
pip3 list
echo "=====\n\n"
if [ ! -f bbot-master/buildbot.tac ]; then
buildbot create-master bbot-master
fi
buildbot start bbot-master
tail -f /bbot/bbot-master/twistd.log
and the pip3 list line, which runs on startup for debugging, shows that I have
buildbot 2.7.0
buildbot-console-view 2.7.0
buildbot-gitea 1.2.0
buildbot-grid-view 2.7.0
buildbot-waterfall-view 2.7.0
buildbot-worker 2.7.0
buildbot-www 2.7.0
edit: checked the JS console in Firefox, and there seems to be a problem connecting to the server via websockets:
Firefox can’t establish a connection to the server at wss://buildbot.mydomain/ws.
From Chrome, this is what I see:
WebSocket connection to 'wss://buildbot.aleph0.info/ws' failed: Error during WebSocket handshake: Unexpected response code: 200
(200? why 200?)
I can't see why it wouldn't work. Apache is configured to do reverse proxying, like this:
RewriteEngine On
RewriteCond ${HTTP:Upgrade} websocket [NC]
RewriteCond ${HTTP:Connection} upgrade [NC]
RewriteRule .* "wss:/localhost:8010/$1" [P,L]
ProxyPass / http://localhost:8010/
ProxyPassReverse / http://localhost:8010/
So... What else can I do to continue debugging this?
(By the way, it does look like the buildbot mailing list is not very active -- after posting this question there I checked the archives, and there is svery low activity. Where do users of Buildbot go these days in order to get and share advice?)
I found the problem!
It was the reverse proxy that wasn't properly configured for websockets.
I used this in my apache virtualhost config,
<Location /ws>
ProxyPass ws://127.0.0.1:8010/ws
ProxyPassReverse ws://127.0.0.1:8010/ws
</Location>
ProxyPass /ws !
ProxyPass / http://localhost:8010/
ProxyPassReverse / http://localhost:8010/
and it works now!
( after searching a lot, I found the solution here:
https://docs.buildbot.net/0.9.2/manual/cfg-www.html )
There it is, in case anyone else needs it.