Useful info
I am building a webapp and using as back-end rocket.rs, with diesel.rs to connect to the Postgres database. I wanto to dockerize this webapp: I'm on Windows 11, and I want to cross-compile for Linux alpine, so I can just add the executable in the container.
Toolchain: nightly-2021-12-05-x86_64-pc-windows-msvc
Target: x86_64-unknown-linux-musl
Cargo.toml file:
[package]
name = "project-api"
version = "1.0.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
rocket = { version = "0.4.10", default-features = false }
diesel = { version = "1.4.8", features = ["postgres"] }
serde = { version = "1.0.131", features = ["derive"]}
dotenv = "0.15.0"
[dependencies.rocket_contrib]
version = "0.4.10"
default-features = false
features = ["json", "serve", "diesel_postgres_pool"]
.cargo/config.toml file:
[target.x86_64-unknown-linux-musl]
linker = "rust-lld"
Compile command: cargo build --target x86_64-unknown-linux-musl --release
In my PATH I already added:
.../PostgreSQL/14/bin
.../PostgreSQL/14/lib
.../PostgreSQL/14/include
The problem
When I compile the project, I get the following error:
error: linking with `rust-lld` failed: exit code: 1
...
= note: rust-lld: error: unable to find library -llibpq
This is the complete error:
Compiling project-api v1.0.0 (D:\projects\project-management-system)
error: linking with `rust-lld` failed: exit code: 1
|
= note: "rust-lld" "-flavor" "gnu" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\rcrt1.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crti.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtbeginS.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.0.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.1.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.10.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.11.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.12.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.13.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.14.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.15.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.2.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.3.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.4.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.5.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.6.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.7.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.8.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.project_api.f9c82bac-cgu.9.rcgu.o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7.22qos1iwh0pu1l3f.rcgu.o" "--as-needed" "-L" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps" "-L" "D:\\projects\\project-management-system\\target\\release\\deps" "-L" "D:/programs/PostgreSQL/14/lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-Bstatic" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdotenv-17787ac5e36ba405.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_contrib-908e672c5a94e5a1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libdiesel-b618b4aafb14a3d4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpq_sys-0deff7ab575185bc.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbitflags-cd7b464110d56932.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libr2d2-17991d433d6b8815.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscheduled_thread_pool-ed3fe870ef2578f4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot-084e1692fb5bda8c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libparking_lot_core-57505ee8fbe2715a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblock_api-c4af51bd9cc3c613.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libscopeguard-d10f930610969f1b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libinstant-5b170197aa126111.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde_json-f02c39ef48f71da7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libryu-62ce0cec239f97c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libitoa-8ec78a5baa91caac.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket-28b96e263956caad.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libatty-11406b7a761cb014.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-d9fe97a8863a409c.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmemchr-1dd949379d427674.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtoml-6efc66767f0cacd7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libserde-4add0bb05fd83ce8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libyansi-077b82a357bfe296.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\librocket_http-9426f414126ee4c8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhyper-5854b3142fe943a8.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmime-41e7a782b7da09c1.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-8290fbf24ebfc650.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblog-52629d6d7adb881b.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcfg_if-b1347d601d516de4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblanguage_tags-1f120d4eb4c64568.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtypeable-37b90a66c0e53fd4.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtraitobject-e42a82912129d07a.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libnum_cpus-2c9ebf56e8506369.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhttparse-5fdd6843d2b90a53.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicase-1ff8862237f0ad7e.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liburl-cb2f1703c2051404.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libidna-2dccc330fa120c40.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_normalization-05c479f908735c18.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec-89542cb3fe5ec50d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtinyvec_macros-0d79dc173afd5963.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_bidi-25a8617db7cb03aa.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libmatches-5a438e594be46ce5.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbase64-1d678e0cfd664cb9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsafemem-7dc2f1cd7551f3b9.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libbyteorder-2481812d3e7c910d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libunicode_xid-d2b897ae1242d958.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libstate-d915de2c2db68eef.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libindexmap-3190e1dd9e2809b7.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libhashbrown-4794792eab7bd8e3.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libcookie-b22759f25ba38a69.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libtime-6373f03850cf300d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\liblibc-d520e3667288b8da.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-ee9e622516b87769.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libsmallvec-2817ef9eba1ed532.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpercent_encoding-dd42f0c66c80313d.rlib" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\libpear-f2f7fd587bab036d.rlib" "--start-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd-84fa96c5a6567c2a.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libpanic_unwind-2694d6b5ee3337f3.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libminiz_oxide-58e7d510a302cb06.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libadler-3b97f36d6f391442.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libobject-7f2f74160dfc1203.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libmemchr-bc62538a6b3d16cf.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libaddr2line-531fc0db5fed2ff4.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libgimli-267be56b670e3626.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libstd_detect-5efe0f28e08ac5c2.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_demangle-5cb8086f7dea7d11.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libhashbrown-26229185d2048651.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_alloc-5e539cff88cca34d.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libunwind-a33879496a1412b2.rlib" "-lunwind" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcfg_if-cb8b44b450e6e086.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liblibc-9946a456dc1d9b49.rlib" "-lc" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\liballoc-7fa17c9ac5f3bbe0.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\librustc_std_workspace_core-d23d1fdf2f9c2e82.rlib" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcore-4927c240c984c2ab.rlib" "--end-group" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\libcompiler_builtins-ff05cc303f3269e0.rlib" "-Bdynamic" "-llibpq" "--eh-frame-hdr" "-znoexecstack" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib" "-L" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained" "-o" "D:\\projects\\project-management-system\\target\\x86_64-unknown-linux-musl\\release\\deps\\project_api-565490f9d632eaa7" "--gc-sections" "-static" "-pie" "--no-dynamic-linker" "-z" "text" "-zrelro" "-znow" "-O1" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtendS.o" "C:\\Users\\user\\.rustup\\toolchains\\nightly-2021-12-05-x86_64-pc-windows-msvc\\lib\\rustlib\\x86_64-unknown-linux-musl\\lib\\self-contained\\crtn.o"
= note: rust-lld: error: unable to find library -llibpq
error: could not compile `project-api` due to previous error
What I tried
I already tried adding the libpq.dll and libpq.lib files to the x86_64-unknown-linux-musl/lib folder, but I got the same error. I tried adding it to the default target too, but it didn't work. I found out it could be a problem of some build.rs file in a library... but I don't know which one. I also tried compiling with the regular nightly toolchain.
I want to write platform independent Makefile's environment variable setup script which can set Makefile's environment variable like CC, CFLAGS, LDFLAGS, LD etc. This is my make file. I want to run it on Window or Linux as per the user need. So instead of setting CC, CFLAGS, LDFLAGS, LD every time, I want to write a script which can set the variable for user depending on which platform they are using.
LDFLAG=-L..\..\test\lib
LIBS=-ltestlibs
INCLUDES = ..\..\test\inc
SRCS = $(wildcard *.c)
OBJS = $(SRCS:.c=.o)
EXECUTABLE = targetImage.exe
.PHONY: clean
all: myprog
myprog: $(OBJS)
$(CC) $(OBJS) $(LDFLAG) $(LIBS) -o $(EXECUTABLE)
$(OBJS): $(SRCS) $(INCLUDES)
$(CC) $(CFLAGS) -I$(INCLUDES) -c $(SRCS)
clean:
$(RM) $(OBJS) $(EXECUTABLE)
I know the .sh script but it can be run on CYGWIN or MINGW.
#!/bin/sh
echo "Finding the current OS type"
echo
osType="$(uname -s)"
#osType=$1
case "${osType}" in
"CYGWIN")
{
echo "Running on CYGWIN."
CURRENT_OS=CYGWIN
export CC=""
} ;;
Linux*)
{
echo "Running on Linux."
CURRENT_OS=Linux
export CC="-g -Wall"
} ;;
"MINGW")
{
echo "Running on MINGW."
CURRENT_OS=MINGW
} ;;
"Thor96")
{
source /opt/fsl-imx-xwayland/4.14-sumo/environment-setup-aarch64-poky-linux
CURRENT_OS=THOR96
};;
*)
{
echo "Unsupported OS:${osType}, exiting"
exit
} ;;
esac
echo ${CURRENT_OS}
But it can run on linux only. So how can I achieve the same using powershell .ps1 scripts ? So same script can work on any platform.
I have drafted this .ps1 file for reference but need to update it as I am not sure it's correct or not. Please guide me with proper solution. I am not able to find the proper solution for this.
Function RunOn-Windows
{
Write-Host 'The Script is Running on a Windows Machine'
$Env:CC = "gcc"
}
Function RunOn-Linux
{
Write-Host 'The Script is Running on a Linux Machine'
$Env:CC = "gcc"
}
Function RunOn-Mac
{
Write-Host 'The Script is Running on a Mac'
}
Function RunOn-Other
{
Write-Host 'The Script is Running on a Other'
$Env:CC = "aarch64-poky-linux-gcc"
}
If ($IsWindows)
{RunOn-Windows}
elseif ($IsLinux)
{RunOn-Linux}
elseif ($IsMacOS)
{RunOn-Mac}
else
{RunOn-Other}
I don't have a Windows box to do tests but if yours has GNU make available there is a chance that you can do what you want from inside your Makefile:
# List supported OS (as returned by uname -s, case sensitive)
SUPPORTED_OS := CYGWIN Linux MINGW Darwin
# Get OS
OS := $(shell uname -s)
# Check if OS is supported
ifneq ($(filter-out $(SUPPORTED_OS),$(OS)),)
$(error Unsupported OS: $(OS))
endif
# Define all OS-dependent make variables as OS_VARIABLE
# Linux:
Linux_CC := gcc
Linux_CFLAGS := -g -Wall
Linux_LDFLAGS := ...
# CIGWIN:
CYGWIN_CC := ...
CYGWIN_CFLAGS := ...
CYGWIN_LDFLAGS := ...
...
# Assign make variables
CC := $($(OS)_CC)
CFLAGS := $($(OS)_CFLAGS)
LDFLAGS := $($(OS)_LDFLAGS)
I have a local readthedocs installation at /readthedocs.org on a Debian buster container. Python 3.6 is installed along with the pip requirements (including virtualenv). I have a single repository added to readthedocs (Openwhisk) with the following basic config:
# In .readthedocs.yml
mkdocs:
configuration: mkdocs.yml
python:
version: 3.6
system_packages: true
# In mkdocs.yml
site_name: Openwhisk
When I run python3.6 manage.py update_repos I get the following error:
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:146[4964]: INFO Running: 'python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest' [$HOME]
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:294[4964]: DEBUG Post response via JSON encoded data: {'id': 41, 'run_time': 0, 'command': 'python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest', 'description': '', 'output': 'usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]\n [--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]\n [--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]\n dest\nvirtualenv: error: unrecognized arguments: /readthedocs.org/user_builds/openwhisk/envs/latest', 'exit_code': 2, 'start_time': '2020-05-27T19:00:01.078364Z', 'end_time': '2020-05-27T19:00:01.326186Z', 'build': 7}
[27/May/2020 19:00:01] readthedocs.vcs_support.utils:129[4964]: DEBUG Lock (openwhisk): Releasing
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:625[4964]: WARNING (Build) [openwhisk:latest] Command python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest failed:
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]
[--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]
[--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]
dest
virtualenv: error: unrecognized arguments: /readthedocs.org/user_builds/openwhisk/envs/latest
Traceback (most recent call last):
File "/readthedocs.org/readthedocs/projects/tasks.py", line 763, in run_build
self.setup_python_environment()
File "/readthedocs.org/readthedocs/projects/tasks.py", line 1139, in setup_python_environment
self.python_env.setup_base()
File "/readthedocs.org/readthedocs/doc_builder/python_environments.py", line 314, in setup_base
cwd='$HOME',
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 638, in run
return super().run(*cmd, **kwargs)
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 420, in run
return self.run_command_class(cls=self.command_class, cmd=cmd, **kwargs)
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 644, in run_command_class
return super().run_command_class(*cmd, **kwargs)
File "/readthedocs.org/readthedocs/doc_builder/environments.py", line 492, in run_command_class
raise BuildEnvironmentWarning(msg)
readthedocs.doc_builder.exceptions.BuildEnvironmentWarning: Command python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest failed:
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--app-data APP_DATA] [--clear-app-data] [--discovery {builtin}] [-p py] [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed]
[--activators comma_sep_list] [--clear] [--system-site-packages] [--symlinks | --copies] [--download | --no-download] [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip]
[--no-setuptools] [--no-wheel] [--symlink-app-data] [--prompt prompt] [-h]
dest
virtualenv: error: unrecognized arguments: /readthedocs.org/user_builds/openwhisk/envs/latest
[27/May/2020 19:00:01] readthedocs.doc_builder.environments:581[4964]: INFO (Build) [openwhisk:latest] Build finished
[27/May/2020 19:00:01] celery.app.trace:124[4964]: INFO Task readthedocs.projects.tasks.send_notifications[508c1b76-28bf-4c97-9afd-5824295c378f] succeeded in 0.007888936903327703s: None
[27/May/2020 19:00:01] readthedocs.projects.tasks:1450[4964]: INFO Skipping build files deletetion for version: 1
[27/May/2020 19:00:01] celery.app.trace:124[4964]: INFO Task readthedocs.projects.tasks.update_docs_task[bf8e5976-c70a-47b8-bf9c-69a31fbd9c24] succeeded in 4.51686664391309s: True
The odd thing is when I run the exact same command from the CLI it works fine:
root#38e93376269d:/readthedocs.org# python3.6 -mvirtualenv /readthedocs.org/user_builds/openwhisk/envs/latest
created virtual environment CPython3.6.10.final.0-64 in 274ms
creator CPython3Posix(dest=/readthedocs.org/user_builds/openwhisk/envs/latest, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Any suggestions?
the use_system_site_packages: true key/value config has been removed and doesn't work anymore.
use system_packages: true instead.
python:
version: 3.6
system_packages: true
To answer my own question: There's currently a bug with readthedocs 5.1 (and possibly older versions) where having use_system_site_packages set to false in .readthedocs.yml will give this error.
To fix, put this in .readthedocs.yml:
python:
use_system_site_packages: true
The error is due to the way that Read the Docs invokes python -m virtualenv, which seems to cause issues with the building environment in some cases (but likely not for Docker environments). I can reproduce this with Debian Buster as well, and it is directly reproducible by adding an empty-string argument: python -mvirtualenv "" my_dest/
It is reported upstream, keep an eye out for a fix there: https://github.com/readthedocs/readthedocs.org/issues/7322
I have installed buildbot -- one docker image with a master, and another with a worker. Inter-container networking is allowed, and they share the same network; I have also a gitea instance, and installed the buildbot_gitea plugin.
So far I got a small project to run make on the worker after a push, and buildbot correctly reports success back to gitea (I can tell form the logs, and gitea also shows the green check image on the repo).
However,
the waterfall view is always empty; console and grid views do not load (they show the "loading" animation and never finish);
in the "Home" buildbot tab, sometimes the list of recent builds show up, sometimes it doesn't. (But the number of recent builds is always correct)
if I click on one of the builds (successful or not, doesn't matter), it shows a build page, but empty (no build steps, no build properties, nothing).
The only things that look strange on the master logs are periodic timeout messages, some connection drop messages:
2020-03-21 12:11:26+0000 [-] Timing out client: IPv4Address(type='TCP', host='172.27.0.1', port=56388)
2020-03-21 12:11:26+0000 [-] Timing out client: IPv4Address(type='TCP', host='172.27.0.1', port=56380)
2020-03-21 12:11:26+0000 [-] Timing out client: IPv4Address(type='TCP', host='172.27.0.1', port=56392)
2020-03-21 12:19:40+0000 [-] dropping connection to peer tcp4:172.27.0.1:56598 with abort=False: None
and this:
2020-03-21 12:10:49+0000 [-] Unhandled error in Deferred:
2020-03-21 12:10:49+0000 [-] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/_threads/_threadworker.py", line 46, in work
task()
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/_threads/_team.py", line 190, in doWork
task()
--- <exception caught here> ---
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/threadpool.py", line 250, in inContext
result = inContext.theWork()
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/bbot/sandbox/lib/python3.8/site-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
File "/bbot/sandbox/lib/python3.8/site-packages/buildbot/buildbot_net_usage_data.py", line 204, in _sendBuildbotNetUsageData
res = _sendWithRequests(PHONE_HOME_URL, data)
File "/bbot/sandbox/lib/python3.8/site-packages/buildbot/buildbot_net_usage_data.py", line 197, in _sendWithRequests
r = requests.post(url, json=data)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/bbot/sandbox/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='events.buildbot.net', port=443): Max retries exceeded with url: /events/phone_home (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff7297704f0>: Failed to establish a new connection: [Errno 110] Operation timed out'))
which seems to happen only once (why is buildbot trying to phone home anyway? there is no mention of events.buildbot.net in any of my config files.
The docker containers have full network access, ipv6, routing and DNS are all fine (tested with the buildbot-master image).
This is my master.cfg:
import os
from twisted.application import service
from buildbot.master import BuildMaster
from buildbot.plugins import *
from buildbot_gitea.auth import GiteaAuth
from buildbot_gitea import *
basedir = '/bbot/bbot-master'
rotateLength = 10000000
maxRotatedFiles = 10
configfile = 'master.cfg'
umask = None
if basedir == '.':
basedir = os.path.abspath(os.path.dirname(__file__))
application = service.Application('buildmaster')
from twisted.python.logfile import LogFile
from twisted.python.log import ILogObserver, FileLogObserver
logfile = LogFile.fromFullPath(os.path.join(basedir, "twistd.log"), rotateLength=rotateLength,
maxRotatedFiles=maxRotatedFiles)
application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
m = BuildMaster(basedir, configfile, umask)
m.setServiceParent(application)
m.log_rotation.rotateLength = rotateLength
m.log_rotation.maxRotatedFiles = maxRotatedFiles# -*- python -*-
# ex: set filetype=python:
from buildbot.plugins import *
c = BuildmasterConfig = {}
####### WORKERS
c['workers'] = [worker.Worker("bbot-worker", "BUILDBOT_PASSWORD")]
c['protocols'] = {'pb': {'port': 9989}}
####### CHANGESOURCES
c['change_source'] = []
c['change_source'].append(changes.PBChangeSource())
####### SCHEDULERS
c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
name="all",
change_filter=util.ChangeFilter(branch='master'),
treeStableTimer=None,
builderNames=["runtests"]))
c['schedulers'].append(schedulers.ForceScheduler(
name="force",
builderNames=["runtests"]))
####### BUILDERS
factory = util.BuildFactory()
factory.addStep(steps.Gitea(repourl='gitea#gitea.mydomain:myself/repo.git',
mode='incremental',
workdir="build",
branch="master",
progress=True,
logEnviron=False,
))
factory.addStep(steps.ShellCommand(command=["make"]))
c['builders'] = []
c['builders'].append(
util.BuilderConfig(name="runtests",
workernames=["bbot-worker"],
factory=factory))
####### BUILDBOT SERVICES
c['services'] = [
reporters.GiteaStatusPush(
baseURL="https://gitea.mydomain/",
token="GITEA_API_ACCESS_TOKEN",
verbose=True)
]
####### PROJECT IDENTITY
c['title'] = "My Domain!"
c['titleURL'] = "https://buildbot.mydomain"
c['buildbotURL'] = "https://buildbot.mydomain/"
c['www'] = dict(port=8010,
plugins=dict(waterfall_view={}, console_view={}, grid_view={}))
c['www']['authz'] = util.Authz(
allowRules = [
util.AnyEndpointMatcher(role="admins")
],
roleMatchers = [
util.RolesFromUsername(roles=['admins'], usernames=['myself'])
]
)
c['www']['auth'] = GiteaAuth(
endpoint="https://gitea.mydomain/",
client_id="MY_CLIENT_ID_FROM_GITEA",
client_secret='MY_CLIENT_SECRET_FROM_GITEA')
c['www']['change_hook_dialects'] = {
'gitea': {
'secret': 'THE_GITEA_WEBHOOK_SECRET',
'onlyIncludePushCommit': True
}
}
####### DB URL
c['db'] = {
'db_url' : "postgresql://buildbot:MY_SECRET_DB_PASSWORD#172.25.0.2/buildbot",
}
The Dockerfile for the master is
FROM alpine:3.11.3
EXPOSE 9989
RUN apk update
RUN apk add python3 bash busybox-extras w3m gcc python3-dev libffi-dev openssl-dev musl-dev postgresql-dev
RUN mkdir /bbot
COPY entrypoint.sh /root/
RUN chmod a+x /root/entrypoint.sh
RUN mkdir /root/.ssh && chmod og-rwx /root/.ssh/
COPY bbot-gitea bbot-gitea.pub /root/.ssh/
RUN chmod og-w /root/.ssh/bbot-gitea*
RUN cd /bbot && \
python3 -m venv sandbox && \
source sandbox/bin/activate && \
pip3 install 'buildbot[bundle]' && \
pip3 install 'requests[security]' && \
pip3 -v install buildbot_gitea && \
pip3 install treq && \
pip3 install psycopg2
RUN apk del gcc python3-dev libffi-dev openssl-dev musl-dev
RUN ls -la /root
RUN cat /root/entrypoint.sh
ENTRYPOINT [ "/root/entrypoint.sh" ]
and the entrypoint does nothing special -- it is this,
#!/bin/bash
cd /bbot
echo " BBOT MASTER ENTRYPOINT"
source sandbox/bin/activate
buildbot upgrade-master bbot-master
# debug: check everything that was pip-installed:
echo "\n\n=====\n"
pip3 list
echo "=====\n\n"
if [ ! -f bbot-master/buildbot.tac ]; then
buildbot create-master bbot-master
fi
buildbot start bbot-master
tail -f /bbot/bbot-master/twistd.log
and the pip3 list line, which runs on startup for debugging, shows that I have
buildbot 2.7.0
buildbot-console-view 2.7.0
buildbot-gitea 1.2.0
buildbot-grid-view 2.7.0
buildbot-waterfall-view 2.7.0
buildbot-worker 2.7.0
buildbot-www 2.7.0
edit: checked the JS console in Firefox, and there seems to be a problem connecting to the server via websockets:
Firefox can’t establish a connection to the server at wss://buildbot.mydomain/ws.
From Chrome, this is what I see:
WebSocket connection to 'wss://buildbot.aleph0.info/ws' failed: Error during WebSocket handshake: Unexpected response code: 200
(200? why 200?)
I can't see why it wouldn't work. Apache is configured to do reverse proxying, like this:
RewriteEngine On
RewriteCond ${HTTP:Upgrade} websocket [NC]
RewriteCond ${HTTP:Connection} upgrade [NC]
RewriteRule .* "wss:/localhost:8010/$1" [P,L]
ProxyPass / http://localhost:8010/
ProxyPassReverse / http://localhost:8010/
So... What else can I do to continue debugging this?
(By the way, it does look like the buildbot mailing list is not very active -- after posting this question there I checked the archives, and there is svery low activity. Where do users of Buildbot go these days in order to get and share advice?)
I found the problem!
It was the reverse proxy that wasn't properly configured for websockets.
I used this in my apache virtualhost config,
<Location /ws>
ProxyPass ws://127.0.0.1:8010/ws
ProxyPassReverse ws://127.0.0.1:8010/ws
</Location>
ProxyPass /ws !
ProxyPass / http://localhost:8010/
ProxyPassReverse / http://localhost:8010/
and it works now!
( after searching a lot, I found the solution here:
https://docs.buildbot.net/0.9.2/manual/cfg-www.html )
There it is, in case anyone else needs it.
I have been trying to run ./hack/update-all.sh script and I am getting this error while updating codegen:
$ ./hack/update-all.sh
Running in the silent mode, run with -v if you want to see script logs.
Running in short-circuit mode; run with -a to force all scripts to run.
Updating generated-protobuf
Updating codegen
# runtime
/usr/local/go/src/runtime/os2_linux_generic.go:12: _SS_DISABLE redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:8
/usr/local/go/src/runtime/os2_linux_generic.go:13: _NSIG redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:9
/usr/local/go/src/runtime/os2_linux_generic.go:14: _SI_USER redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:10
/usr/local/go/src/runtime/os2_linux_generic.go:15: _SIG_BLOCK redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:11
/usr/local/go/src/runtime/os2_linux_generic.go:16: _SIG_UNBLOCK redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:12
/usr/local/go/src/runtime/os2_linux_generic.go:17: _SIG_SETMASK redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:13
/usr/local/go/src/runtime/os2_linux_generic.go:18: _RLIMIT_AS redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:14
/usr/local/go/src/runtime/os2_linux_generic.go:24: sigset redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:20
/usr/local/go/src/runtime/os2_linux_generic.go:26: rlimit redeclared in this block
previous declaration at /usr/local/go/src/runtime/os2_linux.go:22
/usr/local/go/src/runtime/panic1.go:11: paniclk redeclared in this block
previous declaration at /usr/local/go/src/runtime/panic.go:552
/usr/local/go/src/runtime/panic1.go:11: too many errors
!!! Error in /home/peeyush/work/kubernetes/hack/lib/golang.sh:435
'go install "${goflags[#]:+${goflags[#]}}" -ldflags "${goldflags}" "${nonstatics[#]:+${nonstatics[#]}}"' exited with status 2
Call stack:
1: /home/peeyush/work/kubernetes/hack/lib/golang.sh:435 kube::golang::build_binaries_for_platform(...)
2: /home/peeyush/work/kubernetes/hack/lib/golang.sh:574 kube::golang::build_binaries(...)
3: /home/peeyush/work/kubernetes/hack/build-go.sh:26 main(...)
Exiting with status 1
!!! Error in /home/peeyush/work/kubernetes/hack/lib/golang.sh:494
'( kube::golang::setup_env; echo "Go version: $(go version)"; local host_platform; host_platform=$(kube::golang::host_platform); local goflags goldflags; eval "goflags=(${KUBE_GOFLAGS:-})"; goldflags="${KUBE_GOLDFLAGS:-} $(kube::version::ldflags)"; local use_go_build; local -a targets=(); local arg; for arg in "$#";
do
if [[ "${arg}" == "--use_go_build" ]]; then
use_go_build=true;
else
if [[ "${arg}" == -* ]]; then
goflags+=("${arg}");
else
targets+=("${arg}");
fi;
fi;
done; if [[ ${#targets[#]} -eq 0 ]]; then
targets=("${KUBE_ALL_TARGETS[#]}");
fi; local -a platforms=("${KUBE_BUILD_PLATFORMS[#]:+${KUBE_BUILD_PLATFORMS[#]}}"); if [[ ${#platforms[#]} -eq 0 ]]; then
platforms=("${host_platform}");
fi; local binaries; binaries=($(kube::golang::binaries_from_targets "${targets[#]}")); local parallel=false; if [[ ${#platforms[#]} -gt 1 ]]; then
local gigs; gigs=$(kube::golang::get_physmem); if [[ ${gigs} -ge ${KUBE_PARALLEL_BUILD_MEMORY} ]]; then
kube::log::status "Multiple platforms requested and available ${gigs}G >= threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in parallel"; parallel=true;
else
kube::log::status "Multiple platforms requested, but available ${gigs}G < threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in serial"; parallel=false;
fi;
fi; if [[ "${parallel}" == "true" ]]; then
kube::log::status "Building go targets for ${platforms[#]} in parallel (output will appear in a burst when complete):" "${targets[#]}"; local platform; for platform in "${platforms[#]}";
do
( kube::golang::set_platform_envs "${platform}"; kube::log::status "${platform}: go build started"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status "${platform}: go build finished" ) &> "/tmp//${platform//\//_}.build" &
done; local fails=0; for job in $(jobs -p);
do
wait ${job} || let "fails+=1";
done; for platform in "${platforms[#]}";
do
cat "/tmp//${platform//\//_}.build";
done; exit ${fails};
else
for platform in "${platforms[#]}";
do
kube::log::status "Building go targets for ${platform}:" "${targets[#]}"; kube::golang::set_platform_envs "${platform}"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};
done;
fi )' exited with status 1
Call stack:
1: /home/peeyush/work/kubernetes/hack/lib/golang.sh:494 kube::golang::build_binaries(...)
2: /home/peeyush/work/kubernetes/hack/build-go.sh:26 main(...)
Exiting with status 1
!!! Error in ./hack/../hack/update-codegen.sh:32
'"${KUBE_ROOT}/hack/build-go.sh" ${BUILD_TARGETS[*]}' exited with status 1
Call stack:
1: ./hack/../hack/update-codegen.sh:32 main(...)
Exiting with status 1
Updating codegen FAILED
Any idea what could be the reason behind this? Or how to resolve this issue?
Looks like an environment issue. Cleared everything, cloned the repo afresh and everything is working fine.