Add plpython3 Extension to Postgres/timescaledb Alpine Docker Image - postgresql

I try to add the plpython3 extension to my timescaledb/postgres (based on linux alpine) image:
FROM timescale/timescaledb:0.9.0-pg10
RUN set -ex \
&& apk add --no-cache --virtual .plpython3-deps --repository http://nl.alpinelinux.org/alpine/edge/testing \
postgresql-plpython3
When I try to create the extension I get the following error:
postgres=# CREATE EXTENSION plpython3u;
ERROR: could not open extension control file "/usr/local/share/postgresql/extension/plpython3u.control": No such file or directory
But when I search for the files inside my container I can find them within a different directory:
/ # find / -name '*plpy*'
/usr/lib/postgresql/plpython3.so
/usr/share/postgresql/extension/plpython3u.control
/usr/share/postgresql/extension/plpython3u--1.0.sql
/usr/share/postgresql/extension/plpython3u--unpackaged--1.0.sql
How can I install postgresql-plpython3 to a different directory or configure postgres to recognize my added extension?
Update
When I just mv the files to /usr/local/share/postgresql/extension I get the error:
postgres=# CREATE EXTENSION plpython3u;
ERROR: could not access file "$libdir/plpython3": No such file or directory
Update 2
So the issue with $libdir was that pg_config --pkglibdir points to /usr/local/lib/postgresql but plpython3.so is inside /usr/lib/postgresql. When I move everything to the according /usr/local directories I can successfully create the extension.
This leads to the question where I hope to find an answer. How can I install postgresql-plpython3 to /usr/local/... instead of /usr/...?

I am fairly certain that if you use prebuilt packages you are stuck with hardcoded installation paths.
The easiest way around your problem is to create symlinks after installation:
FROM timescale/timescaledb:0.9.0-pg10
RUN set -ex \
&& apk add --no-cache --virtual .plpython3-deps --repository http://nl.alpinelinux.org/alpine/edge/testing \
postgresql-plpython3 \
&& ln -s /usr/lib/postgresql/plpython3.so /usr/local/lib/postgresql/plpython3.so \
&& ln -s /usr/share/postgresql/extension/plpython3u.control /usr/local/share/postgresql/extension/plpython3u.control \
&& ln -s /usr/share/postgresql/extension/plpython3u--1.0.sql /usr/local/share/postgresql/extension/plpython3u--1.0.sql \
&& ln -s /usr/share/postgresql/extension/plpython3u--unpackaged--1.0.sql /usr/local/share/postgresql/extension/plpython3u--unpackaged--1.0.sql

Related

SwiftPM Package building target in docker

Im trying to setup an external library in a SwiftPM Package project. This package (Mockingbird) is only used from within my test target, and therefor my main application target does not depend on this.
On my MacOS host machine, building and executing this test target works perfectly fine. No errors whatsoever.
However, i'm also building this application within a dockerized container within a CI process. Im using the official swift docker images (swift 5.5) for this.
From within this docker container, building the main target itself works fine, because it does not depends on the Mockingbird lib. But building the test target fails when compiling mockingbird.
When starting this Docker container, i always clean the build folder and resolve the packages first before building the targets.
This is the error im getting:
In file included from /app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/MKBTypeFacade.m:1:
/app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/../include/MKBTypeFacade.h:1:9: fatal error: 'Foundation/Foundation.h' file not found
#import <Foundation/Foundation.h>
^~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/MKBTestExpectation.m:1:
/app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/../include/MKBTestExpectation.h:1:9: fatal error: 'XCTest/XCTest.h' file not found
#import <XCTest/XCTest.h>
^~~~~~~~~~~~~~~~~
/app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/MKBTestUtils.m:1:
/app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/../include/MKBTestUtils.h:1:9: fatal error: 'Foundation/Foundation.h' file not found
#import <Foundation/Foundation.h>
^~~~~~~~~~~~~~~~~~~~~~~~~
/app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/MKBMocking.m:1:
/app/.build/checkouts/mockingbird/Sources/MockingbirdFramework/Objective-C/Bridge/sources/../include/MKBMocking.h:1:9: fatal error: 'Foundation/Foundation.h' file not found
#import <Foundation/Foundation.h>
Maybe im missing out on something?
Thanks in advance!
Edit 1: Added dockerfile
FROM swift:5.6.2-focal
WORKDIR /app
RUN apt-get update \
&& apt-get install openssh-client lcov -y
ARG INSTALL_COMMANDS="echo \"No custom install commands provided, proceeding\""
RUN ${INSTALL_COMMANDS}
ARG GITLAB_SSH_PRIVATE_KEY
RUN mkdir -p ~/.ssh \
&& echo "${GITLAB_SSH_PRIVATE_KEY}" > ~/.ssh/id_rsa \
&& chmod 0600 ~/.ssh/id_rsa \
&& eval "$(ssh-agent -s)" \
&& ssh-add ~/.ssh/id_rsa \
&& ssh-keyscan ${GIT_URL} > ~/.ssh/known_hosts
## Copy entire repo into container
COPY ./Package.* ./
RUN swift package resolve
COPY . .
## Remove SSH keys
RUN rm -rf /root/.ssh/
After my image was created containing the source files, i run it like:
docker run ${IMAGE_NAME} \
/bin/sh -c "swift build --target AppTests"

PostgreSQL - HyperLogLog extension not found

Can someone explain in a better way (well, in a way for dummies to understand), or more correctly how to install HyperLogLog hll extension for PostgreSQL on my Mac M1 machine.
When running CREATE EXTENSION hll;
I get:
Query 1 ERROR: ERROR: could not open extension control file "/opt/homebrew/share/postgresql/extension/hll.control": No such file or directory
I am new at this, so this documentation https://github.com/citusdata/postgresql-hll did not helped me a lot.
I installed all other extensions that I need except this one..
When typing which postgres I get:
/opt/homebrew/bin/postgres
And version: postgres (PostgreSQL) 14.3
I saw about configuring PG_CONFIG but I do not understand what exactly I should be doing here?
I will appreciate the help and I hope that this post will be of use for other dummies as I. :)
We can simplify the script above and execute it inline by copying and pasting all of the following into your terminal:
> yes |
#!/bin/bash
# download latest release
curl -s https://api.github.com/repos/citusdata/postgresql-hll/releases/latest \
| grep '"tarball_url":' \
| sed -E 's/.*"([^"]+)".*/\1/' \
| xargs curl -o package.tar.gz -L
# extract to new hll directory
mkdir hll && tar xf package.tar.gz -C hll --strip-components 1
# build and install extension to postgres extensions folder
cd hll
make
make install
# remove hll directory
cd ../
rm -r ./hll
# connect to PostgreSQL and install extension
psql -U postgres -c "CREATE EXTENSION hll;"
I wrote the script for myself to get the last package and install it.
I build it by using make.
# check if Makefile installed
make -v
# download latest release
curl -s https://api.github.com/repos/citusdata/postgresql-hll/releases/latest \
| grep '"tarball_url":' \
| sed -E 's/.*"([^"]+)".*/\1/' \
| xargs curl -o package.tar.gz -L
# extract to hll directory
mkdir hll && tar xf package.tar.gz -C hll --strip-components 1
cd hll
# build and instll extension to postgres extensions folder
make
make install
# remove hll directory
cd ../
rm -r ./hll
# connect to PostgreSQL
psql -U postgres
# install extension in your DB
CREATE EXTENSION hll;

Building postgres from source throws "'utils/errcodes.h' file not found" when called from other makefile

I am currently constructing a Makefile and one of the things it will be doing is downloading and building postgres from source. Before starting to write the makefile file I always used the following set of commands to do this:
curl -LJ https://github.com/postgres/postgres/archive/refs/tags/REL_13_3.zip -o postgres.zip
unzip postgres.zip
rm postgres.zip
cd postgres-REL_13_3
./configure --prefix "`pwd`" --without-readline --without-zlib
make
make install
Executing the above listed commands in the terminal results in the successful installation of postgres. Then I translated these into a makefile which looks as follows:
build:
curl -LJ https://github.com/postgres/postgres/archive/refs/tags/REL_13_3.zip -o postgres.zip
unzip postgres.zip
rm postgres.zip
cd postgres-REL_13_3 \
&& ./configure --prefix "`pwd`" --without-readline --without-zlib \
&& $(MAKE) \
&& $(MAKE) install
Running this Makefile results in the error:
../../src/include/utils/elog.h:71:10: fatal error: 'utils/errcodes.h' file not found
It seems that something about calling the make from another Makefile causes a referencing issue with the files during the build process, but I just can figure out for the life of me what I have to change to fix this.
It appears to be the influence of Makefile enviromental variables. I didn't discover the exact mechanism, but unsetting them helps.
build:
curl -LJ https://github.com/postgres/postgres/archive/refs/tags/REL_13_3.zip -o postgres.zip
unzip postgres.zip
rm postgres.zip
unset MAKELEVEL && unset MAKEFLAGS && unset MFLAGS && cd postgres-REL_13_3 \
&& ./configure --prefix "`pwd`" --without-readline --without-zlib \
&& $(MAKE) \
&& $(MAKE) install

mongo inside ansible-operator image

I need the mongo shell installed inside the ansible-operator image.
My first attempt was to use this Dockerfile:
FROM mongo:4.2.9
FROM quay.io/operator-framework/ansible-operator:v1.0.0
COPY --from=0 /usr/bin/mongo /usr/bin/mongo
COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install -r ${HOME}/requirements.yml \
&& chmod -R ug+rwx ${HOME}/.ansible
COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbooks/ ${HOME}/playbooks/
Unsurprisingly, this didn't worked.
"stderr_lines": ["/usr/bin/mongo: /lib64/libcurl.so.4: no version information available (required by /usr/bin/mongo)", "Failed global initialization: InvalidSSLConfiguration Can not set up PEM key file."]
Can anyone help me?
I finally figured it out...
Just add this to your Dockerfile.
(mongodb-org-4.2.repo)
USER 0
COPY mongodb-org-4.2.repo /etc/yum.repos.d/mongodb-org-4.2.repo
RUN yum -y update \
&& yum install -y mongodb-org-shell \
&& yum clean all \
&& rm -rf /var/cache/yum
RUN rm /etc/yum.repos.d/mongodb-org-4.2.repo
USER ${USER_UID}

Prepare coursier artifact for offline use inside container

I have an sbt project producing my artifact xyz.
I would like to put it along with all its dependencies in the docker container so it can be used using
coursier launch --mode offline xyz
The important part is that preparation should take use of local cursier cache from host.
I tried
executing sbt publishLocal,
then resolving my artifact dependencies (cursier resolve xyz),
then preparing to directories - local & cache - by copying resolved artifact into them
then copying those directories into docker container (as coursier cache and ivy local respectively).
This didn't work because coursier doesn't list .pom and .xml files in its output. I tried copying whole directories (abc/1.0.0 instead of abc/1.0.0/some.jar) but AFAIK there is no reliable way to know how many folders up one has to go because maven and ivy have different dir structures.
while my usecase is not quite identical to yours -- I figure I'd write up my findings and maybe my solution works for you as well!
here's my sample dockerfile, I used this to install scalafmt in an offline-compatible way
FROM ubuntu:jammy
RUN : \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* # */ stackoverflow highlighting bug
ARG CS=v2.1.0-RC4
ARG CS_SHA256=176e92e08ab292531aa0c4993dbc9f2c99dec79578752f3b9285f54f306db572
ARG JDK_SHA256=aef49cc7aa606de2044302e757fa94c8e144818e93487081c4fd319ca858134b
ENV PATH=/opt/coursier/bin:$PATH
RUN : \
&& curl --location --silent --output /tmp/cs.gz "https://github.com/coursier/coursier/releases/download/${CS}/cs-x86_64-pc-linux.gz" \
&& echo "${CS_SHA256} /tmp/cs.gz" | sha256sum --check \
&& curl --location --silent --output /tmp/jdk.tgz "https://download.java.net/openjdk/jdk17/ri/openjdk-17+35_linux-x64_bin.tar.gz" \
&& echo "${JDK_SHA256} /tmp/jdk.tgz" | sha256sum --check \
&& mkdir -p /opt/coursier \
&& tar --strip-components=1 -C /opt/coursier -xf /tmp/jdk.tgz \
&& gunzip /tmp/cs.gz \
&& mv /tmp/cs /opt/coursier/bin \
&& chmod +x /opt/coursier/bin/cs \
&& rm /tmp/jdk.tgz
ENV COURSIER_CACHE=/opt/.cs-cache
RUN : \
&& cs fetch scalafmt:3.6.1 \
&& cs install scalafmt:3.6.1 --dir /opt/wd/bin
the key to offline execution for me was to use cs fetch and set COURSIER_CACHE
here's the offline execution succeeding:
$ docker run --net=none --rm -ti cs /opt/wd/bin/scalafmt --version
scalafmt 3.6.1