Specify RPM build order in mock - postgresql

TLDR: Ensure to keep system OS up-to-date to help ensure consistency with current spec files.
Symptom
When rebuilding PostgreSQL 11.1 SRPM using mock, the build fails with:
BUILDSTDERR: /builddir/build/BUILD/postgresql-11.1/src/bin/psql/command.c:1814 undefined reference to `PQencryptPasswordConn`
NB: PQencryptPasswordConn is a libpq.so function (provided by postgresql-devel-10.3-5.fc27.x86_64 on my system...outside the mock chroot environment). Unless I'm mistaken, the Postgresql SRPM builds the postgresql-devel RPM along with others.
Steps to reproduce
I ran the following to rebuild the SRPM before attempting to apply any patches not already present in the SRPM:
# Obtain SRPM source
git clone https://src.fedoraproject.org/rpms/postgresql.git
cd postgresql
# Download local copies of SRPM sources
wget $(spectool -S *.spec | awk '/^Source.*:\/\//{IFS=" "; print $2}')
# ...check SHAs of downloaded sources...
# Run SRPM-specific prep scripts
./generate-pdf.sh
./generate-sources.sh
# Generate the SRPM
mock --root=fedora-27-x86_64 --resultdir="./SRPMS" --buildsrpm --spec postgresql.spec --sources .
# >>> Everything seems to work fine up to this point <<<
# Build the RPM inside mock chroot
mock --root=fedora-27-x86_64 --rebuild ./SRPMS/postgresql-11.1-4.fc27.src.rpm
# !!! Fail here (with symptom above) !!!
The Problem
I have so far been unable to have mock load the appropriate libpq library headers into the chroot environment to make sure rpmbuild builds against the libpq that contains the PQencryptPasswordConn header (which appears to exist on my system outside the build environment):
grep -lr "PQencryptPasswordConn" /usr/include
# /usr/include/libpq-fe.h
grep -lr "PQencryptPasswordConn" /var/lib/mock/fedora-27-x86_64/root/usr/include
# (Nothing returned)
When reviewing mock's installed_pkgs.log, the following were installed (the latter of which I expect would provide a version of libpq headers):
postgresql-libs-9.6.10-3.fc27.x86_64
postgresql-devel-9.6.10-3.fc27.x86_64
However, I cannot find a way to install the postgresql-* packages into the chroot environment that contain the updated library headers.
The Ask
Since postgresql SRPM is supposed to build postgresql-devel RPM, I think that mock will need to build and install the postgresql-devel RPM in the chroot before rpmbuild attempts to compile psql/command.c so that the latter compilation finds the appropriate library headers (unless the build process is intelligent enough to identify new libraries currently under build).
How can I best accomplish this (would prefer to avoid multiple mock calls for each RPM package built from the SRPM unless that's the only way to go)?
Please note that the build process on my system spawns multiple processes to parallel compilations.
I have also tried to use mockchain —recurse without success.
System Info
Linux 4.16.6-202.fc27.x86_64

First hint, you use the latest postgresql.spec version, but you try to build it against rather old (in fact unsupported nowadays) version 27 of Fedora distribution. I'd encourage you to migrate to a newer version of Fedora, or at least checkout the branch f27 in the same RPM git repository.
Second hint, we changed the layout of PostgreSQL packaging in Fedora 30+. We've cut out the library (libpq.so) into separate package, per announcement.
How to continue; always checkout appropriate branch based on what Fedora you build against, and adjust the spec file appropriately (checkout f27 and update to PostgreSQL 11.1 in this case).
JFTR (might help), there already is a testing modular build of PostgreSQL 11 against Fedora 28+, and the build scripts are maintained in separate branch stream-postgresql-11. With a bit of luck, you would be able to build that branch against old Fedora 27, too. Note that this version of postgresql.spec file is a little bit complicated (it needs to be because we build it against different versions of Fedora).

Related

rpmbuild unable to find the custom installed package

There are plenty of perl packages missing in Centos 8 and Rocky Linux. So, I try to get the rpm spec by cpanspec and build rpm by myself. But, it seems like that rpmbuild could not find the rpm I built.
This is the script for me to build rpm.
cd /root/rpmbuild
cpanspec --packer 'Example <example#example.com>' <Perl-Package-Name>
mkdir SOURCES
cp <Perl-Package-Name>.tar.gz SOURCES
rpmbuild -ba perl-<Package-Name>.spec
Let's say we have two package A and B. A is needed by B.
I try to build both of the packages through the script above. I build A first, switch into /root/rpmbuild/RPMS/noarch and install A.rpm. Then, I try to build package B.
I got
error: Failed build dependencies:
perl(A) is needed by perl-<B>
I try to check the existence of package A.
yum list installed | grep A
and
perldoc -l A
Both of the commands show that A exists.
Did I miss something?
update 2022/06/07
I just gave up and commented the BuildRequires: A in B package. This is not a good approach but it works.

python 3.7.6 pip setup.py test with ContextualVersionConflict

my setup.py is like below:
from setuptools import find_packages, setup
tests_require = ['pytest-env', 'pytest-mock', 'pytest-cov', 'pytest-xdist', 'pytest', 'mock', 'moto<=1.3.10']
setup(
name='repo_name',
version='0.1.0',
description='repo_name',
keywords=['?'],
packages=find_packages('src', exclude=['tests', 'venv']),
package_dir={'': 'src'},
package_data={'': ['schema/*.yaml']},
install_requires=[
'boto3<=1.10.19',
'requests<=2.22.0',
'jsonschema<=3.0.1',
'objectpath<=0.6.1',
'pyyaml<=5.1.1',
'sqlalchemy<=1.3.5',
'psycopg2-binary<=2.8.3',
'auth-client<=1.0.23', # internal package
'policy_client<=1.0.9', # internal package
'audit-client<=1.1.20', # internal package
'flask<=1.1.1',
'click<=7.0',
'Werkzeug<=0.15.5',
'itsdangerous<=1.1.0',
'Jinja2<=2.10.1',
'MarkupSafe<=1.1.1',
'structlog<=19.2.0',
'python-rapidjson<=0.9.1'
],
setup_requires=['pytest-runner'],
tests_require=tests_require,
extras_require={'test': tests_require},
include_package_data=True,
zip_safe=False
)
When I run python setup.py test, it will always reinstall the most of the packages that I already installed into .eggs folder, which I understand. While it's installing the package with different versions that I provided in the install_requires section, which results in below error:
pkg_resources.ContextualVersionConflict: (urllib3 1.25.10 (/Users/***/Desktop/repo/.eggs/urllib3-1.25.10-py3.7.egg), Requirement.parse('urllib3<1.25,>=1.21.1'), {'requests'})
I have to pin the requests version to 2.22.0 which requires urllib3 version no larger than 1.25, which caused my issue. So, is there anyway I can pin the urllib3 version in somewhere so when it runs, it will only install the provided version? I have tried to add the urllib3 version in install_requires, tests_require and in requirements.txt, but no luck so far.
Probably not what you want to hear but... setup_requires and tests_require are deprecated. Although they (probably) still work (mostly) fine, you might want to look for alternative solutions. As far as I know the most common alternative solutions are to use Tox instead of tests_require and PEP 517 to replace setup_requires.
In your case, it seems that your usage of setup_requires is only a consequence of your usage of tests_require. So by switching to something like Tox you get rid of both tests_require and setup_requires.
I got it resolve to simply replace python setup.py test to be pip instll -e . [test], and then pytest directly.
It will still install all the tests_require packages and then run pytest directly. Instead of going through all the list packages and find the best match versions for all packages.
UPDATES:
The real problem is I did not remove the old .eggs/ and venv/ folder when I made the packages version change. So the solution is updating the requests version to be 2.21.0 in the setup.py file, then remove the .eggs/ and venv/ folder and rerun everything.

How do I set up my Dockerfile to use cpanm to install a specific version of a Perl module?

Within my Dockerfile, I am setting up the Perl modules that will be installed when run, like so:
RUN ["cpanm", "Carp", "Carp::Heavy", "Class::Data::Inheritable"]
However, for one module, I need a specific version of a module, not the latest one. How can I specify that version in the above line?
I've been searching online for hours, and haven't turned up anything useful yet.
Instead of specifying a module name, specify a URL. Eg, instead of Class::Data::Inheritable, use https://cpan.metacpan.org/authors/id/T/TM/TMTM/Class-Data-Inheritable-0.06.tar.gz
You can find the applicable URL by going to the module page on metacpan, selecting the version you want, and copying the download link.
PS: You might want to also set PERL_CPANM_OPT=--from https://cpan.metacpan.org/ in the environment so cpanm only downloads using HTTPS.
For anyone who's searching for this same answer in the future, another option can be found here in the documentation for cpanm:
cpanm Plack#0.9990
If you have a long list of modules, consider feeding a cpanfile into cpanm rather than listing them all in the Dockerfile.
The easiest way to specify a particular version number for a module in a cpanfile is like this:
requires 'Text::ParseWords', '==3.1';
The syntax for requesting the latest version of a module is this:
requires 'Text::ParseWords';
Requesting a minimum version: (note the lack of '==')
requires 'Text::ParseWords', '3.1';
The syntax for requesting specific versions in other ways is fairly well-documented here.
Another great write-up of the use of cpanm and a cpanfile can be found
in Installation of cpan modules by cpanm and cpanfile.
To have CPAN install a specific version of a module, you need to provide the full module distribution filename including the author. For example to install the module Set::Object version 1.28, at the command line type:
cpan SAMV/Set-Object-1.28.tar.gz
Same thing apply with Docker, just add
RUN cpan SAMV/Set-Object-1.28.tar.gz
To specify target module version you can use
cpanm MIYAGAWA/Plack-0.99_05.tar.gz # full distribution path
cpanm http://example.org/LDS/CGI.pm-3.20.tar.gz # install from URL
cpanm ~/dists/MyCompany-Enterprise-1.00.tar.gz # install from a local file
See official documentation: https://metacpan.org/dist/App-cpanminus/view/bin/cpanm
But better, to my mind, would be to use cpanfile. See --cpanfile option https://metacpan.org/dist/App-cpanminus/view/bin/cpanm#-cpanfile and format of this file https://metacpan.org/pod/cpanfile
But if you have many modules (like me), I recommend to use cpm. It installs modules in parallel very fast. Also with help of docker we could cache builds, thus rebuilds will takes seconds. Here is my Dockerfile:
## Modules
WORKDIR ${APP_ROOT}
# install modules outside of WORKDIR, so it will not interfere when we do COPY . .
RUN mkdir -p ../modules
RUN ln -s ../modules local
RUN cpanm -n -L ./local App::cpm Carton::Snapshot && rm -rf /root/.cpanm
COPY cpanfile ./
COPY cpanfile.snapshot ./
RUN \
--mount=type=cache,target=/root/.perl-cpm \
cpm install -w 16 --no-test -L ./local \
--with-develop
# regenerate cpanfile.snapshot
# https://github.com/miyagawa/Carmel#cpm
# cpm doesn't have the ability to manage cpanfile.snapshot file on its own.
RUN carton install
# You can copy snapshot from container by running:
# docker cp <container_name>:${APP_ROOT}/cpanfile.snapshot.latest ./cpanfile.snapshot

Building spark-jobserver Using SBT and Scala

Can anyone suggest me a better documentation about spark-jobserver. I have gone through the url spark-jobserver but unable to follow the same. It will be great if some one explain step by step instruction on how to use spark-jobserver.
Tools used in building the project.
sbt launcher version 0.13.5
Scala code runner version 2.11.6
With the above mentioned tools I am getting errors while building the spark-jobserver.
The documentation provided in the jobserver repo is indeed confusing.
Here's the steps I followed to manually build and run Spark Job Server on a local machine.
1. git clone https://github.com/spark-jobserver/spark-jobserver
2. sudo mkdir -p /var/log/job-server
3. sudo chown user1:user1 /var/log/job-server
4. cd spark-jobserver
5. sbt job-server/assembly
6. cd config
7. cp local.sh.template abc.sh # Note that the same name 'abc' is used in steps 8 and 10 as well
8. cp ec2.conf.template abc.conf
9. cd .. # The jobserver root dir
10. ./bin/server_package.sh abc # This script copies the files and packages necessary to run job server into a single dir [ default - /tmp/job-server]
11. cd /tmp/job-server [This is where the files and packages necessary to run job server are published by default]
12. ./server_start.sh
13. Run ./server_stop.sh to stop the server
Hope this helps
Here are the steps that I used to install:
Clone the jobserver repo.
Get sbt using wget https://dl.bintray.com/sbt/native-packages/sbt/0.13.8/sbt-0.13.8.tgz
Move "sbt-launch.jar" in sbt/bin to /bin
Create a script /bin/sbt, contents found here, making sure to change the pointer to java if necessary
Make the above script executable
Now cd into the spark jobserver directory, and run sbt publish-local
Assuming the above was successful, run sbt in the same directory
Finally, use the command re-start, and if it succeeds the server is now running!

Install GD Library on RedHat machine for twiki

My ultimate goal is to run a twiki website for my research group.
I have space on RedHat server that is running Apache, etc., but upon which I do not have root access. Since I cannot install perl modules with the current permissions, I've decided to manually install a local version of perl. Got that working no problem. The following modules are required to get twiki to work:
FreezeThaw - http://search.cpan.org/~ilyaz/FreezeThaw
CGI::Session - http://search.cpan.org/~markstos/CGI-Session
Error - http://search.cpan.org/~shlomif/Error
GD - http://search.cpan.org/~lds/GD
HTML::Tree - http://search.cpan.org/~petek/HTML-Tree
Time-modules - http://search.cpan.org/~muir/Time-modules
I have installed FreezeThaw, CGI, Error, and it fails on GD with the following error:
UNRECOVERABLE ERROR Could not find gdlib-config in the search path.
Please install libgd 2.0.28 or higher. If you want to try to
compile anyway, please rerun this script with the option --ignore_missing_gd.
In searching for how to get around this newest obstacle, I found a previous SO question: How to install GD library with Strawberry Perl asked about installing this and the top answer suggested manually compiling gdlib. You'll note, however, that that link is broken. The base site: http://www.libgd.org/ is basically down saying to go to the project's bitbucket page.
So I got the tarball from that page and am trying to install it. The following problems occur when I follow the instructions included. README.TXT says: "If the sources have been fetched from CVS, run bootstrap.sh [options]."
Running bootstrap.sh yields:
configure.ac:64: warning: macro `AM_ICONV' not found in library
configure.ac:10: required directory ./config does not exist cp: cannot
create regular file `config/config.guess': No such file or directory
configure.ac:11: installing `config/config.guess' configure.ac:11:
error while copying cp: cannot create regular file
`config/config.sub': No such file or directory configure.ac:11:
installing `config/config.sub' configure.ac:11: error while
copying cp: cannot create regular file `config/install-sh': No such
file or directory configure.ac:28: installing `config/install-sh'
configure.ac:28: error while copying cp: cannot create regular
file `config/missing': No such file or directory configure.ac:28:
installing `config/missing' configure.ac:28: error while copying
configure.ac:577: required file `config/Makefile.in' not found
configure.ac:577: required file `config/gdlib-config.in' not found
configure.ac:577: required file `test/Makefile.in' not found
Makefile.am:14: Libtool library used but `LIBTOOL' is undefined
Makefile.am:14: The usual way to define `LIBTOOL' is to add
AC_PROG_LIBTOOL' Makefile.am:14: to configure.ac' and run
aclocal' and autoconf' again. Makefile.am:14: If `AC_PROG_LIBTOOL'
is in `configure.ac', make sure Makefile.am:14: its definition is in
aclocal's search path. cp: cannot create regular file
`config/depcomp': No such file or directory Makefile.am: installing
`config/depcomp' Makefile.am: error while copying Failed
And it says I should also install the following 3rd party libraries:
zlib, available from http://www.gzip.org/zlib/
Data compression library
libpng, available from http://www.libpng.org/pub/png/
Portable Network Graphics library; requires zlib
FreeType 2.x, available from http://www.freetype.org/
Free, high-quality, and portable font engine
JPEG library, available from http://www.ijg.org/
Portable JPEG compression/decompression library
XPM, available from http://koala.ilog.fr/lehors/xpm.html
X Pixmap library
Which I am ignoring for now.
Switching to the generic instructions it says follow the advice in the INSTALL file; which says: "cd to the directory containing the package's source code and type ./configure to configure the package for your system." Which flat does not work: I've cd'ed into every directory of the tarball and running that command does nothing.
So, trying to install twiki required me to install perl, which required me to install the perl modules: FreezeThaw, CGI, Error, HTML, Time-modules, and GD -- which itself required me to install gdlib -- which further suggested I install zlib, libpng, FreeType 2.x, JPEG library, and XPM. And of course, I'm stuck at the installing gdlib stage.
My question is: what other process can possibly demean humanity to such a level? I cannot fathom the depths of cruelty that lay ahead of me as I dive ever deeper into this misery onion. Should I just end it all? Can meaning be brought from this madness? Will the sun come up tomorrow, and if so, does it even matter?
But seriously, any suggestions on what to do differently/better would be much appreciated -- I can't remember what a child's laughter sounds like anymore.
Install the package gd-devel, it contains /usr/bin/gdlib-config.
This should work:
sudo apt-get -y install libgd2-xpm-dev build-essential