I have one project I'd like to publish as packages targeting two Python versions (3.6 and 3.8).
What I understand:
How to install and activate different python versions using pyenv.
How to get poetry to create virtual environments that correspond to the chosen Python version.
How to setup pyproject.toml to specify the python version, manage dependencies, and publish a package using this configuration.
What I do not understand: how can I publish the same package for more than one Python version? I can't be the only one with this use-case right?
Does need two pyproject.toml files? (one for each python version and set of corresponding dependencies...)
Are there established ways of doing this with Poetry, or are other tools/workflows necessary?
Edit
Doing a bit more digging, I found this https://python-poetry.org/docs/versions/#multiple-constraints-dependencies which looks like it might be relevant.
Here's the example at the link above.
[tool.poetry.dependencies]
foo = [
{version = "<=1.9", python = "^2.7"},
{version = "^2.0", python = "^3.4"}
]
I've also found you can specify the Python version using poetry add like this...
poetry add cleo --python 3.6.10
Which adds dependencies in pyproject.toml like this...
cleo = {version = "^0.8.1", python = "3.6.10"}
Going to experiment and see if this works.
You probably need something like that in your pyproject.toml:
[tool.poetry.dependencies]
python = '3.6 || 3.8'
But I am not sure on the exact notation, it's a bit vague.
It seems to generate a setup.py with the following:
'>=3.6, !=2.7.*, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.7.*'
So that would include 3.9, 3.10, etc. and this is incorrect.
No. You don't need to create multiple pyproject.toml files or otherwise create separate workflows for each Python version you're targeting (for this specific situation targeting similar versions at least).
You can simply use dependency syntax to say you want to target >=3.6<4.0 like this...
[tool.poetry.dependencies]
python = '^3.6'
And then add dependencies similarly...
poetry add <dependency> python ^3.6
Which results in something like this...
[tool.poetry.dependencies]
python = '^3.6'
cleo = {version = "^0.8.1", python = "^3.6"}
pyyaml = {version = "^5.4.1", python = "^3.6"}
...
This worked, though I further went and made some of dependencies less specific to avoid incompatibilities on certain hosts.
pyyaml = {version = "^5.0", python = "^3.6"}
...
Related
Pip supports the pyproject.toml file but so far all practical usage of the new schema requires a 3rd party tool that auto-generates these files (e.g., poetry and pip). Unlike setup.py which is already human-writeable, pyproject.toml is not (yet).
From setuptools docs,
[build-system]
requires = [
"setuptools >= 40.9.0",
"wheel",
]
build-backend = "setuptools.build_meta"
However, this file does not include package dependencies (as outlined in PEP 621). Pip does support installing packages using pyproject.toml but nowhere does pep specify how to write package dependencies in pyproject.toml for the official build system setuptools.
How do I write package dependencies in pyproject.toml?
Related StackOverflow Questions:
How to init the pyproject.toml file
This question asks for a method to auto-generate pyproject.toml, my question differ because I ask for a human-written pyproject.toml.
my question differ because I ask for a human-written pyproject.toml
First, the pyproject.toml file is always "human-writable".
Then, it is important to know that in this context setuptools and Poetry take the role of what are called "PEP 517 build back-ends", and there are many such back-ends available today, setuptools and Poetry are just two examples of them.
As of today, it seems like most (if not all) of the build back-ends I know of expect their configuration (including dependencies) to be written in pyproject.toml.
PEP 621
There is a standard called PEP 621 that specifies how a project's metadata, including dependencies, should be laid out in the pyproject.toml file.
Here is a list of build back-ends I know of that have support for PEP 621:
enscons
flit_core (see flit)
hatchling (see hatch)
pdm-pep517 (see pdm)
setuptools (experimental support since version 61.0.0)
trampolim
whey
For all PEP 621 compatible build back-ends, the dependencies should be written in pyproject.toml file like in the following:
[project]
name = "Thing"
version = "1.2.3"
# ...
dependencies = [
"SomeLibrary ~= 2.2",
]
References:
https://peps.python.org/pep-0621/
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (experimental support for PEP 621 has been added to setuptools version 61.0.0, released 2022-03-24)
setuptools (before version 61.0.0)
In setuptools before version 61.0.0 there is no support for writing the configuration in pyproject.toml (in other words: no PEP 621 support). You have to either write a setup.cfg, or a setup.py, or a combination of both.
My recommendation is to write as much as possible in setup.cfg. Such a setup.cfg could look like this:
[metadata]
name = Thing
version = 1.2.3
[options]
install_requires =
SomeLibrary ~= 2.2
packages = find:
and in most cases the setup.py can be omitted completely or it can be as short as:
import setuptools
setuptools.setup()
References about the dependencies specifically:
https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html
https://www.python.org/dev/peps/pep-0508/
https://www.python.org/dev/peps/pep-0440/
Again, note that in most cases it is possible to omit the setup.py file entirely, one of the conditions is that the setup.cfg file and a pyproject.toml file are present and contain all the necessary information. Here is an example of pyproject.toml that works well for a setuptools build backend:
[build-system]
build-backend = 'setuptools.build_meta'
requires = [
'setuptools >= 43.0.0',
]
poetry
In poetry everything is defined in pyproject.toml, but it uses poetry-specific sections. In other words, Poetry does not currently use the PEP 621 standard, but there are some plans to move to this standard in the future.
This file can be hand-written. As far as I can tell, there is no strict need to ever explicitly install poetry itself (commands such as pip install and pip wheel can get you far enough).
The pyproject.toml file can be as simple as:
[tool.poetry]
name = 'Thing'
version = '1.2.3'
[tool.poetry.dependencies]
python = '^3.6'
SomeLibrary = '~2.2'
[build-system]
requires = ['poetry-core~=1.0']
build-backend = 'poetry.core.masonry.api'
References:
https://python-poetry.org/docs/pyproject/
https://python-poetry.org/docs/dependency-specification/
With tools like npm we can install a specific version
npm install foo#1.2.0
How do you install a specific version using spago install?
Firstly, that's not what spago install does. Instead of "adding a package to your project", spago install downloads all packages that are currently referenced in your spago.dhall file.
Secondly, the idea with Spago is that you don't choose a specific package version. Instead what you choose is a "snapshot", which is a collection of certain versions of all available packages that are guaranteed to compile and work together. This is a measure intended to prevent version conflicts and versioning hell (and this is similar to how Haskell stack works)
The snapshot is defined in your packages.dhall file, and then you specify the specific packages that you want to use in spago.dhall. The version for each package comes from the snapshot.
But if you really need to install a very specific version of a package, and you really know what you're doing, then you can modify the snapshot itself, which is described in packages.dhall.
By default your packages.dhall file might look something like this:
let upstream =
https://raw.githubusercontent.com/purescript/package-sets/psc-0.13.5-20200103/src/packages.dhall sha256:0a6051982fb4eedb72fbe5ca4282259719b7b9b525a4dda60367f98079132f30
let additions = {=}
let overrides = {=}
in upstream // additions // overrides
This is the default template that you get after running spago new.
In order to override the version for a specific package, add it to the overrides map like this:
let overrides =
{ foo =
upstream.foo // { version = "v1.2.0" }
}
And then run spago install. Spago should pull in version 1.2.0 of the foo package for you.
tl;dr I want to ship a package.json with my Perl library, run yarn install (or npm install during the installation) and install the downloaded JavaScript dependencies with the Perl modules.
I have the following dist.ini:
name = Foobar
version = 1.2.3
license = Perl_5
copyright_holder = Yours Truly
copyright_year = 2018
[#Filter]
-bundle = #Basic
-remove = GatherDir
[Git::GatherDir]
[Web::FileHeader]
header_filename = EMM-include.pm
file_match = ^Makefile\.PL$
The file EMM-include.pm contains a MY::postamble method:
package MY;
use strict;
use Cwd qw(abs_path);
use File::Spec;
sub postamble {
my ($self) = #_;
my $here = Cwd::abs_path();
my $libdir = File::Spec->catdir($here, 'lib', 'Foobar');
chdir $libdir or die;
0 == system 'yarn', 'install' or die;
chdir $here or die;
return '';
}
The plug-in [Web::FileHeader] takes the file and patches it to the beginning of Makefile.PL.
Then there is a lib/Foobar/package.json:
{
"name": "foobar",
"version": "1.2.3",
"main": "index.js",
"dependencies": {
"ajv": "^6.5.4"
}
}
The MY::postamble section from EMM-include.pm invokes yarn install (replace it with npm install if you don't have yarn) and populate the directory lib/Foobar/node_modules with ajv and its dependencies.
Finally, there must be a module lib/Foobar.pm:
package Foobar;
# ABSTRACT: Just a test.
1;
That almost works as intended: The distribution can be created with dzil build. In the distribution directory, perl Makefile.PL invokes yarn install, the directory lib/Foobar/node_modules gets populated but the files in there are not installed with make install.
If I run perl Makefile.PL a second time, everything works, the JavaScript dependencies make it into blib/ and make install would install the JavaScript modules alongside the Perl modules.
Shipping the JavaScript dependencies with the distribution is not an option. They are already too many and they may have conflicting licenses (I am using GPLv3 here). Downloading the deps at runtime, after the installation will mostly fail because of missing privileges.
True, this has not that much to do with Dist::Zilla, it's rather a problem with ExtUtils::MakeMaker. But I am actually using Dist::Zilla here.
In case it matters, the real distribution is https://github.com/gflohr/qgoda and the last commit at the time of this writing is https://github.com/gflohr/qgoda/commit/3f34a3dfec8da665061432c3a8f7bd8eef28b95e.
First, instead of using [Web::FileHeader] to alter your Makefile.PL, replace [MakeMaker] (used by #Basic) with [MakeMaker::Awesome], which allows you to modify Makefile.PL directly, and correctly enables dynamic_config since your distribution needs it. Also, don't give your include-file a .pm extension since it's not a perl module, and exclude it from being gathered into the resulting distribution so it doesn't accidentally get installed.
[#Filter]
-bundle = #Basic
-remove = GatherDir
-remove = MakeMaker
[Git::GatherDir]
exclude_filename = EMM-include
[MakeMaker::Awesome]
header_file = EMM-include
I strongly suggest using my #Starter bundle instead of the outdated #Basic, but if not at least add [MetaJSON] so you have modern metadata.
[#Starter::Git]
revision = 3
installer = MakeMaker::Awesome
Git::GatherDir.exclude_filename[] = EMM-include
MakeMaker::Awesome.header_file = EMM-include
Regarding what needs to be done at install time. First I will caution that requiring an internet connection to install is not something you can always rely on, nor is having yarn available of course. But the Alien series of modules for installing external libraries does this sort of thing often. Since you don't need to compile this code you probably don't need the whole Alien::Build/Alien::Base setup, but it may turn out to be an easier way to solve your problem than Makefile hacking described below. Basically you would first release an Alien distribution which installs your javascript library if necessary, and then this distribution could depend on that to load the library. If you decide to pursue this direction, check out Alien::Build, and the IRC channel #native on irc.perl.org.
The postamble section for ExtUtils::MakeMaker is not for running arbitrary code; it's for adding custom rules to the Makefile it generates; this is the way you need to influence the make process. I know very little about Makefiles so I can't help you further here, all I can suggest is to read all of the EUMM docs and note that the postamble is a function from MM_Any which you override to add your text, among other options from MM_Any and MM_Unix. You may be able to find people to help you in this direction on the IRC channel #toolchain on irc.perl.org.
I'm trying to build a toolchain for my hobby kernel, but I'm running into problems when building Newlib. Whenever I try to run autoreconf in my kernels directory under newlib/libc/sys/ I get an error:
configure.in:5: error: support for Cygnus-style trees has been removed
Here is the content of configure.in (basically, taken from the below tutorial):
AC_PREREQ(2.59)
AC_INIT([newlib], [NEWLIB_VERSION])
AC_CONFIG_SRCDIR([crt0.S])
AC_CONFIG_AUX_DIR(../../../..)
NEWLIB_CONFIGURE(../../..)
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
and the source for Makefile.am (again mostly from tutorial):
AUTOMAKE_OPTIONS = cygnus
INCLUDES = $(NEWLIB_CFLAGS) $(CROSS_CFLAGS) $(TARGET_CFLAGS)
AM_CCASFLAGS = $(INCLUDES)
noinst_LIBRARIES = lib.a
if MAY_SUPPLY_SYSCALLS
extra_objs = $(lpfx)syscalls.o
else
extra_objs =
endif
lib_a_SOURCES =
lib_a_LIBADD = $(extra_objs)
EXTRA_lib_a_SOURCES = syscalls.c crt0.S
lib_a_DEPENDENCIES = $(extra_objs)
lib_a_CCASFLAGS = $(AM_CCASFLAGS)
lib_a_CFLAGS = $(AM_CFLAGS)
if MAY_SUPPLY_SYSCALLS
all: crt0.o
endif
ACLOCAL_AMFLAGS = -I ../../..
CONFIG_STATUS_DEPENDENCIES = $(newlib_basedir)/configure.host
Yes, I have tried removing the AUTOMAKE_OPTIONS=cygnus.
I've Googled around and been trying to understand this, and as far as I can tell, it is because of the version of autotools I'm using. According to the tutorial I used originally (OSDev - OS Specific Toolchain), I need an older version. My problem is that I'm using Kubuntu, which uses the apt package manager, and that version is not available to fall back to even temporarily. There has to be some fix for this. Either Newlib is outdated (this release is from December of 2013...) or the developers are crazy for depending on an outdated autotools version.
The only other thing I can think is that this is a message from the newlib configuration scheme itself in which case I have no idea how to modify my configure.in and Makefile.am to align with the new newlib configure format. That tutorial is the only one I've found that didn't use libgloss (which I'd prefer not to do) so far and the documentation of adding a new target is rather lacking in the documentation for newlib (or I missed something).
Here is some version information:
System: Kubuntu 14.04
Automake: 1.14.1
Autoconf: 2.69
Newlib: 2.1.0
Unfortunately I'm afraid using automake 1.12 or earlier is your only choice. Ubuntu has an Automake1.11 separate package to help you there, if I'm not mistaken, since the compatibility between 1.12 and 1.14 is generally good, but before that it was spotty.
I am writing this answer for people struggling with the tutorial described here.
I am in the same situation you are (or were), I am building a kernel from scratch and I wanted to port newlib to my toolchain. Unfortunately I think the tutorial has become out of date because I followed the instructions EXACTLY, even installing the correct software with the proper versions (including the correct newlib version). The accepted solution above didn't work for me but I found another solution that might work for others:
Step 1 - get the correct software
Acquire Automake (v1.12) and Autoconf (v2.65) from here:
http://ftp.gnu.org/gnu/automake/automake-1.12.tar.gz
http://ftp.gnu.org/gnu/autoconf/autoconf-2.65.tar.gz
Step 2 - build process
Untar both of the archives:
tar xf automake-1.12.tar.gz
tar xf autoconf-2.65.tar.gz
Create a destination folder:
mkdir ~/bin
Create a build folder:
mkdir build
cd build
Configure automake first:
../automake-1.12/configure --prefix="~/bin"
Make and install
make && make install
Now lets configure autoconf
../autoconf-2.65/configure --prefix=~/bin
Then make and install:
make && make install
You should now have the proper binaries in ~/bin!
Step 3 - update PATH
To add these binaries to your path temporarily (recommended):
export PATH=~/bin:$PATH
Once you update your path, rerun autoconf and autoreconf and it should complete.
I am familiar with using package.json with node.js, Gemfile for Ruby, Podfile for Objective-C, et al.
What is the equivalent file for Perl and what is the syntax used?
I've installed a couple packages using cpanm and would like to save the package names and version in a single file that can be executed by team members.
For simple use cases, writing a cpanfile is a good choice. A sample file might look like
requires 'Marpa::R2', '2.078';
requires 'String::Escape', '2010.002';
requires 'Moo', '1.003001';
requires 'Eval::Closure', '0.11';
on test => sub {
requires 'Test::More', '0.98';
};
That is, it's actually a Perl script, not a data format. The dependencies can then be installed like
$ cd /path/to/your/module
$ cpanm --installdeps .
This does not install your module! But it makes sure that all dependencies are satisfied, so we can do:
use lib '/path/to/your-module/lib'; # add the location as a module search root
use Your::Module; # works! yay
This is usually sufficient e.g. for a git repository which you want others to tinker with.
If you want to create a tarball that can be distributed and installed easily, I'd recommend Dist::Zilla (although it's geared towards CPAN releases). Instead of a cpanfile we use a dist.ini:
name = Your-Module
version = 1.2.3
author = Your Self <you#example.com>
license = GPL_3
copyright_holder = Your Self
[#Basic]
[Prereqs]
Marpa::R2 = 2.078
String::Escape = 2010.002
Moo = 1.003001
Eval::Closure = 0.11
[Prereqs / TestRequires]
Test::More = 0.98
Then:
$ dzil test # sanity checks, and runs your tests
$ dzil build # creates a tarball
Dist::Zilla takes care of creating a Makefile.PL and other infrastructure that is needed to install the module.
You can then distribute that tarball, and install it like cpanm Your-Module-1.2.3.tar.gz. Dependencies are resolved, your packages are copied to a permanent location, and you can now use Your::Module in any script without having to specify the location.
Note that you should adhere to the standard directory layout for Perl modules:
./
lib/
Your/
Module.pm # package Your::Module
Module/
Helper.pm # package Your::Module::Helper
t/ # tests to verify the module works on the target syste,
foo.t
bar.t
xt/ # optional: Author tests that are not run on installation
baz.t
bin/ # optional: scripts that will later end up in the target system's $PATH
command-line-tool
Makefile.PL usually (along with a few other files; Perl has had packages for longer then any of the other languages you mention and suffers from a bit of inelegance here).
Module Starter is a sensible way to start writing a package. It has a getting started guide.