Setuptools supports dynamic metadata for project properties in pyproject.toml, and as a PEP517 backend, it also has the option to specify build requirements by implementing get_requires_for_build_wheel. But I cannot figure out whether it uses the chance and does implement a way to specify build requirements based on configuration options, and if so, how to specify it in the pyproject.toml.
I naively tried
[build-system]
requires = {file = "requirements-build.txt"}
but that understandably leads to pip complaining “This package has an invalid build-system.requires key in pyproject.toml. It is not a list of strings.” And adding
[project]
dynamic = ["build-system.requires"]
also doesn't work, because the possible options of dynamic are explicitly enumerated. I would be somewhat surprised if there wasn't an option for this, given that all the infrastructure elements are available, but how do I specify it?
As far as I know, it is not possible.
If it is really necessary for your use case, and you think it is worth the cost, maybe it is possible to add some dynamic behavior here by (mis-)using the "in-tree build backends" feature.
Related
Context
So, I'm trying to create a new python package following this guideline: https://packaging.python.org/en/latest/tutorials/packaging-projects/
As a guideline says - in my pyproject.toml I should have this structure:
[project]
name = "example_package_YOUR_USERNAME_HERE"
version = "0.0.1"
authors = [
{ name="Example Author", email="author#example.com" },
]
description = "A small example package"
but, when I've created this file with poetry init I have this structure:
[tool.poetry]
name = "example_package_YOUR_USERNAME_HERE"
version = "0.0.1"
authors = [
{ name="Example Author", email="author#example.com" },
]
description = "A small example package"
The main difference between this two is project and tool.poetry headers for sections.
I also see, that poetry can't do anything with project, when there is no [tool.poetry] in pyproject.toml
So my questions is:
What the difference between this two?
Should I have only one or both at the same time in my pyproject.toml? And what should contain what if I should keep both?
If there should be only [tool.poetry] - so I need to follow same rules for [project] sections naming? So [project.urls] will be renamed to [tool.poetry.urls]?
What is best for future publishing on PyPI? Or there is no difference?
Is changing [build-system] from poetry-core to setuptools is a good idea? Or I should keep poetry-core?
1. What the difference between this two?
The [project] section is standardized (also known as PEP-621). But Poetry is older than the creation of this standard, so it started by using its own section [tool.poetry]. Poetry is planning to add support for the standardized [project] (see python-poetry/poetry/issues/3332 and python-poetry/roadmap/issues/3), but it takes time.
The differences between the two are quite small, they are basically different notations for the same package metadata.
2. Should I have only one or both at the same time in my pyproject.toml? And what should contain what if I should keep both?
You should have only one. You have to choose a build back-end. If your build back-end is poetry-core then you need the [tool.poetry] section. If you choose a build back-end that requires [project] (which is the case of setuptools), then that is what you should have.
3. If there should be only [tool.poetry] - so I need to follow same rules for [project] sections naming? So [project.urls] will be renamed to [tool.poetry.urls]?
This is not exactly one-to-one equivalent, there are some differences. Follow Poetry's documentation if you use Poetry. Or the [project] specification if you use something else (setuptools, etc.).
4. What is best for future publishing on PyPI? Or there is no difference?
There is not much difference. You could argue that choosing a build back-end that follows the [project] standard is better, but really it is not what you should base your choice on. There are many other criteria you should base your choice on.
For example:
https://sinoroc.gitlab.io/kb/python/packaging_tools_comparisons.html#development-workflow-tools
https://sinoroc.gitlab.io/kb/python/packaging_tools_comparisons.html#build-back-ends
5. Is changing [build-system] from poetry-core to setuptools is a good idea? Or I should keep poetry-core?
Poetry the "development workflow tool" does not allow using any other build back-end than poetry-core. So if you want to keep using Poetry for your project, you have no choice but to keep using poetry-core as build back-end.
The [project] section is mandatory in pyproject.toml. If the entry is missing, the build tool (defined in [build-system] section) have to add it dynamically. I guess that's exactly what poetry does.
From the documentation:
The keys defined in this specification MUST be in a table named [project] in pyproject.toml. No tools may add keys to this table which are not defined by this specification. For tools wishing to store their own settings in pyproject.toml, they may use the [tool] table as defined in the build dependency declaration specification. The lack of a [project] table implicitly means the build back-end will dynamically provide all keys.
So you don't need the [project] while you are using poetry. If you change the build system, you must convert your pyproject.toml to be PEP 621 compliant.
Background
I'm trying to remove resize-observer-polyfill from an nx workspace that I'm working on because it's natively supported in the browsers that we are targeting. Once I removed the polyfill, I needed to add #types/resize-observer-browser because the workspace currently uses typescript#4.0.5 and my understanding is that TypeScript does not have a "native" type for ResizeObserver until v4.2 which I'd love to update to, but can't atm.
Problem
In order to make TypeScript happy, it seems like I have to go in and manually add "resize-observer-browser" to individual tsconfig compilerOptions.types entries. This didn't seem that bad to me at first. I just updated the tsconfig.lib.json file of the libraries that happened to utilize ResizeObserver. However, I soon realized I needed to also add it to the tsconfig.spec.json of the libraries so that the unit tests could run, and then I also needed to add it to the tsconfig.app.json of any applications that happened to import those libraries.
Question
Is there an easier way in an nx workspace to handle this sort of problem?
I think that I could remove the default types overrides in each of the tsconfig files, since that would let TypeScript just utilize everything that exists under node_modules/#types when compiling. I didn't want to take that path since I assume there is a good reason for the default nx library/app generators to add the types override (I assume it's to force you to be explicit and not accidentally get away with accidental imports of test code from business logic).
The docs seem to recommend against this for #types packages, but /// <reference types="..." /> (e.g. /// <reference types="resize-observer-browser" />) can be also be used to include types, and might be easier to manage if the type is only used in a few places.
Docs: https://www.typescriptlang.org/docs/handbook/triple-slash-directives.html#-reference-types-
Are #babel/runtime and #babel/plugin-transform-runtime supposed to be on the same version (e.g. both 7.2.0 exactly)? Or can I (as a library author) specify #babel/runtime dependency as ^7.0.0, whilst having the latest #babel/plugin-transform-runtime?
I'm aware that during the beta versions of Babel 7, there was a breaking change in beta.56 (see https://stackoverflow.com/a/51686837/2148762), but I'm guessing this should no longer be the case with the current stable version?
The reason I ask this is I'd ideally want the helpers from #babel/runtime to be shared across different packages, and to me leaving the version range open seems like a good idea. But at the same time, I'm not sure how low I should go (^7.0.0 or ^7.2.0), and whether there's an implicit contract between #babel/runtime and #babel/plugin-transform-runtime with regards to version numbers.
By default, #babel/plugin-transform-runtime is only allowed to output references to #babel/runtime that work on ^7.0.0 because it does not know what version you'd otherwise want to use, and doing anything else would cause lots of issues for users. This means that what you want to do is safe. The downside of this is that, if we add new helpers moving forward, your code would not be able to take advantage of the #babel/runtime version of them (because you might still be using a #babel/runtime version that doesn't have them.
Users can specify the version in the arguments for the transform, if you want to specifically make use of helpers that may have been added to Babel since 7.0.0, e.g.
{
"plugins": [
["#babel/plugin-transform-runtime", { version: "^7.2.0" }],
]
}
would then require you to have "#babel/runtime": "^7.2.0" in your package.json.
For instance, since support for the newer decorators proposal didn't land until Babel 7.1.5, if you use transform-runtime and non-legacy decorators, the decorators helper will still be inserted into every file where you use decorators, instead of importing it from #babel/runtime. To get the shared helper, you need to specify version: "^7.1.5" in your options for transform-runtime.
Can I (as a library author) specify #babel/runtime dependency as ^7.0.0, whilst having the latest #babel/plugin-transform-runtime?
Yes, this is safe.
I'm guessing this should no longer be the case with the current stable version?
Correct, that issue was because people failed to take the beta versioning into account.
I have a custom machine layer based on https://github.com/jumpnow/meta-wandboard.
I've upgraded the kernel to 4.8.6 and want to add X11 to the image.
I'm modifying to image recipe (console-image.bb).
Since wandboard is based on i.MX6, I want to include the xf86-video-imxfb-vivante package from meta-fsl-arm.
However, it fails complaining about inability to build kernel-module-imx-gpu-viv. I believe that happens because xf86-video-imxfb-vivante DEPENDS on imx-gpu-viv which in turn RDEPENDS on kernel-module-imx-gpu-viv.
I realize that those dependencies have been created with meta-fsl-arm BSP and vanilla Poky distribution. But those things are way outdated for wandboard, hence I am using the custom machine layer with modern kernel.
The kernel is configured to include the Vivante DRM module and I really don't want the kernel-module-imx-gpu-viv package to be built.
Is there a way to exclude it from RDEPENDS? Can I somehow swear my health to the build system that I will take care of this specific run-time dependency myself?
I have tried blacklisting 'kernel-module-imx-gpu-viv' setting PNBLACKLIST[kernel-module-imx-gpu-viv] in my local.conf, but that's just a part of a solution. It helps avoid build failures, but the packaging process still fails.
IIUC you problem comes from these lines in img-gpu-viv recipe:
FILES_libgal-mx6 = "${libdir}/libGAL${SOLIBS} ${libdir}/libGAL_egl${SOLIBS}"
FILES_libgal-mx6-dev = "${libdir}/libGAL${SOLIBSDEV} ${includedir}/HAL"
RDEPENDS_libgal-mx6 += "kernel-module-imx-gpu-viv"
INSANE_SKIP_libgal-mx6 += "build-deps"
I would actually qualify this RDEPENDS as a bug, usually kernel module dependencies are specified as RRECOMMENDS because most modules can be compiled into the kernel thus making no separate package at all while still providing the functionality. But that's another issue.
There are several ways to fix this problem, the first general route is to tweak RDEPENDS for the package. It's just a bitbake variable, so you can either assign it some other value or remove some portion of it. In the first case it's going to look somewhat like this:
RDEPENDS_libgal-mx6 = ""
In the second one:
RDEPENDS_libgal-mx6_remove = "kernel-module-imx-gpu-viv"
Obviously, these two options have different implications for your present and future work. In general I would opt for the softer one which is the second, because it has less potential for breakage when you're to update meta-fsl-arm layer, which can change imx-gpu-viv recipe in any kind of way. But when you're overriding some more complex recipe with big lists in variables and you're modifying it heavily (not just removing a thing or two) it might be easier to maintain it with full hard assignment of variables.
Now there is also a question of where to do this variable mangling. The main option is .bbappend in your layer, that's what appends are made for, but you can also do this from your distro configuration (if you're maintaining your own distro it might be easier to have all these tweaks in one place, rather than sprayed into numerous appends) or from your local.conf (which is a nice place to quickly try it out, but probably not something to use in longer term). I usually use .bbappend.
But there is also a completely different approach to this problem, rather than fixing package dependencies you can also fix what some other package provides. If for example you have a kernel configured to have imx-gpu-viv module built right into the main zimage you can do
RPROVIDES_kernel-image += "kernel-module-imx-gpu-viv"
(also in .bbappend, distro configuration or local.conf) and that's it.
In any case your approach to fixing this problem should reflect the difference between your setup and recipe assumptions. If you do have the module, but in a different package, then go for RPROVIDES, if you have some other module providing the same functionality to libgal-mx6 package then fix libgal-mx6 dependencies (and it's better to fix them, meaning not only drop something you don't need, but also add things that are relevant for your setup).
I have a question related to V4L-DVB drivers. Following the
Building/Compiling the Latest V4L-DVB Source Code link, there are 3 ways to
compile. I am curious about the last approach (More "Manually
Intensive" Approach). It allows me to choose the components that I
wish to build and install using the "make menuconfig". Some of these components (i.e. "CONFIG_MEDIA_ATTACH") are used in pre-processor directives that define a function in one shape if defined, and a function in another if not defined (i.e.
dvb_attach, dvb_detach) in the resulting modules (i.e. dvb_core.ko)
that will be loaded by most of the DVB drivers. What happens if there are two
drivers (*.ko modules) on the same host machine, one that needs dvb_core.ko with
CONFIG_MEDIA_ATTACH defined and another that needs dvb_core.ko with
CONFIG_MEDIA_ATTACH undefined, is there a clean way to handle this?
What is also not clear to me is: Since the V4L compilation environment seems very customizable (by setting the .config file), if I develop a driver using V4L-DVB structures, there is a big chance that it has conflicts with other drivers since each driver has its own custom settings. Is my understanding correct?
Thanks!
Dave