2023-01-21 Thoughts on the Python packaging ecosystem

PyPA and Conda

Within this spectrum, I think the relationship between Conda and PyPA projects is definitely collaborative.

The two groups of maintainers have worked together to solve problems that affect both groups.

Conda packages are often built up from Python packages that are built with PyPA tools. Heck, at the time of writing, one of the founders of the PyPA currently sits on the Conda Steering Council.

Python build-backends

The goal of enabling pyproject.toml-based builds2 was to move Python packaging from a collaborative (or at least, cordial) model, to a competitive model for the build mechanisms.

The stated goal of PEP 517 is:

The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems.

If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods.

Moving to a competitive model for build mechanisms was intended to enable the ecosystem to move away from the “only one way” of building Python packages (the quote is from the Zen of Python) because the implementation we had of “only one way” was exceedingly difficult to evolve.

Poetry / PDM / Hatch / PyFlow / etc

These tools are firmly in a competitive model.

They’re competing for users. They’re competing to be the “best” solution to the “workflow” problem. They’re, arguably, even competing for contributors.

Other than the obvious sign that these tools can’t be used together on the same codebase (mostly), this can be seen in other aspects of these projects:

  • the way they’re marketed/documented they have an incentive to invest in this, because they’re competing for users and good-looking/flashy documentation is a good way to attract users.

  • the way they do community management: some have dedicated community Discord servers, mention $tool-specific ecosystems, have their own $tool plugin ecosystems, etc.

  • the way their users advocate for them as the one-true-solution on the internet :)

This competition also leads to incentives for projects to do things like implementing draft standards that are not accepted or settled and claims that standards are implemented, even when the implementation does not match the standard.