2023-07

2023-07-31 Excellent tutorial about async programming : “Bridging the Python async gap from the other side” by Adrian ( https://github.com/lemon24 )

So, you’re doing some sync stuff.

But you also need to do some async stuff , without making everything async.

Maybe the sync stuff is an existing application.

Maybe you still want to use your favorite sync library .

Or maybe you need just a little async, without having to pay the full price.

Of course, you can run a coroutine with asyncio.run(), and blocking sync code from a coroutine with asyncio.to_thread(), but the former isn’t granular enough, and the latter doesn’t solve async code being at the top.

As always, there must be a better way.

2023-07-30 What’s up, Python? The GIL removed, a new compiler, optparse deprecated…

  • https://www.bitecode.dev/p/whats-up-python-the-gil-removed-a

  • Python without the GIL, for good

  • LPython: a new Python Compiler

  • Pydantic 2 is getting usable

  • PEP 387 defines “Soft Deprecation”, getopt and optparse soft deprecated

  • Cython 3.0 released with better pure Python support

  • PEP 722 – Dependency specification for single-file scripts

  • Python VSCode support gets faster

  • Paint in the terminal

LPython: a new Python Compiler

That’s the news I didn’t see coming. In “What’s the deal with CPython, Pypy, MicroPython, Jython…?” we talked about Python compilers, and I thought I did a pretty good job about listing everything that mattered.

Well, the team behind LPython decided to take this list and .append() on it.

LPython is a new BSD 3 compiler that takes Python code and translate it for the following for LLVM, C, C++ or WASM.

It doesn’t aim to compile the entire program, although it can, but rather, like numba and cython, to let you speed up numerical bottle neck.

The benchmarks are very promising and the ability to switch between Ahead-of-Time and Just-in-Time very convenient, although you will still need the entire compilation chain installed on the machine.

LPython likes raw Python code, so if you call a Python function inside your snippet, you must explicitly mark it as such with a decorator. So most will likely use it for very specific snippets.

Pydantic 2 is getting usable

I’ve been pitching the coming of the version 2 of Pydantic for some time, because I, and many people, use it a lot for data validation / schema definition, and the new version is much faster.

Yes, it came out as stable last month, but if you read “Relieving your Python packaging pain” you know I don’t encourage people to use the last version of anything except for testing or having fun.

Indeed, even a stable major version is still something that is guaranteed to need refinement, and still has little community support.

But now two things have happened:

  • Pydantic 2.1 has been released, the first wave of nasty bugs have been eradicated.

  • Fast API announced support of Pydantic 2. Since it’s the biggest driver of Pydantic usage, it’s a milestone.

I will now proceed with giving it a try in one personal project, and if it works, move it into professional projects in a few months.

2023-07-28 Security Developer-in-Residence – Weekly Report #5

CVE Numbering Authority (CNA) registration

I’ve continued working on having the Python Software Foundation registered as a CNA. We have at this point submitted our registration containing a list of contacts, a disclosure policy, location of advisories and an advisory database.

We’ve received an onboarding meeting date (August 21st) where we’ll meet with the CNA partner team to ask questions about the process and fill out some example CVE records.

The Python Steering Council have approved having Python scoped under the PSF CNA! 🥳

Part of being a CNA is creating and updating CVE records which can be tough to do by hand to large JSON documents.

To make managing CVE records a more guided experience there are tools like Vulnogram which provide a UI to editing a CVE record.

Vulnogram lets you create template files that you can load into the program, so it’s possible to pre-fill Python’s security release schedule into the “Affected versions” fields for CVE records along with other fields like vendor, CPEs, product name, etc:

Vulnogram also has the ability to render human-readable advisories from the input information which should make the manual aspects of CVE disclosure easier. There’s a YouTube video all about Vulnogram and how to use it with CVE Services.

I attended the OpenSSF Vulnerability Disclosures working group call and discussed the PSF CNA motivations and my desire to create materials for other open source projects looking to become CNAs for their own project.

The folks on the call agreed that the documentation for a “minimum viable CNA” geared towards maintainers of open source projects would be a valuable resource.

Look forward to updates here as the PSF progresses through the CNA onboarding process.

Sigstore signed Python releases

Last week I reported that Sigstore signatures for Python releases were being worked on in order to have a consistent and well-documented experience as a downstream consumer of release artifact signatures.

This week I can report that all Python releases with signatures are now consistent with the documented signing methods.

My announcement spawned a conversation about downstream distributions consumption of Sigstore signatures compared to GPG signatures, mostly that there is no way to have a previously configured trust configuration (for Sigstore that is a identity and provider pair, for GPG that would be a key) continue to work and be extended to a new trust configuration (ie adding a new identity/provider for Sigstore or GPG key).

For Sigstore a consumer would need to check the documented release manager in order to know which identity to verify for a given Python release stream.

Sigstore bundles

When Sigstore signing was first applied to Python releases there were two files created, a certificate (.crt) and a signature (.sig) for each release artifact.

Some time later the concept of a Sigstore “bundle” emerged as a way to have a single file with all materials required to verify an artifact (the certificate, signature, and the Rekor transparency log entry).

By default, the Sigstore Python client creates both sets of verification materials when signing an artifact.

This meant that the downloads page would need to document both ways of verifying Python release artifacts which is already confusing, but then there would need to be an additional decision for consumers as well!

Old releases may only contain a crt/sig combination and not a bundle.

I raised this issue in the Sigstore Slack and was recommended by William Woodruff that this functionality may be useful for others looking to migrate to the newer improved Sigstore bundles.

Together we created an issue describing the problem and I authored a pull request that adds the internal APIs required in order to transform crt/sig files into a bundle.

This will allow back-filling bundles to previous Python release artifacts without needing to resign them.

2023-07-28 WOW! IT’S HAPPENING: the Steering Council posted their “intent to accept PEP 703”, in other words to allow for a version of #Python without the #GIL

WOW! IT’S HAPPENING: the Steering Council posted their “intent to accept PEP 703” in other words to allow for a version of #Python without the #GIL.

2023-07-25 Weekly Report, EuroPython 2023 by Łukasz Langa

Our new Security Developer in Residence is out and about, and publishes weekly updates on what he’s up to.

That inspires me to resume doing the equivalent of those updates.

And what better opportunity to do that than on the heels of EuroPython 2023!

This year EuroPython was held in Prague, which is an easy drive from Poznań, where I’m currently located. Apart from the event, I was able to do some sightseeing with my family, and experience some of the less touristy Prague as well. I could see myself living there, seems like a great city. The metro in particular is pretty impressive.

Sprints after the conference

There was no official core developer presence at the sprint last year and I wanted to fix that this time around. I announced the sprint on the website , and again at my talk, and at the Sprint Introductions after the main conference closed.

Hugo and I expected a few people, 10 tops, to join us and do some light coding over the weekend. We grossly underestimated interest.

On Saturday we’ve had over 30 people show up to code with us, for a room of 24 people that we shared with BeeWare. Russell kindly switched rooms to allow us to grow (thank you!) and we still had to overflow people to other rooms.

Amazingly, no-one reported build problems to me this time around, which is a first for any sprints I took part in.

The combination of the devguide , devcontainer integration in VScode, as well as the , (comma) key in GitHub UI on the Web made hacking on Python more accessible than ever.

And people did hack! We merged over 140 PRs over the weekend, ranging from trivial documentation fixes to changes deep in the C code of the interpreter. My favorite rabbit holes involved investigating how reference counting in Python interacts with C pointer indirection when using ctypes, a slew of quick well-thought-through patches to gettext, and a still-ongoing investigation of dtrace/systemtap probes.

Of course, these are just a few examples of a variety of improvements to Python made over the weekend. I was overwhelmed by people’s dedication and excitement for improving our favorite little interpreter. We’re so lucky.

Other highlights

Although I wasn’t myself involved in those, I would like to point out a few other favorite moments from the conference:

The 5-star Czech Slovak Restaurant

Finally, I would be remiss if I did not mention that the 5-star Czech Slovak Restaurant serves a tasting menu that takes over 4 hours to get through, as me and those fine people in the picture below can attest.

Highly recommended!

../_images/restaurant.png

Mind is empty, heart is full

This conference was intense! Countless positive interactions, deeply technical conversations, and the notoriously vegetable-free Czech food (+beer!) combined into a very positive experience.

While we don’t know yet if EuroPython 2024 will be held in Prague again, we know that we will be returning for sure.

2021-07-21 Sigstore signed Python releases by Seth Larson

Ever since the 3.11.0 release of Python, all Python release tarballs are signed using Sigstore.

You can read this page for more information about Sigstore signatures and how to verify the signatures using the Python Sigstore client.

The gist is each Release Manager for a given Python release (ie 3.7, 3.8, etc) has an identity (email address with @python.org) defined in the Sigstore information page.

This is the identity used to verify who signed the release.

Sigstore uses OpenID Connect which means we also need to designate an identity provider (IdP) to verify signatures.

# Currently documented example of how to verify Python signatures 

$ python -m sigstore verify identity \
  --certificate Python-3.11.0.tgz.crt \
  --signature Python-3.11.0.tgz.sig \
  --cert-identity pablogsal@python.org \
  --cert-oidc-issuer https://accounts.google.com \
  Python-3.11.0.tgz

It was reported that the instructions for verifying signatures weren’t consistent with the reality of how the artifacts had been signed.

I put together a few simple scripts which downloaded and attempted to verify every Python release artifact against its signatures and published the results. These scripts found a few issues:

  • Ned Deily and Łukasz Langa used GitHub’s IdP and Pablo Galindo Salgado and Thomas Wouters used Google’s IdP.

  • Łukasz’s identity for all signatures was lukasz @ langa . pl , instead of the documented lukasz @ python . org .

  • Python 3.11.4 was signed with Pablo’s correct identity, but with GitHub’s IdP instead of Google’s IdP.

  • Python 3.7.14 was signed by Pablo, but Ned was the release manager for 3.7.

  • Python 3.8.14 and 3.9.14 had signatures generated but weren’t accessible from python.org/download due to permission issues.

  • Python 3.10.1 and 3.10.7 onwards were signed, but 3.10.0 and 3.10.2-3.10.6 weren’t signed. This is fine though as only 3.10.7 and onwards are documented as being signed.

With these findings the release managers took the following steps to make the signature verifications consistent for consumers:

  • Ned resigned the 3.7.14 release with his own identity.

  • Pablo resigned 3.11.4 with the Google IdP.

  • Łukasz fixed the permissions for 3.8.14 and 3.9.14 making the signatures available.

  • The CDN cache had to be purged for the updates to be available.

You can see the change in state in this commit to the above dataset.

From here all that’s needed is fixing Łukasz’s identity on signatures which may prove difficult as using GitHub as an IdP only allows the primary email address as an identity rather than all configured email addresses.

William Woodruff opened an issue after discussion of this any other GitHub IdP use-cases on Sigstore’s Slack channel.

I’ve also opened a PR in Python’s release-tool to keep Sigstore signing consistent with the documentation for future Python releases.

2023-07-15 Security Developer-in-Residence – Weekly Report #3 by Seth Larson

Python wheel build numbers

Python wheel distributions have a concept called “build numbers” or “build tags” which are kind-of part of the package version, but also not! They don’t appear in the version of the release on PyPI but build numbers do have an impact on resolution of packages like when installing via pip.

Higher build numbers are favored by pip over lower (or non-existent) build numbers.

The primary use-case for build numbers is to be able to publish new wheels to an existing release without needing to rebuild all the wheels again which would be necessary if a new release was created.

Now why are build numbers interesting from a security POV? The problem is that unlike the “partial rebuild” mechanism for many ecosystems (like downstream repackaging) the build number is not a part of the version.

This means that without using hashes or direct URLs there’s no way to force pip to install a wheel with a specific build number.

This also means that it’s difficult to reference the newly released distributions from outside the ecosystem which typically use versions and version ranges (like CVEs, OSV, Package URLs, etc).

I’ve created a discussion thread on this topic and will create some guidance for build numbers’ use-case.

Relatedly: Back at PyCon US 2023 I was nerd-sniped by William Woodruff into writing an entire post about PEP 440 and quirks in Python versioning.

That piece is nearly complete and will hopefully be coming soon. Stay tuned!

2023-07 A new tutorial: Data analysis with SQLite and Python

I presented this as a 2hr45m tutorial at PyCon a few months ago.

The video is now available, and I like to try to turn these kinds of things into more permanent documentation.

The Datasette website has a growing collection of tutorials, and I decided to make that the final home for this one too.

Data analysis with SQLite and Python now has the full 2hr45m video plus an improved version of the handout I used for the talk. The written material there there should also be valuable for people who don’t want to spend nearly three hours watching the video!

As part of putting that page together I solved a problem I’ve been wanting to figure out for a long time: I figured out a way to build a custom Jinja block tag that looks like this: 

{% markdown %}
# This will be rendered as markdown

- Bulleted
- List
{% endmarkdown %}

I released that in datasette-render-markdown 2.2.

I also wrote up a TIL on Custom Jinja template tags with attributes describing the pattern I used.

One bonus feature for that tutorial: I decided to drop in a nested table of contents, automatically derived from the HTML headers on the page.

2023-07 Python packages with pyproject.toml and nothing else

I’ve been using setuptools and setup.py for my Python packages for a long time: I like that it works without me having to think about installing and learning any additional tools such as Flit or pip-tools or Poetry or Hatch.

pyproject.toml is the new (or not so new, the PEP is dated June 2020) standard for Python packaging metadata.

Today I figured out how to package a project with a single pyproject.toml file, using just pip and build to install and build that package.

(Note that the approach described in this document likely only works for pure Python packages. If your package includes any binary compiled dependencies you likely need to use a different approach.)

2023-07-07 Python bestpractices.coreinfrastructure.org

hyper-bump-it

stamina

2023-07-04 Création du groupe https://framagit.org/gdevops/python

Création du groupe https://framagit.org/gdevops/python

Avec 4 sous-projets: