Generating beautiful Python API documentation with Sphinx AutoAPI and Jinja (a template engine) by Antoine Beyeler

../../../_images/twitter_announce.png

https://fediverse.org/abey79/status/1524067831154892801?s=20&t=o_xiUFXNuqK9KE3qOuKBlQ

Specifications

It would be useful to have an option that causes Sphinx to automatically create a TOC entry for:

  • every function ,

  • every class ,

  • every method .

In the absence of this, tables of contents are of limited value.

Antoine Beyeler

Antoine Beyeler, @abey79 2022-05-04 15h @mariatta @pradyunsg

Would it be possible to update this tutorial with the steps required for the class/members to show up in the TOC with autodoc? I haven’t been able to achieve this to this day, though it’s likely something anyone would want.

I have yet to find a suitable workaround. autosummary is far from it out-of-the-box.

Careful template crafting helps, but there is hardly any reference material on those.

Generating beautiful Python API documentation with Sphinx AutoAPI and Jinja (a template engine)

Despite #Sphinx ubiquity in the #Python world, it is surprisingly hard to produce nice looking and functional API reference docs, with cross-ref’d summaries and proper TOC support.

I spent days on this for #vsketch ( https://github.com/abey79/vsketch/pull/255 ), a nd wrote a tutorial with my findings:

../../../_images/twitter_announce.png

https://fediverse.org/abey79/status/1524067831154892801?s=20&t=o_xiUFXNuqK9KE3qOuKBlQ

../../../_images/discord_announce.png

https://discord.com/channels/499297341472505858/748589023731122277/974060638416556056

../../../_images/vsketch.png

https://vsketch.readthedocs.io/en/latest/autoapi/vsketch/index.html

../../../_images/intro.png

https://bylr.info/articles/2022/05/10/api-doc-with-sphinx-autoapi/

Following a recent discussion on Twitter , I decided to take yet another deep dive in my Python projects” documentation and fix once and for all the issues I had with it.

../../../_images/how_it_started.png

I first focused on the automatically-generated API reference section and this article details the results of my finding.

Specifically, I’m using vsketch’s API reference , which I recently updated, as an example documentation source.

This article addresses the following objectives:

  • Produce a beautiful API documentation based on the code docstrings that is both nice to look at and easy to navigate.

  • Support for a proper table of content navigation down to each class/module’s member.

  • Nice looking summary tables listing modules” and classes” contents.

Implementation

conf.py 

# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html


# -- Project information -----------------------------------------------------

project = "vsketch"
copyright = "2020-2022, Antoine Beyeler"
author = "Antoine Beyeler"

# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
    "sphinx.ext.intersphinx",
    "sphinx.ext.napoleon",
    "myst_parser",
    "sphinx_copybutton",
    "autoapi.extension",
]


# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "venv", ".*"]

# -- Global options ----------------------------------------------------------

# Don't mess with double-dash used in CLI options
smartquotes_action = "qe"

# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = "furo"

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ["_static"]

# -- Intersphinx options
intersphinx_mapping = {
    "shapely": ("https://shapely.readthedocs.io/en/latest/", None),
    "vpype": ("https://vpype.readthedocs.io/en/latest/", None),
    "python": ("https://docs.python.org/3/", None),
    "numpy": ("https://numpy.org/doc/stable/", None),
}

# -- Napoleon options
napoleon_include_init_with_doc = False

# -- autoapi configuration ---------------------------------------------------

autodoc_typehints = "signature"  # autoapi respects this

autoapi_type = "python"
autoapi_dirs = ["../vsketch"]
autoapi_template_dir = "_templates/autoapi"
autoapi_options = [
    "members",
    "undoc-members",
    "show-inheritance",
    "show-module-summary",
    "imported-members",
]
# autoapi_python_use_implicit_namespaces = True
autoapi_keep_files = True
# autoapi_generate_api_docs = False


# -- custom auto_summary() macro ---------------------------------------------


def contains(seq, item):
    """Jinja custom test to check existence in a container.

    Example of use:
    {% set class_methods = methods|selectattr("properties", "contains", "classmethod") %}

    Related doc: https://jinja.palletsprojects.com/en/3.1.x/api/#custom-tests
    """
    return item in seq


def prepare_jinja_env(jinja_env) -> None:
    """Add `contains` custom test to Jinja environment."""
    jinja_env.tests["contains"] = contains


autoapi_prepare_jinja_env = prepare_jinja_env

# Custom role for labels used in auto_summary() tables.
rst_prolog = """
.. role:: summarylabel
"""

# Related custom CSS
html_css_files = [
    "css/label.css",
]


# noinspection PyUnusedLocal
def autoapi_skip_members(app, what, name, obj, skip, options):
    """
        set_what={'method', 'package', 'function', 'class', 'data', 'exception', 'attribute', 'module'}
    """

    # skip submodules
    if what == "module":
        skip = True
    elif what == "data":
        if obj.name in ["EASING_FUNCTIONS", "ParamType"]:
            skip = True
    elif what == "function":
        if obj.name in ["working_directory"]:
            skip = True
    elif "vsketch.SketchClass" in name:
        if obj.name in [
            "vsk",
            "param_set",
            "execute_draw",
            "ensure_finalized",
            "execute",
            "get_params",
            "set_param_set",
        ]:
            skip = True
    elif "vsketch.Param" in name:
        if obj.name in ["set_value", "set_value_with_validation"]:
            skip = True
    return skip


def setup(app):
    app.connect("autoapi-skip-member", autoapi_skip_members)

skip members, python mmapper 

set_what = set()
def autoapi_skip_members(app, what, name, obj, skip, options):
    """
    """
    skip = False
    if name in [
        "intranet.conftest",
        "intranet.manage",
        "intranet.manage_dev",
        "intranet.manage_prod",
    ]:
        skip = True
        # print(f"autoapi_skip_members() {what=} {name=}")

    if "intranet.sorl" in name:
        skip = True

    set_what.add(what)
    return skip


def setup(app):
    app.connect("autoapi-skip-member", autoapi_skip_members)

def print_etat_objects():
    print(f"{set_what=}")

import atexit

atexit.register(print_etat_objects)

Results 

set_what={'method', 'package', 'function', 'class', 'data', 'exception', 'attribute', 'module'}

AutoAPI objects

Understanding the Sphinx AutoAPI objects is key to customising jinja templates . They are one of the major difference with respect to autodoc/autosummary.

In order to generate the API documentation, the autodoc loads your actual code and uses Python’s introspection capabilities to extract the required information from your module and class objects.

In contrast, Sphinx AutoAPI parses your Python code and builds a collection of so-called mapper objects which describe your code and its structure.

These objects are then passed on as context to the Jinja templating engine .

Oddly, the documentation doesn’t provide a reference about them , but their implementation is easy to read.

readthedocs/sphinx-autoapi/master/autoapi/mappers/python/__init__.py 

 1 from .mapper import PythonSphinxMapper
 2 from .objects import (
 3     PythonClass,
 4     PythonFunction,
 5     PythonModule,
 6     PythonMethod,
 7     PythonPackage,
 8     PythonAttribute,
 9     PythonData,
10     PythonException,
11 )

readthedocs/sphinx-autoapi/master/autoapi/mappers/python/mapper.py 

 1 class PythonSphinxMapper(SphinxMapperBase):
 2
 3     """Auto API domain handler for Python
 4
 5     Parses directly from Python files.
 6
 7     :param app: Sphinx application passed in as part of the extension
 8     """
 9
10     _OBJ_MAP = {
11         cls.type: cls
12         for cls in (
13             PythonClass,
14             PythonFunction,
15             PythonModule,
16             PythonMethod,
17             PythonPackage,
18             PythonAttribute,
19             PythonData,
20             PythonException,
21         )
22     }

Autoapi Jinja templates (vsketch/tree/master/docs/_templates/autoapi)

Description

Jinja is a fast, expressive, extensible templating engine. Special placeholders in the template allow writing code similar to Python syntax. Then the template is passed data to render the final document.

It includes:

  • Template inheritance and inclusion.

  • Define and import macros <macros_vsketch> within templates.

  • HTML templates can use autoescaping to prevent XSS from untrusted user input.

  • A sandboxed environment can safely render untrusted templates.

  • AsyncIO support for generating templates and calling async functions.

  • I18N support with Babel.

  • Templates are compiled to optimized Python code just-in-time and cached, or can be compiled ahead-of-time.

  • Exceptions point to the correct line in templates to make debugging easier.

  • Extensible filters, tests, functions, and even syntax.-

Jinja’s philosophy is that while application logic belongs in Python if possible, it shouldn’t make the template designer’s job difficult by restricting functionality too much.

Autoapi jinja Python templates 

sphinx-autoapi/autoapi/templates
    └── python
        ├── attribute.rst
        ├── class.rst
        ├── data.rst
        ├── exception.rst
        ├── function.rst
        ├── method.rst
        ├── module.rst
        └── package.rst

vsketch jinja vsketch/docs/_templates

Redefinition of Autoapi class and module . 

vsketch/docs/_templates

 └── autoapi
     ├── index.rst
     ├── macros.rst
     └── python
         ├── class.rst
         └── module.rst

vsketch jinja _template.autoapi.index.rst 

 1 API Reference
 2 =============
 3
 4 This page contains auto-generated API reference documentation.
 5
 6 .. toctree::
 7    :titlesonly:
 8
 9    {% for page in pages %}
10    {% if page.top_level_object and page.display %}
11    {{ page.include_path }}
12    {% endif %}
13    {% endfor %}

vsketch jinja _template.autoapi.macros.rst (Macro Jinja macros.auto_summary ) 

 1 {# AutoApiSummary replacement macro #}
 2 {#
 3 The intent of this macro is to replace the autoapisummary directive with the following
 4 improvements:
 5
 6 1) Method signature is generated without typing annotation regardless of the value of
 7    `autodoc_typehints`
 8 2) Properties are treated like attribute (but labelled as properties).
 9 3) Label are added as summary prefix to indicate if the member is a property, class
10    method, or static method.
11
12 Copyright: Antoine Beyeler, 2022
13 License: MIT
14 #}
15
16 {# Renders an object's name with a proper reference and optional signature.
17
18 The signature is re-generated from obj.obj.args, which is undocumented but is the
19 only way to have a type-less signature if `autodoc_typehints` is `signature` or
20 `both`. #}
21 {% macro _render_item_name(obj, sig=False) -%}
22 :py:obj:`{{ obj.name }} <{{ obj.id }}>`
23      {%- if sig -%}
24        \ (
25        {%- for arg in obj.obj.args -%}
26           {%- if arg[0] %}{{ arg[0]|replace('*', '\*') }}{% endif -%}{{  arg[1] -}}
27           {%- if not loop.last  %}, {% endif -%}
28        {%- endfor -%}
29        ){%- endif -%}
30 {%- endmacro %}
31
32
33 {# Generates a single object optionally with a signature and a labe. #}
34 {% macro _item(obj, sig=False, label='') %}
35    * - {{ _render_item_name(obj, sig) }}
36      - {% if label %}:summarylabel:`{{ label }}` {% endif %}{% if obj.summary %}{{ obj.summary }}{% else %}\-{% endif +%}
37 {% endmacro %}
38
39
40
41 {# Generate an autosummary-like table with the provided members. #}
42 {% macro auto_summary(objs, title='') -%}
43
44 .. list-table:: {{ title }}
45    :header-rows: 0
46    :widths: auto
47    :class: summarytable {#- apply CSS class to customize styling +#}
48
49   {% for obj in objs -%}
50     {#- should the full signature be used? -#}
51     {%- set sig = (obj.type in ['method', 'function'] and not 'property' in obj.properties) -%}
52
53     {#- compute label -#}
54     {%- if 'property' in obj.properties -%}
55       {%- set label = 'prop' -%}
56     {%- elif 'classmethod' in obj.properties -%}
57       {%- set label = 'class' -%}
58     {%- elif 'abstractmethod' in obj.properties -%}
59       {%- set label = 'abc' -%}
60     {%- elif 'staticmethod' in obj.properties -%}
61       {%- set label = 'static' -%}
62     {%- else -%}
63       {%- set label = '' -%}
64     {%- endif -%}
65
66     {{- _item(obj, sig=sig, label=label) -}}
67   {%- endfor -%}
68
69 {% endmacro %}

vsketch jinja _template.autoapi.python.class.rst 

 1 {% import 'macros.rst' as macros %}
 2
 3 {% if obj.display %}
 4 .. py:{{ obj.type }}:: {{ obj.short_name }}{% if obj.args %}({{ obj.args }}){% endif %}
 5 {% for (args, return_annotation) in obj.overloads %}
 6    {{ " " * (obj.type | length) }}   {{ obj.short_name }}{% if args %}({{ args }}){% endif %}
 7 {% endfor %}
 8
 9
10    {% if obj.bases %}
11    {% if "show-inheritance" in autoapi_options %}
12    Bases: {% for base in obj.bases %}{{ base|link_objs }}{% if not loop.last %}, {% endif %}{% endfor %}
13    {% endif %}
14
15
16    {% if "show-inheritance-diagram" in autoapi_options and obj.bases != ["object"] %}
17    .. autoapi-inheritance-diagram:: {{ obj.obj["full_name"] }}
18       :parts: 1
19       {% if "private-members" in autoapi_options %}
20       :private-bases:
21       {% endif %}
22
23    {% endif %}
24    {% endif %}
25    {% if obj.docstring %}
26    {{ obj.docstring|indent(3) }}
27    {% endif %}
28    {% if "inherited-members" in autoapi_options %}
29    {% set visible_classes = obj.classes|selectattr("display")|list %}
30    {% else %}
31    {% set visible_classes = obj.classes|rejectattr("inherited")|selectattr("display")|list %}
32    {% endif %}
33    {% for klass in visible_classes %}
34    {{ klass.render()|indent(3) }}
35    {% endfor %}
36    {% if "inherited-members" in autoapi_options %}
37    {% set visible_attributes = obj.attributes|selectattr("display")|list %}
38    {% else %}
39    {% set visible_attributes = obj.attributes|rejectattr("inherited")|selectattr("display")|list %}
40    {% endif %}
41    {% if "inherited-members" in autoapi_options %}
42    {% set visible_methods = obj.methods|selectattr("display")|list %}
43    {% else %}
44    {% set visible_methods = obj.methods|rejectattr("inherited")|selectattr("display")|list %}
45    {% endif %}
46
47    {% if visible_methods or visible_attributes %}
48    .. rubric:: Overview
49
50    {% set summary_methods = visible_methods|rejectattr("properties", "contains", "property")|list %}
51    {% set summary_attributes = visible_attributes + visible_methods|selectattr("properties", "contains", "property")|list %}
52    {% if summary_attributes %}
53    {{ macros.auto_summary(summary_attributes, title="Attributes")|indent(3) }}
54    {% endif %}
55
56    {% if summary_methods %}
57    {{ macros.auto_summary(summary_methods, title="Methods")|indent(3) }}
58    {% endif %}
59
60    .. rubric:: Members
61
62    {% for attribute in visible_attributes %}
63    {{ attribute.render()|indent(3) }}
64    {% endfor %}
65    {% for method in visible_methods %}
66    {{ method.render()|indent(3) }}
67    {% endfor %}
68    {% endif %}
69 {% endif %}

vsketch jinja _template.autoapi.python.module.rst 

  1 {% import 'macros.rst' as macros %}
  2
  3 {% if not obj.display %}
  4 :orphan:
  5
  6 {% endif %}
  7 {{ obj.name }}
  8 {{ "=" * obj.name|length }}
  9
 10 .. py:module:: {{ obj.name }}
 11
 12 {% if obj.docstring %}
 13 .. autoapi-nested-parse::
 14
 15    {{ obj.docstring|indent(3) }}
 16
 17 {% endif %}
 18
 19 {% block subpackages %}
 20 {% set visible_subpackages = obj.subpackages|selectattr("display")|list %}
 21 {% if visible_subpackages %}
 22 Subpackages
 23 -----------
 24 .. toctree::
 25    :titlesonly:
 26    :maxdepth: 3
 27
 28 {% for subpackage in visible_subpackages %}
 29    {{ subpackage.short_name }}/index.rst
 30 {% endfor %}
 31
 32
 33 {% endif %}
 34 {% endblock %}
 35 {% block submodules %}
 36 {% set visible_submodules = obj.submodules|selectattr("display")|list %}
 37 {% if visible_submodules %}
 38 Submodules
 39 ----------
 40 .. toctree::
 41    :titlesonly:
 42    :maxdepth: 1
 43
 44 {% for submodule in visible_submodules %}
 45    {{ submodule.short_name }}/index.rst
 46 {% endfor %}
 47
 48
 49 {% endif %}
 50 {% endblock %}
 51 {% block content %}
 52 {% if obj.all is not none %}
 53 {% set visible_children = obj.children|selectattr("display")|selectattr("short_name", "in", obj.all)|list %}
 54 {% elif obj.type is equalto("package") %}
 55 {% set visible_children = obj.children|selectattr("display")|list %}
 56 {% else %}
 57 {% set visible_children = obj.children|selectattr("display")|rejectattr("imported")|list %}
 58 {% endif %}
 59 {% if visible_children %}
 60 Overview
 61 --------
 62
 63 {% set visible_classes = visible_children|selectattr("type", "equalto", "class")|list %}
 64 {% set visible_functions = visible_children|selectattr("type", "equalto", "function")|list %}
 65 {% set visible_attributes = visible_children|selectattr("type", "equalto", "data")|list %}
 66 {% if "show-module-summary" in autoapi_options and (visible_classes or visible_functions) %}
 67 {% block classes scoped %}
 68 {% if visible_classes %}
 69 {{ macros.auto_summary(visible_classes, title="Classes") }}
 70 {% endif %}
 71 {% endblock %}
 72
 73 {% block functions scoped %}
 74 {% if visible_functions %}
 75 {{ macros.auto_summary(visible_functions, title="Function") }}
 76 {% endif %}
 77 {% endblock %}
 78
 79 {% block attributes scoped %}
 80 {% if visible_attributes %}
 81 {{ macros.auto_summary(visible_attributes, title="Attributes") }}
 82 {% endif %}
 83 {% endblock %}
 84 {% endif %}
 85
 86 {% if visible_classes %}
 87 Classes
 88 -------
 89 {% for obj_item in visible_classes %}
 90 {{ obj_item.render()|indent(0) }}
 91 {% endfor %}
 92 {% endif %}
 93
 94 {% if visible_functions %}
 95 Functions
 96 ---------
 97 {% for obj_item in visible_functions %}
 98 {{ obj_item.render()|indent(0) }}
 99 {% endfor %}
100 {% endif %}
101
102 {% if visible_attributes %}
103 Attributes
104 ----------
105 {% for obj_item in visible_attributes %}
106 {{ obj_item.render()|indent(0) }}
107 {% endfor %}
108 {% endif %}
109
110
111 {% endif %}
112 {% endblock %}

vsketch CSS

_static.css.label.css 

.summarylabel {
    background-color: var(--color-foreground-secondary);
    color: var(--color-background-secondary);
    font-size: 70%;
    padding-left: 2px;
    padding-right: 2px;
    border-radius: 3px;
    vertical-align: 15%;
    padding-bottom: 2px;
    filter: opacity(40%);
}


table.summarytable {
    width: 100%;
}