Integrate cffi-buildtool into CFFI itself by ngoldbaum · Pull Request #241 · python-cffi/cffi

ngoldbaum · 2026-04-20T19:42:36Z

Closes #47. Revives #76.

This adds a new subpackage for cffi named cffi.buildtool and a new CLI script distributed by CFFI named gen-cffi-src. Also adds new tests and documentation. See the new documentation and tests for usage details.

I realize that this is a big change that makes a number of opinionated choices. I'm very happy to adjust things to suit the maintainers' taste.

My ultimate goal here is to make it possible for projects using CFFI to use arbitrary build tools and remove the perceived tight coupling between CFFI and setuptools/distutils. If there isn't appetite to add the code in cffi-buildtool to CFFI itself, I'd also happily document how to depend on and use cffi-buildtool instead.

See inklesspen/cffi-buildtool#2 where I got @inklesspen's blessing to do this.

The is based on code and documentation that were originally written by Rose Davidson (@inklesspen on GitHub) for the cffi-buildtool project: https://github.com/inklesspen/cffi-buildtool.

@inklesspen

This adds code and documentation that was originally written by Rose Davidson (@inklesspen on GitHub) for the cffi-buildtool project: https://github.com/inklesspen/cffi-buildtool

inklesspen · 2026-04-20T19:46:05Z

Just commenting here to confirm I give my full support for this upstreaming, and after a release is cut which includes these changes, I will mark my existing project as deprecated.

Also I'm available to answer any questions about the original code upon which this is based.

ngoldbaum

Some thoughts that occurred to me since yesterday and a typo fix

mattip · 2026-04-21T16:38:35Z

There is a clang-tsan failure

ngoldbaum · 2026-04-21T16:41:21Z

@mattip see #242 and #243, the failure is unrelated.

ngoldbaum · 2026-05-19T18:30:53Z

Sorry for taking so long to come back to this. I think this is ready for another round of review now.

mattip

This seems very helpful, and will provide a path to move away from distutils/setuptools only cffi support.

Some of my comments might miss the mark. In general I think the concept of "build" is not clear enough. A cffi out-of-line module has three parts: generate C code, compile the C code into a c-extension, and import the module. Does "build" address all three?

An in-line mode module (no compiler used) is different, I assume this does not address the inline-mode usage.

mattip · 2026-06-08T17:28:35Z

+.. contents::
+
+CFFI ships a small subpackage, :mod:`cffi.buildtool`, together with a
+command-line program, ``gen-cffi-src``. Both produce the same output as


Is the command line program installed with an entry point?

Yes; see the change in pyproject.toml.

On further reflection, I think it simplifies things to make the only UI be python -m cffi.buildtool, so I've removed the gen-cffi-src script and entry point in the latest version of the PR.

mattip · 2026-06-08T17:31:12Z

+a CPython extension module. What they add is two convenient front-ends
+-- one that executes an existing "build" Python script, and one that
+reads a ``cdef`` and C prelude from two files. This tool enables


I don't understand. Can you rework the "They add" with more words?

Suggested change

a CPython extension module. What they add is two convenient front-ends

-- one that executes an existing "build" Python script, and one that

reads a ``cdef`` and C prelude from two files. This tool enables

a CPython extension module. They add is two convenient front-ends

-- one that executes an existing "build" Python script, and one that

reads a ``cdef`` and C prelude from two files. This tool enables

I'm applying your suggestion but I'm going to push a subsequent comment that rewords this.

This paragraph ended up getting dropped completely. I think the point that cffi.buildtool is only for generating C sources is a clearer now.

mattip · 2026-06-08T17:34:15Z

+.. py:function:: find_ffi_in_python_script(pysrc, filename, ffivar)
+
+   Execute a Python build script and return the :class:`cffi.FFI`
+   object it defines. ``pysrc`` is the text of the script,


Does the build script always compile a c-extension module? Is the lib object also returned somehow?

No, the build script doesn't build an extension module at all. It generates the FFI binding object (a cffi.api.FFI instance) which then is used to generate the C extension sources.

I guess calling it a Python build script is confusing. How about Calling it "a script that dynamically defines a set of FFI bindings"?

I'll try to rewrite these docs to make that point clearer.

mattip · 2026-06-08T17:35:06Z

+
+.. note::
+
+   When you drive the build from a build backend, the


What does "drive the build" mean in this context?

I mean when CFFI is used without CFFI's built-in distutils integration. Specifically via functionality recompiler.py. I'll rephrase this to make that clearer.

mattip · 2026-06-08T17:36:31Z

+
+This mode takes the Python build script you would normally run by
+hand -- the one the CFFI docs show under "Main mode of usage" -- and
+generates the ``.c`` source for you. For example, given this


Does it stop after generating the C source, or does it also compile the c-extension module?

It stops after generating the C source; it is the job of a c compiler to compile the c extension module. (Build tools such as meson already have support for this.)

I'll try to make this whole set of documentation clearer that there are no compilation steps involved - this is all about generating the C source code for a CFFI extension at build time, without relying on setuptools or CFFI's recompiler and setuptools integration.

mattip · 2026-06-08T17:39:12Z

+
+   This covers how to create wrapper modules. See :ref:`buildtool_docs`
+   for instructions on how to integrate with a Python build backend and
+   distribute wrapper modules.


What is a wrapper module? A definition would be nice.

mattip · 2026-06-08T17:41:13Z

+* :func:`make_ffi_from_sources` -- construct an :class:`cffi.FFI`
+  from a ``cdef`` string and a C source prelude.
+* :func:`generate_c_source` -- emit the generated C source for an
+  :class:`cffi.FFI` as a string.


In the documentation it mentioned two modes for the subpackage: (1) generate the c code and (2) compile the c-extension and import the module. Is that mentioned somewhere here?

Sorry if the language about build scripts was confusing. There's no compilation happening. The two modes of operation are either based on a Python script that defines the FFI bindings or a pair of text files that define the FFI bindings for the C API to be wrapped.

The second mode is based on the read_sources function that's currently not exposed in the top-level API here. I guess I might as well expose that as well if we're going to expose a Python API here.

I think we could also probably choose not to expose a public Python API here. On reflection a couple months after doing this PR initially, I lean towards not exposing a Python API and only documenting the command-line interface.

rgommers · 2026-06-08T19:49:21Z

+   ``extra_compile_args=`` etc. arguments you pass to
+   :meth:`FFI.set_source` are *ignored*. Link and include settings are
+   the build backend's responsibility; for meson-python you express
+   them through the ``dependencies:`` / ``include_directories:``


extra_compile_args maps to c_args, might want to add that for completeness

rgommers · 2026-06-08T19:54:37Z

+      squared_ext_src,
+      subdir: 'squared',
+      install: true,
+      dependencies: [square_dep, py.dependency()],


Drop py.dependency here, it's never needed inside a py.extension_module call. It may be needed inside a static_library call (as above), if that used CPython C API rather than it being a pure C library (EDIT: but if it were, you'd see it in CI as a build failure).

rgommers · 2026-06-08T20:02:29Z

 and *if* the "stuff" part is big enough that import time is a concern,
 then rewrite it as described in `the out-of-line but still ABI mode`__
-above.  Optionally, see also the `setuptools integration`__ paragraph.
+above.  Optionally, see also the :ref:`build backend and distrubution


typo distrubution (also higher up, search the diff for it)

rgommers · 2026-06-08T20:11:14Z

@@ -0,0 +1,58 @@
+# Integrated from the cffi-buildtool project by Rose Davidson
+# (https://github.com/inklesspen/cffi-buildtool), MIT-licensed.


Technically this is incorrect, it misses the required attribution. Easiest way to do this correctly is to add the upstream LICENSE file as buildtool/LICENSE in this PR. The main cffi license file already says "Except when otherwise stated (look for LICENSE files in directories or
information at the beginning of each file) ...", so nothing else is needed.

I licensed this code under the MIT license because that was the license used for CFFI at the time that I wrote this tool. I will be happy to relicense it under the current "MIT No Attribution" license that is now used by CFFI, if that makes matters at all simpler.

No worries, I think I'll just add the license file you have now to the buildtool subdirectory as Ralf suggested. That seems cleanest to me from an attribution point of view.

In the latest version of the PR the whole implementation is in one _buildtool.py file which now includes the original MIT license verbatim in the source header.

rgommers · 2026-06-08T20:15:05Z

I had a read through out of interest; overall this LGTM. The CLI taking the two main build modes and source as input, generated C code as output is useful and matches how Cython, f2py and other such codegen/binding tools are integrated into meson-python and scikit-build-core; the pattern should indeed work for other build backends that are able to build extension modules from C code as well.

Thanks for the review comments! I'm going to apply the review suggestions that are pending now, pull, and address the remaining comments. Co-authored-by: Matti Picus <matti.picus@gmail.com>

…ol CLI

ngoldbaum · 2026-06-09T20:19:35Z

I think I've responded to all the review comments. I'd appreciate another look :)

Integrate cffi-buildtool into CFFI itself

92345d6

This adds code and documentation that was originally written by Rose Davidson (@inklesspen on GitHub) for the cffi-buildtool project: https://github.com/inklesspen/cffi-buildtool

ngoldbaum mentioned this pull request Apr 20, 2026

Document how to use meson to build a cffi cextension #47

Open

ngoldbaum commented Apr 21, 2026

View reviewed changes

Comment thread doc/source/buildtool.rst Outdated

Comment thread doc/source/buildtool.rst

Comment thread testing/cffi1/buildtool_example2/src/csrc/square.c Outdated

ngoldbaum added 3 commits May 19, 2026 11:45

Merge branch 'main' into integrate-buildtool

7f161ff

typo fix

9293037

Use symlinks and give test folders better names

5dad224

ngoldbaum mentioned this pull request Jun 8, 2026

Initial support for abi3t.abi3 wheels. #232

Open

mattip reviewed Jun 8, 2026

View reviewed changes

rgommers reviewed Jun 8, 2026

View reviewed changes

ngoldbaum and others added 4 commits June 9, 2026 11:42

Apply suggestions from code review

e25ff27

Thanks for the review comments! I'm going to apply the review suggestions that are pending now, pull, and address the remaining comments. Co-authored-by: Matti Picus <matti.picus@gmail.com>

Remove Python interface and consolidate functionality in cffi.buildto…

dec64ee

…ol CLI

Respond to code review comments

6df3f77

stop referring to 'build scripts'

f8e2c2e

		@@ -0,0 +1,58 @@
		# Integrated from the cffi-buildtool project by Rose Davidson
		# (https://github.com/inklesspen/cffi-buildtool), MIT-licensed.

Conversation

ngoldbaum commented Apr 20, 2026

Uh oh!

inklesspen commented Apr 20, 2026

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattip commented Apr 21, 2026

Uh oh!

ngoldbaum commented Apr 21, 2026

Uh oh!

ngoldbaum commented May 19, 2026

Uh oh!

mattip left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgommers Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgommers commented Jun 8, 2026

Uh oh!

ngoldbaum commented Jun 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

rgommers Jun 8, 2026 •

edited

Loading