Bring over PEPs 517, 518, and 660 to the specs section

[brettcannon]

As of right now various specifications reference those PEPs but there is no complete view of what they contain (e.g. the sdist spec references PEP 518 for what pyproject.toml although technically it should also reference PEP 517 for being to actually build an sdist from a source tree). We should probably have a single specification that covers how to build sdists and wheels from a source tree which the sdist and wheel format specs can reference on how to end up with those binary artifacts.

[CAM-Gerlach]

We should probably have a single specification that covers how to build sdists and wheels from a source tree which the sdist and wheel format specs can reference on how to end up with those binary artifacts.

Would this involve unifying PEP 517 and PEP 660 in the resulting PyPA spec, with PEP 518 specifying the pyproject.toml format, or are you suggesting globbing that in there two? I'd think the former might be more sensible to keep the specs focused and at a manageable size, though I'd obviously defer to your judgement here.

[brettcannon]

Would this involve unifying PEP 517 and PEP 660 in the resulting PyPA spec, with PEP 518 specifying the pyproject.toml format, or are you suggesting globbing that in there two?

Probably all of that into a single spec. the build-system table in pyproject.toml is the unifying bit here. Basically we should have a spec for specifying the build tool for a distribution. That covers PEP 518, 517, and 660 as none of those can be specified without the previous PEP.

But I'm not doing the work and I can also see an argument to potentially do it separately.

[JDLH]

…We should probably have a single specification that covers how to… [details elided]…

I agree. Please see related discussion at Discuss: What new Reference or Explanation would usefully reduce confusion?.

My thought is that there should be two specifications, one for the behaviour of tools and one for the format and location of the pyproject.toml file:

The main need I see is for a Reference (in the Diàtaxis sense) about the interaction of tools for building source distributions and wheels with declarative project metadata. Right now, PEP 517’s Build backend interface seems to be the authority on this. PEP 517’s Terminology and goals contributes. Part of what is not clear in PEP 517 is that the scope of “build” is narrow, limited to sdists and wheels. This confused me and some others.

Also, I don’t see a PyPA Specification which really describes the pyproject.toml file. There is a PyPA Declaring build system dependencies specification, but it is a stub which refers out to PEP 518. PEP 518 defines the pyproject.toml file overall, and the [build-system] and [tool] tables therein. PEP 517 defines the build-backend key in the [build-system] table. The Pypa Declaring project metadata specification describes the metadata parts of pyproject.toml well, but not the basic format or the [build-system] and [tool] tables.

I suggest a need for a Reference specifying everything about pyproject.toml files in one place. Maybe the shortest path is to add to the existing Declaring project metadata Specification everything from PEP 518 and PEP 517 about pyproject.toml that is not already there. It could continue with the existing format. The metadata heading would become a section of the overall Reference. It would need a more general title. Alternatively, there could be a new Specification about pyproject.toml overall, and it could delegate the metadata fields specification to the existing document. In either case, the Declaring build system dependencies stub gets replaced by another document.

In the thread at Discuss, I ask if we have a consensus on the desirability of creating this/these specification(s). Do we?

[CAM-Gerlach]

In the thread at Discuss, I ask if we have a consensus on the desirability of creating this/these specification(s). Do we?

Migrating the specs in these PEPs to the PyPA specifications site is uncontroversial, formally approved and well understood to be beneficial and necessary in the long term. The major reason it hasn't happened so far is simply due no one with the relevant skills/background and motivation to do it having yet been able to find the time.

It might be possible to guide someone else without a strong background in the packaging ecosystem in what to do, since its mostly an editorial and mechanical task, and we could help specify the high-level organization. However, the person would need to have a strong grasp of reST/Sphinx syntax, roles, directives and other constructs, and be a solid technical writer, and I'd be concerned that the time it would take to guide the contributor and address issues during review would be more time-consuming for both parties involved than me just dedicating a day to it and knocking it out.

On the other hand, it would be uniquely valuable to be a user like yourself relatively new to the details of packaging but interested and motivated to understand it better, when it comes to helping revise the existing user-focused PyPUG "Tutorial" and "Explanation" type content, and in particular @cameron-simpson 's proposed high-level overview of the packaging ecosystem.

[CAM-Gerlach]

It seems there are four different approaches suggested this far:

Summary of proposed approaches (click to expand)

• @brettcannon 's proposal, combining PEP 517, 518 and 660 into one document, keeps everything build-system related in one place. However, taken as-is, this also would mean the high-level spec of the pyproject.toml file and its top-level tables would be embedded within the build system spec, when only one of those tables is build-system related, and covered by other specs.
• My original proposal, moving PEP 518 into one spec and PEP 517/660 into another, nominally separates the file format from the backend interface. However, taken as-is, this bifurcates the description of the build-system table between two specs, creating a circular dependency and makes things hard to follow.
• The first proposal above, putting everything pyproject.toml-related in one spec, keeps everything related to that file in one place. However, it requires moving existing content, mixes a bunch of different concerns and could become unmanagably long to navigate.
• The second proposal, an overall document for pyproject.toml that delegates to the existing specification for the project table, avoids most of these issue, but its not clear what it proposes to do about the build backend interface specs, which are the main thrust of PEP 517/660; including them in the same document basically reduces to @brettcannon 's proposal

After thinking about these ideas some more, re-reading the PEPs and mocking up some outlines, I propose migrating the aforementioned PEPs via the following structure. This includes a dedicated high-level document for pyproject.toml that delegates and links the standardized top-level tables to individual specifications, which makes it an easily discoverable, navigatable and maintainable one-stop shop without becoming either unmanagably long or buried in another not-directly-related spec, avoids requiring any modification to existing migrated specs, maintains consistency, parallelism and extensibility with each top-level table corresponding to one spec document.

Further, the below proposal separates the specification of the frontend-backend Python hooks and interface from the declaration of the build-system table options, since these are related but separate concerns with distinct audiences (author-specified static configuration vs. internal frontend-backend runtime interaction), make the resulting documents shorter and more focused (to wit, I tried and failed to come up with a concise but descriptive title that covered both), and roughly parallel the separation of the project table format from the underlying Core Metadata fields and semantics. However, if required, those can be combined into the same document, with the former being a top-level section ("The build-requires table") of the latter.

Proposed structure with mapping from existing PEP sections (click to expand)

• The pyproject.toml project configuration file
  • FIle format and location (PEP 518 section)
  • tool table (PEP 518 section)
  • build-system table (high-level statement of purpose from the PEP 518 section; links to Specifying a project's build system spec for the actual details)
  • project table (very brief statement of purpose, which links to the Declaring project [source] metadata spec)
• Declaring a project's build system
  • requires key (PEP 518 build-system section and PEP 517 Build Requirements section
  • build-backend key (PEP 517 Source Trees section [which only contains ≈1 sentence about source trees])
  • backend-path key (PEP 517 In-tree Build Backends section)
  • JSON schema (requires specified by the PEP 518 JSON Schema section; would be expanded to include the other keys)
• Build backend interface specification
  • Sdist hooks
    • build_sdist (mandatory) (PEP 517 section)
    • get_requires_for_build_sdist (PEP 517 section)
  • Wheel hooks
    • build_wheel (mandatory) (PEP 517 section)
    • get_requires_for_build_wheel (PEP 517 section)
    • prepare_metadata_for_build_wheel (PEP 517 section)
  • Editable hooks
    • build_editable (PEP 660 section)
    • get_requires_for_build_editable (PEP 660 section)
    • prepare_metadata_for_build_wheel (PEP 660 section)
  • Config settings (PEP 517 section)
  • Frontend responsibilities
    • Build environment (PEP 517 section)
      • Recommendations (PEP 517 section)
    • Editable installs (PEP 660 Frontend Requirements section)
• PyPA Glossary (for terminology, linked in the above documents on first use)
  • Build backend, Build backend, Integration frontend, Source tree (PEP 517 terminology section)
  • Editable installation (PEP 660 terminology section)

[JDLH]

It would be good for a pyproject.toml specification to answer the question, is a project permitted to add extra stanzas and keys not defined by the spec? Or is everything forbidden unless specifically permitted?

Source: this Discuss question: Adding extra fields in the pyproject.toml authors/maintainers list.

…it ocurred to me that adding other identifiers (like ORCID ) is something that makes sense for our use case, where all contributors to the codebase are added as authors to scientific papers describing the software.…

[CAM-Gerlach]

As mentioned on #1093 , I hope to have a draft of the above proposed structure up as a PR here by sometime this week, presuming no major objections (though I've given due consideration to such alternatives, it is relatively flexible to adapt if reviewers feel that, e.g. "Declaring a project's build system" should be a subsection of "Build backend interface specification", or that the hooks in the latter should be organized by required/optional rather than build type).

It would be good for a pyproject.toml specification to answer the question, is a project permitted to add extra stanzas and keys not defined by the spec? Or is everything forbidden unless specifically permitted?

The process of migrating the specifications described in this issue is strictly an editorial one; to that end, I've been careful to map existing atomic normative sections in the approved PEPs as directly as practicable to those of the proposed unified specifications, aside from any changes from subsequent approved PEPs (e.g. explicitly adding the project table from PEP 621/Declaring Project [Source] Metadata to the top-level pyproject.toml spec.

Any non-trivial changes to the normative content need to follow the PyPA specification update process, which depending on the scope of the change, range from a normal pull request on this repo (once the spec is migrated) for trivial corrections, typos and similar; a Discourse discussion for small but potentially content-relevant changes, or a new PEP for significant changes.

In any case, the question being asked there concerns keys in the project table, which are defined by the Declaring project [source] metadata specification already hosted here, which explicitly states that "No tools may add fields to this table which are not defined by this specification" and that the tool table must be used instead, as it is explicitly reserved in PEP 518 for such. I've replied on that thread with more details and suggested approaches directly addressing the OP's question.

Also, note that PEP includes "additionalProperties": false in in its JSON schema, which specifies that no new non-standardized top-level tables should be added to pyproject.toml either (since again, tool exists for exactly that purpose).

[pradyunsg]

Or is everything forbidden unless specifically permitted?

Pretty much. If you want to have additional information, you can add it by making the corresponding key dynamic and adding in the information via a separate tool-specific table.

[pradyunsg]

Ah, @CAM-Gerlach responded to your question on https://discuss.python.org/t/adding-extra-fields-in-the-pyproject-toml-authors-maintainers-list/16848/3 as well. I'll mark the preceding discussion as off-topic, since it's more of a question about the spec, rather than related to actually moving them over.

[CAM-Gerlach]

As an on-topic portion of my comment was hidden as well (I should have initially, collapsed the off-topic portion with <details>), I will repeat it again here:

As mentioned on #1093 , I hope to have a draft of the above proposed structure up as a PR here by sometime this week, presuming no major objections (though I've given due consideration to such alternatives, it is relatively flexible to adapt if reviewers feel that, e.g. "Declaring a project's build system" should be a subsection of "Build backend interface specification", or that the hooks in the latter should be organized by required/optional rather than build type).

Ah, @CAM-Gerlach responded to your question on discuss.python.org/t/adding-extra-fields-in-the-pyproject-toml-authors-maintainers-list/16848/3 as well.

Well, actually, I answered the Discourse thread OP's question over there :) ; I still suggest @JDLH read my collapsed reply here, as it attempts to clear up some broader confusion about the what is in and out of scope for this proposed spec migration, and what he can do if he would like to change the content of the specs themselves, as it appeared he was requesting previously on the other Discourse thread.

[JDLH]

Thank you for the replies. I intended my comment as a test case for the usefulness of the new specification, not as a proposal for a substantive change. Thus I sort of think it is on-topic.

This issue talks about writing a specification document that will explain to app developers about what goes into pyproject.toml. The Discuss thread has an app developer asking about what goes into pyproject.toml. Will the new specification proactively answer that question? Or does it omit the answer due to imperfect wording, but the original PEP has the answer? (In which case, an editorial improvement to the spec is desireable.) Or does the original PEP not answer the question? (In which case, this process has uncovered a gap in the substance, and a PyPA specification update is desirable.)

[CAM-Gerlach]

This issue talks about writing a specification document that will explain to app developers about what goes into pyproject.toml.

Well, not quite—this issue is about moving existing specification content regarding how build frontends and backends interact to build sdists and wheels, to the PyPA spec section to serve as reference material for packaging tool implementors, which is not really the same thing as a new "explanation"-oriented document aimed at regular Python developers. If you think such a document would be useful (and there certainly seems to be plenty of room for such), I encourage you to open an issue.

However, the particular question being asked was about the content of the [project] table, already covered in its own PEP and PyPA specification, rather than the build-system table defined in PEP 518, and is in fact specific to packaging tool maintainers and thus to the specification itself, as the specific feature requires a custom tool to implement it, which would then need to determine (and inform users in its own documentation) where and how to handle the new metadata in the tools table (unless it would be standardized, in which case it would no longer be a "key not defined by the spec" as in your original question).

Will the new specification proactively answer that question?

While this is a question that is already answered by an existing specification and not actually part of the three core PEPs in question, the structure I propose to implement here should greatly help the important discoverability problem you and the OP highlighted in how it was rather non-obvious to find, underscoring my rationale for it, as it includes a dedicated top-level document for the pyproject.toml specifications named as such, which in turn links to the specifications for each of the top-level tables. Therefore, those interested would be able to easily find the top-level pyproject.toml spec and then quickly navigate to the specification for the top-level table they are interested in. As such, I'm further convinced that is the right approach here, as opposed to burying the pyproject.toml spec in a unitary build spec.

[JDLH]

this issue is about moving existing specification content …to the PyPA spec section to serve as reference material

I think we agree that we are talking about reference material, what Diàtaxis calls a Specification, not an Explanation.

…regarding how build frontends and backends interact to build sdists and wheels … for packaging tool implementors …

Which entity is expected to type in values into the fields of the [project] table in pyproject.toml? A build frontend? In my — admittedly, very limited — experiments, my frontend was "python -m build". I did not see it modify the pyproject.toml file. I would have to type in name=Jimomatic into the [project] table myself. If it is expected that app developers edit pyproject.toml directly, to declare their project metadata, then surely app developers are also an audience for this reference material?

Or is it specified that build frontends shall collect project metadata from app developers, and write it in pyproject.toml on behalf of the app developer? In that case, certainly, the app developer does not need to refer to this material, only the build frontend developer. But also, the specification document should state that interaction.