fix(pdf): draw tracked text with the spacing in the glyph widths

CHANGELOG.md

@@ -239,13 +239,36 @@ follow semantic versioning; release dates are ISO 8601.
   The default is `DocumentLetterSpacing.NONE`, which resolves to zero at every font size, so
   a document that never asks for tracking renders exactly as it did.
 
-  **PDF honours it natively.** The advance comes from the PDF `Tc` operator, not from spaces
-  pushed into the string, so a spaced-caps headline still reads as `JANE DOE` to search,
-  copy/paste, text extraction and ATS parsers — one glyph per character, the original text.
-  Tracked runs also state their own text via `ActualText`, because an extractor decides
-  where words are by how far apart glyphs sit and tracking is the act of moving them apart;
-  without that statement a widely tracked line comes back as `J A N E  D O E` from a file
-  that is otherwise perfectly correct.
+  **PDF honours it natively.** The advance is drawn by the PDF itself — in the glyph widths
+  of the font, or with the `Tc` operator — not from spaces pushed into the string, so a
+  spaced-caps headline still reads as `JANE DOE` to search, copy/paste, text extraction and
+  ATS parsers — one glyph per character, the original text.
+
+  Placing the letters is only half of it. An extractor decides where the words are from the
+  gaps between glyph boxes, and a box is as wide as the font says its glyph is: drawn with
+  `Tc` alone, every tracked letter sits in the right place and still stands apart from the
+  next by the whole tracking, and pdf.js (past about a tenth of the font size) and
+  pdfplumber (past three points) read a widely tracked line as `J A N E  D O E`. So a
+  positively tracked run is drawn with a second font resource over the same embedded font
+  program — its FontFile2, ToUnicode, CIDToGIDMap and descriptor shared by reference — whose
+  widths include the tracking. The glyphs land where `Tc` puts them, each box reaches the
+  next, and no font program is duplicated or modified. The widths are whole thousandths of
+  an em, because PDFium truncates fractional CID widths; the rest of the tracking, at most
+  half a thousandth of the font size, stays in `Tc`. One resource serves each face and
+  tracking per em for the whole document, on every page.
+
+  Tracked runs still state their own text via `ActualText`, for the readers that honour it.
+  Negative tracking, Standard 14 faces, text drawn in visual order (a right-to-left run, every
+  line of a right-to-left table cell, a highlight chip whose text needs bidi) and text a
+  face's GSUB substitutions would rewrite keep drawing with `Tc` alone; a table cell decides
+  once for all its lines, so a single line the face cannot serve keeps the whole cell on `Tc`.
+  The raised widths differ from the font program's own advances on purpose, which ISO 32000
+  allows and PDF/A and PDF/UA do not; the backend has no output mode that claims either. A
+  custom paragraph or table handler draws tracked text the same way through
+  `PdfRenderEnvironment.letterSpacedFont(font, fontSize, letterSpacing, text)`, which ships
+  `@Beta` with its result record `LetterSpacedFont`: the glyph positions and text layer it
+  produces are settled, while the shape of the call — a nullable result, a face bound to one
+  size — may still move in a minor release.
 
   Measurement and drawing use one rule, measured off PDFBox rather than assumed: one spacing
   unit per Unicode **code point** of the string actually drawn, the trailing unit included.
@@ -1832,6 +1855,32 @@ follow semantic versioning; release dates are ISO 8601.
 
 ### Tests
 
+- **Letter spacing carried in widths is held to the picture `Tc` draws.**
+  `PdfLetterSpacedFontTest` compares every glyph origin of a tracked run with the same run
+  drawn by PDFBox with `Tc` from the same point — at 0.18em, at 0.02em, and at a points
+  tracking across three sizes, to 0.01pt, the trailing advance included — and requires the
+  gap a reader measures between two tracked letters to be no more than 0.005pt, in a
+  paragraph, a table cell and a highlight chip alike. It reads the text layer back from
+  each; checks that the letter-spaced resource draws with the base font's own program,
+  ToUnicode and CIDToGIDMap with every width raised by the tracking and no `Tc` carrying
+  it; opens a password-protected document and finds the resource still there; and holds
+  one resource per face and tracking per em across styles, sizes, pages and the sections of
+  one document, with a face handed back standing for its base instead of stacking a second
+  tracking on it. `PdfLetterSpacedFontFallbackTest` pins each run that keeps `Tc`: negative
+  tracking, a Standard 14 face, a right-to-left run, a highlight chip whose text needs
+  bidi, tracking below half a thousandth of an em, and Devanagari that Poppins'
+  substitutions rewrite — while the same face still carries a Latin heading's spacing in
+  its widths. A table cell decides once from the text it draws, so one line the face cannot
+  serve keeps the whole cell on `Tc`, and a line drawn in visual order goes back to `Tc`
+  inside a cell whose other lines keep the widths. Verified by mutation: sending every run
+  back to `Tc`, widening by one unit too many, skipping the substitution check, excluding
+  every face that keeps substitutions, handing a table cell no text, not unwrapping a face
+  handed back, and sending a highlight chip or a table cell back to `Tc` or leaving the
+  full `Tc` on top of its widths each turn the tests that describe them red.
+  `LetterSpacingAcrossBackendsTest` and `TrackingFixedLayoutParityTest` read the tracking a
+  PDF declares as the width raise over its embedded program plus the `Tc` in force, and
+  turn red when the widths are one unit too wide or the residual `Tc` is dropped.
+
 - **The schedule fixtures no longer carry a real venue's staff.** The weekly-schedule
   test fixture and the example data factory were written from a real bar's rota and kept
   its people's names, its venue name, and two third parties named in the day notes.

docs/api-stability.md

@@ -57,17 +57,22 @@ matrix.
 > Geometry identity with the PDF backend is a design invariant and will not
 > change; the API shape around it may still move in a minor release.
 >
-> Four members of the otherwise-Stable **PDF backend** also carry `@Beta`. The
+> Six members of the otherwise-Stable **PDF backend** also carry `@Beta`. The
 > package is not Experimental — these are:
 > `PdfFixedLayoutBackend.renderSections` / `writeSections`, the low-level seam
 > that concatenates several sections into one document, where
 > `MultiSectionDocument` via `GraphCompose.documents()` is the settled entry
-> point most callers want instead; and
+> point most callers want instead;
 > `PdfFixedLayoutBackend.Builder.deterministic` in both overloads, which pins
 > `CreationDate` / `ModDate` and derives the `/ID` from metadata so a document
 > renders byte-identically across runs. Determinism is off by default, and what
 > reproducible builds depend on is the *behaviour* — it is the shape of the
-> opt-in that may still move.
+> opt-in that may still move; and
+> `PdfRenderEnvironment.letterSpacedFont` with its result record
+> `PdfRenderEnvironment.LetterSpacedFont`, the seam a render handler uses to
+> draw tracked text with the spacing in the glyph widths rather than in `Tc`
+> gaps. The glyph positions and text layer it produces are settled; the shape of
+> the call — a nullable result, a face bound to one size — may still move.
 
 ### What each tier promises
 

knowledge/api/backends.json

@@ -22,10 +22,10 @@
     "graph-compose-testing:sources"
   ],
   "counts": {
-    "types": 69,
-    "methods": 371,
+    "types": 70,
+    "methods": 375,
     "constants": 18,
-    "generated": 189
+    "generated": 192
   },
   "packages": [
     {
@@ -1033,6 +1033,33 @@
           ],
           "artifact": "graph-compose-render-pdf",
           "members": [
+            {
+              "kind": "method",
+              "name": "letterSpacedFont",
+              "static": false,
+              "origin": "source",
+              "typeParameters": null,
+              "returns": "PdfRenderEnvironment.LetterSpacedFont",
+              "params": [
+                {
+                  "type": "PDFont",
+                  "name": "font"
+                },
+                {
+                  "type": "double",
+                  "name": "fontSize"
+                },
+                {
+                  "type": "double",
+                  "name": "letterSpacing"
+                },
+                {
+                  "type": "String",
+                  "name": "text"
+                }
+              ],
+              "stability": "beta"
+            },
             {
               "kind": "method",
               "name": "document",
@@ -1149,6 +1176,57 @@
               ]
             }
           ]
+        },
+        {
+          "name": "PdfRenderEnvironment.LetterSpacedFont",
+          "binaryName": "com.demcha.compose.document.backend.fixed.pdf.PdfRenderEnvironment$LetterSpacedFont",
+          "kind": "record",
+          "modifiers": [
+            "final"
+          ],
+          "artifact": "graph-compose-render-pdf",
+          "stability": "beta",
+          "members": [
+            {
+              "kind": "constructor",
+              "name": "LetterSpacedFont",
+              "static": false,
+              "origin": "generated",
+              "typeParameters": null,
+              "returns": null,
+              "params": [
+                {
+                  "type": "PDFont",
+                  "name": null
+                },
+                {
+                  "type": "float",
+                  "name": null
+                }
+              ],
+              "stability": "beta"
+            },
+            {
+              "kind": "method",
+              "name": "font",
+              "static": false,
+              "origin": "generated",
+              "typeParameters": null,
+              "returns": "PDFont",
+              "params": [],
+              "stability": "beta"
+            },
+            {
+              "kind": "method",
+              "name": "characterSpacing",
+              "static": false,
+              "origin": "generated",
+              "typeParameters": null,
+              "returns": "float",
+              "params": [],
+              "stability": "beta"
+            }
+          ]
         }
       ]
     },

knowledge/api/backends.md

@@ -28,7 +28,7 @@ note: "Generated from the pinned artifact's class files. Authoritative closed se
 
 **GraphCompose version:** 2.4.0-SNAPSHOT
 
-Types: 69 · methods: 371 · constants: 18 · compiler-generated members: 189
+Types: 70 · methods: 375 · constants: 18 · compiler-generated members: 192
 
 ## com.demcha.compose.document.backend.fixed
 
@@ -115,6 +115,7 @@ Types: 69 · methods: 371 · constants: 18 · compiler-generated members: 189
 - `PdfHeaderFooterOptions toPdf(DocumentHeaderFooter entry)`
 
 ### PdfRenderEnvironment (class)
+- `PdfRenderEnvironment.LetterSpacedFont letterSpacedFont(PDFont font, double fontSize, double letterSpacing, String text)   [beta]`
 - `PDDocument document()`
 - `FontLibrary fonts()`
 - `PDPageContentStream pageSurface(int pageIndex)`
@@ -125,6 +126,11 @@ Types: 69 · methods: 371 · constants: 18 · compiler-generated members: 189
 - `void markReorderedText()`
 - `void registerAnchor(PlacedFragment fragment, String anchor)`
 
+### PdfRenderEnvironment.LetterSpacedFont (record)   [beta]
+- `new LetterSpacedFont(PDFont, float)   [beta]`
+- `PDFont font()   [beta]`
+- `float characterSpacing()   [beta]`
+
 ## com.demcha.compose.document.backend.fixed.pdf.handlers
 
 ### PdfAnchorMarkerRenderHandler (class)

qa/src/test/java/com/demcha/compose/document/api/LetterSpacingAcrossBackendsTest.java

@@ -28,6 +28,7 @@
 import java.util.regex.Pattern;
 
 import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.within;
 
 /**
  * One authored style, all three backends.
@@ -40,8 +41,10 @@
  *
  * <p>The three units are genuinely different numbers for the same distance:
  * 2.4pt is {@code spc="240"} in DrawingML's hundredths, {@code w:val="48"} in
- * Word's twentieths, and a {@code Tc} of 2.4 in the PDF's points. A test that
- * asserted one number across all three would be asserting a bug.</p>
+ * Word's twentieths, and 2.4 points of advance per glyph in the PDF, which states
+ * it in the widths of the font it draws with (120 thousandths of the 20pt em) and
+ * in {@code Tc} for any remainder. A test that asserted one number across all
+ * three would be asserting a bug.</p>
  */
 class LetterSpacingAcrossBackendsTest {
 
@@ -84,12 +87,13 @@ void docxReceivesItInTwentiethsOfAPoint() throws Exception {
     }
 
     @Test
-    void pdfReceivesItAsPointsOfCharacterSpacing() throws Exception {
+    void pdfReceivesItAsPointsOfAdvancePerGlyph() throws Exception {
         byte[] pdf = render(session -> session.render(new PdfFixedLayoutBackend()));
 
-        // Tc is written in points, so the operator carries the resolved value
-        // itself rather than a converted one.
-        assertThat(contentStream(pdf)).containsPattern("2\\.4\\d*\\s+Tc");
+        // Read off the file, not off the backend: what the glyph widths carry
+        // beyond the embedded program's own, plus Tc, is the distance every glyph
+        // moves, and it is the resolved value itself.
+        assertThat(PdfDeclaredTracking.ofFirstRun(pdf)).isCloseTo(EXPECTED_POINTS, within(1e-6));
         assertThat(pdfText(pdf)).isEqualTo(NAME);
     }
 
@@ -192,11 +196,4 @@ private static String pdfText(byte[] pdf) throws Exception {
             return new PDFTextStripper().getText(document).trim();
         }
     }
-
-    private static String contentStream(byte[] pdf) throws Exception {
-        try (PDDocument document = Loader.loadPDF(pdf)) {
-            return new String(document.getPage(0).getContents().readAllBytes(),
-                    java.nio.charset.StandardCharsets.ISO_8859_1);
-        }
-    }
 }

qa/src/test/java/com/demcha/compose/document/api/PdfDeclaredTracking.java

@@ -0,0 +1,110 @@
+package com.demcha.compose.document.api;
+
+import org.apache.fontbox.ttf.TrueTypeFont;
+import org.apache.pdfbox.Loader;
+import org.apache.pdfbox.contentstream.operator.Operator;
+import org.apache.pdfbox.cos.COSArray;
+import org.apache.pdfbox.cos.COSBase;
+import org.apache.pdfbox.cos.COSName;
+import org.apache.pdfbox.cos.COSNumber;
+import org.apache.pdfbox.cos.COSString;
+import org.apache.pdfbox.pdfparser.PDFStreamParser;
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.apache.pdfbox.pdmodel.PDPage;
+import org.apache.pdfbox.pdmodel.font.PDCIDFontType2;
+import org.apache.pdfbox.pdmodel.font.PDFont;
+import org.apache.pdfbox.pdmodel.font.PDType0Font;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.List;
+
+/**
+ * The letter spacing a PDF declares for the first text on its first page, in points per glyph.
+ *
+ * <p>A PDF has two places to state tracking, and the PDF backend uses both: the widths of the font
+ * resource a run is drawn with, raised above the widths its embedded program gives the same
+ * glyphs, and the {@code Tc} operator for what whole thousandths of an em cannot state. A reader's
+ * pen moves by their sum beyond the glyph's own width, so the sum is what the file declares. It is
+ * read off the file — the shown glyph's width against its embedded program, plus the {@code Tc} in
+ * force — so it holds whichever way the backend divides the distance between the two.</p>
+ */
+final class PdfDeclaredTracking {
+
+    private PdfDeclaredTracking() {
+    }
+
+    static double ofFirstRun(byte[] pdf) throws IOException {
+        try (PDDocument document = Loader.loadPDF(pdf)) {
+            PDPage page = document.getPage(0);
+            PDFStreamParser parser = new PDFStreamParser(page);
+            List<COSBase> operands = new ArrayList<>();
+            Deque<TextState> saved = new ArrayDeque<>();
+            TextState state = new TextState(null, 0f, 0f);
+            Object token;
+            while ((token = parser.parseNextToken()) != null) {
+                if (token instanceof COSBase operand) {
+                    operands.add(operand);
+                    continue;
+                }
+                switch (((Operator) token).getName()) {
+                    case "q" -> saved.push(state);
+                    case "Q" -> state = saved.pop();
+                    case "Tf" -> state = new TextState((COSName) operands.get(0),
+                            ((COSNumber) operands.get(1)).floatValue(), state.characterSpacing());
+                    case "Tc" -> state = new TextState(state.font(), state.size(),
+                            ((COSNumber) operands.get(0)).floatValue());
+                    case "Tj", "TJ" -> {
+                        byte[] shown = firstString(operands);
+                        if (shown.length > 0) {
+                            PDFont font = page.getResources().getFont(state.font());
+                            int code = font.readCode(new ByteArrayInputStream(shown));
+                            return widthRaise(font, code) * (double) state.size() / 1000.0 + state.characterSpacing();
+                        }
+                    }
+                    default -> {
+                    }
+                }
+                operands.clear();
+            }
+        }
+        throw new AssertionError("the first page shows no text");
+    }
+
+    /**
+     * How many thousandths of an em the font resource's width for {@code code} exceeds the width a
+     * PDF writer states for the same glyph of the embedded program, which is its advance rounded to
+     * a whole thousandth of an em.
+     */
+    private static long widthRaise(PDFont font, int code) throws IOException {
+        if (!(font instanceof PDType0Font type0) || !(type0.getDescendantFont() instanceof PDCIDFontType2 cid)) {
+            return 0;
+        }
+        TrueTypeFont program = cid.getTrueTypeFont();
+        int programWidth = Math.round(program.getAdvanceWidth(cid.codeToGID(code))
+                * (1000f / program.getUnitsPerEm()));
+        return Math.round(font.getWidth(code)) - programWidth;
+    }
+
+    private static byte[] firstString(List<COSBase> operands) {
+        for (COSBase operand : operands) {
+            if (operand instanceof COSString string) {
+                return string.getBytes();
+            }
+            if (operand instanceof COSArray array) {
+                for (COSBase element : array) {
+                    if (element instanceof COSString string) {
+                        return string.getBytes();
+                    }
+                }
+            }
+        }
+        return new byte[0];
+    }
+
+    private record TextState(COSName font, float size, float characterSpacing) {
+    }
+}