Consolidate bundle schema annotations into a single tree-structured file#5574
Merged
Conversation
Drop the checked-in annotations_openapi.yml: the schema generator now derives those annotations from .codegen/cli.json in memory on every run. Merge annotations.yml and annotations_openapi_overrides.yml into a single annotations.yml whose structure mirrors the bundle configuration tree. Docs absent from the file inherit from cli.json; entries override them. The generated jsonschema.json and jsonschema_for_docs.json are unchanged. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 231eefc
22 interesting tests: 15 SKIP, 7 KNOWN
Top 25 slowest tests (at least 2 minutes):
|
Replace the "_" entry inside "fields" with an explicit "type" key on the node, sibling to "fields". For map and sequence fields it documents each entry; for scalar fields it documents the value's type (enum values live here). Purely a file-format spelling change; the in-memory representation and the generated schema are unchanged. Co-authored-by: Isaac
Fold the near-identical inline-descriptor and type-docs serialization paths into one descriptorToMap helper and hoist the shared key order to a package var. Document RootTypeKey as the in-memory sentinel. No change to the generated annotations file or schema. Co-authored-by: Isaac
The in-memory annotations keyed each type's docs by field name plus a
magic "_" entry for the type itself, forcing "_" to dodge real field
names (config has fields named type, fields, description). Model it
instead as TypeAnnotation{Self, Fields}: Self is the type's own docs
(applied to its $defs entry; also where enum values live), Fields is the
per-field docs. The "_" sentinel and RootTypeKey are gone.
No change to the generated annotations file or schema.
Co-authored-by: Isaac
structFieldNames re-implemented the generator's readonly/internal/ exported/json-tag skip rules and then cross-checked the result against the emitted schema. The schema's property set is already authoritative, so order the names by struct declaration and keep only those the schema emitted; reflection now supplies order alone. A count check still catches a struct walk that fails to reach a property. Co-authored-by: Isaac
A config field named "type" (artifacts.*.type, variables.*.type) was told apart from the structural type-docs key only by nesting level. Prefix the two structural keys with "$" — matching the JSON-Schema convention this file documents — so they can never be confused with or collide with a field of the same name, which always appears as a bare key inside "$fields". Also tidy the saver: a takeSelf accessor mirrors takeField, and the ordering line numbers are named constants. No change to the generated schema. Co-authored-by: Isaac
It wrapped clijson.Parse with an empty-schemas check and was called once. Fold it into generateSchema and drop cli_json.go. Co-authored-by: Isaac
The type graph was built from the unpruned schema, so fields the generator deletes (jobs.GitSource.git_snapshot, resources.Pipeline.dry_run) stayed valid graph edges and kept stale PLACEHOLDER entries alive in annotations.yml even though they can never appear in the schema. Build the graph with the structural prune transforms (removeJobsFields, removePipelineFields) applied, via a shared configTypeGraph used by both the generator and TestNoDetachedAnnotations, so the file only documents fields the schema actually emits. The two stale entries are dropped. Also: surface all type-graph drift errors via errors.Join instead of keeping only the last; add a direct test for annotation.File's SetField/SetSelf; document why descriptorToMap/descriptorEmpty rely on a nil reference; and align the two "Dropping annotation" warnings. No change to the generated schema. Co-authored-by: Isaac
Reconciles #5443 (expose launch stage/release status in the schema) with the consolidated single-file annotation system on this branch: - Brought in unchanged: annotation.Descriptor's new LaunchStage / EnumLaunchStages / EnumDescriptions fields, the PreviewTag helper (preview.go), jsonschema.Extension's enumDescriptions, and assignAnnotation's launch-stage description prefixing + enumDescriptions emission (these merged cleanly onto our annotations.go). - Reapplied in our in-memory extractor: parser.go now populates the launch stage on each field and the enum launch stages/descriptions on the type's Self descriptor via SetField/SetSelf, instead of the deleted annotations_openapi.yml. The launch-stage data is inherited from cli.json at generation time and never persisted to annotations.yml. - Kept annotations_openapi.yml deleted (our system computes that layer in memory). bundle/schema/jsonschema.json and jsonschema_for_docs.json are byte-for-byte identical to origin/main after regeneration. Co-authored-by: Isaac
pietern
changed the title
pietern
requested review from
janniklasrose
and removed request for
janniklasrose
pietern
requested review from
andrewnester,
denik,
janniklasrose and
shreyas-goenka
Conflicts came from two origin/main changes landing on top of the annotation-storage overhaul: - #5611 (launch stage as the single preview source of truth) refactored the same codegen the branch rewrote. Resolved parser.go to keep the branch's in-memory annotation.File (SetSelf/SetField, no overrides file I/O) while adopting #5611's typed clijson.LaunchStage, ParseLaunchStage validation, and stageErr accumulation; dropped the now-removed Preview field and previewFromLaunchStage. annotations.go, descriptor.go, preview.go and extension.go auto-merged into the same shape. - #5467 (postgres_roles resource) added entries to the old multi-file annotations. Reproduced its manual descriptions for the PostgresRole fields in the new tree-format annotations.yml (the embedded RoleRoleSpec is flattened inline, so these can't be derived from cli.json); the RoleAttributes/RoleAuthMethod/etc. openapi docs regenerate in memory. The deleted annotations_openapi*.yml stay deleted. Verified: jsonschema.json and jsonschema_for_docs.json regenerate byte-identical to origin/main, annotations.yml is byte-idempotent, and go test ./bundle/internal/schema/... ./internal/clijson/... plus the refschema acceptance test pass. Co-authored-by: Isaac
janniklasrose
approved these changes
Jun 17, 2026
janniklasrose
added a commit
that referenced
this pull request
Jun 17, 2026
PR #5574 (the annotation consolidation this branch was stacked on) landed on main, superseding the local consolidation commit. Resolved conflicts by taking main's consolidation wholesale and re-applying the PLACEHOLDER-shadowing fix, adapted to main's restructured types: annotation.File is now map[string]TypeAnnotation{Self,Fields} and Descriptor dropped Preview and gained LaunchStage/EnumLaunchStages/EnumDescriptions. Regenerated jsonschema.json, jsonschema_for_docs.json, and annotations.yml. Co-authored-by: Isaac
pietern
added a commit
that referenced
this pull request
Jun 17, 2026
Conflict in bundle/internal/schema/annotations.yml: #5574 consolidated the annotations into a single tree-structured generated file. Resolved to main's structure and regenerated so postgres_databases is re-added in the new form. Also reordered the postgres entries in acceptance/bundle/invariant/test.toml to match main (project, branch, endpoint, catalog, synced_table, role) with postgres_databases appended after postgres_roles. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: ace5f45
648 interesting tests: 595 MISS, 36 FAIL, 8 KNOWN, 4 flaky, 3 PANIC, 2 SKIP
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
annotations_openapi.yml,annotations_openapi_overrides.yml,annotations.yml) with oneannotations.ymlstructured like the bundle config tree..codegen/cli.jsonin memory at generation time, instead of checking in ~7k lines of generated YAML. The file now holds only CLI-owned docs and overrides.$type/$fields; absent entries inherit from cli.json, anddescription: PLACEHOLDERflags undocumented fields.task generate-schemakeeps the file in lockstep with the config structure: it adds placeholders for new fields and drops entries for fields the schema no longer emits.Why
The old files duplicated cli.json data that went stale between regenerations and were keyed by Go type name, which hid where a field actually lives. One structured file gives a single place for CLI-owned docs, no duplicated upstream content, and a layout that reads like the config it documents.
Tests
bundle/schema/jsonschema.jsonandjsonschema_for_docs.jsonare byte-for-byte identical tomain— this is purely an internal representation change.This PR was written by Isaac.