Skip to content

[C++][Packaging] Add native DEB, RPM, Homebrew, and Windows package artifacts - #964

Open
ColinLeeo wants to merge 27 commits into
developfrom
feat/native-package-artifacts
Open

ColinLeeo wants to merge 27 commits into
developfrom
feat/native-package-artifacts

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a native packaging pipeline for the C++ implementation and ties it to a single Maven-driven build hierarchy. The workflow builds the C++ SDK once per platform, reuses those SDK artifacts for downstream Go and Python checks, produces the native package families, and assembles one checksummed release bundle.

The pipeline deliberately stops before publishing: it produces verified artifacts plus a machine-readable manifest, and leaves JFrog uploads, mutable latest pointers, tags, signing, and final-release behavior to a separate change.

Background

TsFile already has Python wheel workflows, but there was no single CI path that produced publishable C++ package artifacts and also verified the install/consume contract for Linux, macOS, and Windows. This change makes Maven the only native build entry point and makes artifact flow explicit between build, test, and aggregation jobs.

Related: #914 (earlier native packaging work in this direction).

What this changes

Maven build hierarchy

  • -Pwith-cpp package is the C++ build entry point for native packaging.
  • -Pwith-python-only package builds the Python extension and wheel from an existing C++ SDK instead of rebuilding the C++ core.
  • Native package jobs pass -Dtsfile.version.sync.skip=true, so VersionUpdater.groovy does not rewrite cpp/CMakeLists.txt or Python metadata during package builds. Package identity therefore remains tied to the checkout-derived version generated by native_package_versions.py.
  • Maven forwards the package identity to CMake: TSFILE_ARCHIVE_VERSION, TSFILE_DEBIAN_PACKAGE_VERSION, TSFILE_RPM_PACKAGE_VERSION, and TSFILE_RPM_PACKAGE_RELEASE.
  • -Denable.cpack=ON exposes CPack generation through Maven without forcing CPack on ordinary developer builds.

CMake install and consume contract

  • Relocatable CMake package config (cpp/cmake/TsFileConfig.cmake.in) and pkg-config metadata (cpp/cmake/libtsfile.pc.in).
  • Explicit public header closure (cpp/cmake/TsFilePublicHeaders.cmake) mirroring the includes the current C++ implementation uses; narrowing to a stable public API is intentionally deferred.
  • Static SDK support: TsFileStaticDependencies.cmake(.in) re-exports bundled codec archives so TsFile::tsfile carries the full link line.
  • Installed-consumer fixtures used by CI: a CMake consumer and a pkg-config consumer, both built out-of-tree against the installed prefix.

CPack packaging

  • Components map to native package names: runtime / development / tools become tsfile, tsfile-dev (tsfile-devel on RPM), and tsfile-tools.
  • Development and tools packages pin the matching runtime build.
  • RPM/DEB filenames use native package naming instead of component-style archive names.
  • Defaults preserve local developer behavior; the packaging workflow supplies immutable development versions explicitly.

Native packaging workflow

.github/workflows/native-packages.yml:

  • workflow_dispatch only, permissions: contents: read, no secrets, and no publishing credentials.
  • prepare derives one build identity from TsFile_CPP_VERSION (which must match MAJOR.MINOR.PATCH.dev), the UTC build date, run_number, run_attempt, and the short commit SHA.
  • Layer 1 builds one C++ SDK per platform through Maven:
    • Ubuntu 22.04 x86_64, also producing DEBs.
    • AlmaLinux 9 x86_64, also producing RPMs.
    • Windows MSVC x86_64 with /MT, also producing the combined ZIP.
  • Layer 2 consumes those SDKs instead of rebuilding C++:
    • Go runs go vet, go test, and go test -race against the Ubuntu SDK.
    • Python builds and smoke-tests the Linux wheel with -Pwith-python-only against the Ubuntu SDK.
    • DEB and RPM installation tests run from clean target containers and build an external consumer.
    • Homebrew bottles the development Formula on arm64 and Intel macOS, then merges both bottle tags.
  • The final assemble job depends on all validation jobs and fails unless every required artifact family is present. It emits tsfile-native-packages-<archive_version> containing:
    • The Ubuntu, AlmaLinux, and Windows SDK archives.
    • The Ubuntu Python wheel.
    • DEBs, RPMs, merged Homebrew Formula/bottles, and the Windows ZIP.
    • A sorted SHA256SUMS and manifest.json with family, platform, size, SHA-256, target repository/path/properties, source identity, and generated versions.
  • Homebrew source coordinates now use ${{ github.repository }} with the immutable workflow commit.

Docs

  • packaging/README.md documents the Maven entry points, SDK layout, downstream consumer flow, per-family packaging commands, and the final bundle contract.

Version identity

For a checkout declaring TsFile_CPP_VERSION of 2.3.2.dev:

Family Version
logical 2.3.2.dev0+20260910.123.1.gabcdef1
DEB 2.3.2~dev0+20260910.123.1.gabcdef1-1
RPM 2.3.2 / release 0.dev0.20260910.123.1.gabcdef1.el9
archive 2.3.2-dev0.20260910.123.1.gabcdef1
Homebrew 2.3.2.dev0.20260910.123.1.gabcdef1

Not included

  • Uploading anything to JFrog/Artifactory or updating mutable release pointers.
  • Git tags, GitHub Releases, RC/final release versions, or signing.
  • sdist packaging.
  • Replacing the existing MinGW-based Windows Python wheel workflow.
  • Linux ARM64, macOS SDK archives, Windows ARM64, or Linuxbrew in this iteration.

How it was tested

  • python3 -m unittest discover -s packaging/tests -p 'test_*.py' -v passes 34 tests (5 installation tests skip unless TSFILE_RUN_INSTALL_TESTS=1 and a dependency cache are provided).
  • ruby packaging/tests/check_native_workflow.rb passes and checks the manual/read-only workflow contract, Maven entry points, artifact consumers, final aggregation, and shell syntax.
  • Maven validation confirms both -Pwith-cpp and -Pwith-python-only skip source-version rewrites when -Dtsfile.version.sync.skip=true.
  • -Pwith-cpp generate-test-sources -Denable.cpack=ON confirms all four version properties reach CMake and that the generated CPackConfig.cmake uses the expected archive version.
  • End-to-end validation comes from the manual workflow itself: clean-container package install tests, external consumers, Go/Python SDK consumers, Windows runtime inspection, and Homebrew build/test/bottle on both macOS architectures.

GitHub only exposes workflow_dispatch for workflows that already exist on the default branch, so a pre-merge run cannot be dispatched through the workflow definition added by this PR. After this workflow lands on develop, run Build native package artifacts and attach the resulting run as post-merge validation.

Reviewer notes

  • The wide public header closure is intentional for now; it is broader than a stable API and is documented as such.
  • packaging/**/*.py is outside the Python spotless scope, so those files are not currently Black-formatted.
  • The Go job stages the SDK into cpp/target/build because the current cgo bridge hardcodes that layout. Building against an installed tsfile-dev package through pkg-config remains a follow-up.

@ColinLeeo
ColinLeeo force-pushed the feat/native-package-artifacts branch from 54b42b0 to 8341c6b Compare September 17, 2026 08:22
@ColinLeeo
ColinLeeo force-pushed the feat/native-package-artifacts branch from 8341c6b to 72abe30 Compare September 17, 2026 10:52
@ColinLeeo ColinLeeo changed the title Feat/native package artifacts [C++][Packaging] Add native DEB, RPM, Homebrew, and Windows package artifacts Sep 17, 2026
- pom.xml: add the with-python-only profile plus the shared check default
- python/pom.xml: resolve tsfile.cpp.build, pass it to setup.py via
  TSFILE_CPP_BUILD, and validate the C++ artifacts at the validate phase
- python/check_cpp_build.py: fail fast with an actionable message
- wheels.yml: drop the explicit -Denable.lzma2=ON so Python wheels follow
  the single LZMA2 default owned by cpp/pom.xml
Let the build tree produced by '-Pwith-cpp package' also generate the
CPack configuration, so native packages can be produced from that same
tree without reconfiguring or rebuilding it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant