FEAT: Add Conda release and publish tooling for mssql-python - #720
Conversation
Adds conda recipes for mssql-python and its mssql-python-odbc companion, a local build/test-before-live harness, and a draft OneBranch conda publish stage. Azure SDK deps resolve from the lean 'microsoft' Anaconda channel under --strict-channel-priority so conda-forge's azure-core recipe (which over-declares flask/six -> celery/boto3/botocore) does not bloat the environment. See conda-forge/azure-core-feedstock#71.
There was a problem hiding this comment.
Pull request overview
This PR adds initial conda packaging assets for mssql-python (and its mssql-python-odbc companion) so the project can be published to the Microsoft-owned microsoft Anaconda channel, alongside the existing PyPI wheel distribution.
Changes:
- Adds conda recipes for
mssql-pythonandmssql-python-odbcthat repackage existing wheels (no compilation). - Adds a local PowerShell harness to build both recipes, create a local channel, and validate import / optional live-connect.
- Adds a draft OneBranch stage to build/test/publish conda artifacts from signed release artifacts.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| conda/onebranch-publish-conda-stage.yml | Draft OneBranch stage to build from signed artifacts, gate on smoke tests, and publish to the microsoft Anaconda channel. |
| conda/mssql-python/meta.yaml | Conda recipe for repackaging the mssql-python wheel and depending on a version-locked mssql-python-odbc. |
| conda/mssql-python-odbc/meta.yaml | Conda recipe for repackaging the proprietary driver wheel into a companion conda package. |
| conda/build_and_test_local.ps1 | Local “test-before-live” harness to build, index, install, and smoke test the conda packages. |
Suppressed comments (1)
conda/onebranch-publish-conda-stage.yml:133
- Same string-vs-boolean condition issue here: quoting the template expression turns it into a string, which can cause the publish step to be skipped unexpectedly even when
publishToCondais true.
- task: PowerShell@2
displayName: 'Publish to anaconda.org/microsoft'
condition: and(succeeded(), eq('${{ parameters.publishToConda }}', true))
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Recipes now honor ARTIFACTS_PY/ARTIFACTS_ODBC to install signed wheels offline (--no-index --find-links) instead of always hitting PyPI. build_and_test_local.ps1 adds --override-channels for reproducible solves and its header no longer claims an offline/no-microsoft-channel run. Draft publish stage condition compares the boolean parameter directly (eq(param, true)) instead of a quoted string.
- Single-source versions via MSSQL_PYTHON_VERSION/MSSQL_ODBC_VERSION env (wired from the publish-stage params) so the package version and the companion pin can't drift. - Point mssql-python-odbc license_file at the actual ODBC Driver 18 EULA + VC++ license (was MIT-primary root LICENSE); remove the resolved TODO. - Drop the no-op azure-identity >=1.12.0 floor (microsoft channel ships CalVer). - Re-assert the wheel platform floor via __glibc/__osx virtual-package run constraints. - Add conda/driver_load_probe.py + run it in the gate and local harness so we prove the native ODBC driver loads, not just the Python shim. - Publish stage: require signed wheels (no PyPI fallback), publish companion-first with --skip-existing and a #706 pair guard, and document the required resources.pipelines declaration.
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changesNo lines with coverage information in this diff. 📋 Files Needing Attention📉 Files with overall lowest coverage (click to expand)mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 61.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.row.py: 77.6%
mssql_python.pybind.ddbc_bindings.cpp: 77.7%
mssql_python.pybind.connection.connection_pool.cpp: 81.8%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.py_type_cache.hpp: 91.6%🔗 Quick Links
|
Port the productionized conda pipeline from the ADO conda-publish-pipeline branch onto GitHub conda-onboarding, replacing the earlier draft prototype:
- conda recipes: meta.yaml + build.sh + bld.bat for mssql-python and mssql-python-odbc, vendored ODBC/VC++ EULA text, and .gitattributes (LF for shell scripts).
- OneBranchPipelines conda glue: scripts/build-conda-packages.{sh,ps1}, steps/conda-build-validate-step{,-posix}.yml, steps/conda-publish-step.yml, steps/conda-release-step.yml, jobs/consolidate-conda-artifacts-job.yml.
- Wire conda legs into build/release pipelines (buildConda params on win-64/osx/linux legs + ConsolidateConda stage), preserving GitHub-only signWindowsBinaries content.
- Remove superseded draft prototype: conda/onebranch-publish-conda-stage.yml and conda/build_and_test_local.ps1.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.
Suppressed comments (4)
OneBranchPipelines/scripts/build-conda-packages.sh:173
- The verify env name only includes the Python version. On macOS this script is invoked twice on the same agent (native osx-arm64, then cross-target osx-64), so the second
conda create -n verify_...can fail with an existing prefix or reuse the wrong-arch env. Include the effective subdir (native vs $CONDA_SUBDIR) in the env name to avoid collisions.
for py in $pyvers; do
envName="verify_${py//./}"
echo "=== [py $py] create verify env from local channel ==="
OneBranchPipelines/stages/build-linux-single-stage.yml:116
- This apt install runs before the later
apt-get updatein the job, so it can fail on agents with stale apt indices (common on hosted Ubuntu). Add anapt-get update(and ideally noninteractive) before installing qemu-user-static/libc6-arm64-cross.
# qemu-user-static: run aarch64 ELF binaries on the x86_64 host.
# libc6-arm64-cross: the aarch64 glibc runtime (loader + libc/libm/...)
# under /usr/aarch64-linux-gnu so qemu can resolve /lib/ld-linux-aarch64.so.1
# for the emulated aarch64 conda build/verify (QEMU_LD_PREFIX points here).
sudo apt-get install -y qemu-user-static libc6-arm64-cross
OneBranchPipelines/steps/conda-publish-step.yml:91
- Installing
anaconda-clientfrom PyPI without pinning a version can make releases non-reproducible and may unexpectedly break publishing if a new release introduces behavioral changes. Consider pinning to a vetted version (or using a constraints file) so publish behavior is stable.
$ErrorActionPreference = 'Stop'
python -m pip install --upgrade pip
python -m pip install anaconda-client
# anaconda-client installs the `anaconda` console script onto PATH.
anaconda --version
OneBranchPipelines/steps/conda-build-validate-step.yml:126
- The error message references
$links(the wheel find-links directory), but conda packages are searched under${{ parameters.outputDir }}/bld. If this trips, the message will mislead troubleshooting.
if (-not $built) { Write-Error "No conda packages were produced under $($links)"; exit 1 }
…uildAll on Windows The Windows mssql-python-odbc companion conda is now built ONCE as a Python-agnostic package in the ODBC_BuildAll stage (no python in host, wheel extracted via tar in bld.bat), mirroring the single py3-none-win_* PyPI wheel, instead of once per Python on every binding leg. The per-Python mssql-python binding legs seed that prebuilt companion into their local channel (-Package binding -DriverCondaDir) so the version-locked dependency still resolves. ConsolidateConda now also pulls the ODBC_BuildAll companion, and the #706 release/publish gates use presence-pairing (keeping strict 1:1 only for per-Python companions, c>1). macOS/Linux stay per-Python (unchanged).
P0-1: invert conda/driver_load_probe.py from a fail-OPEN denylist to a fail-CLOSED allowlist. A repackaged native ODBC driver that fails to load now FAILS the DB-less pre-publish gate instead of passing on any unrecognized exception. Only a clean connect or a connection-stage diagnostic the loaded msodbcsql driver alone can emit (ODBC branding, network provider, TLS, auth) counts as PASS. Defer 'import mssql_python' into main() so the classifier is unit-testable without the compiled extension. Adds tests/test_026 (22 no-DB tests). P0-3: gate the conda release on package METADATA, not folder names/counts. New conda/validate_conda_release.py reads each package's authoritative info/index.json (zstd) and validates real subdir == folder, allowed subdirs, the full (subdir x Python) binding matrix, exact/consistent versions, and #706 binding<->companion pairing. Catches a mislabeled subdir and the 8e7f217 dropped-win-64-variant regression the count gate missed. Rewires OneBranchPipelines/steps/conda-release-step.yml to call it. Adds tests/test_027 (11 tests incl. a real .conda round-trip).
…P0-2) macos-latest is an Intel Mac and the arm64 Python cannot execute there (no reverse Rosetta), so the previous osx-arm64 leg silently built NATIVE osx-64 packages and staged them under osx-arm64 (mislabeled). Cross-build for real: - conda/*/build.sh: when the host-env Python is not executable (non-emulated cross-build), extract the universal2 wheel into \ with unzip instead of pip -- mirrors the Windows bld.bat tar path; the arm64 slice comes from the universal2 wheel. Native + QEMU-emulated legs keep the pip install. - conda/*/meta.yaml: skip_compile_pyc on macOS so conda-build does not run the non-runnable arm64 Python for .pyc byte-compilation (Python regenerates it). - build-conda-packages.sh: section-7 verify auto-skips the runtime import when the target Python can't run on the host (osx-arm64 on Intel); the static arm64-slice audit is the stand-in. Native/QEMU legs still import for real. - build-macos-single-stage.yml: add condaTargetSubdir: osx-arm64 + continueOnError to the arm64 leg; mark osx-64 as the native/blocking leg; add a BLOCKING static arm64 slice audit (lipo/otool/file) asserting the shipped ddbc_bindings ext and macos/arm64 dylibs really contain arm64 Mach-O. Fix the misleading 'Apple Silicon, native' comments.
The vendored ODBC Driver 18 links crypto/auth libs that must receive conda's security updates instead of being frozen into the payload -- mirroring conda-forge libpq (declares openssl + krb5, vendors neither): - openssl # [not win]: the driver dlopen's libssl/libcrypto for TLS; it is not an ELF NEEDED entry so overlinking can't see it. Windows uses SChannel. - krb5 # [linux]: libmsodbcsql NEEDs libkrb5.so.3 + libgssapi_krb5.so.2 (not bundled). macOS uses Kerberos.framework, Windows uses SSPI. - vc14_runtime # [win]: msodbcsql18.dll imports VCRUNTIME140.dll but the vendored vcredist ships only msvcp140.dll; declare the serviced conda runtime.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.
Suppressed comments (2)
OneBranchPipelines/scripts/build-conda-packages.sh:184
- The verify env name is only based on the Python version (e.g. "verify_310"). On macOS this script is invoked twice on the same agent (osx-64 and osx-arm64) sharing the same Miniforge install/outputDir, so the second invocation will fail when it tries to create an env that already exists. Include the target subdir in the env name (and/or delete any existing env before creating it) so the two runs don’t collide.
for py in $pyvers; do
envName="verify_${py//./}"
echo "=== [py $py] create verify env from local channel ==="
# -c microsoft (ahead of conda-forge) so azure-core/azure-identity/msal resolve from the
# lean `microsoft` channel, NOT conda-forge whose azure-core recipe over-declares flask/six
# -> celery/boto3/botocore (~9 MB); see conda-forge/azure-core-feedstock#71.
# --strict-channel-priority keeps the freshly built local companion + binding authoritative.
"$conda" create -y -n "$envName" -c "$bld" -c microsoft -c conda-forge --strict-channel-priority --override-channels "python=$py" mssql-python
conda/validate_conda_release.py:67
- read_index_json() uses next(...) to locate the info-*.tar.zst member inside a .conda. If the archive is malformed (missing that member), this will raise StopIteration and produce a stack trace rather than a clear validation failure message. Handle the empty case and raise a ValueError with a helpful message instead.
with zipfile.ZipFile(path) as zf:
info_name = next(
n for n in zf.namelist() if n.startswith("info-") and n.endswith(".tar.zst")
)
info_blob = zf.read(info_name)
libodbcinst.so.2 has NEEDED libltdl.so.7 but no RUNPATH, so a minimal glibc Linux base throws 'OSError: libltdl.so.7: cannot open shared object file' on import. macOS already vendors libltdl.7.dylib; Linux was the inconsistent outlier. Two parts: 1) eng/scripts/patch-linux-odbc-libs.sh (new): maintainer/CI tool, run in a manylinux_2_28 container, that sources the glibc libltdl.so.7, copies it next to libodbcinst.so.2, and patchelf --set-rpath '\' so the driver resolves it from its own dir. Skips Alpine/musl by design. 2) build-odbc-all-stage.yml: the wheel content verifier now REQUIRES libltdl.so.7 in both manylinux_2_28 payloads (fail-closed). This gate stays red until a maintainer runs the patch script and commits the produced libltdl.so.7 + rpath-patched libodbcinst.so.2 under mssql_python_odbc/libs/linux/debian_ubuntu/<arch>/lib/ (cannot be produced on the Windows-only odbc build host).
Decision 2(b): Alpine is a supported, PR-tested platform, so its wheels must be
fixed too (not dropped, not documented-as-limitation). Extends the libltdl
self-contained-payload work to musl:
- eng/scripts/patch-linux-odbc-libs.sh now auto-detects libc + arch and patches
the matching distro subtrees: a glibc-built libltdl (manylinux, dnf) for
debian_ubuntu/rhel/suse, a musl-built libltdl (Alpine, apk add libtool) for
alpine. One build per (libc, arch) serves all that arch's distro subtrees. It
also drops a per-dir LIBLTDL_LGPL_LICENSE.txt notice.
- mssql_python_odbc/libs/LICENSING: document libltdl (GNU Libtool, LGPL-2.1-or-
later, dynamically linked) covering the existing macOS libltdl.7.dylib and the
new Linux libltdl.so.7 (compliance; precedent = macOS already vendors it).
- build-odbc-all-stage.yml: musllinux_1_2_{x86_64,aarch64} wheels now also REQUIRE
libltdl.so.7 (fail-closed), since the Alpine test leg's system libltdl masks
the OSError today.
- eng/scripts/audit_bundled_binaries.py (new): allowlist-driven ELF/Mach-O/PE
dependency audit (gate step 3). Every dep must be BUNDLED, BASE (OS/libc), or
DECLARED (openssl/krb5 [linux], vc14_runtime [win]); anything else fails. ELF
binaries that need a bundled sibling must carry an \ RUNPATH; macOS
absolute non-system install names (e.g. /opt/homebrew) fail regardless of
basename; --require-arch asserts Mach-O slices (the Intel-agent substitute for
the arm64 runtime import). Validated locally: Windows PASS, Linux FAIL on the
missing libltdl.so.7, macOS FAIL on libodbc.2.dylib's /opt/homebrew libltdl.
Move the sole probe into eng.conda_tools, separate typed execution and reporting, and retain direct target-file isolation. Update callers, normal-import tests and documentation without changing native classification or command outcomes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Validate installed component identities, pins, ownership and RS transport selection at both release boundaries. Extract the public wheel audit into the shared CLI with exact published hashes and explicit historical-profile reporting; reuse native build metadata and target policies. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> # Conflicts: # conda/README.md
Keep target-runtime verification on the absolute standalone file while adding a lazy source CLI route with reporting, exit, argument and working-directory controls. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Combine the eight unpublished Conda refactor, metadata validation, RS ownership/delivery, documentation and driver-probe updates into one commit. Preserve the validated candidate tree and the already-integrated main ancestry, without rewriting the existing public PR commits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve the validated release implementation and all existing history. Match the published parent CI baseline so the dependent PR contains no unrelated revert. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
One or more issues must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
conda/README.md:80
- This status text says publication/provenance tooling is proposed in a separate PR, but this change adds
eng.conda_tools.provenance,publication, the standalone release pipeline, and an explicitpublishToConda=trueproduction path. The same page documents invokingprovenanceandpromote, so the caveat is self-contradictory and can cause operators to treat the shipped release path as unavailable; update it to describe the explicit, manually gated path instead.
Run this existing build workflow only in a disposable isolated installation: shared-environment
ownership hardening is outside this change. Publication/provenance tooling is proposed
in a separate release-additions PR that follows this native/tooling foundation; see the
[release status and qualification caveats](../README.md#installation).
- Files reviewed: 20/20 changed files
- Comments generated: 3
- Review effort level: Lite
Preserve the release and shared archive tooling while integrating the merged PR781 binding-selection, metadata-spacing and archive-cardinality fixes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Five moderate findings remain, including unavailable audit inputs and uncontrolled archive or provenance error paths.
Review details
Suppressed comments (5)
.github/workflows/conda-audit.yml:89
- This PR's current checkout is version 1.15.0 and
fetch-wheelsderives the requested versions fromsetup.py, so this step will query PyPI for the 1.15.0 binding/ODBC pair (and the current RS provider when required). The PR description explicitly says that verified public 1.15.0 wheel metadata/hashes are not available yet and that the 1.14.0 pair is only an older control; consequently the PR-triggered audit will fail before building the package until those public inputs exist. Gate this job on an available qualification profile or add an explicit readiness condition instead of unconditionally fetching the unpublished current release.
python -m eng.conda_tools fetch-wheels \
--wheel-dir "$PWD/wheels" --requirements-file "$PWD/wheel-inputs.txt" \
--python-tag cp311 --conda-subdir linux-64
eng/conda_tools/archive.py:209
read_release_index()usesdecompress_index()for.condametadata, but this helper returns backendZstdErrorexceptions unchanged, unlikezstd_decompress(). A corrupt compressed info component can therefore bypass the release gate'sValueErrorhandling and produce an uncontrolled failure; normalize decoder errors toValueErrorhere.
def decompress_index(raw: bytes) -> bytes:
decode, _ = _zstd_decoder("metadata")
return decode(raw)
eng/conda_tools/provenance.py:179
json.load(response)is returned without checking its type. A syntactically valid but malformed ADO response such as a JSON array then reaches_verify_run, where.get(...)raisesAttributeError;provenance.cli()intentionally does not catch that class, so the pipeline emits a traceback instead of the single-line controlled error promised by this boundary. Validate that the decoded payload is a dict and raiseValueErrorbefore returning it.
with build_opener(_NoRedirect()).open(request, timeout=60) as response:
return json.load(response)
eng/conda_tools/release.py:430
collect_packages()reaches the archive readers, which can raisezipfile.BadZipFile,tarfile.TarError,EOFError, andKeyErrorfromarchive.READ_ERRORS, but this boundary catches onlyValueError. A corrupt or truncated release artifact therefore escapespython -m eng.conda_tools validatewith a traceback instead of the controlled non-zero gate result; catch the full archive error tuple here.
except ValueError as exc:
eng/conda_tools/release.py:475
- The installed-input pass reads each package payload again, so malformed tar/zip payloads can raise the same
archive.READ_ERRORSexceptions here. Because this second boundary also catches onlyValueError,validate --release-versionscan still terminate with an uncaught traceback after the metadata phase succeeds; use the archive error tuple for this handler as well.
except ValueError as exc:
- Files reviewed: 20/20 changed files
- Comments generated: 0 new
- Review effort level: Lite
Handle expected archive and receipt errors at both release CLI boundaries, and normalize known zstd data errors without backend fallback. Keep imported readers raising and unexpected programming errors visible. Clarify metadata-driven public RS selection without changing pins or availability gates, and document the included release tooling. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical wheel-target and payload-ownership validation issues, plus a provenance ID issue, remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (4)
.github/workflows/conda-audit.yml:45
- The path filter omits
setup_odbc.pyandmssql_python_odbc/**, even though the Conda build consumes the ODBC wheel produced bysetup_odbc.py(OneBranchPipelines/stages/build-odbc-all-stage.yml:104-115) and vendors that package tree. A PR changing the ODBC packager or payload can therefore skip this audit. Include the ODBC build script and package tree in the trigger paths.
- 'setup.py'
- 'mssql_python/__init__.py'
- 'mssql_python_odbc/__init__.py'
.github/workflows/conda-audit.yml:91
fetch-wheelsderives its versions from the current checkout, so this job requests the current 1.15.0 binding/ODBC pair from PyPI on every matching PR. The PR description says only the older public 1.14.0 pair is currently hash-qualified and that verified 1.15.0 wheel metadata/hashes are still pending; until those artifacts are available, this workflow fails at the fetch step for every relevant PR. Gate the job on a verified public profile or explicitly skip it with a visible status until the 1.15.0 inputs are available.
python -m eng.conda_tools fetch-wheels \
--wheel-dir "$PWD/wheels" --requirements-file "$PWD/wheel-inputs.txt" \
--python-tag cp311 --conda-subdir linux-64
conda/README.md:102
- This new section says publication/provenance tooling is included, but the same document still states at lines 84-85 that it is proposed in a separate release-additions PR. That contradiction can send maintainers to obsolete release instructions; update the stale status text as part of this change.
Release matrix and Python-admissibility policy live in `release.py`, recorded
Azure DevOps source checks in `provenance.py`, source-bound component inputs and
public wheel fetching in `inputs.py`, and staged publication/recovery in `publication.py`. Both release
and native auditing use `archive.py`; release validation retains its stricter
container and index rules rather than weakening them to the generic audit policy.
eng/conda_tools/provenance.py:127
resources.pipelines.<alias>.pipeline.idis the referenced pipeline definition ID; the selected execution ID is inrunId(the YAML already distinguishespipelineIDfromrunIDatOneBranchPipelines/conda-release-pipeline.yml:142-143). Treatingpipeline.idaswheel_run_idmakes this guard query the definition ID as if it were a run, so the real provenance check will fail or inspect the wrong wheel run. Read and validaterunId, and separately requirepipeline.id == _WHEEL_PIPELINE_ID; the fixture needs to model that response shape too.
# ADO records the selected run ID in the nested pipeline.id and its build number
# in version. The authoritative Build response supplies the actual definition ID.
wheel_resource = run.get("resources", {}).get("pipelines", {}).get("buildPipeline", {})
wheel_run_id = _positive_id(wheel_resource.get("pipeline", {}).get("id"))
- Files reviewed: 20/20 changed files
- Comments generated: 2
- Review effort level: Lite
Reject mismatched binding/ODBC targets and unrecorded or cross-owned core payloads before staging. Reuse the same ownership policy for installed release and native audits, preserving historical providers and target selection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved publication, provenance, and release-validation findings block safe approval.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (4)
OneBranchPipelines/steps/conda-publish-step.yml:203
BINSTAR_CONFIG_DIRis configured as the client’s isolated configuration directory, but the file is written below an extradatasubdirectory. The followingget_config()check therefore does not prove that the pinned client loaded this file and can fall back to ambient agent configuration, defeating the endpoint/SSL isolation claimed by this gate. Writeconfig.yamldirectly underBINSTAR_CONFIG_DIR(or pass the explicit file path to the pinned client).
$clientConfigDir = Join-Path $env:BINSTAR_CONFIG_DIR 'data'
New-Item -ItemType Directory -Force -Path $clientConfigDir | Out-Null
@('url: https://api.anaconda.org', 'ssl_verify: true') |
Set-Content -Path (Join-Path $clientConfigDir 'config.yaml') -Encoding ASCII
python -c "from binstar_client.utils import get_config; c=get_config(); assert c['url']=='https://api.anaconda.org'; assert c['ssl_verify'] is True; print('Anaconda endpoint configuration verified')"
eng/conda_tools/provenance.py:129
resources.pipelines.buildPipeline.pipeline.idis the referenced wheel pipeline definition ID, not the selected wheel run ID (the wheel definition is 2199 inOneBranchPipelines/conda-build-pipeline.yml:69-74and the resource carries the selected build number inversion). Treating it aswheel_run_idmakes the normal path query build/run 2199 instead of the recorded upstream run, so provenance will reject or validate the wrong build. Resolve the run by the referenced definition plus build number, or persist/pass the upstream run ID explicitly before calling the Runs API.
# ADO records the selected run ID in the nested pipeline.id and its build number
# in version. The authoritative Build response supplies the actual definition ID.
wheel_resource = run.get("resources", {}).get("pipelines", {}).get("buildPipeline", {})
wheel_run_id = _positive_id(wheel_resource.get("pipeline", {}).get("id"))
wheel_build = get_json(f"build/builds/{wheel_run_id}?api-version=7.1")
if not wheel_resource.get("version") or wheel_resource["version"] != wheel_build.get(
eng/conda_tools/release.py:149
- The ABI scan only recognizes
*_cpXY; a non-normal ABI pin such aspython_abi 3.13.* *_cp313tis ignored while thepy313build token still identifies the package as 3.13. The release matrix can therefore accept a free-threaded package in the normal CPython cell. Reject unsupportedpython_abibuild pins (or explicitly parse and reject thetform) and add a regression case.
match = _PY_ABI_RE.fullmatch(str(dep).strip())
if match:
abi_minors.add(f"{match.group(1)}.{match.group(2)}")
abi_minors.add(f"{match.group(3)}.{match.group(4)}")
eng/conda_tools/release.py:160
- The patch sampler starts with only patch 0, so a two-component lower-bound/exclusion such as
python >3.12orpython !=3.12is evaluated only at 3.12.0 and is incorrectly reported as excluding the entire minor. The existing requirement cases expect these constraints to admit a later stable 3.12 patch; include a post-zero candidate (or otherwise sample the interval after a two-component boundary).
patches = {0}
# For the supported comparisons, truth can change only at a named patch
- Files reviewed: 22/22 changed files
- Comments generated: 1
- Review effort level: Lite
Require an explicit nonblank ANACONDA_API_TOKEN for API promotion and recovery. Remove BINSTAR_API_TOKEN only from the publishing task process so CLI uploads use the same credential. Preserve token-free local checks and existing upload/recovery behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Gaurav Sharma (bewithgaurav)
left a comment
There was a problem hiding this comment.
the earlier Python compatibility issue is addressed. the added packaging protection is non-blocking hardening. approving.
There was a problem hiding this comment.
🟡 Changes recommended
Critical archive, audit, and wheel-validation gaps plus provenance and publishing issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
OneBranchPipelines/steps/conda-publish-step.yml:204
BINSTAR_CONFIG_DIRis the directory anaconda-client uses forconfig.yaml, but this writes the file under a nesteddatadirectory. Therefore the followingget_config()check does not verify the file just written, and the upload CLI can fall back to ambient/default configuration instead of the intended endpoint/SSL policy. Writeconfig.yamldirectly under$env:BINSTAR_CONFIG_DIR(or set the environment variable to the parent ofdata).
$clientConfigDir = Join-Path $env:BINSTAR_CONFIG_DIR 'data'
New-Item -ItemType Directory -Force -Path $clientConfigDir | Out-Null
@('url: https://api.anaconda.org', 'ssl_verify: true') |
Set-Content -Path (Join-Path $clientConfigDir 'config.yaml') -Encoding ASCII
eng/conda_tools/provenance.py:128
resources.pipelines.<alias>.pipeline.idis the upstream pipeline definition ID; the selected run is carried inrunId. Readingpipeline.idaswheel_run_idmakes the real release querybuild/builds/2199(the wheel definition) instead of the selected run (for example, 173176), so_verify_runrejects the artifact before publication. Read and validaterunIdseparately from the definition ID, and update the fixture to model that response shape.
wheel_resource = run.get("resources", {}).get("pipelines", {}).get("buildPipeline", {})
wheel_run_id = _positive_id(wheel_resource.get("pipeline", {}).get("id"))
wheel_build = get_json(f"build/builds/{wheel_run_id}?api-version=7.1")
tests/test_036_conda_provenance.py:538
- This fixture encodes the same incorrect contract as the implementation: it places the wheel run ID in
pipeline.id. In the ADO pipeline-resource response,pipeline.idshould be the wheel definition (2199) and the selected run should be inrunId; otherwise the test cannot catch the production lookup failure.
"buildPipeline": {"pipeline": {"id": 173176}, "version": "26250.2"}
- Files reviewed: 22/22 changed files
- Comments generated: 3
- Review effort level: Lite
Require one installed RECORD per distribution and binding metadata before a package audit can pass. Reuse the canonical wheel metadata policy for public and staged inputs, rejecting nested and aliased metadata before extraction. Preserve native diagnostics and update valid fixtures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Three critical and one moderate review findings remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
OneBranchPipelines/steps/conda-publish-step.yml:204
anaconda-clienttreatsBINSTAR_CONFIG_DIRas the directory containingconfig.yaml, but this writes the file under an extradatasubdirectory. The client therefore does not use the endpoint/SSL settings prepared here and may fall back to an ambient/default configuration, defeating the isolation this publishing boundary relies on. Writeconfig.yamldirectly under$env:BINSTAR_CONFIG_DIR(and verify that path).
$clientConfigDir = Join-Path $env:BINSTAR_CONFIG_DIR 'data'
New-Item -ItemType Directory -Force -Path $clientConfigDir | Out-Null
@('url: https://api.anaconda.org', 'ssl_verify: true') |
Set-Content -Path (Join-Path $clientConfigDir 'config.yaml') -Encoding ASCII
- Files reviewed: 24/24 changed files
- Comments generated: 3
- Review effort level: Lite
Sumit Sarabhai (sumitmsft)
left a comment
There was a problem hiding this comment.
The current release path has unresolved provenance, publication-concurrency, bounded-archive-processing, and audit-reproducibility gaps. These should be addressed before enabling production Conda publication.
Recommendation: Request changes.
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved moderate and critical findings block approval.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
.github/workflows/conda-audit.yml:66
- This PR gate installs an unpinned
pytestand a version-onlyzstandardfrom public PyPI, so the code that decides whether the audit passes is not reproducible or hash-verified. The existing build/test workflows use--require-hasheswith generated locks (for exampleeng/requirements-test-linux.txt), so this workflow should use a reviewed hash-locked test dependency set instead of resolving mutable artifacts.
python -m pip install --quiet --only-binary=:all: pytest "zstandard==0.23.0"
OneBranchPipelines/steps/conda-publish-step.yml:205
BINSTAR_CONFIG_DIRis the directory in whichbinstar_client.utils.get_config()looks forconfig.yaml, but this writes the file under$BINSTAR_CONFIG_DIR/data/config.yaml. The verification at the next line therefore does not validate the isolated file (and publication may fall back to defaults), so the endpoint/configuration isolation promised by this task is not actually established. Write directly toJoin-Path $env:BINSTAR_CONFIG_DIR 'config.yaml'.
$clientConfigDir = Join-Path $env:BINSTAR_CONFIG_DIR 'data'
New-Item -ItemType Directory -Force -Path $clientConfigDir | Out-Null
@('url: https://api.anaconda.org', 'ssl_verify: true') |
Set-Content -Path (Join-Path $clientConfigDir 'config.yaml') -Encoding ASCII
python -c "from binstar_client.utils import get_config; c=get_config(); assert c['url']=='https://api.anaconda.org'; assert c['ssl_verify'] is True; print('Anaconda endpoint configuration verified')"
eng/conda_tools/provenance.py:76
- This API boundary assumes
build.get("definition", {})is always a mapping. A valid JSON response withdefinition: nullor a scalar raisesAttributeError, whichprovenance.cli()does not classify as an expected API/input failure, so the pipeline emits a traceback instead of a controlled refusal. Validate nested response shapes before dereferencing them.
def _verify_run(build: dict, run: dict, pipeline_id: int, run_id: int) -> dict:
if build.get("id") != run_id or build.get("definition", {}).get("id") != pipeline_id:
raise ValueError(f"Build API identity mismatch for pipeline {pipeline_id}, run {run_id}.")
- Files reviewed: 24/24 changed files
- Comments generated: 1
- Review effort level: Lite
Stream and bound archive processing, reject installed and wheel metadata aliases, and hash-lock the audit dependency closure. Publication serialization remains an unresolved operational prerequisite; this change does not enable publication. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical and moderate findings remain in the release pipeline, path validation, provenance handling, release-version logic, and audit workflow.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (3)
.github/workflows/conda-audit.yml:93
- This job derives the requested wheel versions from the checkout and then requires those exact versions to already exist on PyPI. The repository's release procedure bumps
setup.pyandmssql_python/__init__.pyin the release PR before publication, so a normal release PR will hit a PyPI 404 here and fail the required check. Exclude release PRs from this PyPI-repackaging job or audit the last published version instead.
python -m eng.conda_tools fetch-wheels \
--wheel-dir "$PWD/wheels" --requirements-file "$PWD/wheel-inputs.txt" \
--python-tag cp311 --conda-subdir linux-64
eng/conda_tools/provenance.py:130
- This reads the Azure DevOps pipeline-resource definition ID as the upstream wheel run ID. In the Runs API,
resources.pipelines.<alias>.pipeline.ididentifies the pipeline definition, while the selected run is under the resource'srun.id; with the current code this can query build2199(the wheel pipeline definition) as though it were a run and make provenance validation fail or inspect the wrong record. Validate the definition ID against_WHEEL_PIPELINE_IDand extract the run ID fromrun.idbefore calling the Build API.
wheel_resource = run.get("resources", {}).get("pipelines", {}).get("buildPipeline", {})
wheel_run_id = _positive_id(wheel_resource.get("pipeline", {}).get("id"))
wheel_build = get_json(f"build/builds/{wheel_run_id}?api-version=7.1")
if not wheel_resource.get("version") or wheel_resource["version"] != wheel_build.get(
"buildNumber"
eng/conda_tools/release.py:421
--release-versionsmay be supplied without a value andmssqlPythonVersionis documented as optional, but this unconditional comparison makes that source-bound mode fail whenever no override is passed. The laterexpected_versionsconstruction also omits the verified binding version in that case, so simply removing the comparison would allow the Conda index version to drift from the verified source version. Derive the expected binding version fromversionswhenever source-bound validation is enabled, or explicitly require the override and align the template/help with that contract.
if args.release_versions is not None:
versions = inputs.parse_release_versions(args.release_versions)
if args.mssql_python_version != versions["mssql-python"]:
raise ValueError(
"Binding release version differs from verified component versions."
- Files reviewed: 27/27 changed files
- Comments generated: 2
- Review effort level: Lite
Document the administrator-owned main-only branch-control check, fail-closed protection verification, and impact on all group consumers. No ADO controls are configured by this documentation; publication trust remediation remains pending external approval. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The PR audit mishandles unreleased release-PR versions, and provenance uses the pipeline definition ID instead of the selected run ID.
Review details
Suppressed comments (2)
.github/workflows/conda-audit.yml:93
- This PR-triggered job derives the exact versions from the checked-out
setup.pyand__init__.py, then immediately requires those versions to exist on PyPI. The documented release flow updates those files in a release PR before publication, so the release PR will receive a 404 here and cannot pass the audit; add an unpublished-version/release-PR path or defer this fetch until the wheels are published.
python -m eng.conda_tools fetch-wheels \
--wheel-dir "$PWD/wheels" --requirements-file "$PWD/wheel-inputs.txt" \
--python-tag cp311 --conda-subdir linux-64
eng/conda_tools/provenance.py:127
- The Azure DevOps pipeline-resource payload exposes the selected upstream run as
runId;pipeline.idis the referenced pipeline definition. Usingpipeline.idhere makes the next Build API request target the definition ID rather than the selected wheel run, so the identity/build-number checks will reject the real release resource. ReadrunIdfor the wheel run and keep_WHEEL_PIPELINE_IDas the definition ID for the Runs API call.
wheel_run_id = _positive_id(wheel_resource.get("pipeline", {}).get("id"))
- Files reviewed: 27/27 changed files
- Comments generated: 0 new
- Review effort level: Lite
Select native sequential locking on the production stage consuming Anaconda Publishing, keeping upload, snapshot, promotion, rollback and cleanup in that stage. Validate-only runs omit the protected resource. Document native-check setup and protected-stage recovery; no custom lock client or acquired-state assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical archive ownership bypass, incomplete archive reads, and dry-run plan mismatch remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
OneBranchPipelines/conda-release-pipeline.yml:226
- This dry-run is not the exact production plan it claims to show: it prints an upload directly to the target label (
$env:CONDA_LABEL), whileconda-publish-step.ymluploads tomain_staging_<BuildId>and then runspromoteto add the public label and remove staging. A reviewer can therefore approve a plan that differs from the production mutations; print the generated staging label and the subsequent promotion/cleanup operations instead.
foreach ($p in ($pkgs | Sort-Object FullName)) {
$subdir = Split-Path -Leaf (Split-Path -Parent $p.FullName)
$sha256 = (Get-FileHash -LiteralPath $p.FullName -Algorithm SHA256).Hash.ToLowerInvariant()
Write-Host " SHA256: $sha256 $subdir/$($p.Name)"
Write-Host " WOULD RUN: anaconda upload --user $env:CONDA_CHANNEL --label $env:CONDA_LABEL --skip-existing $subdir/$($p.Name)"
eng/conda_tools/archive.py:113
_read_bytesonly enforces an upper limit and stops on the first empty read; it never verifies that a TAR member's declaredmember.sizewas actually consumed. Consequently a truncated payload (or aninfo/index.jsonwhose valid prefix ends before its declared size) can be returned as if it were complete. Pass the declared size through these TAR readers and reject short reads before yielding/parsing the member.
reader = _LimitedReader(source, limit, description)
with io.BytesIO() as result:
while data := reader.read(_CHUNK_BYTES):
result.write(data)
return result.getvalue()
- Files reviewed: 27/27 changed files
- Comments generated: 1
- Review effort level: Lite
Sumit Sarabhai (sumitmsft)
left a comment
There was a problem hiding this comment.
Reviewed the PR for correctness, security, reliability, performance, test coverage, repository conventions, and applicable architecture and design specifications. No actionable issues were identified. The implementation is consistent with repository standards and the applicable approved design requirements.
The findings raised in earlier review rounds have been verified as resolved at this head:
- The dead
CONDA_ALLOW_UNSIGNED_PATCH"assertion-only" guard and its misleading documentation have been removed. - The Miniforge installer is now verified fail-closed against
MINIFORGE_SHA256(or the release's published.sha256sidecar) before execution. - Conda release provenance and publication checks are enforced.
Recommendation: Approve
Work Item / Issue Reference
AB#47315
Summary
This pull request introduces a new GitHub Actions workflow for auditing conda package builds and adds a standalone OneBranch pipeline for validating and publishing conda releases. These changes ensure that conda packages are properly built, audited for security, and published only after passing rigorous validation, with a dry-run mode for safe verification before production releases.
New conda package auditing and release validation:
Adds
.github/workflows/conda-audit.ymlto automatically build and audit linux-64 conda packages on pull requests. This workflow fetches PyPI wheels, builds the package, and runs multiple audit tests to catch regressions in dependencies, binary layout, or provenance before merging.Standalone OneBranch pipeline for conda releases:
Introduces
OneBranchPipelines/conda-release-pipeline.yml, a decoupled pipeline that validates the conda package set for completeness, provenance, and correctness before publishing to Anaconda.org. The pipeline supports a dry-run mode (default) that prints the exact upload plan without publishing, and a production mode for real releases.The pipeline defaults to a dry-run (validate-only) mode for safety, only uploading to Anaconda.org when explicitly enabled. This helps prevent accidental releases and allows teams to review the release plan in advance.
The pipeline verifies the source and build provenance of the conda artifacts, ensuring they match the expected versions and were produced by the correct upstream pipeline, further increasing release integrity.
These additions strengthen the security, traceability, and reliability of the conda package release process.