Auto cast values on export part and partition just like insert select by arthurpassos · Pull Request #1779 · Altinity/ClickHouse

arthurpassos · 2026-05-11T19:40:21Z

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Applies to part and partition export the same auto cast ClickHouse INSERT ... SELECT does.

Current rules in this PR:

Schema column count must match
Check is done by column position, not by name.
Lossy casts are allowed if export_merge_tree_part_allow_lossy_cast=1

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-05-11T19:41:33Z

Workflow [PR], commit [5161c90]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 99002b9740

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-11T19:55:41Z


+        /// Align the pipeline header with the destination's non-materialized sample block,
+        /// using the same `makeConvertingActions(Position)` call INSERT SELECT performs.
+        addExportConvertingActions(plan_for_part, *destination_storage, local_context);


Convert partition values before creating the import sink

When the destination is partitioned object storage and the partition column type changes in a castable way (for example source year String, destination year UInt32, both PARTITION BY year), this new row conversion happens only after destination_storage->import has already computed the S3/Hive partition path from block_with_partition_values using the source part's minmax block. That means a schema pair now accepted by the synchronous validation can either fail inside partition-key evaluation with the destination actions seeing source-typed inputs, or write files under a path based on the uncast source value while the exported rows were cast, so the export is not equivalent to INSERT SELECT.

Useful? React with 👍 / 👎.

…patibility

arthurpassos · 2026-05-22T12:33:33Z

Hm... the approach of following the same INSERT ... SELECT rules for casting is somewhat problematic for partition exports because the partition column results might be different, and the export operation does not repartition.

…patibility

arthurpassos · 2026-06-16T10:34:51Z

@codex Review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5693b6068

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-16T10:50:44Z


+        /// Align the pipeline header with the destination's non-materialized sample block,
+        /// using the same `makeConvertingActions(Position)` call INSERT SELECT performs.
+        addExportConvertingActions(plan_for_part, *destination_storage, local_context);


Convert exported partition values as well

When the partition column itself needs a cast (for example a source UInt16 partition key exported to an Iceberg table whose partition column is String, which canBeSafelyCast accepts without opt-in), this step only converts the row stream. StorageObjectStorage::import still receives the source-typed block_with_partition_values, and the Iceberg commit paths still use the source part's partition.value, so the data files are written with destination types while the object path/manifest partition value remains source-typed; for Iceberg this can fail at commit or record an incorrectly typed partition. Please apply the same conversion to the partition values used for import/commit whenever partition-key columns are cast.

Useful? React with 👍 / 👎.

It is expected, at least for now.

arthurpassos · 2026-06-17T12:28:02Z

The commit 308159a aims to address the following: I was previously not casting the partition key and using the source partition key since the partition expressions were equal and the data was supposed to belong to the same partition.

On the other hand, even if the partition exp is the same, if we r dealing with different data types, chances are the partition key will be different depending on the transform used. For instance:

(id Int32, year UInt16) partition by year
x
(id Int32, year String) partition by year

or

(id Int32, year UInt16) partition by icebergBucket(5, year)
x
(id Int32, year String) partition by icebergBucket(5, year) -- the result of icebergBucket is different

arthur :) SELECT icebergBucket(5, '50');

SELECT icebergBucket(5, '50')

Query id: 828c5a2b-cbd0-4dc8-a36c-81a19f9ad4b1

   ┌─icebergBucket(5, '50')─┐
1. │                      2 │
   └────────────────────────┘

1 row in set. Elapsed: 0.006 sec. 

arthur :) SELECT icebergBucket(5, 50);

SELECT icebergBucket(5, 50)

Query id: 48ab602b-5cdf-4e5e-b65b-5aec174c1e0d

   ┌─icebergBucket(5, 50)─┐
1. │                    3 │
   └──────────────────────┘

1 row in set. Elapsed: 0.006 sec. 

arthur :)

mkmkme

LGTM!

mkmkme · 2026-06-17T15:38:57Z

+
+
+def test_export_part_with_castable_narrowing_values_fit(cluster):
+    """A lossy narrowing (id Int64 -> Int32) succeeds once the user opts in via


Perhaps a test that fails EXPORT PART when the setting is not set and there is a lossy conversion would be nice

We do have those for export partition. You mean one for plain export part as well?

At that moment I hadn't yet seen the tests for export partition. But yeah since we're mostly mirroring the tests between export partition/export partition, would be great to have one for export part as well

mkmkme · 2026-06-17T15:41:38Z

+    )
+
+
+def test_export_partition_lossy_cast_rejected_without_optin(cluster):


without_**optin** :)

DimensionWieldr · 2026-06-19T16:08:48Z

AI Triage, edited for conciseness:

CI Failures Analysis

Related to this PR (regression test updates needed)

/s3/minio/export tests/export part/sanity/mismatched columns — expects exitcode 122, got 20 (NUMBER_OF_COLUMNS_DOESNT_MATCH) after auto-cast behavior change
/s3/minio/export tests/export partition/sanity/mismatched columns — same
/iceberg/export partition/.../source-only schema drift is rejected at EXPORT — expects INCOMPATIBLE_COLUMNS (122), got NUMBER_OF_COLUMNS_DOESNT_MATCH (20)
/iceberg/.../system tables partition sorting keys — stale regression pin (5b1ccf771); needs 0e1dbef8b+ for PR 1874 expectations
All upstream checks (Stateless, Integration, Stress, Fuzzer) passed.

Infrastructure Issues

Grype scan: CVE-2026-45447 in Ubuntu 22.04 base image — fixed by Antalya-26.3: Upgrade libssl3 in Ubuntu server Docker image #1925

cast like insert select

99002b9

chatgpt-codex-connector Bot reviewed May 11, 2026

View reviewed changes

arthurpassos added antalya port-antalya PRs to be ported to all new Antalya releases antalya-26.3 labels May 11, 2026

svb-alt added the antalya-26.3.10.20001 label May 21, 2026

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-cast-com…

2fffd64

…patibility

svb-alt removed the antalya-26.3.10.20001 label May 22, 2026

svb-alt added the roadmap Key features and improvements for Antalya project label Jun 7, 2026

arthurpassos added 9 commits June 8, 2026 08:58

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-cast-com…

5271dc1

…patibility

tmp

305a318

fix build and tests

63d8f7e

yet another impl

267dd33

fix tests and background setting

2534320

fix tests

8f869ac

Merge branch 'antalya-26.3' into feature/antalya-26.3/export-cast-com…

7cd3ee9

…patibility

some more test fixes

a5c01b8

add docs

c5693b6

chatgpt-codex-connector Bot reviewed Jun 16, 2026

View reviewed changes

arthurpassos added 2 commits June 16, 2026 09:59

improve docs with a warning

a289a6a

hmm... maybe incomplete, gotta continue it tomorrow

308159a

add tests covering corner tricky case

5161c90

mkmkme approved these changes Jun 17, 2026

View reviewed changes



		def test_export_part_with_castable_narrowing_values_fit(cluster):
		"""A lossy narrowing (id Int64 -> Int32) succeeds once the user opts in via

		)


		def test_export_partition_lossy_cast_rejected_without_optin(cluster):

Conversation

arthurpassos commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 11, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented May 22, 2026

Uh oh!

arthurpassos commented Jun 16, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos commented Jun 17, 2026

Uh oh!

mkmkme left a comment

Choose a reason for hiding this comment

Uh oh!

mkmkme Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

mkmkme Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

mkmkme Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

DimensionWieldr commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI Failures Analysis

Related to this PR (regression test updates needed)

Infrastructure Issues

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

arthurpassos commented May 11, 2026 •

edited

Loading

github-actions Bot commented May 11, 2026 •

edited

Loading

DimensionWieldr commented Jun 19, 2026 •

edited

Loading