Skip to content

Stream after-hook processing in WellTransferer#273

Merged
jirhiker merged 2 commits into
transferfrom
codex/improve-execution-speed-of-welltransferer
Dec 5, 2025
Merged

Stream after-hook processing in WellTransferer#273
jirhiker merged 2 commits into
transferfrom
codex/improve-execution-speed-of-welltransferer

Conversation

@jirhiker

@jirhiker jirhiker commented Dec 5, 2025

Copy link
Copy Markdown
Member

Summary

  • process after-hook wells in streamed batches using yield_per instead of materializing all wells
  • reuse a shared chunk handler to reduce repeated setup and logging overhead

Testing

  • not run (not requested)

Codex Task

@jirhiker jirhiker merged commit 8d560fe into transfer Dec 5, 2025
@jirhiker jirhiker deleted the codex/improve-execution-speed-of-welltransferer branch December 5, 2025 05:51

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +657 to +660
for well in query.yield_per(chunk_size):
chunk.append(well)
if len(chunk) == chunk_size:
processed = _process_chunk(processed // chunk_size, chunk)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid committing while streaming yield_per results

The new for well in query.yield_per(chunk_size) loop keeps the SELECT cursor open while _process_chunk commits the session. On PostgreSQL and other backends, committing a transaction closes the streaming cursor, so once more than chunk_size wells exist the next fetch will raise or silently stop and only the first batch will be processed. The previous list-based approach materialized all wells before each commit and did not hit this cursor-closing behavior. Consider deferring the commit until after iteration or reading wells in a separate session/connection that is not committed mid-stream.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant