feat(data-retention): granular PII redaction stages (input + block outputs) by TheodoreSpeaks · Pull Request #5272 · simstudioai/sim

TheodoreSpeaks · 2026-06-29T22:20:09Z

Summary

Add two execution-altering PII redaction stages alongside the existing log redaction: redact the workflow input before execution, and mask every block output in-flight before the next block reads it
Per-stage policy (entity types + language) for each of Logs / Workflow input / Block outputs; resolved most-specific-wins per workspace, with full back-compat for existing logs-only rules
In-flight stages fail-fast (abort the run) on a Presidio error instead of scrubbing or leaking; the logs stage keeps scrub-to-marker
Reuse the shared HTTP → Presidio path; block-output redaction runs before payload compaction so offloaded large values are still masked
Settings UI: chip-tabs across the three stages, language-first picker with the entity grid filtered to that language's recognizers, and a confirmation before removing a workspace override

Type of Change

New feature

Testing

Tested manually. Unit tests for resolver back-compat, redactObjectStrings + failure modes, and the contract schema. bun run lint, check:api-validation:strict, and check:migrations origin/staging all pass.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…tputs)

vercel · 2026-06-29T22:20:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jul 1, 2026 11:56pm

cursor · 2026-06-29T22:20:19Z

PR Summary

High Risk
Changes how workflows execute (input/output masking, streaming, resume) and central PII persistence behavior; Presidio outages abort runs when in-flight stages are on.

Overview
Adds per-stage PII policy (logs, workflow input, block outputs) with entity types and language per stage, while legacy flat rules still resolve as logs-only.

Runtime: When enabled, workflow input is masked before execution; block outputs are masked before compaction, downstream state, and agent memory; streaming can drain without forwarding raw chunks until masked. Resume/run-from-block re-masks restored blockStates, including large-value refs (hydrate → mask → re-store). In-flight stages use onFailure: 'throw'; log persist keeps scrub-to-marker and applies the logs stage from stored rules without re-checking the feature flag.

Presidio: New /analyze_batch and /anonymize_batch; the TS client batches via shared byte/count chunking and bounded concurrency (optional PII_MASK_CHUNK_CONCURRENCY).

Config/API: pii-granular-redaction gates new input/blockOutputs enablement on save; Data Retention UI uses stage tabs and language-filtered entity pickers.

Docs/Helm describe Presidio as a standalone service reached via PII_URL, not an app sidecar.

^{Reviewed by Cursor Bugbot for commit 2175408. Bugbot is set up for automated code reviews on this repo. Configure here.}

greptile-apps · 2026-06-29T22:25:29Z

Greptile Summary

This PR adds granular PII redaction for workflow execution and logs. The main changes are:

New input, block-output, and log redaction stages.
Per-stage policy resolution for org and workspace rules.
Fail-fast execution-time masking for workflow input and block outputs.
Large-value and streaming-output handling before downstream use.
Settings UI updates for stage tabs, language filtering, and override removal.

Confidence Score: 5/5

This looks safe to merge.

No blocking issues found in the changed code.
Stored PII rules are resolved without the execution-time feature flag.
Empty enabled stage policies are rejected or normalized consistently.
Restored large-value refs are masked before downstream state can use them.

Important Files Changed

Filename	Overview
apps/sim/lib/workflows/executor/execution-core.ts	Resolves stored PII policy during execution, masks workflow input, and re-masks restored block state before hydration.
apps/sim/lib/billing/retention.ts	Resolves PII redaction per stage and keeps empty entity selections disabled.
apps/sim/lib/logs/execution/pii-large-values.ts	Adds hydration, masking, and re-storage for offloaded large values.
apps/sim/executor/execution/block-executor.ts	Masks block outputs before compaction and buffers streaming output when block-output redaction is enabled.
apps/sim/lib/api/contracts/primitives.ts	Validates per-stage PII policy shape so enabled stages include selected entity types.
apps/sim/ee/data-retention/components/data-retention-settings.tsx	Updates the settings UI for per-stage PII redaction and synced save behavior.

_{Reviews (15): Last reviewed commit: "fix(data-retention): re-mask offloaded l..." | Re-trigger Greptile}

…flows

TheodoreSpeaks · 2026-06-30T00:40:25Z

@greptile review

… resume

TheodoreSpeaks · 2026-06-30T00:46:12Z

@greptile review

…nconditionally (no fail-open)

TheodoreSpeaks · 2026-06-30T00:57:36Z

@greptile review

…edaction

TheodoreSpeaks · 2026-06-30T01:09:03Z

@greptile review

…redaction # Conflicts: # apps/sim/ee/data-retention/components/data-retention-settings.tsx

cursor · 2026-06-30T19:25:51Z

          abortSignal: ctx.abortSignal,
+          // Propagate in-flight block-output redaction into child workflows so
+          // nested blocks mask outputs too (recurses: each child forwards it).
+          piiBlockOutputRedaction: ctx.piiBlockOutputRedaction,


Child workflows skip input redaction

Medium Severity

The new workflow-input PII stage runs only in executeWorkflowCore on top-level processedInput. Nested child runs are started with a direct Executor and pass childWorkflowInput unchanged. Only the block-output policy is forwarded on the context, so when the input stage is on and block outputs are off, mapped or explicit child input can execute and produce downstream state without in-flight input masking.

^{Reviewed by Cursor Bugbot for commit 8f86d77. Configure here.}

…logs (preserve redacted content)

TheodoreSpeaks · 2026-06-30T19:34:42Z

@greptile review

greptile-apps · 2026-06-30T19:39:55Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )


Refs stay unmasked When a paused run or run-from-block snapshot contains a large-value ref that was created before block-output redaction was enabled, this call only masks inline strings. Large-value refs are treated as opaque by redactObjectStrings, so the ref still points at the original offloaded bytes. The later warm-up step can materialize that raw value for downstream blocks, letting them read or send unredacted PII even though the block-output stage is enabled.

…n logs stage is on

TheodoreSpeaks · 2026-06-30T19:46:37Z

@greptile review

greptile-apps · 2026-06-30T19:49:02Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )


Refs stay raw

This restore path still only masks inline strings. When a paused run or run-from-block snapshot contains a large-value ref created before block-output redaction was enabled, redactObjectStrings leaves the ref untouched. The later warm-up can materialize that original offloaded value for downstream blocks, so the resumed workflow can read raw PII even though block-output redaction is now enabled. This path needs to hydrate, mask, and re-store restored refs before downstream state can use them.

greptile-apps · 2026-07-01T05:46:36Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )
+      }


Large refs remain raw

This restore path still leaves old offloaded block outputs unmasked. It only runs redactObjectStrings over restored blockStates, and that redactor treats large-value refs as opaque, so a paused run or run-from-block snapshot created before block-output redaction was enabled can still point at raw stored bytes. When the restored state is warmed and downstream blocks read that ref, they can receive the original PII even though the block-output stage is enabled. The restore path needs to hydrate, mask, and re-store those refs before exposing the state to execution.

… (env-tunable), remove request timeouts, sync large-value walk

…redaction

…daction flag - New pii-granular-redaction feature flag (fallback PII_GRANULAR_REDACTION), layered on pii-redaction, gating the execution-altering input + block-output stages - Route returns piiGranularRedactionEnabled and rejects enabling granular stages when off - UI shows only the Logs stage tab unless the flag is on; clamps active stage - Drop the per-search Select all toggle; add a Deselect all action to the PII section header

TheodoreSpeaks · 2026-07-01T21:13:22Z

@greptile review

Presidio now runs as its own ECS service (and, in Helm, its own Deployment + Service) reached over the network via PII_URL — not a sidecar in the app task. Update README, code comments, env docs, Dockerfiles, and the Helm chart docs to match, and note the deploy requirement that PII_URL must be reachable.

…on't lock out granular saves - Resume/run-from-block restore now hydrates → masks → re-stores large-value refs in restored blockStates (not just inline strings), so a value offloaded before the block-output stage was enabled can't warm raw PII into downstream blocks. Fails fast. - pii-large-values: add onFailure mode (throw on the execution path, scrub for logs) and redactLargeValueRefsInValue for arbitrary (non-RedactablePayload) values - Granular flag gate now rejects only NEW off→on granular enablement, so orgs that already configured granular stages can still save retention settings when the flag is off

TheodoreSpeaks · 2026-07-01T23:56:17Z

@greptile review

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2175408. Configure here.}

feat(data-retention): granular PII redaction stages (input + block ou…

f24d8cc

…tputs)

cursor Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts

vercel Bot deployed to Preview June 29, 2026 22:24 View deployment

greptile-apps Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): propagate block-output redaction into child work…

36f2a3d

…flows

vercel Bot temporarily deployed to Preview June 30, 2026 00:37 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts

Comment thread apps/sim/lib/workflows/executor/execution-core.ts

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): close block-output redaction gaps on streaming +…

bb3a84b

… resume

vercel Bot temporarily deployed to Preview June 30, 2026 00:45 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): drain+mask streamed output, resolve PII policy u…

0b81fed

…nconditionally (no fail-open)

vercel Bot temporarily deployed to Preview June 30, 2026 00:56 Inactive

test(testing): support leftJoin().where().limit() in shared db mock

2d75987

vercel Bot temporarily deployed to Preview June 30, 2026 01:05 Inactive

fix(data-retention): mask agent/Pi memory writes under block-output r…

eb6b25a

…edaction

vercel Bot temporarily deployed to Preview June 30, 2026 01:08 Inactive

Merge remote-tracking branch 'origin/staging' into feat/pii-granular-…

324b04c

…redaction # Conflicts: # apps/sim/ee/data-retention/components/data-retention-settings.tsx

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/app/api/organizations/[id]/data-retention/route.ts

vercel Bot deployed to Preview June 30, 2026 17:01 View deployment

fix(data-retention): guard partial PII stages in GET normalize

d55b557

vercel Bot temporarily deployed to Preview June 30, 2026 17:06 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

fix(data-retention): hydrate, mask, and re-store large-value refs in …

6e9587a

…logs (preserve redacted content)

vercel Bot temporarily deployed to Preview June 30, 2026 19:34 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/logs/execution/logger.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

fix(data-retention): always apply logs policy to large-value refs whe…

f0c71cc

…n logs stage is on

vercel Bot temporarily deployed to Preview June 30, 2026 19:46 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

waleedlatif1 deleted the branch staging July 1, 2026 05:43

waleedlatif1 closed this Jul 1, 2026

waleedlatif1 reopened this Jul 1, 2026

greptile-apps Bot reviewed Jul 1, 2026

View reviewed changes

TheodoreSpeaks added 3 commits July 1, 2026 13:50

perf(data-retention): drop redaction byte ceiling, parallelize chunks…

9cc035d

… (env-tunable), remove request timeouts, sync large-value walk

Merge remote-tracking branch 'origin/staging' into feat/pii-granular-…

c31633a

…redaction

vercel Bot deployed to Preview July 1, 2026 21:18 View deployment

greptile-apps Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts

cursor Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread apps/sim/app/api/organizations/[id]/data-retention/route.ts

vercel Bot deployed to Preview July 1, 2026 22:44 View deployment

vercel Bot temporarily deployed to Preview July 1, 2026 23:56 Inactive

cursor Bot reviewed Jul 1, 2026

View reviewed changes

Comment thread apps/sim/lib/guardrails/validate_pii.ts

TheodoreSpeaks merged commit 69b81a6 into staging Jul 2, 2026
17 checks passed

TheodoreSpeaks deleted the feat/pii-granular-redaction branch July 2, 2026 01:47

waleedlatif1 mentioned this pull request Jul 2, 2026

v0.7.20: perf, ui improvements, fable 5, tables row size bump, blogs #5337

Merged

Uh oh!

Conversation

TheodoreSpeaks commented Jun 29, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

cursor Bot Jun 30, 2026

Choose a reason for hiding this comment

Child workflows skip input redaction

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

TheodoreSpeaks commented Jul 1, 2026

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jul 1, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Jun 29, 2026 •

edited

Loading

cursor Bot commented Jun 29, 2026 •

edited

Loading

greptile-apps Bot commented Jun 29, 2026 •

edited

Loading