Add V3 streaming event ingestion pipeline - #2370
Conversation
1a60290 to
b3ce174
Compare
# Conflicts: # src/Exceptionless.Core/Configuration/AppOptions.cs # src/Exceptionless.Web/Api/Handlers/EventHandler.cs
|
Follow-up feedback and thermo-nuclear audit complete. Feedback inventory and classification:
Thermo-nuclear full-diff result:
Verification:
No concrete external blocker remains. PR draft/admin state was intentionally left unchanged. |
…stion # Conflicts: # src/Exceptionless.Core/Models/Organization.cs # src/Exceptionless.Web/Program.cs # tests/Exceptionless.Tests/Api/OpenApiSnapshotTests.cs
|
| await Task.WhenAll(organizationProcessedTask, projectProcessedTask); | ||
| var organizationProcessed = await organizationProcessedTask; | ||
| var projectProcessed = await projectProcessedTask; | ||
| bool isOrganizationProcessed = organizationProcessed.HasValue && organizationProcessed.Value; |
There was a problem hiding this comment.
[P1] Include the durable applied-bucket marker before reusing this bucket
These booleans consult only Redis. If SavePendingOrganizationUsageAsync saves LastAppliedUsageBucketUtc but its following processed-key write fails (an explicitly supported retry path), a late settlement can arrive here, see false, and be added back to the closed bucket. The saver retry then sees LastAppliedUsageBucketUtc >= pendingBucketUtc and deletes that counter without applying it; the project path can diverge the same way. In this locked slow path, also consult the persisted organization/project LastAppliedUsageBucketUtc values (or make the handoff atomic) before choosing bucket.Key, and add a regression test for marker-write failure followed by a late settlement.
| @@ -837,6 +844,232 @@ public async Task GetUsageAsync_PendingDeleted_IncludesInCurrentUsage() | |||
| Assert.Equal(5, projectUsageResponse.CurrentHourUsage.Deleted); | |||
| } | |||
|
|
|||
| [Fact] | |||
| public async Task SavePendingUsageAsync_OrganizationCleanupFailsMidBucket_RetriesFailedAndRemainingOrganizations() | |||
There was a problem hiding this comment.
These V3 saver retry/idempotency cases and their DispatchProxy fixture push UsageServiceTests.cs from 872 to 1,201 lines even though this class is already partial and the quota cases were previously split by concern. Please extract this new block into a focused partial test file (for example, UsageServicePersistenceTests.cs) so the original fixture stays scannable and below the review skill’s 1k threshold.
| Task TrackDiscardedAsync(string organizationId, string projectId, int eventCount); | ||
| } | ||
|
|
||
| public sealed class IngestionQuotaService(UsageService usageService) : IIngestionQuotaService |
There was a problem hiding this comment.
This is currently an identity wrapper while the V3 reservation/settlement implementation and its Redis/idempotency dependencies push UsageService.cs from 668 to 1,009 lines. Please move ownership of V3 reserve/commit/release behind this service (or an equivalent focused collaborator) so UsageService retains the legacy usage aggregation boundary and this abstraction earns its indirection. That should also bring the existing file back below the review skill’s 1k threshold without a cosmetic partial-file split.
2 participants
Summary
HttpRequest.BodyReaderusing source-generated System.Text.Json metadataMicrosoft.OpenApi 2.7.5to remediate CVE-2026-49451 discovered during dependency validationAPI and compatibility
V3 is a new breaking-contract surface:
POST /api/v3/eventsPOST /api/v3/projects/{projectId}/eventsContent-Type: application/x-ndjsonThe V2 controller, payload formats, queued handoff, and response contract remain available and unchanged. V3 is disabled by default and can be rolled out by project or organization allowlist.
Verification
dotnet build Exceptionless.slnx --no-restore --disable-build-servers -m:1 -v:minimal— 0 warnings, 0 errorsgit diff --check— cleanRollout
Keep this PR in draft while reviewing the architecture and running the documented one-, two-, four-, and eight-instance load matrix against the intended Redis/Elasticsearch topology. Production activation remains a configuration-only, reversible step.
Closes #2368