perf(service): parallelize intra-task tile merge (MAPCO-11325) - #260
perf(service): parallelize intra-task tile merge (MAPCO-11325)#260shimoncohen wants to merge 1 commit into
Conversation
The service merged tiles one coord at a time in a sequential x/y loop. Merge is the hot path and each coord is independent, so run each batch's coords through Parallel.ForEach, one chunk at a time. - Coords are enumerated per bounds and processed in chunks of batchMaxSize. Each chunk is merged in parallel into a ConcurrentBag, then flushed to the target before the next chunk starts, so peak memory stays bounded to ~one chunk (count-based; with the default batchMaxSize=1000 this is ~the previous 20MB byte cap for jpeg tiles). - Within a chunk the target is only read (writes happen after the parallel phase), and GetCorrespondingTile is thread-safe per data source, so the merge relies on the thread-safe GpkgClient connection from MAPCO-11320. - Degree of parallelism is configurable via GENERAL:parallel:numOfThreads (0/unset -> Environment.ProcessorCount). Depends on and is stacked over MAPCO-11320 (logic-1-gpkgclient-perf). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Check that Parallel doesn't call other parallel's |
Checked. There is one real nest: this PR's outer The pre-existing S3 Fixed the inner parallel in #264 (sequential highest-zoom-first, short-circuits): #264 |
SVC-1 of MAPCO-11318. Stacked on #255 (MAPCO-11320) — base branch is
logic-1-gpkgclient-perf, not master, so this diff shows only the SVC-1 change. Retarget to master after #255 merges.Change
The service merged tiles sequentially in a nested x/y loop. Merge is the hot path and each coord is independent, so:
batchMaxSize, merging each chunk's coords withParallel.ForEachinto aConcurrentBag, then flushing the chunk to the target before starting the next.batchMaxSize=1000, that's ≈ the previous 20 MB byte cap for jpeg tiles. Per the design decision, the precise byte cap is approximated by the count cap.GetCorrespondingTileis thread-safe per data source — so this relies on the thread-safeGpkgClientconnection from perf(gpkg): connection reuse + prepared bulk-insert + keyset paging (MAPCO-11320) #255.GENERAL:parallel:numOfThreads(0/unset →Environment.ProcessorCount).Scope — delivers P9 of MAPCO-11325 only
MAPCO-11325 scopes four points; this PR implements P9 (parallelize the intra-task merge). The rest remain open in that ticket and are intentionally out of scope here:
MainRunner: one task at a time process-wide → per-type bounded workers). Job-state/heartbeat complexity; separate change.Separately, the count-based memory cap here is an approximation of the old byte-accurate 20 MB bound; tracked as its own follow-up in MAPCO-11367.
Testing
Full suite: 1142 passed, 0 failed. The batch-flush test now uses dense bounds and asserts count-based (chunked) flushing.
Depends on
🤖 Generated with Claude Code