fix(replay): Release MediaMuxer when the encoder fails to start

[runningcode]

📜 Description

The MediaMuxer is opened eagerly in SimpleVideoEncoder's constructor, but its release() was only reachable on paths that assume start() succeeded. Two cases leaked it when the encoder failed to start:

• ReplayCache.createVideoOf constructed the encoder and called start() in a single expression (SimpleVideoEncoder(...).also { it.start() }). If start() threw, the assignment to the encoder field never happened, so the only caller of release() could never run.
• SimpleVideoEncoder.release() released the muxer as the last statement of the try block, after draining and stopping the codec. Draining a codec that never started throws, which jumped to the catch and skipped the muxer release.

The fix releases the encoder if start() throws (then rethrows), and always releases the muxer from a finally block so it is freed even when draining/stopping the codec fails.

💡 Motivation and Context

This surfaced as a CloseGuard "A resource was acquired at attached stack trace but never released" warning pointing at the MediaMuxer allocation in SimpleMp4FrameMuxer. It is a real resource leak on devices where the encoder fails to start, not just a test artifact. Complements #5583, which closed the no-frames and never-started paths.

💚 How did you test it?

Added a regression test (ReplayCacheTest.releases the muxer when the encoder fails to start) that forces start() to fail via the test MediaCodec shadow and asserts, through ShadowCloseGuard.getErrors(), that no MediaMuxer leak is reported. Verified the test fails without the fix (with the exact Explicit termination method 'release' not called error) and passes with it. The full ReplayCacheTest suite is green.

📝 Checklist

• I added tests to verify the changes.
• No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
• No breaking change or entry added to the changelog.
• No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

None.

[sentry]

📲 Install Builds

Android

🔗 App Name	App ID	Version	Configuration
SDK Size	io.sentry.tests.size	8.50.0 (1)	release

⚙️ sentry-android Build Distribution Settings

[github-actions]

Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	345.80 ms	378.31 ms	32.51 ms
Size	0 B	0 B	0 B

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
22f4345	325.23 ms	454.66 ms	129.43 ms
22f4345	312.78 ms	347.40 ms	34.62 ms
8558cac	306.16 ms	355.24 ms	49.09 ms
bb0ff41	317.76 ms	384.66 ms	66.90 ms
7c1a728	289.46 ms	368.15 ms	78.69 ms
d501a7e	314.55 ms	343.34 ms	28.79 ms
4fc476b	280.63 ms	363.04 ms	82.42 ms
6727e14	337.22 ms	373.94 ms	36.71 ms
ae7fed0	293.84 ms	380.22 ms	86.38 ms
ee747ae	400.46 ms	423.61 ms	23.15 ms

App size

Revision	Plain	With Sentry	Diff
22f4345	1.58 MiB	2.29 MiB	719.83 KiB
22f4345	1.58 MiB	2.29 MiB	719.83 KiB
8558cac	0 B	0 B	0 B
bb0ff41	0 B	0 B	0 B
7c1a728	0 B	0 B	0 B
d501a7e	0 B	0 B	0 B
4fc476b	0 B	0 B	0 B
6727e14	1.58 MiB	2.28 MiB	718.64 KiB
ae7fed0	1.58 MiB	2.12 MiB	551.77 KiB
ee747ae	1.58 MiB	2.10 MiB	530.95 KiB

Previous results on branch: no/fix-replay-muxer-leak-on-start-failure

Startup times

Revision	Plain	With Sentry	Diff
23ac7bc	304.46 ms	364.53 ms	60.07 ms
5f1e9d5	327.00 ms	385.27 ms	58.27 ms

App size

Revision	Plain	With Sentry	Diff
23ac7bc	0 B	0 B	0 B
5f1e9d5	0 B	0 B	0 B

[romtsn]

sorry this slipped, looks great!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 938d944. Configure here.}

[romtsn]

nice!