Skip to content

perf(windows): keep WGC recording frames on the GPU #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
YoneRai12 wants to merge 1 commit into from
Closed

perf(windows): keep WGC recording frames on the GPU #114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions electron/ipc/handlers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1052,6 +1052,7 @@ function readNativeWindowsEncoderSelection(output: string) {
try {
return JSON.parse(lastLine) as {
video?: string;
frameTransport?: "gpu-zero-copy" | "cpu-readback";
preferSoftwareEncoder?: boolean;
};
} catch {
Expand Down Expand Up @@ -1767,6 +1768,7 @@ export function registerIpcHandlers(
path: outputPath,
helperPath,
videoEncoderSelection: encoderSelection?.video ?? null,
videoFrameTransport: encoderSelection?.frameTransport ?? null,
};
} catch (error) {
console.error("Failed to start native Windows recording:", error);
Expand Down
7 changes: 5 additions & 2 deletions electron/native/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,9 @@ Current V2 JSON shape:

The current helper implementation supports display/window video capture, system audio loopback, selected-microphone capture, Media Foundation webcam capture, and a DirectShow webcam fallback for virtual cameras that are not exposed through Media Foundation. Webcam frames are currently composed into the primary MP4 as a bottom-right picture-in-picture overlay. Browser `deviceId` values do not always map to Media Foundation symbolic links or WASAPI endpoint IDs, so the renderer passes both browser IDs and user-visible device names. For microphones, the helper tries the requested WASAPI endpoint ID first, then resolves an active capture endpoint by `microphoneDeviceName`, then falls back to the default endpoint. For webcams, Electron resolves a matching DirectShow filter CLSID for the selected label; the helper uses Media Foundation first, then that exact DirectShow filter when the requested camera is absent from Media Foundation.

Encoder selection: by default the helper keeps the existing sink-writer path first. If that path fails while setting up H.264, it retries with the Microsoft software H.264 encoder (`mfh264enc.dll`). The key of this retry is registering that encoder locally in the helper process via `MFTRegisterLocalByCLSID`, which makes a software H.264 encoder available even when the machine's hardware encoders are missing or broken; hardware transforms are disabled for the retry only as a secondary guard so the sink writer prefers the locally registered software encoder, not as the fallback mechanism itself. Set `preferSoftwareEncoder: true` in the helper JSON, or set `OPENSCREEN_WGC_PREFER_SOFTWARE_ENCODER=true` before launching Electron, to force the software path from the first attempt.
Encoder selection: by default the helper explicitly enables Media Foundation hardware transforms, supplies the capture D3D11 device through `MF_SINK_WRITER_D3D_MANAGER`, and passes DXGI texture-backed samples to the sink writer. A four-texture GPU ring keeps each texture alive until `IMFSinkWriter::PlaceMarker` confirms that Media Foundation consumed it. This avoids the former per-frame staging-texture `Map` and BGRA memory copy. If the GPU-backed writer cannot be created, or the DXGI frame path fails while recording, the helper keeps recording through the CPU readback path. If H.264 setup itself fails, it retries with the Microsoft software H.264 encoder (`mfh264enc.dll`). The key of this retry is registering that encoder locally in the helper process via `MFTRegisterLocalByCLSID`, which makes a software H.264 encoder available even when the machine's hardware encoders are missing or broken; hardware transforms are disabled for the retry only as a secondary guard so the sink writer prefers the locally registered software encoder, not as the fallback mechanism itself. Set `preferSoftwareEncoder: true` in the helper JSON, or set `OPENSCREEN_WGC_PREFER_SOFTWARE_ENCODER=true` before launching Electron, to force the software path from the first attempt.

The helper reports the outcome through the `encoder-selection` stdout event (`video` is `default`, `software-preferred`, or `software-fallback`). When the app sees `software-fallback` — the default encoder failed and the helper switched on its own — it shows a small dismissible notice in the recording HUD with a "Don't show again" option, because software encoding can raise CPU usage. An explicit `software-preferred` selection shows no notice, and the event stays available for diagnostics either way.
The helper reports the outcome through the `encoder-selection` stdout event. `video` is `hardware` when the selected H.264 transform exposes the Media Foundation hardware URL attribute, `software-default` when the hardware-enabled path selected a software encoder, `software-preferred` or `software-fallback` for the explicit software paths, and `default` only when the sink writer does not expose enough information to classify the transform. `frameTransport` is `gpu-zero-copy` when video frames stay in D3D11/DXGI memory, or `cpu-readback` when the compatibility path is active. The final `encoder-summary` event includes the effective frame transport, GPU/CPU frame counts, elapsed time, and helper process CPU time. The app shows its dismissible software-encoder notice for `software-default` and `software-fallback`, because CPU usage can be higher. An explicit `software-preferred` selection shows no notice, and the event stays available for diagnostics either way.

Encoder diagnostic on final sink-writer failure: when the final `MFCreateSinkWriterFromURL` attempt fails, the helper logs the registered H.264 video encoder MFT count (via `MFTEnumEx`), the registered AAC encoder count when audio was requested, and the hex HRESULT. If no H.264 encoder is registered, it additionally emits the four-bullet actionable error (missing Media Feature Pack / GPU driver registration / empty `HKLM:\SOFTWARE\Microsoft\Windows Media Foundation\Transforms` / reboot). If an H.264 encoder IS registered but the sink writer still failed, it logs a hint pointing at invalid output path, missing MP4 mux, or GPU driver incompatibility. There is still no fail-fast pre-flight gate because `MFTEnumEx` and `MFCreateSinkWriterFromURL` can disagree about which H.264 encoders are available in non-interactive / Session 0 contexts.

Expand All @@ -96,13 +96,16 @@ Smoke-test the helper with:
npm run test:wgc-helper:win
npm run test:wgc-helper:win -- --software-encoder
npm run test:wgc-helper:win -- --software-fallback
npm run test:wgc-helper:win -- --gpu-frame-fallback
npm run test:wgc-window:win
npm run test:wgc-audio:win
npm run test:wgc-mic:win
npm run test:wgc-mixed-audio:win
npm run test:wgc-webcam:win
```

Set `OPENSCREEN_WGC_TEST_FPS=60` to change the smoke-test frame rate, and set `OPENSCREEN_WGC_TEST_REQUIRE_GPU_ZERO_COPY=true` to fail the test unless a hardware H.264 encoder consumes every screen frame through the DXGI path. `OPENSCREEN_WGC_TEST_DISPLAY_X`, `OPENSCREEN_WGC_TEST_DISPLAY_Y`, `OPENSCREEN_WGC_TEST_DISPLAY_WIDTH`, and `OPENSCREEN_WGC_TEST_DISPLAY_HEIGHT` can select a monitor by its physical Win32 bounds for multi-monitor coverage.

`--software-encoder` keeps testing the explicit `software-preferred` path with
`preferSoftwareEncoder: true`. `--software-fallback` keeps
`preferSoftwareEncoder: false` and sets
Expand Down
90 changes: 84 additions & 6 deletions electron/native/wgc-capture/src/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,29 @@ struct CaptureControl {
}
};

uint64_t currentProcessCpuTimeHns() {
FILETIME creationTime{};
FILETIME exitTime{};
FILETIME kernelTime{};
FILETIME userTime{};
if (!GetProcessTimes(
GetCurrentProcess(),
&creationTime,
&exitTime,
&kernelTime,
&userTime)) {
return 0;
}

ULARGE_INTEGER kernel{};
kernel.LowPart = kernelTime.dwLowDateTime;
kernel.HighPart = kernelTime.dwHighDateTime;
ULARGE_INTEGER user{};
user.LowPart = userTime.dwLowDateTime;
user.HighPart = userTime.dwHighDateTime;
return kernel.QuadPart + user.QuadPart;
}

std::wstring utf8ToWide(const std::string& value) {
if (value.empty()) {
return {};
Expand Down Expand Up @@ -404,6 +427,14 @@ int main(int argc, char* argv[]) {
const bool injectDefaultSinkWriterFailureOnce =
injectDefaultSinkWriterFailureLength == 1 &&
injectDefaultSinkWriterFailure[0] == '1';
char injectGpuFrameFailure[2]{};
const DWORD injectGpuFrameFailureLength = GetEnvironmentVariableA(
"OPENSCREEN_WGC_TEST_INJECT_GPU_FRAME_FAILURE_ONCE",
injectGpuFrameFailure,
static_cast<DWORD>(sizeof(injectGpuFrameFailure)));
const bool injectGpuFrameFailureOnce =
injectGpuFrameFailureLength == 1 &&
injectGpuFrameFailure[0] == '1';

std::cout << "{\"event\":\"ready\",\"schemaVersion\":2}" << std::endl;

Expand Down Expand Up @@ -435,9 +466,9 @@ int main(int argc, char* argv[]) {
return 1;
}

// WGC owns the captured texture size. Encoding must use that exact size
// until a dedicated GPU scaling pass is introduced; CopyResource requires
// matching resource dimensions.
// H.264 requires even dimensions. WGC window textures can be odd-sized, so
// keep the largest even rectangle and crop at most one pixel on the right
// or bottom during the GPU copy below.
int width = session.captureWidth();
int height = session.captureHeight();
width = (std::max(2, width) / 2) * 2;
Expand Down Expand Up @@ -516,6 +547,8 @@ int main(int argc, char* argv[]) {
MFEncoderOptions encoderOptions{};
encoderOptions.preferSoftwareEncoder = config.preferSoftwareEncoder;
encoderOptions.injectDefaultSinkWriterFailureOnce = injectDefaultSinkWriterFailureOnce;
encoderOptions.injectGpuFrameFailureOnce = injectGpuFrameFailureOnce;
encoderOptions.allowGpuFrameTransport = !config.webcamEnabled || writeSeparateWebcam;

MFEncoder encoder;
if (!encoder.initialize(
Expand All @@ -533,13 +566,17 @@ int main(int argc, char* argv[]) {
}
std::cout << "{\"event\":\"encoder-selection\",\"schemaVersion\":2,\"video\":\""
<< encoder.videoEncoderSelection()
<< "\",\"frameTransport\":\""
<< encoder.videoFrameTransport()
<< "\",\"preferSoftwareEncoder\":"
<< (config.preferSoftwareEncoder ? "true" : "false")
<< "}" << std::endl;
MFEncoder webcamEncoder;
if (writeSeparateWebcam) {
MFEncoderOptions webcamEncoderOptions = encoderOptions;
webcamEncoderOptions.injectDefaultSinkWriterFailureOnce = false;
webcamEncoderOptions.injectGpuFrameFailureOnce = false;
webcamEncoderOptions.allowGpuFrameTransport = false;
const int webcamPixels = std::max(1, webcamCapture.width()) * std::max(1, webcamCapture.height());
const int webcamBitrate = webcamPixels >= 1280 * 720 ? 8'000'000 : 4'000'000;
if (!webcamEncoder.initialize(
Expand Down Expand Up @@ -579,6 +616,8 @@ int main(int argc, char* argv[]) {
if (!latestFrameTexture) {
D3D11_TEXTURE2D_DESC desc{};
texture->GetDesc(&desc);
desc.Width = static_cast<UINT>(width);
desc.Height = static_cast<UINT>(height);
desc.BindFlags = 0;
desc.CPUAccessFlags = 0;
desc.MiscFlags = 0;
Expand All @@ -590,7 +629,23 @@ int main(int argc, char* argv[]) {
}
}

session.context()->CopyResource(latestFrameTexture.Get(), texture);
const D3D11_BOX sourceBox{
0,
0,
0,
static_cast<UINT>(width),
static_cast<UINT>(height),
1,
};
session.context()->CopySubresourceRegion(
latestFrameTexture.Get(),
0,
0,
0,
0,
texture,
0,
&sourceBox);
Comment on lines +632 to +648

@coderabbitai coderabbitai Bot Jul 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Clamp the subresource box to the incoming texture dimensions.

If the captured window is resized smaller than its original dimensions, the incoming WGC texture will be smaller than width or height. When sourceBox extends beyond the source texture's bounds, D3D11's CopySubresourceRegion silently drops the operation, causing the recording to freeze.

Fetch the incoming texture's actual description and clamp the copy bounds to prevent this.

🐛 Proposed fix to prevent frozen frames on window resize 
-        const D3D11_BOX sourceBox{
-            0,
-            0,
-            0,
-            static_cast<UINT>(width),
-            static_cast<UINT>(height),
-            1,
-        };
+        D3D11_TEXTURE2D_DESC srcDesc{};
+        texture->GetDesc(&srcDesc);
+        const D3D11_BOX sourceBox{
+            0,
+            0,
+            0,
+            std::min(static_cast<UINT>(width), srcDesc.Width),
+            std::min(static_cast<UINT>(height), srcDesc.Height),
+            1,
+        };
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const D3D11_BOX sourceBox{
0,
0,
0,
static_cast<UINT>(width),
static_cast<UINT>(height),
1,
};
session.context()->CopySubresourceRegion(
latestFrameTexture.Get(),
0,
0,
0,
0,
texture,
0,
&sourceBox);
D3D11_TEXTURE2D_DESC srcDesc{};
texture->GetDesc(&srcDesc);
const D3D11_BOX sourceBox{
0,
0,
0,
std::min(static_cast<UINT>(width), srcDesc.Width),
std::min(static_cast<UINT>(height), srcDesc.Height),
1,
};
session.context()->CopySubresourceRegion(
latestFrameTexture.Get(),
0,
0,
0,
0,
texture,
0,
&sourceBox);
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@electron/native/wgc-capture/src/main.cpp` around lines 632 - 648, Update the
copy setup surrounding sourceBox and CopySubresourceRegion to retrieve the
incoming texture’s actual D3D11 description, then clamp the box width and height
to its dimensions before copying. Preserve the existing copy behavior while
ensuring sourceBox never exceeds texture’s bounds during window resizing.

latestFrameTimestampHns = timestampHns;
if (!firstFrameWritten.exchange(true)) {
control.cv.notify_all();
Expand Down Expand Up @@ -834,6 +889,8 @@ int main(int argc, char* argv[]) {
if (audioMixer) {
audioMixer->beginTimeline();
}
const auto recordingWallStart = std::chrono::steady_clock::now();
const uint64_t recordingCpuStartHns = currentProcessCpuTimeHns();
startVideoWriter();

std::cout << "{\"event\":\"recording-started\",\"schemaVersion\":2}" << std::endl;
Expand Down Expand Up @@ -869,10 +926,14 @@ int main(int argc, char* argv[]) {
logStopStep("wgc-session-close");
{
std::scoped_lock lock(mutex);
encoder.finalize();
if (!encoder.finalize()) {
encodeFailed = true;
}
logStopStep("encoder-finalize");
if (writeSeparateWebcam) {
webcamEncoder.finalize();
if (!webcamEncoder.finalize()) {
encodeFailed = true;
}
logStopStep("webcam-encoder-finalize");
}
}
Expand All @@ -886,6 +947,23 @@ int main(int argc, char* argv[]) {
return 1;
}

const auto recordingElapsedMs = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::steady_clock::now() - recordingWallStart).count();
const uint64_t recordingCpuEndHns = currentProcessCpuTimeHns();
const uint64_t recordingCpuTimeMs = recordingCpuEndHns >= recordingCpuStartHns
? (recordingCpuEndHns - recordingCpuStartHns) / 10'000ULL
: 0;

std::cout << "{\"event\":\"encoder-summary\",\"schemaVersion\":2,\"video\":\""
<< encoder.videoEncoderSelection()
<< "\",\"frameTransport\":\""
<< encoder.videoFrameTransport()
<< "\",\"gpuFrames\":" << encoder.gpuFramesWritten()
<< ",\"cpuFrames\":" << encoder.cpuFramesWritten()
<< ",\"elapsedMs\":" << recordingElapsedMs
<< ",\"processCpuTimeMs\":" << recordingCpuTimeMs
<< "}" << std::endl;

std::cout << "{\"event\":\"recording-stopped\",\"schemaVersion\":2,\"screenPath\":\""
<< jsonEscape(config.outputPath) << "\"";
if (writeSeparateWebcam) {
Expand Down
Loading
Loading