Skip to content

fix(linux): avoid memory leak from unnecessary encoder re-probing - #5404

Merged
ReenigneArcher merged 1 commit into
LizardByte:masterfrom
neatnoise:fix-vulkan-device-leak
Jul 12, 2026
Merged

fix(linux): avoid memory leak from unnecessary encoder re-probing#5404
ReenigneArcher merged 1 commit into
LizardByte:masterfrom
neatnoise:fix-vulkan-device-leak

Conversation

@neatnoise

@neatnoise neatnoise commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Description

On Linux, needs_encoder_reenumeration() unconditionally returns true, causing full encoder re-probing (h264, hevc, av1) on every client reconnect. Each probe cycle triggers FFmpeg CBS (Coded Bitstream) operations that allocate ~7.9 MB per HEVC VPS clone for validation. With 2 HEVC + 2 AV1 probes per reconnect, this leaks ~20 MB per cycle that is never fully reclaimed.

Fix by caching the render device path and only returning true when the GPU actually changes (hotplug, driver reload). This reduces per-reconnect memory growth from ~20 MB to <1 MB.

Measured results (AMD RX 9070 XT, RADV, Vulkan encoder):

Reconnect Before After
1st +18 MB +0.6 MB
2nd +18 MB +0.4 MB
3rd +18 MB +0.4 MB
4th +18 MB +0.4 MB
5th +18 MB +0.4 MB

The residual ~0.4-0.6 MB is malloc arena fragmentation (confirmed reclaimable via malloc_trim).

Root cause analysis

  • needs_encoder_reenumeration() always returned true on Linux
  • probe_encoders() is called from nvhttp.cpp on every launch/resume
  • Each probe creates codec contexts for h264/hevc/av1, triggering CBS cbs_h265_replace_vps which clones H265RawVPS (7.9 MB struct) via av_refstruct
  • Confirmed via heaptrack: 43 MB of CBS allocations marked as leaked at process exit, growing with each reconnect cycle

Issues Fixed or Closed

Screenshot

Type of Change

  • feat: New feature (non-breaking change which adds functionality)
  • fix: Bug fix (non-breaking change which fixes an issue)
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code
  • refactor: Code change that neither fixes a bug nor adds a feature
  • perf: Code change that improves performance
  • test: Adding missing tests or correcting existing tests
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Other changes that don't modify src or test files
  • revert: Reverts a previous commit
  • BREAKING CHANGE: Introduces a breaking change

Checklist

  • Code follows the style guidelines of this project
  • Code has been self-reviewed
  • Code has been commented, particularly in hard-to-understand areas
  • Code docstring/documentation-blocks for new or existing methods/components have been added or updated
  • Unit tests have been added or updated for any new or modified functionality

AI Usage

  • None: No AI tools were used in creating this PR
  • Light: AI provided minor assistance (formatting, simple suggestions)
  • Moderate: AI helped with code generation or debugging specific parts
  • Heavy: AI generated most or all of the code changes

@neatnoise
fix(linux): avoid memory leak from unnecessary encoder re-probing
aecd402 
On Linux, needs_encoder_reenumeration() unconditionally returned true,
causing full encoder re-probing (h264, hevc, av1) on every client
reconnect. Each probe cycle allocates ~20 MB of FFmpeg CBS buffers
for HEVC/AV1 VPS validation that are not fully reclaimed, leading to
unbounded memory growth.

Fix by tracking the render device path and only re-probing when it
actually changes (e.g. GPU hotplug or driver reload).
@neatnoise neatnoise mentioned this pull request Jul 12, 2026
Merged
21 tasks
@sonarqubecloud

sonarqubecloud Bot commented Jul 12, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Bundle Report

Bundle size has no change ✅

@Kishi85

Kishi85 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

This PR improves things for me but I'm still seeing a +8MB leak on each capture re-init when using Vulkan (on CachyOS latest).
It's no longer re-probing encoders, which seems to help and reduce the size but a leak when using Vulkan encoding is still present (this is likely the same issue that can be reproduced with ffmpeg directly as described in #5360 (comment)).

@codecov

codecov Bot commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 26.70%. Comparing base (3377f7a) to head (aecd402).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/platform/linux/misc.cpp 0.00% 4 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5404      +/-   ##
==========================================
- Coverage   26.71%   26.70%   -0.01%     
==========================================
  Files         111      111              
  Lines       25324    25329       +5     
  Branches    11194    11198       +4     
==========================================
- Hits         6766     6765       -1     
+ Misses      15538    15333     -205     
- Partials     3020     3231     +211
Flag Coverage Δ
Archlinux 0.00% <0.00%> (ø)
FreeBSD-aarch64 ?
FreeBSD-amd64 13.19% <0.00%> (-0.02%) ⬇️
Homebrew-macos-14 20.35% <ø> (ø)
Homebrew-macos-15 20.52% <ø> (ø)
Homebrew-macos-26 20.64% <ø> (ø)
Homebrew-ubuntu-24.04 12.93% <0.00%> (-0.01%) ⬇️
Linux-AppImage 12.27% <0.00%> (-0.01%) ⬇️
Windows-AMD64 15.20% <ø> (ø)
Windows-ARM64 13.24% <ø> (-0.01%) ⬇️
macOS-arm64 17.69% <ø> (ø)
macOS-x86_64 18.37% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/platform/linux/misc.cpp 15.62% <0.00%> (-0.12%) ⬇️

... and 53 files with indirect coverage changes

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3377f7a...aecd402. Read the comment docs.

@neatnoise

neatnoise commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

@Kishi85 thank you for the tests, you might try to build with FFmpeg CBS fix too here LizardByte/build-deps#727 . It requires the custom build of FFmpeg with the patch file:

cd ~/build-deps
mkdir -p build/dist
cmake -B build -S . -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=$(pwd)/build/dist
make -C build -j$(nproc) install

and build sunshine with option -DFFMPEG_PREPARED_BINARIES=~/build-deps/build/dist/ffmpeg

@ReenigneArcher
ReenigneArcher merged commit 1d9ab7b into LizardByte:master Jul 12, 2026
70 of 71 checks passed
@Kishi85

Kishi85 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@Kishi85 thank you for the tests, you might try to build with FFmpeg CBS fix too here LizardByte/build-deps#727 . It requires the custom build of FFmpeg with the patch file:

cd ~/build-deps
mkdir -p build/dist
cmake -B build -S . -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=$(pwd)/build/dist
make -C build -j$(nproc) install

and build sunshine with option -DFFMPEG_PREPARED_BINARIES=~/build-deps/build/dist/ffmpeg

Unfortunately still seeing the 8MB leak per re-init (tested with display switch) with #5407 and this one applied. No leak with VAAPI so it's definitely a Vulkan specific issue.

@neatnoise

neatnoise commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@Kishi85 thank you for the tests, you might try to build with FFmpeg CBS fix too here LizardByte/build-deps#727 . It requires the custom build of FFmpeg with the patch file:

cd ~/build-deps
mkdir -p build/dist
cmake -B build -S . -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=$(pwd)/build/dist
make -C build -j$(nproc) install

and build sunshine with option -DFFMPEG_PREPARED_BINARIES=~/build-deps/build/dist/ffmpeg

Unfortunately still seeing the 8MB leak per re-init (tested with display switch) with #5407 and this one applied. No leak with VAAPI so it's definitely a Vulkan specific issue.

Hhhm, I can't reproduce it after FFmpeg/sunshine fixes. I checked with clients' reconnect, changing monitors, changing resolutions. Are you sure the compiled FFmpeg with fixes is used?

@Kishi85

Kishi85 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@Kishi85 thank you for the tests, you might try to build with FFmpeg CBS fix too here LizardByte/build-deps#727 . It requires the custom build of FFmpeg with the patch file:

cd ~/build-deps
mkdir -p build/dist
cmake -B build -S . -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=$(pwd)/build/dist
make -C build -j$(nproc) install

and build sunshine with option -DFFMPEG_PREPARED_BINARIES=~/build-deps/build/dist/ffmpeg

Unfortunately still seeing the 8MB leak per re-init (tested with display switch) with #5407 and this one applied. No leak with VAAPI so it's definitely a Vulkan specific issue.

Hhhm, I can't reproduce it after FFmpeg/sunshine fixes. I checked with clients' reconnect, changing monitors, changing resolutions. Are you sure the compiled FFmpeg with fixes is used?

PROJECT_FQDN: dev.lizardbyte.app.Sunshine
PROJECT_NAME: Sunshine
PROJECT_VERSION: 2026.712.223218.r26.g5b0ec1a
PROJECT_VERSION_MAJOR: 2026
PROJECT_VERSION_MINOR: 712
PROJECT_VERSION_PATCH: 223218
CMAKE_PROJECT_VERSION: 2026.712.223218.r26.g5b0ec1a
CMAKE_PROJECT_VERSION_MAJOR: 2026
CMAKE_PROJECT_VERSION_MINOR: 712
CMAKE_PROJECT_VERSION_PATCH: 223218
RC_VERSION_BUILD: 2232
RC_VERSION_REVISION: 18
PROJECT_YEAR: 1990
PROJECT_MONTH: 01
PROJECT_DAY: 01
-- Could NOT find boost_system (missing: boost_system_DIR)
-- Boost include dirs: /usr/include
-- Boost libraries: Boost::filesystem;Boost::log;Boost::program_options;Boost::locale
-- Could NOT find Boost: missing: system (found /usr/lib/cmake/Boost-1.91.0/BoostConfig.cmake (found suitable version "1.91.0", minimum required is "1.53.0"))
CMake Warning at third-party/libdisplaydevice/src/CMakeLists.txt:12 (message):
  Linux is not supported yet.


-- Could NOT find WrapVulkanHeaders (missing: Vulkan_INCLUDE_DIR) 
-- Using FFmpeg from build-deps tag: v2026.713.1051
-- Using existing FFmpeg binaries at /home/user/tmp/sunshine-git/src/build/_deps/ffmpeg

According to my CMake output it's using v2026.713.1051 which should include your patch as listed in #5407.

I'm also seeing 2 leaking VkImageView when using the ffmpeg command compared to the single leaking VkImageView in #5360 (comment) maybe that's a hint to what's going on but I don't even have a clue how I'd debug this.

@neatnoise

neatnoise commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@Kishi85 does it help? LizardByte/build-deps#728

@Kishi85

Kishi85 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@Kishi85 does it help? LizardByte/build-deps#728

That seems to fix it. Running this I'm getting a very stable 128MB memory usage when running with Vulkan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@neatnoise @Kishi85 @ReenigneArcher