Media: Add UltraHDR (ISO 21496-1) gain map support#74873
Open
adamsilverstein wants to merge 27 commits intotrunkfrom
Open
Media: Add UltraHDR (ISO 21496-1) gain map support#74873adamsilverstein wants to merge 27 commits intotrunkfrom
adamsilverstein wants to merge 27 commits intotrunkfrom
Conversation
adamsilverstein
requested review from
ajitbohra,
gziolo,
nerrad,
ntwb and
ryanwelcher
as code owners
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Unlinked AccountsThe following contributors have not linked their GitHub and WordPress.org accounts: @kleisauke, @gregbenz. Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases. If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
adamsilverstein
added
[Type] Enhancement
github-project-automation
Bot
moved this to 🔎 Needs Review
in WordPress 7.0 Editor Tasks
|
Flaky tests detected in 1edd5d7. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/25594199702
|
|
Size Change: +228 kB (+2.87%) Total Size: 8.15 MB 📦 View Changed
ℹ️ View Unchanged
|
Contributor
andrewserong
left a comment
andrewserong
left a comment
There was a problem hiding this comment.
Nice, this is testing well for me! I was going to ask how heavy the upstream library is, and then I noticed that it's from your own repo, so I assume you're quite comfortable with it 😄
The main question I have is to do with the idea that if most uploaded JPEGs are not UltraHDR, how expensive is the operation to detect it? If it's very cheap, then this seems like a great addition. I've left a comment, but before I looked at the code, my assumption is that this kind of operation would flag if it succeeds / does something, but wouldn't warn if it can't find an UltraHDR data.
What do you think?
This is all still behind an experiment of course, so don't let my questions here stop you from progressing! (Also I'll be AFK early next week, so apologies if my reply is delayed until mid-week)
Comment on lines
+460
to
+464
| } catch ( error ) { | ||
| // If UltraHDR detection fails, continue with regular upload | ||
| // eslint-disable-next-line no-console | ||
| console.warn( 'UltraHDR detection failed:', error ); | ||
| } |
Contributor
There was a problem hiding this comment.
I'm getting this warning for uploading normal (non UltraHDR) jpegs from my desktop. Given that the majority of jpegs won't be ultra HDR, should we just fail silently here?
Member
Author
There was a problem hiding this comment.
yes, this is mainly for debugging purposes and I will remove before merging to trunk.
Member
Author
I am comfortable with it working based on the tests, although I also plan to ask for help doing manual testing to verify it properly handles real world images.
That is a great question and worth investigating more. I will review the detection logic to see if it can be optimized. Also, I'm still not certain about what we should output. My assumption is that UltraHDR JPEGs are decoder compatible with existing JPEGs so we would want to output UltraHDR for uploaded UltraHDR. Then again, some users may prefer to use the SDR version for their website exclusively because they want smaller file sizes and don't care about HDR. We might need a new way for developers to implement that since the mime type is still image/jpeg and out=r
Hopefully for something fun! |
|
@adamsilverstein libultrahdr includes a probe() method to validate that required components are present without decoding. I haven't tested it for performance, but it may serve as inspiration. Gain maps require an auxiliary image. You could check the header and reject if that's missing. If it exists, you could then further check for a supported gain map (ie a more robust check, but you've already quickly skipped most images that aren't relevant). There are two kinds of map encoding supported by libultra: (a) ISO standard gain maps and (b) gain maps encoded with the Android XMP spec. The ISO encoding is widely used for new images now and has been for a while in most encoders. Many now dual encode (as the auxiliary image is the same and you're just writing redundant data - binary in the aux image for ISO and standard XMP for the Android spec). The only thing you'd miss by skipping XMP would be older images captured on Android or exported with old versions of Adobe software (Apple has a proprietary method they used pre-ISO and it is not supported by libultra). I believe skipping XMP would generally be ok to do, as it is a very limited edge case (and the impact would be that transcoding falls back to SDR rather than total failure). |
|
@adamsilverstein If the source includes a supported gain map, the output should preserve it by default. SharpJS is working towards a keepGainMap() method which should do this, however, I'm not sure how much guidance libvips needs. Coordinating with SharpJS may make sense here, as both efforts are likely chasing the same goals for default transcoding (ie ability to do basic crop/resize/compress while retaining an output which shows high fidelity to the original). Questions for transcoding will inevitably come up around tuning (compression of the base image / map). Testing to make sure the final result is ok based on the compression applied to both the base and map will be an important step to validate quality vs size objectives. Existing approaches for the base image probably translate well, and this is probably mostly a question of how to compress the map. Aside from that, most of the options chosen for encoding should remain as they were in the source. The metadata should be unchanged (HDR capacity, offsets, ranges, etc). The map scaling (1:1 vs 1:2 or 1:4) and number of channels (1 for luminosity only map vs 3 for full color) should remain the same. It should not alter sub-sampling (ie keep 444 if that's how it was encoded as the map is not a color image and the assumptions behind sub-sampling are not applicable in this domain, and can cause artifacts if altered). An affordance for aggressive compression of the image (including loss of HDR) may be useful for some users / workflows, but is not required and could get complicated. There are also other ways beyond stripping the map which can compress the image without stripping HDR, but they involve quality tradeoffs that would need careful assessment. For example, you could downgrade the map from 1:1 resolution in full color to a 1:2 map with luminosity only. That will cause loss of high frequency detail in the HDR rendition, as well as color error (which may be a small or significant issue depending on the relationship of the SDR to the HDR rendition). Due to the effort, complexity, and risk of confusion here, it may be best that initial support just preserve the gain map and not use more complex options to compress the image further. That can be managed now via the encoding of the file uploaded to WordPress (upload SDR if you only want SDR, phone captures are already low resolution luminosity maps, etc). Advanced transcoding for compression might best be left for 3rd-party plugins to address. |
Member
Author
great, thanks for the tip - i will take a look at that.
👍🏼
Right, be default we will always try to use the uploaded format for the output (so uploaded UltraHDR should output UltraHDR). My pondering was more about how would enable developers to choose to extract only the SDR image for output/front end if thats what they wanted. The current output mapping won't work, so we may need a more explicit filter to choose HDR or SDR encoding when an UltraHDR is uploaded. |
Member
Author
I created an issue to add a probe mode to lib-open-ultrahdr: |
Member
Author
Thanks again for the tip @gregbenz - I added the probe feature in adamsilverstein/lib-open-ultrahdr#6 based on the libultrahdr approach |
Member
Author
Thanks again for the question @andrewserong - it turns out libultrahdr has a |
adamsilverstein
removed request for
ajitbohra,
gziolo,
nerrad,
ntwb and
ryanwelcher
adamsilverstein
added
the
[Status] In Progress
Narrow the try/catch in detectUltraHdr to only wrap the probe logic, making it clearer what errors are being caught. Move the UltraHDR detection result check outside the try block. Add comment explaining why operations are replaced (to skip transcoding that would strip HDR gain map data). Remove unused CreateUltraHdr enum value.
Revert the webpack asset/resource rule for .wasm files and instead inline the UltraHDR WASM binary at build time using the existing esbuild wasmInlinePlugin, matching the pattern used by vips. This avoids separate file downloads and MIME type serving issues. Use the new setWasmUrl() API from open-ultrahdr 0.1.2 to pass the inlined data URL directly to the WASM initialization.
4 tasks
When an UltraHDR/ISO 21496-1 image is uploaded, sub-sizes (thumbnails, scaled versions) now retain the gain map data. During detection, the gain map and metadata are decoded and cached. Each sub-size is resized normally via vips, then re-encoded as UltraHDR by reconstructing HDR pixel data from the SDR base and gain map using the ISO 21496-1 formula. Format transcoding is skipped for UltraHDR images since it would strip the gain map. Cache entries are cleaned up when the parent is finalized.
Use .buffer to get the underlying ArrayBuffer from the Uint8Array when constructing a Blob, satisfying stricter TypeScript BlobPart type.
Wrap cached.gainMap.buffer with new Uint8Array() to convert ArrayBufferLike (which includes SharedArrayBuffer) to a valid BlobPart type accepted by the Blob constructor.
TypeScript 5.9 treats Uint8Array.buffer as ArrayBufferLike which includes SharedArrayBuffer, but Blob only accepts ArrayBuffer. Use explicit cast since decoded JPEG data is always ArrayBuffer.
Member
Author
Note: this PR currently uses open-ultrahdr due to license incompatibility with libultrahdr. However, Google has agreed to dual license the Once google/libultrahdr#386 is reviewed and merged, we can switch this PR to using that library. Since the libraries offer similar functional signatures, the PR should be relatively minor for this change. |
Resolve conflict in packages/upload-media/src/store/private-actions.ts by keeping the PR branch's UltraHDR-aware changes.
Preserve trunk's vipsBatchResizeImage() optimization for non-UltraHDR uploads while keeping the per-thumbnail fallback as the primary path for UltraHDR images. The batch routine uses thumbnailImage() which strips gain maps, so UltraHDR sub-sizes must be resized individually with EncodeUltraHdr to preserve HDR data.
Member
Author
|
So far, this PR is very light. If the bundle size calculation increases, it might be worth exploring following the "canonical plugin" approach I am taking with JXL (#76990) and GIF->Video support with ffmpeg (#76964). The idea is the first time a user tries to access one of these formats/features, the code required (a canonical plugin) is side-loaded into the system. From that point on the feature/format is supported and the plugin can also update in the background in case we need to ship new versions. The primary goal of shipping media format support this way is keeping the core bundle size as small as possible. Since features may only be needed by a small subset of users, it makes sense to ship them as needed/on the fly as a one-time download As pointed out by @swissspidy in this comment, it might even be worth building a dedicated API for this. |
Resolves conflicts in packages/upload-media/src/store/private-actions.ts by rebuilding the file on top of trunk's architecture (subsize accumulation and dimension group deduplication) and re-applying UltraHDR-specific changes: DetectUltraHdr/EncodeUltraHdr operations, ultraHdrCache for gain-map reuse across sub-sizes, UltraHDR branches in generateThumbnails, and cache cleanup in finalizeItem. Drops the branch's transient vipsBatchResizeImage integration since trunk removed that utility; UltraHDR sub-sizes are generated via the per-thumbnail path with EncodeUltraHdr, which is where gain-map preservation has to happen anyway.
Member
Author
I verified the upstream wasm is relatively tiny - ~150 KB gzipped - I think its fine just to bundle support directly in this case. |
Adopt trunk's attachment.image_output_format/image_save_progressive response fields in place of the old settings.imageOutputFormats lookup, and keep the UltraHDR guard so gain map images skip sub-size transcoding (which would strip the gain map).
open-ultrahdr 0.2.0 swaps the in-house Rust UltraHDR reimplementation for an Emscripten/embind build of upstream google/libultrahdr, now that upstream is dual-licensed Apache-2.0 OR MIT. The WASM filename changes from open_ultrahdr_bg.wasm (wasm-pack convention) to open_ultrahdr.wasm (Emscripten convention), so the inline import path moves accordingly. Public JS API is unchanged. Tracks adamsilverstein/lib-open-ultrahdr#14 / PR #15. Note: package-lock.json regen will be needed once open-ultrahdr@0.2.0 is published to npm; CI installs will fail until then.
|
wasm-vips v0.0.18 is expected to be released in ~2 weeks and includes updated licensing info for libultrahdr (see commit kleisauke/wasm-vips@33a1975). I think using that instead (e.g. via the |
open-ultrahdr@0.2.0 and open-ultrahdr-wasm@0.2.0 are now published on npm (Apache-2.0 OR MIT, backed by upstream libultrahdr via Emscripten). Regenerate the lockfile entries so npm ci picks up the new tarballs and integrity hashes.
Member
Author
Excellent! Thank you for changing that :) I'll hold off on adding support until that ships. |
Bump wasm-vips to ^0.0.17 to pick up libvips's native UltraHDR (ISO 21496-1 gain map) support, which was introduced upstream once libultrahdr dual-licensed under Apache-2.0 OR MIT. Add a getUltraHdrInfo helper that probes a buffer via uhdrloadBuffer and returns dimensions plus HDR headroom in stops, and an isUltraHdr flag on resizeImage so callers can route through uhdrloadBuffer + uhdrsaveBuffer. The thumbnail path uses thumbnailImage on the already-loaded image so the gain map is downsized in lockstep with the base image and re-embedded on save.
Drop the open-ultrahdr and open-ultrahdr-wasm dependencies in favor of the UltraHDR pipeline now built into wasm-vips. libultrahdr's recent relicensing to Apache-2.0 OR MIT made it compatible with the GPL-2.0 WordPress codebase, removing the original reason to maintain a separate in-house WASM build. Replace the open-ultrahdr probe + decode + manual ISO 21496-1 re-encode round-trip with a single isUltraHdr flag passed through vipsResizeImage. libvips loads the source via uhdrload, resizes the base image and gain map together via thumbnailImage, and saves via uhdrsave, eliminating the OffscreenCanvas pixel reconstruction path along with the EncodeUltraHdr operation type and the per-item gain-map cache.
Member
Author
|
Thanks @kleisauke! Updated this PR to use What changed:
Both packages type-check, lint, and unit-test clean (209 tests). Net diff: −276 / +67 in I'll plan a follow-up bump to |
# Conflicts: # packages/vips/CHANGELOG.md
Cover the new UltraHDR pipeline at three levels.
Vips unit (packages/vips/src/test/ultrahdr.ts): mock wasm-vips and
verify getUltraHdrInfo returns dims + log2-stop capacity for valid
inputs, returns null on missing gain map / decode failure, and falls
back to zero capacity when the metadata field is absent. Verify
resizeImage with isUltraHdr=true routes through uhdrloadBuffer +
thumbnailImage + uhdrsaveBuffer instead of newFromBuffer +
thumbnailBuffer + writeToBuffer.
Upload Media unit (packages/upload-media/src/store/test/
private-actions.js): mock vipsGetUltraHdrInfo and exercise the
detectUltraHdr thunk. Cover: positive detection writes
attachment.meta.ultrahdr + hdr_capacity, no metadata change when not
UltraHDR, graceful pass-through when probing throws, normalization
when REST returns meta as [] (rather than {}), and early return
when the queue item is missing.
E2E (test/e2e/specs/editor/various/client-side-media-processing.spec
.js): upload a synthesized 1024x768 UltraHDR JPEG fixture, fetch the
medium sub-size from REST, and re-probe it through wasm-vips to
confirm the gain map survived the resize. The fixture is generated
deterministically by .gen-ultrahdr-fixture.mjs (committed alongside
so it can be regenerated without vendoring a third-party binary).
kleisauke
reviewed
May 9, 2026
| smartCrop = false, | ||
| quality = 0.82 | ||
| quality = 0.82, | ||
| isUltraHdr = false |
There was a problem hiding this comment.
You don't need this. Since uhdrload*() has a higher priority than jpegload*(), you can continue using vips.Image.newFromBuffer() (or vips.Image.thumbnailBuffer()). Likewise, jpegsave*() automatically delegates to uhdrsave* when the image contains a gainmap or uses the scRGB color space.
Here's an example to help you get started:
https://gist.github.com/kleisauke/f90c05b839ef871cbb97b294e9435bf7
Member
Author
There was a problem hiding this comment.
Fantastic, thank you!
There was a problem hiding this comment.
This also needs to crop the gainmap, if present. An example is available here:
https://www.libvips.org/API/current/uhdr.html#a-la-carte-processing
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
Address kleisauke's review on PR #74873: - Drop the explicit isUltraHdr branching in resizeImage. uhdrload* has higher priority than jpegload* in libvips, so newFromBuffer and thumbnailBuffer auto-detect UltraHDR JPEGs and decode the gain map alongside the base image. jpegsave* delegates to uhdrsave* on output when a gain map is attached, so writeToBuffer handles the save side too. Removes the parameter from vipsResizeImage at every layer. - Crop the gain map alongside the main image on positional crop. Scale crop coordinates to the gain map's resolution, since the gain map can be smaller than the main image. Follows libvips's a-la-carte processing example: https://www.libvips.org/API/current/uhdr.html#a-la-carte-processing - Clean up the keep:'icc|gainmap' save option formatting that was applied as a raw suggestion in the previous commit. Extend the ForeignKeep type to include 'gainmap' and pipe-combined values. Update tests to cover the simplified path, the new gain-map crop branch, and the keep:'icc|gainmap' save option.
Member
Author
|
Thanks again for the review @kleisauke — addressed all three points in f17ac14:
Tests in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@wordpress/vipsworker — no extra WASM module is bundledCloses #74874
Similar to adamsilverstein/client-side-media-experiments#15
Approach
UltraHDR support comes from
wasm-vips(libvips compiled to WebAssembly), which gained native UltraHDR (uhdrload/uhdrsave) in0.0.17. This was unblocked bygoogle/libultrahdr#386, which dual-licensed libultrahdr underApache-2.0 OR MITand made it compatible with WordPress's GPLv2-or-later codebase.@wordpress/vipsis bumped towasm-vips@^0.0.17and gains:getUltraHdrInfo(buffer)helper that probes a JPEG for an embedded gain map and reports HDR headroomisUltraHdrflag onresizeImagethat routes throughuhdrloadBuffer+uhdrsaveBufferso the gain map is downsized in lockstep with the base image and re-embedded on save@wordpress/upload-mediauses these helpers in its existingDetectUltraHdroperation and tags sub-size resizes withisUltraHdrso the resize step transparently preserves the gain map. Thanks to @kleisauke for landing native UltraHDR support inwasm-vips.Removed
open-ultrahdrandopen-ultrahdr-wasmdependencies are no longer needed.EncodeUltraHdrqueue operation type is gone — there's no separate re-encode step; resize handles it inline.encodeUltraHdrItem) is removed.Follow-up
wasm-vips@0.0.18will land a docs-only update toTHIRD-PARTY-NOTICES.mdto make the dual-licensing explicit. We can bump to^0.0.18when it's published.Test plan