[mono][wasm] Fix pinvoke with 64-bit enum argument

[lewing]

Summary

Fixes #112262. A P/Invoke whose argument is an enum with a 64-bit underlying type (e.g. [Flags] enum Flags : ulong) crashes on wasm at the call site with RuntimeError: null function or function signature mismatch.

Root cause

interp_type_as_ptr() (interp/transform.c) decides whether a native call can take the fast MINT_CALLI_NAT_FAST → do_icall path, which passes every argument as a pointer-sized gpointer. It treated every enum as pointer-compatible without checking its underlying type:

if (tp->type == MONO_TYPE_VALUETYPE && m_class_is_enumtype (...))
    return TRUE;   // ignores underlying type

Raw long/ulong are already correctly excluded on 32-bit targets via the #if SIZEOF_VOID_P == 8 guard (which is why the documented workaround — casting the enum to ulong — works), but a 64-bit enum slipped through. do_icall then called the native void(int64_t) as void(int32_t) on wasm32 → call_indirect signature mismatch.

Fix

Only a 64-bit enum can be misclassified as pointer-sized on a 32-bit target, so for enums with an I8/U8 underlying type defer to the underlying type (which is SIZEOF_VOID_P-guarded); all other enums keep their existing classification. Behavior only changes for enums with a 64-bit underlying type on 32-bit targets such as wasm32; enum:int/enum:uint/smaller enums and all enums on 64-bit hosts are unchanged.

interp_type_as_ptr is interp-transform-only; the AOT (type_to_c) and jiterpreter (mono_type_to_ldind) signature paths already reduce enums to their underlying type, so no parallel change is needed.

Test

Extended the existing WASM ABI pinvoke test (AbiRules.cs + wasm-abi.c) with enum:ulong and enum:long direct pinvokes — the two cases the fix changes. EnsureWasmAbiRulesAreFollowedInInterpreter (the interpreter path where the bug lives) exercises the regression.

Verification

Built the browser Mono runtime with the fix and ran the WASM ABI interpreter test end-to-end:

• Without the fix: void(enum:ulong) pinvoke → function signature mismatch, app aborts before running.
• With the fix: test passes; native round-trips the full 64-bit value (eu (eu)=18374966859414961921, ei (ei)=-2).

Also verified out-of-band that a pinvoke argument of every enum width — byte/sbyte/short/ushort/int/uint/long/ulong — round-trips correctly on wasm with this change.

Note

This pull request was authored with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR fixes a WASM Mono interpreter native-call fast-path issue where P/Invokes with 64-bit underlying enum arguments could be treated as pointer-sized and lead to a call_indirect signature mismatch. It also extends the existing WASM ABI test coverage to validate 64-bit enum round-tripping.

Changes:

• Update interp_type_as_ptr to consider an enum’s underlying type when deciding pointer-compatibility for the fast icall/native-call path.
• Extend the WASM ABI native library and managed test entrypoint with enum:ulong and enum:long P/Invoke coverage.
• Add test assertions validating expected 64-bit enum values are observed in the test output.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/mono/mono/mini/interp/transform.c	Changes enum pointer-compatibility logic used for selecting fast native-call/icall signatures.
src/mono/wasm/testassets/EntryPoints/PInvoke/AbiRules.cs	Adds managed enums and P/Invoke calls that exercise 64-bit enum argument passing.
src/mono/wasm/testassets/native-libs/wasm-abi.c	Adds native functions that accept/return 64-bit integers for the new enum P/Invokes.
src/mono/wasm/Wasm.Build.Tests/PInvokeTableGeneratorTests.cs	Adds assertions to validate the new enum-based ABI outputs are present.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[github-actions]

Workflow state for the Holistic Review Orchestrator.

{
  "version": 5,
  "last_dispatched_commit": "d885c28b812da24dfe636706a34d2982b770fafa",
  "last_dispatched_base_ref": "main",
  "last_dispatched_base_sha": "13c03a4a9a854291f77636b1e15038e65919e68b",
  "last_reviewed_commit": "d885c28b812da24dfe636706a34d2982b770fafa",
  "last_reviewed_base_ref": "main",
  "last_reviewed_base_sha": "13c03a4a9a854291f77636b1e15038e65919e68b",
  "last_recorded_worker_run_id": "29689452087",
  "review_attempt_commit": "",
  "review_attempt_base_ref": "",
  "review_attempt_count": 0,
  "max_review_attempts": 5,
  "review_history_format": "holistic-review-disclosure-v1",
  "review_history": [
    {
      "commit": "d885c28b812da24dfe636706a34d2982b770fafa",
      "review_id": 4730869949
    }
  ]
}

[github-actions]

Holistic Review

Motivation: A P/Invoke whose argument is an enum with a 64-bit underlying type (e.g. enum : ulong) traps on wasm32 with RuntimeError: null function or function signature mismatch. The interpreter's interp_type_as_ptr() classified every enum as pointer-compatible, routing the call through the fast do_icall path that passes each argument as a pointer-sized gpointer. On a 32-bit target that turns a native void(int64_t) into void(int32_t), producing a call_indirect signature mismatch. This is a real, reproducible correctness bug (issue #112262).

Approach: The fix narrows the enum branch: for enums whose underlying type is I8/U8, it defers to interp_type_as_ptr(base_type), which is already width-guarded by #if SIZEOF_VOID_P == 8. All other enums keep their previous classification. This is minimal and well-targeted: behavior only changes for 64-bit enums on 32-bit hosts; smaller enums and all enums on 64-bit hosts are unchanged. The recursion terminates because the underlying type is a primitive. The base_type && guard correctly handles a potential NULL from mono_class_enum_basetype_internal. The claim that AOT (type_to_c) and jiterpreter paths already reduce enums to their underlying type is consistent with those paths not being implicated in the reported crash. The regression test is a sensible extension of the existing WASM ABI test: it adds enum:ulong/enum:long direct pinvokes exercised by the interpreter path where the bug lives, and asserts the full 64-bit round-trip values (18374966859414961921 = 0xFF00FF00FF00FF00 + 1; -2 = -3 + 1), both of which check out.

Summary: LGTM. The root cause analysis is accurate, the fix is minimal and correctly guarded, and the regression coverage exercises exactly the changed cases through the affected (interpreter) path. No actionable findings. As with any Mono interp change, final confidence rests on the WASM CI legs (EnsureWasmAbiRulesAreFollowedInInterpreter) passing.

Note

This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.

Generated by Holistic Review · 64 AIC · ⌖ 10.4 AIC · ⊞ 10K

[maraf]

The milestone is set to 10.0.x, but the PR targets main.

[lewing]

/backport to release/10.0

[github-actions]

Started backporting to release/10.0 (link to workflow run)