introduce a wasm jit#151
Open
kevodwyer wants to merge 153 commits into
Open
Conversation
Replaces /Users/james/Boxedwine2/... absolute paths with relative paths (../../../platform/sdl/...) so the project builds regardless of where the repo is cloned.
Adds a new JIT backend (source/emulation/cpu/wasm/) that compiles each
basic block to a standalone WebAssembly module at runtime, instantiates
it via WebAssembly.Module + WebAssembly.Instance (synchronous, as in the
QEMU wasm64 TCG backend), and stores the resulting function as a
wasmTable index in DecodedOp::pfnJitCode.
Architecture
------------
- WasmEmitter (wasmEmitter.h/.cpp): low-level WASM binary builder.
Handles type, import, function, export and code sections; LEB128
encoding; and all instruction opcodes needed for x86 emulation.
- JitWasmCodeGen (jitWasmCodeGen.h/.cpp): implements the full JitCodeGen
virtual interface. Key design decisions:
* Registers: 16 WASM local variables (i32). Locals 1-8 hold GP regs
eax-edi; loaded from the CPU struct on first use and written back on
block exit / explicit sync.
* CPU state access: i32.load / i32.store with cpu_ptr (local 0) as
base and offsetof() offsets — no MMU indirection needed.
* Common arithmetic and logic (add, sub, and, or, xor, shl, shr,
sar, rot, neg, not, cmp, mov) are inlined as WASM instructions.
* Complex operations (rcl/rcr, shld/shrd, mul/div, bswap, FPU/SSE)
fall back to the existing emulateSingleOp() helper.
* Memory read/write uses C++ helper functions imported via the
wasmTable, mirroring the QEMU helper-import pattern.
- boxedwine_wasm_instantiate / boxedwine_wasm_call_block: EM_JS
functions that compile the generated binary and call the result.
- wasmStartJITOp: the static OpCallback installed as
process->startJITOp; reads the table index from pfnJitCode and calls
boxedwine_wasm_call_block.
Build system
------------
New `jit` target in project/emscripten/Makefile:
make jit
Adds -DBOXEDWINE_WASM_JIT, -sALLOW_TABLE_GROWTH, -sALLOW_MEMORY_GROWTH
and the wasm/ source files. boxedwine.h automatically sets
BOXEDWINE_JIT when BOXEDWINE_WASM_JIT is defined.
- Fix getZF()/getCF() to delegate to JitCodeGen (CPU has no .zf field) - Fix DF_MASK -> DF (the correct define in cpu.h) - Make appendULEB128/appendSLEB128 public in WasmEmitter (needed by the static emitMemArg helper in wasmEmitter.cpp) - Fix Makefile source selection: use filtered find to exclude the native JIT backends (x32/, armv8/) from WASM JIT builds, and exclude wasm/ from normal builds, to avoid duplicate startNewJIT() definitions - Remove incReg/decReg overrides (JitCodeGen base provides them) - Add MarkJumpLocation/Goto back to header (pure virtual in Jit base) - Add clearJitBlock() for WASM (releases wasmTable entries on eviction)
Mirrors the existing test target (-D__TEST, shelltest.html, --emrun) but
adds -DBOXEDWINE_WASM_JIT so the CPU test suite exercises JIT-compiled
blocks rather than the normal interpreter.
make testJit → Build/TestJit/boxedwine.html
The WASM JIT was corrupting upper register bits on 8/16-bit writes, miscompiling AH/CH/DH/BH as ESP/EBP/ESI/EDI, returning garbage for getCF() (nakedCall is a no-op under WASM), and double-freeing shared wasmTable indices when invalidating a compiled block. After these fixes, 136 CPU tests pass under testJit via node. - popToReg takes a JitWidth and merges with the existing local for b8/b16 writes so the unreferenced half of a register is preserved. - pushRegValue honors JitReg::isHigh (shr 8) so AH references extract the correct byte. - getReg8/getReadOnlyReg8/getTmpReg8 map reg index 4-7 to the low-half local with isHigh=true, matching CPU::offsetofReg8 semantics. - getCF/getZF override the JitCodeGen versions (which rely on nakedCall) to call C++ helpers that use the lazy-flag machinery. - commitJIT marks all block ops with OP_FLAG_JIT and routes only the first op through startJITOp; clearJitBlock dedupes the shared table index. The Makefile change wires up testJit to pull test sources instead of the main SOURCES list so the CPU test suite actually compiles in.
Three issues surfaced after the first round of fixes: 1. Lazy m_gpLoaded/m_segLoaded tracks compile-time state but doesn't respect dynamic control flow — a local loaded inside one branch leaves the other branch reading uninitialized. Added branchBoundary() which syncs dirty regs and invalidates both caches; called at the entry of every IfXxx and at StartElse/EndIf so each branch sees fresh state. 2. getCondition was returning cpu->flags directly, ignoring lazy flags, so setcc/conditional-jumps saw stale CF/OF/etc. Added 16 per-condition helpers (wasmHelper_cond_O..NLE) that call the C++ getXF() accessors and stash 0/1 in cpu->tmpReg; getCondition emits a single call and loads the result. Using one helper per condition rather than a single parameterised helper avoids clobbering lazy-flag scratch fields (src.u32/dst.u32/result.u32) with the condition index. Also added wasmHelper_fillFlags + fillFlags override since nakedCall no-ops can't drive common_fillFlags. 3. JumpIfCondition and JumpInBlock both treated the in-block jump as a pure block exit without writing the target EIP, which hung testJO (dispatcher re-entered the same block forever). Both now writeEip() before the blockExit so fetchNextOp resumes at the right op. testJit now runs past the previous Pop-Sp/JO stalls. 154 tests pass, then a remaining bug around JO 270 trips a page fault; separate fix.
…ails When boxedwine_wasm_instantiate returned -1 we were returning from commitJIT without touching op->pfn, which leaves it as firstDynamicOp — on the next run that re-enters startNewJIT (which sees runCount > 0 and skips), then calls op->pfn == firstDynamicOp again. Infinite loop. Set op->pfn to the normal interpreter's implementation for every op in the block on instantiation failure so the dispatcher runs the op directly instead of looping back through the JIT entry point.
Three related fixes for conditional-jump correctness and memory behavior: - JumpIfCondition/JumpInBlock were writing the *linear* target address (CS.base + offset) straight into cpu->eip.u32, which stores only the offset — at the next dispatch getEipAddress() re-added CS.base and overflowed 32-bit, landing at 0xA0000004 for test-harness code at 0xD0000000. Subtract cpu->seg[CS].address (known at compile time in tests) before writing. JO 070 and friends now run past the crash. - storeLazyFlagsDest/Src/Result widened to JitWidth::b8 for isHigh regs so that a high-byte operand writes a single byte into dst/src/result — matching what the lazy-flag calculators read via ->u8. - Added KMemory::clearOpCache() JIT-sweep: every newInstruction() call clears the op cache, but nothing was reclaiming the wasmTable indices the deleted ops pointed at, so each test leaked one compiled WASM module. Added DecodedOpCache::collectAllJitBlocks() to gather the pfnJitCode values and hand them to clearJitBlock() before the ops get freed. - Bumped testJit MAXIMUM_MEMORY to 3 GiB so remaining memory growth in V8's internal module state doesn't trip default 2 GiB ceiling.
Pushes testJit from 154/67 to 421/2. Remaining failures: FPU 2d9/2df. - `direct_cmp`/`direct_test` now stage lazy flags (dst/src/result/lazyFlagType) since WASM has no host EFLAGS to consume afterwards. Fixes all J-cc tests. - `getTmpSegAddress` returns a fresh scratch instead of aliasing the cached seg local, so callers can `addValue` without corrupting the cache. - `read`/`write` on `MemPtr` with emulated addresses now flatten (rm + sib*lsl + offset) into a scratch and call the mem helper, instead of falling through to `readHost` which loads from linear memory at the virtual x86 address. - `readHost`/`writeHost` respect the MemPtr `offset` field in the WASM memarg, so `readHost(op + offsetof(DecodedOp, next))` etc. work. - `blockNext1`/`blockNext2` emit a proper block exit (writeEip + call fetchNextOp + return) and subtract CS.base, mirroring the base class. - `sarValue`/`sarReg` for b8/b16 sign-extend the source before `i32.shr_s` so signed right shifts work on sub-word widths (CWD, etc.). - Route shift/rotate, 32-bit string ops (MOVSB/STOS/LODS/CMPS/SCAS), IMul/Dimul, PushA/PopA, and LoopNZ/LoopZ through `emulateSingleOp()`. For the branch loops (LoopNZ/LoopZ), also emit `blockExit()` afterwards so the JIT doesn't keep running the next compiled op after the branch. - Add per-width mem helpers and dedicated `cpu->memHelperAddr`/` memHelperValue` scratch fields so helper calls don't trample lazy-flag state in `src.u32`/`dst.u32`.
Pushes testJit from 421/2 to 787/11. The remaining failures are: FPU 2d9/2df (pre-existing 32-bit FPU), CallJw/CallJd/JmpJw/JmpJd (branches now run via normal CPU but trip a separate stack assertion), XADD 1c0/3c0/1c1/3c1, CMPXCHG8B 3C7. The previous CallJw kpanic that prevented running ~400 later tests is gone. The base codegen for these branch ops uses blockNext1/JumpInBlock with the linear target. WASM doesn't have host EIP, so the JIT block doesn't update cpu->eip.u32 between ops; runNextSingleOp would then decode at a stale EIP. Set EIP to this op's offset before emulateSingleOp, then blockExit so the dispatcher resolves the next op fresh from whatever EIP the normal CPU left. Also applies the same writeEip-then-emulate-then-blockExit shape to LoopNZ/LoopZ.
Pushes testJit from 787/11 to 791/7. The base codegen for these does dynamic_RM_WriteM/dynamic_RR_WriteBoth with a writeback path that doesn't round-trip cleanly to WASM. Routing them through normal CPU clears 4 XADD failures; CMPXCHG8B 3C7 still fails (the 64-bit variant needs more than the per-op emulation can provide here).
The real root cause of the CallJw/JmpJw/Loop branch crashes was that JitWasmCodeGen::postCompile didn't call JitCodeGen::postCompile, so this->currentEip never advanced between ops in a block. Every branch that computed its target as currentEip + op->len + op->imm therefore used the block-start currentEip and landed N bytes too early — for testCallJw0x0e8 with 3 bytes of add ax,10 before the call, that meant the call target was 3 bytes shy and the dispatcher decoded into the 0x123-byte sea of 0xcd padding (INT 0xcd → SIGILL → kpanic). Also: getIfJumpSize() now returns 0 (WASM uses structural control flow, the validator catches mismatched if/end on its own) so the base class postCompile's open-if-statement check passes. With this fix the CallJw/CallJd/JmpJw/JmpJd/LoopNZ/LoopZ overrides that fell back to emulateSingleOp() are no longer needed; their native codegen is correct now. testJit goes from 791/7 to 799/5. The remaining failures (FPU 2d9/2df, CMPXCHG8B 3C7, two Self Modifying Code variants) are real bugs to chase separately.
…verrides for div/mul/idiv/bsf/bsr Pushes testJit from 799/5 to 803/1 — only "Self Modifying Code Same Block(Next)" remains. Two fixes: 1. emulateSingleOp now writes cpu->eip.u32 to the current op's CS-offset when this isn't the first op in the block. The JIT's inline ops don't update cpu->eip mid-block (only block exit does), so a subsequent emulateSingleOp's runNextSingleOp would call getNextOp() against a stale eip and decode the wrong instruction. testFRNDINT/testFSTPFloat in 32-bit and CMPXCHG8B 3C7 hit this — `setRounding` emits inline JIT AND/OR mem ops between FNSTCW and FLDCW; without writeEip, FLDCW's helper re-decodes at the AND-mem position. Conditional on `currentEip != startingEip` so pure-emulate blocks (e.g. FPU tests that go all-helpers) don't pay the per-op writeEip cost — and so tests like Xchg that compile pure-JIT pushf/pop don't hit a perf regression that pushed them past the OOM line. 2. Op-level overrides for MUL/DIV/IDIV/BSF/BSR. The JIT's helper-style `mulReg`/`divRegRegWithRemainder`/`bsfReg`/etc. were all stubbed as emulateSingleOp by an earlier commit, but they're called *inline* from `Jit::div8` (etc.) inside the same op — running runNextSingleOp each time would dispatch the entire instruction multiple times. Override the dynamic_*R8/E8/etc. ops directly so the whole op goes via emulateSingleOp once.
Detect writes that clobber the bytes of the currently-running JIT block
and exit before stale compiled code runs. blockEnter helper captures the
active block at entry; checking write helpers notice when
removeCodeBlock cleared our pfnJitCode and set wasmJitBailout; codegen
emits an if(bailout){writeEip(next); blockExit();} after every inline
write. commitJIT now records blockLen/blockOpCount so removeCodeBlock
can locate the block during iterateOps.
All 822 tests pass under emscripten testJit.
Drop emulateSingleOp overrides for the b32 shift/rotate ops and the b8/b16 shl/shr/sar — the base class already emits native i32.shl/shr_u/ shr_s/rotl/rotr with proper lazy-flag plumbing and 5-bit count masking. WASM rotates only stay emulated for b8/b16 because i32.rotl operates on the full 32-bit word and produces wrong results when narrowed; sar already sign-extends in sarReg/sarValue. Also clarifies why string ops still emulate: base-class movsr uses Goto for a backward branch which is a no-op in the WASM JIT. 32 stub overrides removed, no test regressions (still 822/822).
Adds a section to the file header listing the categories of ops that are routed through emulateSingleOp() and the constraint each one paper- over (b8/b16 rotate width, missing WASM backward branch, lazy CF chaining, atomics, etc.). Saves the next person from rediscovering why each fallback is load-bearing.
Project-level guidance for Claude Code: build commands per platform, test workflow, CPU emulation overview (normal + JIT, including the new WebAssembly JIT backend with BOXEDWINE_WASM_JIT and the testJit target), key compilation flags, and debugging tips.
Default impls forward to MarkJumpLocation / no-op so x86/ARM behavior is unchanged. The WASM JIT will override these to wrap loop bodies in structural `loop` ... `end` pairs since WASM has no arbitrary backward branches.
Replace MarkJumpLocation()+Goto() in movsr/cmpsr/stosr/lodsr/scasr with LoopBegin()+Goto()+LoopEnd(). Behavior is unchanged for x86/ARM because LoopBegin defaults to MarkJumpLocation and LoopEnd defaults to no-op. Sets up the WASM JIT to override these with structural loop/end pairs.
WasmEmitter tracks open structural blocks via m_ctrlDepth (incremented on emitIf/emitBlock/emitLoop, decremented on emitEnd, reset in beginFunction). LoopBegin emits `loop` and returns a tagged token encoding the loop frame's depth. Goto checks the tag: if it's a LoopBegin token, emit `br <currentDepth - loopDepth>` for a backward branch; otherwise (legacy MarkJumpLocation buffer offset), do nothing as before. LoopEnd emits the matching `end`. This gives the WASM JIT a real backward-branch primitive while leaving the existing MarkJumpLocation/Goto path intact for callers that still use the byte-offset patching API. String-op overrides still route through emulateSingleOp; the next commits will remove them one variant at a time.
Final string-op family. The header doc block can now drop the "string ops" entry from "still emulated" since all of movs/cmps/stos/ lods/scas (all widths, both ea16 and ea32, rep'd and non-rep'd) route through native LoopBegin/Goto/LoopEnd code now.
Adds two parallel U32[K_NUMBER_OF_PAGES] arrays in KMemoryData and a matching pair of U32 fields on CPU (wasmReadPageBaseArray / wasmWritePageBaseArray). The arrays hold a 32-bit linear-memory offset to each page's RAM base, or 0 if the page can't be directly accessed (CodePage, RO/no-perm, on-demand-not-yet-allocated). They're populated in onPageChanged alongside the existing readCache/writeCache (the host-exception-based cache). wasmHelper_blockEnter caches the array bases on the CPU struct so JIT codegen can emit `i32.load (wasmReadPageBaseArray + (addr>>12)*4)` without re-resolving cpu->memory->data per access. CodePages set canWriteRam=false → wasmWritePageBase[CodePage] = 0, so the future inline fast-write path will always slow-path writes to JIT'd code, preserving the SMC bailout. No codegen change yet — just the data plumbing. testJit suite still 0 tests FAILED.
JitWasmCodeGen::read now emits an inline check before the helper call:
entry = wasmReadPageBase[addr >> 12]
if (entry == 0 || (addr & 0xfff) > 0x1000 - width) {
slow path: existing helper-based load
} else {
tmp = i32.load{8u,16u,_}(entry + (addr & K_PAGE_MASK))
}
Boundary check is omitted for b8 (every byte is in some page). Saves
the dirty/loaded register trackers around the if so the no-fast-path
arm doesn't compile-time-clear m_gpDirty. customMemoryOp callbacks
still slow-path unconditionally.
The encoding is `entry = host_base` (raw linear-mem offset of the
page's RAM, or 0 for no-access). Adding `addr & K_PAGE_MASK` gives
the host pointer to the requested byte. We can't use the
host-exception cache trick of `(host_base - virt_page_base)` because
under WASM there's no SIGSEGV to catch the sentinel-decoded low
addresses.
testJit suite: 0 tests FAILED.
JitWasmCodeGen::write now inlines the same entry/boundary check as
read but uses wasmWritePageBaseArray. On a hit, emits
i32.store{8,16,_}(entry + (addr & K_PAGE_MASK), src) and skips the
bailout-checking helper.
SMC same-block bailout is preserved by construction:
CodePage::canWriteRam returns false → wasmWritePageBase[CodePage] == 0
→ writes to JIT'd code pages always slow-path → existing
writeCheckHelper + emitBailoutCheck runs unchanged. Plain RWPage
writes can fast-path because they cannot land on an active block.
testJit suite: 0 tests FAILED. All SMC, Xchg, and string-op variants
pass.
Dispatch WASM JIT warmup Callback ops through the normal opcode table, and refetch nextOp from the live decoded-op cache before NormalCPU dereferences it.
Track active wasm JIT blocks so self-modifying-code invalidation can request a bailout, including before branch ops where execution must resume at the branch itself. Restrict native JIT in-block jumps to decoded instruction boundaries and add a regression test for overlapping direct jump targets. Also guard native x86 lock helpers for non-multithreaded builds, force tests to one worker in single-threaded builds
Collaborator
Author
|
JIT target now achieves 155 in benchmark. |
…ite-invalidated code bytes are marked dynamic immediately and stay in the interpreter. That keeps the JVM’s generated code out of the ST WASM JIT. As MT WASM JIT can run Solitaire and is only 20% slower than ST WASM JIT, that is another option for running JVM programs.
…o contain all 4 targets
…x jenkins server firefox runs out of memory. I changed jenkinsfile to use chrome and that worked on the linux jenkins server but only when not using headless. I'm not sure why headless messes it up, but at least its a start
…a larger wasm fie.
…. It was running out of memory because of the large number of modules. Seems like Firefox has more overhead per module. Will now batch wasm compiled blocks so that more than one jit block can be used in a wasm module. For MDK Perf, the number of wasm modules went from 20k to 5k. Currently this is only for single threaded jit.
…. It was running out of memory because of the large number of modules. Seems like Firefox has more overhead per module. Will now batch wasm compiled blocks so that more than one jit block can be used in a wasm module. For MDK Perf, the number of wasm modules went from 20k to 5k. Currently this is only for single threaded jit.
…e this code went in as a fix but it was actually a band aid. By reverting it performance has improved on native and wasm platforms. I tested all the demos with all 4 wasm builds, including java and everything still works, so the real fix must have been applied later.
…s and will also try to combine multiple jits into a single module. Now MDK Perf runs on Firefox without running out of memory when using the multi-threaded JIT. Module usage was reduced by about 80%
# Conflicts: # source/test/cpu/testCPU.cpp # source/test/cpu/testCPU.h
…ecoded operations are not cleared. Retire owner broker/table slots safely, preserve generated CPU offsets, and add regression coverage.
…Jit targets. Will now run all 4 wasm targets through automation with abiword in jenkins.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A wasm JIT for single-threaded and multi-threaded emscripten builds.
makefile targets: jit and multiThreadedJit. test target: testJit
There are ample opportunities for future improvements
AOT mode - adding url param jit-record=true will add a button to ui. Clicking said button will create a zip file of generated wasm modules. This can then be run through the script boxedwine-wasm-jit-cache-pipeline.mjs to produce an optimised zip file of wasm modules that can be used alongside other server files in subsequent runs (rename back to app-jit-modules.zip).
ie
flat mode - run binary_en.js to optimise wasm modules
node boxedwine-wasm-jit-cache-pipeline.mjs --flat app-jit-modules.zip app-jit-modules.flat.zip
piped - try to group related wasm modules together and call internal functions directly (better optimisations coming)
node boxedwine-wasm-jit-cache-pipeline.mjs app-jit-modules.zip app-jit-modules.piped.zip
Summary by CodeRabbit
Release Notes