Overview
ZJIT now inlines methods and does so by default. This has provided the compiler with more context to make better optimization decisions. However, runtime performance has been limited by having heavy inline frames. gen_push_inline_frame writes out env slots (cme/specval/flags), cfp->self, cfp->ep, the caller's JITFrame and SP publish, the callee entry JITFrame install, SP/CFP register bumps, and spills locals. And, between the push and pop, we need to adjust ec->cfp twice for every inlined frame. This is 12 - 15 memory stores (~1.0–1.2 ns/call on my machine) per inlined ISEQ, most of which we don't need most of the time. Notably, this is nearly the same set of stores a non-inlined call performs; inlining today removes call/return overhead but almost none of the frame cost.
To reduce the overhead of inline frames, I propose we adopt virtual frames. With virtual frames we wouldn't push anything onto the stack. Instead, we'll store compile-time metadata on the JITFrame that will allow us to materialize the frames as needed (e.g., frame-walking code or side-exit to the interpreter). This will drastically reduce the overhead in executing an inlined ISEQ as long as we stay on the happy path. However, frame materialization is not cheap and due to the way CRuby expects the stack to be laid out, we need to materialize the entire chain if even one inlined ISEQ needs to materialize.
None of this is intended to be user-visible. Backtraces and profilers (rb_profile_frames consumers such as stackprof and vernier) continue to report the logical call graph, with inlined callees appearing as ordinary frames, same as today. Anything that observes frames in-thread (exceptions, caller, binding, etc.) will trigger materialization and see real frames. With that said, I think we should make an exception for crash reports, where we could annotate virtual frame use to make debugging the VM easier.
Status
We're very much at the "does this make sense?" stage and exploring different ideas. I've been pushing my work to a branch periodically. What's up there isn't necessarily the most recent work because I have multiple branches locally exploring different options. To get to the meat of the problem I needed to satisfy the VM in unsatisfactory ways — there's a lot of "this lets me make forward progress even if it's broken in cases X, Y, and/or Z". Feedback at this stage is most useful on the overall approach and the unresolved questions below, rather than on the code itself. Once we settle the big questions, I'll start the real implementation and push to a draft PR.
Related Work
TruffleRuby uses virtual frames and materializes frames should they escape. So, we know that it's possible to fully handle Ruby semantics using this model. Consequently, I looked at Truffle's VirtualFrame for inspiration. While I think it's a useful guide at a high level, the underlying design of ZJIT and Graal differ such that we'd need to implement them differently. In particular, Truffle's VirtualFrame is a heap-allocated object that can be elided via partial escape analysis. We currently do not have any robust form of escape analysis in ZJIT and we need to consider the needs of the CRuby interpreter. Another big difference is that Truffle prohibits cross-thread stack inspection outside of safepoints so frame data only needs to be coherent at polls. CRuby, on the other hand, runs rb_profile_frames in a signal handler and our frame data must be accurate to any observer.
Design-wise, I think we'd do well to hew closely to how Truffle and Graal deopt inlined frames. But, in implementation, I think we should follow what HotSpot does by way of compiledVFrame and ScopeDesc compile-time scope descriptors.
@XrXr has a draft PR for storing locals in a stack map so that we don't need to spill them. This is complementary work that will be crucial to fully virtualize frames.
Experiments
I did a spike on a rough implementation. On an inline-call microbenchmark, calls drop from ~3.5 ns to ~2.7 ns. This was measured on a release build of ZJIT run on an M4 Pro MacBook Pro. The reduction is real, but the magnitude should be taken with a grain of salt. With the lobsters benchmark we see ~32.5 M inlined sends per iteration. The ~1 ns savings on each call would amount to an 8% performance improvement.
Alas, what I actually saw was a 7.5% reduction in performance in lobsters. I'm still investigating and I'm not sure everything is actually working as expected. E.g., I found the number of recompilations shot through the roof. It turned out we were reading a stale CFP, which led to reading the incorrect profile, which led to the recompilation being targeted at the wrong ISEQ. The fix there could be applied to master so I extracted it and opened a PR (NB: the 7.5% drop is without this fix applied).
Early results show that we need to rethink what's expensive and what's acceptable in a world with virtual frames. E.g., dynamic sends require the calling frame's CFP so any send that remains unspecialized inside an inlined ISEQ will have to materialize its chain of frames back to the root, every time it's invoked. The simplest solution is to not inline such ISEQs, but then we lose the benefit inlining provides today: additional context for optimization.
Currently, when we side-exit, we update the profile but don't track anything else we have learned. Should the root method become eligible for compilation a second time we will inline the same set of methods we did the first time (modulo threshold/budget cutoffs from refined type info). If we recompiled because an inlined ISEQ unconditionally side-exits, we will inline that ISEQ again and repeat the same mistake.
I've prototyped a new compilation approach: at the end of the optimization pipeline, scan for any dynamic sends inside of an inlined method, and if one appears use physical frames. That's effectively what we have today. Otherwise, we should speculate that virtual frames will work. If that assumption is wrong and there's a need to recompile, we make a note to compile with physical frames on the next pass. In the spike, ~89% of the ISEQs forced onto physical frames this way were at a single level of inlining, so the fallback usually matches today's compilation exactly.
Unresolved Questions
Special Variable Routing
$~ and $_ are broken in my initial implementation, since they attach to a frame's svar slot through ec->cfp. C functions that set the backref (regexp matches, String#match) from inside a virtual frame are writing to the physical frame's svar instead of the logical callee's. With virtual frames, that's the root method's frame rather than the inlined ISEQ's frame, which means the values are visible in places they would not have been if no inlining had occurred. That's an incompatible behavioral change.
We can scan for getspecial/setspecial and use physical frames in those cases. But, the write side is challenging because that all happens on the C side and may be reached through direct sends. I don't yet know what the solution should look like. An ugly, but simple, solution may be to check the target of a direct C call to see if it's one of the handful of methods that invoke rb_backref_set or rb_lastline_set. Of course, that falls apart if native extensions are directly invoking those functions as well (my kingdom for a real extension API). Indirect calls aren't a problem because we already use physical frames in that case.
Recompilation Churn
The initial spike showed shape-guard side exits inflating ~16x with virtual frames enabled, almost entirely from inlined code. The stale-CFP fix in #18108 likely accounts for some or all of this, but the benchmark numbers above predate it and need re-measuring once it lands.
Design
To be decided pending exploration outcomes.
Overview
ZJIT now inlines methods and does so by default. This has provided the compiler with more context to make better optimization decisions. However, runtime performance has been limited by having heavy inline frames.
gen_push_inline_framewrites out env slots (cme/specval/flags),cfp->self,cfp->ep, the caller's JITFrame and SP publish, the callee entry JITFrame install, SP/CFP register bumps, and spills locals. And, between the push and pop, we need to adjustec->cfptwice for every inlined frame. This is 12 - 15 memory stores (~1.0–1.2 ns/call on my machine) per inlined ISEQ, most of which we don't need most of the time. Notably, this is nearly the same set of stores a non-inlined call performs; inlining today removes call/return overhead but almost none of the frame cost.To reduce the overhead of inline frames, I propose we adopt virtual frames. With virtual frames we wouldn't push anything onto the stack. Instead, we'll store compile-time metadata on the
JITFramethat will allow us to materialize the frames as needed (e.g., frame-walking code or side-exit to the interpreter). This will drastically reduce the overhead in executing an inlined ISEQ as long as we stay on the happy path. However, frame materialization is not cheap and due to the way CRuby expects the stack to be laid out, we need to materialize the entire chain if even one inlined ISEQ needs to materialize.None of this is intended to be user-visible. Backtraces and profilers (
rb_profile_framesconsumers such as stackprof and vernier) continue to report the logical call graph, with inlined callees appearing as ordinary frames, same as today. Anything that observes frames in-thread (exceptions,caller,binding, etc.) will trigger materialization and see real frames. With that said, I think we should make an exception for crash reports, where we could annotate virtual frame use to make debugging the VM easier.Status
We're very much at the "does this make sense?" stage and exploring different ideas. I've been pushing my work to a branch periodically. What's up there isn't necessarily the most recent work because I have multiple branches locally exploring different options. To get to the meat of the problem I needed to satisfy the VM in unsatisfactory ways — there's a lot of "this lets me make forward progress even if it's broken in cases X, Y, and/or Z". Feedback at this stage is most useful on the overall approach and the unresolved questions below, rather than on the code itself. Once we settle the big questions, I'll start the real implementation and push to a draft PR.
Related Work
TruffleRuby uses virtual frames and materializes frames should they escape. So, we know that it's possible to fully handle Ruby semantics using this model. Consequently, I looked at Truffle's
VirtualFramefor inspiration. While I think it's a useful guide at a high level, the underlying design of ZJIT and Graal differ such that we'd need to implement them differently. In particular, Truffle'sVirtualFrameis a heap-allocated object that can be elided via partial escape analysis. We currently do not have any robust form of escape analysis in ZJIT and we need to consider the needs of the CRuby interpreter. Another big difference is that Truffle prohibits cross-thread stack inspection outside of safepoints so frame data only needs to be coherent at polls. CRuby, on the other hand, runsrb_profile_framesin a signal handler and our frame data must be accurate to any observer.Design-wise, I think we'd do well to hew closely to how Truffle and Graal deopt inlined frames. But, in implementation, I think we should follow what HotSpot does by way of
compiledVFrameandScopeDesccompile-time scope descriptors.@XrXr has a draft PR for storing locals in a stack map so that we don't need to spill them. This is complementary work that will be crucial to fully virtualize frames.
Experiments
I did a spike on a rough implementation. On an inline-call microbenchmark, calls drop from ~3.5 ns to ~2.7 ns. This was measured on a release build of ZJIT run on an M4 Pro MacBook Pro. The reduction is real, but the magnitude should be taken with a grain of salt. With the lobsters benchmark we see ~32.5 M inlined sends per iteration. The ~1 ns savings on each call would amount to an 8% performance improvement.
Alas, what I actually saw was a 7.5% reduction in performance in lobsters. I'm still investigating and I'm not sure everything is actually working as expected. E.g., I found the number of recompilations shot through the roof. It turned out we were reading a stale CFP, which led to reading the incorrect profile, which led to the recompilation being targeted at the wrong ISEQ. The fix there could be applied to master so I extracted it and opened a PR (NB: the 7.5% drop is without this fix applied).
Early results show that we need to rethink what's expensive and what's acceptable in a world with virtual frames. E.g., dynamic sends require the calling frame's CFP so any send that remains unspecialized inside an inlined ISEQ will have to materialize its chain of frames back to the root, every time it's invoked. The simplest solution is to not inline such ISEQs, but then we lose the benefit inlining provides today: additional context for optimization.
Currently, when we side-exit, we update the profile but don't track anything else we have learned. Should the root method become eligible for compilation a second time we will inline the same set of methods we did the first time (modulo threshold/budget cutoffs from refined type info). If we recompiled because an inlined ISEQ unconditionally side-exits, we will inline that ISEQ again and repeat the same mistake.
I've prototyped a new compilation approach: at the end of the optimization pipeline, scan for any dynamic sends inside of an inlined method, and if one appears use physical frames. That's effectively what we have today. Otherwise, we should speculate that virtual frames will work. If that assumption is wrong and there's a need to recompile, we make a note to compile with physical frames on the next pass. In the spike, ~89% of the ISEQs forced onto physical frames this way were at a single level of inlining, so the fallback usually matches today's compilation exactly.
Unresolved Questions
Special Variable Routing
$~and$_are broken in my initial implementation, since they attach to a frame's svar slot throughec->cfp. C functions that set the backref (regexp matches,String#match) from inside a virtual frame are writing to the physical frame's svar instead of the logical callee's. With virtual frames, that's the root method's frame rather than the inlined ISEQ's frame, which means the values are visible in places they would not have been if no inlining had occurred. That's an incompatible behavioral change.We can scan for
getspecial/setspecialand use physical frames in those cases. But, the write side is challenging because that all happens on the C side and may be reached through direct sends. I don't yet know what the solution should look like. An ugly, but simple, solution may be to check the target of a direct C call to see if it's one of the handful of methods that invokerb_backref_setorrb_lastline_set. Of course, that falls apart if native extensions are directly invoking those functions as well (my kingdom for a real extension API). Indirect calls aren't a problem because we already use physical frames in that case.Recompilation Churn
The initial spike showed shape-guard side exits inflating ~16x with virtual frames enabled, almost entirely from inlined code. The stale-CFP fix in #18108 likely accounts for some or all of this, but the benchmark numbers above predate it and need re-measuring once it lands.
Design
To be decided pending exploration outcomes.