Skip to content

Commit 2d26d5b

Browse files
committed
ZJIT: Specialize bar(...) in standalone forwardable ISEQs
`Function::specialize_send_forward` could only expand a `bar(...)` whose enclosing `def foo(...)` frame had been inlined, because that is what makes the forwarded callinfo -- the `...` local -- a compile-time value. A forwardable ISEQ that is *entered* megamorphically is never inlined, so on lobsters 673K of the 776K dynamic `sendforward`s per run came from 186 sites with no inlining context at all. Those are the ActiveRecord reflection delegators, and every one of them was reaching `rb_vm_sendforward`. Profile the callinfo instead. `vm_caller_setup_fwd_args` reads it as `TOPN(0)`, and a site that only ever forwards one *packed* callinfo can guard it with a single comparison: a packed one carries `RUBY_FIXNUM_FLAG`, so it is an immediate whose bits are the whole of `(mid, flag, argc)` with no keyword table and nothing for the GC to keep alive. Heap `imemo_callinfo`s -- what a keyword-carrying caller produces -- would need a root, and are recorded as `None` so that a site mixing the two reads as polymorphic rather than stable. Past the guard the merge is the one the inlined path already does. The difference is where the forwarded arguments come from: they are not values this compilation produced, they sit in the frame extension below the local table, which is where `vm_adjust_stack_forwarding` reads them from and where they stay for the life of the frame. The guarded callinfo fixes their count, so they can be loaded relative to the frame's local EP. Two things a standalone frame cannot know statically: * Whether it was given a block. The expansion asks for the conservative answer first -- "was given one", which keeps `&nil`-rejecting callees and callees that would warn about an unused block off the direct send -- and falls back to guarding that the frame's block handler is `VM_BLOCK_HANDLER_NONE`. That second form is what makes the common blockless delegator specializable, and it is not redundant with the callinfo guard: a literal block at the call site clears `VM_CALL_ARGS_SIMPLE` and so shows up in the callinfo, but a block arriving through `public_send` does not. * Nothing, for chained forwarding. `bar(...)` whose target is itself a `def bar(...)` stays dynamic and now says so with its own counter: the callee's `...` local has to *receive* a callinfo, and no `rb_callinfo` describes the merged argument list. A guard miss exits to the `sendforward` itself with the site's original stack, which is exactly what `vm_adjust_stack_forwarding` expects, and asks for a recompile so a site that turns out to be polymorphic stops speculating. Where the newly expanded sends land is worth knowing, so `sendforward_target_not_specialized` is split into the three reasons the merged call can still fall back for. On lobsters: ccall_rb_vm_sendforward 775,633 -> 495,311 (-36%) sendforward_not_specialized 673,409 -> 113,662 ...target_not_specialized 102,224 -> 0 ...recv_type - -> 156,491 ...method_type - -> 208,420 ...complex_args - -> 16,738 dynamic_send_count 10,678,566 -> 10,429,465 code_region_bytes 29,290,496 -> 29,491,200 280K forwarding calls per run become direct sends. The largest remaining bucket is a merged call whose target is a C function rather than an ISEQ, which this pass does not build a `CCall` for yet. Ported from zjit/all (0fa6d74). Adaptations for this branch: * `IseqProfile`'s side tables are plain `HashMap` fields here, not `Option<Box<..>>`, so `forwarded_cis` is one too and `forwarded_cis_mut` hands out `&mut self.forwarded_cis`. There is no `heap_size()` accounting on this branch, so the memory-reporting hunk is dropped; the two-bucket `FORWARDED_CI_DISTRIBUTION_SIZE` is kept, since it is what makes "did this site ever forward a second callinfo" answerable. * `FieldName::StackSlot`, which only names the load for HIR dumps, comes from an excluded commit and is added here. * `SendDirectData` has no `guard_state` field on this branch. [reorder port note] Cherry-picked from stack position 49 onto position 12, so the diff carried context from ~37 commits that are not underneath it here. Dropped as foreign context: the `super` specialization helpers (`emit_super_call_guards`, `load_super_lep`, `load_frame_method_entry`, `load_super_block_handler`, `emit_super_chain`, `emit_specialized_super`) and `get_super_method_entries`/`super_cme_mut`; the block-handler and block-fallback profiling tables (`block_handlers`, `block_fallbacks`, `observe_block_fallback`, `get_block_handlers`, `block_handlers_mut`); the `send`/`__send__` method-name tables (`send_mid_mut`, `get_send_method_names`); `splat_lengths_mut` (this tree reaches the field directly); and the `SplatLengthChanged`/`SplatLastRuby2Keywords` side-exit reasons and their counters. `FieldName::StackSlot` already exists here, so its doc comment was kept as the tree had it. Adapted: `assume_cme_for_send`/`assume_no_singleton_classes_for_send` and their `ancestor_class` argument do not exist, so the commit step is the tree's explicit `assume_no_singleton_classes` plus an `Insn::PatchPoint { Invariant::MethodRedefined { .. } }`, as the commit below this one does. Weakened, deliberately and for the same reason the commit below this one rejects a block-carrying `bar(...)`: `build_send_direct_args` takes four arguments here (no `block_arg_passthrough`), `SendDirectData` has no `block_arg` field, and `BlockHandler::BlockArg` is `unreachable!()` in `gen_send_iseq_direct`, so a run-time block handler cannot be handed to a `SendDirect`. The `candidates: &[bool]` restructuring is kept, but `true` is never a candidate: an inlined frame that was given a block gets an empty candidate list and keeps its `SendForward` (the pre-existing rejection, now reported as `SendForwardComplexArgs`), and the standalone case only ever takes the blockless form -- which is the one the commit's own design guards with `VM_BLOCK_HANDLER_NONE`, so the common blockless delegator still specializes. Restoring the passthrough arm is a one-line change to `candidates` once `SendDirectData` grows a `block_arg`. `codegen_tests.rs` is kept whole; only the comment on `test_standalone_forwarder_carries_a_block` was rewritten, because on this branch that case expands and then always misses the block guard instead of passing the handler through. [reorder port note 2] On zjit-lobsters4: `FieldName::StackSlot` no longer exists upstream -- the frame-extension loads use this base's `FieldName::Stack(i)` with the forwarded argument's index. `build_send_direct_args` calls pass this base's fifth `block_arg_passthrough: false` argument. profile.rs keeps the front block's `block_handlers` table alongside the new `forwarded_cis` at every keep-both site.
1 parent 7fbf7f0 commit 2d26d5b

5 files changed

Lines changed: 613 additions & 34 deletions

File tree

zjit/src/codegen_tests.rs

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -789,6 +789,294 @@ fn test_forwardable_callee_super() {
789789
"#), @r#"[["base", [1], {k: 2}]]"#);
790790
}
791791

792+
/// Run `program` and assert that `counter` moved by the end of it. Forwarding tests need this:
793+
/// the dynamic `sendforward` computes the same answer as the expansion, so without an assertion
794+
/// on what the compiler actually did they would all pass with the specialization switched off.
795+
#[track_caller]
796+
fn assert_forward_counter(program: &str, name: &str, counter: impl Fn(&crate::stats::Counters) -> u64) -> String {
797+
with_rubyvm(|| {
798+
let counters = crate::state::ZJITState::get_counters();
799+
let before = counter(counters);
800+
let result = assert_compiles_allowing_exits(program);
801+
let counters = crate::state::ZJITState::get_counters();
802+
assert!(counter(counters) > before, "expected {name} to increase, but it did not");
803+
result
804+
})
805+
}
806+
807+
/// Assert that a `bar(...)` site in a standalone-compiled forwardable ISEQ expanded against a
808+
/// profiled, guarded callinfo.
809+
#[track_caller]
810+
fn assert_expands_standalone_forward(program: &str) -> String {
811+
assert_forward_counter(program, "send_forward_expanded_profiled_count",
812+
|c| c.send_forward_expanded_profiled_count)
813+
}
814+
815+
/// The inverse: the site had to stay on the dynamic `sendforward`, for the reason `counter`
816+
/// names. Checking the reason and not just the absence of an expansion is what keeps these tests
817+
/// from passing vacuously if the site stops being compiled at all.
818+
#[track_caller]
819+
fn assert_keeps_standalone_forward(program: &str, name: &str, counter: impl Fn(&crate::stats::Counters) -> u64) -> String {
820+
let expanded_before = with_rubyvm(||
821+
crate::state::ZJITState::get_counters().send_forward_expanded_profiled_count);
822+
let result = assert_forward_counter(program, name, counter);
823+
let expanded_after = with_rubyvm(||
824+
crate::state::ZJITState::get_counters().send_forward_expanded_profiled_count);
825+
assert_eq!(expanded_after, expanded_before,
826+
"expected the `bar(...)` site to stay dynamic, but it expanded");
827+
result
828+
}
829+
830+
// A forwardable ISEQ that is *entered* megamorphically is never inlined, so its `bar(...)` has no
831+
// compile-time callinfo. The profiler records the packed one the `...` local held every time and
832+
// the compiled site guards it, reading the forwarded arguments back out of the frame extension.
833+
#[test]
834+
fn test_standalone_forwarder_megamorphic_entry() {
835+
assert_snapshot!(assert_expands_standalone_forward(r#"
836+
class Target; def bar(a, b) = a + b; end
837+
module Delegate
838+
def fwd(...) = @t.bar(...)
839+
end
840+
target = Target.new
841+
objs = 40.times.map do
842+
klass = Class.new do
843+
include Delegate
844+
define_method(:initialize) { |t| @t = t }
845+
end
846+
klass.new(target)
847+
end
848+
total = 0
849+
50.times { objs.each { |o| total += o.fwd(1, 2) } }
850+
total
851+
"#), @"6000");
852+
}
853+
854+
// The same site reached with a different number of forwarded arguments fails the callinfo guard.
855+
// The exit lands on the `sendforward` itself, where `vm_adjust_stack_forwarding` rebuilds the
856+
// argument list from the frame extension, so the interpreter finishes the call unaided.
857+
#[test]
858+
fn test_standalone_forwarder_guard_miss_on_a_different_call_shape() {
859+
assert_snapshot!(assert_expands_standalone_forward(r#"
860+
class Target; def bar(*a) = a.sum; end
861+
class Fwd
862+
def initialize(t) = @t = t
863+
def fwd(...) = @t.bar(...)
864+
end
865+
f = Fwd.new(Target.new)
866+
200.times { f.fwd(1, 2) }
867+
[f.fwd(1, 2), f.fwd(1, 2, 3), f.fwd, f.fwd(4, 5)]
868+
"#), @"[3, 6, 0, 9]");
869+
assert!(crate::state::ZJITState::get_counters().exit_send_forward_callinfo_changed > 0,
870+
"expected the callinfo guard to have missed at least once");
871+
}
872+
873+
// Same, but the callinfo changes because the *method name* did: two forwarders reaching the same
874+
// packed callinfo would compare equal, so the guard has to see the whole word.
875+
#[test]
876+
fn test_standalone_forwarder_guard_miss_on_a_different_caller_name() {
877+
assert_snapshot!(assert_expands_standalone_forward(r#"
878+
class Target; def bar(a) = a * 2; end
879+
class Fwd
880+
def initialize(t) = @t = t
881+
def one(...) = @t.bar(...)
882+
def two(...) = one(...)
883+
end
884+
f = Fwd.new(Target.new)
885+
200.times { f.one(3) }
886+
[f.one(3), f.two(4)]
887+
"#), @"[6, 8]");
888+
}
889+
890+
// A keyword-carrying caller needs a keyword table, which no packed callinfo has, so the `...`
891+
// local holds a heap `imemo_callinfo`. Holding one across a compilation would need a GC root, so
892+
// the site stays on the dynamic path.
893+
#[test]
894+
fn test_standalone_forwarder_kwargs_caller_stays_dynamic() {
895+
assert_snapshot!(assert_keeps_standalone_forward(r#"
896+
class Target; def bar(a, b:) = [a, b]; end
897+
class Fwd
898+
def initialize(t) = @t = t
899+
def fwd(...) = @t.bar(...)
900+
end
901+
f = Fwd.new(Target.new)
902+
200.times { f.fwd(1, b: 2) }
903+
f.fwd(1, b: 2)
904+
"#, "send_forward_reject_ci_not_packed", |c| c.send_forward_reject_ci_not_packed), @"[1, 2]");
905+
}
906+
907+
// Chained forwarding: the target is itself a `def bar(...)`, whose `...` local has to *receive* a
908+
// callinfo. No `rb_callinfo` describes the merged argument list, so the site is rejected outright
909+
// rather than expanded into a call that could not fill the callee's `...`.
910+
#[test]
911+
fn test_standalone_forwarder_chained_forwarding_stays_dynamic() {
912+
assert_snapshot!(assert_forward_counter(r#"
913+
class Target; def bar(a, b) = a - b; end
914+
class Fwd
915+
def initialize(t) = @t = t
916+
def inner(...) = @t.bar(...)
917+
def outer(...) = inner(...)
918+
end
919+
f = Fwd.new(Target.new)
920+
200.times { f.outer(9, 4) }
921+
f.outer(9, 4)
922+
"#, "send_forward_reject_chained", |c| c.send_forward_reject_chained), @"5");
923+
}
924+
925+
// `bh = VM_ENV_BLOCK_HANDLER(GET_LEP())`: a block given to the forwardable frame goes on to the
926+
// target. A standalone site cannot know statically whether there is one. Upstream either passes
927+
// the frame's handler through or guards that there is none; this branch has no `block_arg` on
928+
// `SendDirectData`, so only the guard form exists (see the `[reorder port note]`) and a frame
929+
// that really was given a block exits to the interpreter, which forwards it unaided.
930+
#[test]
931+
fn test_standalone_forwarder_carries_a_block() {
932+
assert_snapshot!(assert_expands_standalone_forward(r#"
933+
class Target; def bar(x) = yield(x); end
934+
class Fwd
935+
def initialize(t) = @t = t
936+
def fwd(...) = @t.bar(...)
937+
end
938+
f = Fwd.new(Target.new)
939+
200.times { f.fwd(4) { |v| v * 2 } }
940+
f.fwd(4) { |v| v + 1 }
941+
"#), @"5");
942+
}
943+
944+
// The blockless case takes the other branch: the target neither yields nor takes a block
945+
// parameter, so passing the frame's handler on would keep it off the direct send. The site guards
946+
// that the frame has no block instead, and a call that does have one exits to the interpreter.
947+
//
948+
// The block has to arrive through `public_send` for the guard to be the thing that catches it. A
949+
// literal block written at the call site clears `VM_CALL_ARGS_SIMPLE` in that site's callinfo, so
950+
// the *callinfo* guard already tells the two calls apart; `public_send` builds one callinfo for
951+
// both and carries the block beside it.
952+
#[test]
953+
fn test_standalone_forwarder_block_guard_miss() {
954+
assert_snapshot!(assert_expands_standalone_forward(r#"
955+
class Target; def bar(a) = a * 3; end
956+
class Fwd
957+
def initialize(t) = @t = t
958+
def fwd(...) = @t.bar(...)
959+
end
960+
f = Fwd.new(Target.new)
961+
200.times { f.public_send(:fwd, 2) }
962+
[f.public_send(:fwd, 2), f.public_send(:fwd, 2) { :ignored }]
963+
"#), @"[6, 6]");
964+
assert!(crate::state::ZJITState::get_counters().exit_send_forward_block_given > 0,
965+
"expected the block-handler guard to have missed at least once");
966+
}
967+
968+
// A literal block at the call site changes the caller's callinfo, so a forwarder warmed up
969+
// without one and then called with one misses the callinfo guard rather than reaching the target
970+
// with a block the expansion did not plan for.
971+
#[test]
972+
fn test_standalone_forwarder_literal_block_appearing_late() {
973+
assert_snapshot!(assert_expands_standalone_forward(r#"
974+
class Target; def bar(a) = block_given? ? yield(a) : a * 3; end
975+
class Fwd
976+
def initialize(t) = @t = t
977+
def fwd(...) = @t.bar(...)
978+
end
979+
f = Fwd.new(Target.new)
980+
200.times { f.fwd(2) }
981+
[f.fwd(2), f.fwd(2) { |v| v + 1 }]
982+
"#), @"[6, 3]");
983+
}
984+
985+
// A guard *inside* the expansion -- here the receiver class guard on `@t` -- exits to the
986+
// `sendforward` with the site's original stack, `[recv, ...]`, still on it. That is what
987+
// `vm_adjust_stack_forwarding` expects: it clobbers the `...` slot with the forwarded arguments
988+
// it copies back out of the frame extension.
989+
#[test]
990+
fn test_standalone_forwarder_side_exit_resumes_the_sendforward() {
991+
assert_snapshot!(assert_expands_standalone_forward(r#"
992+
class A; def m(a, b, c) = [:a, a, b, c]; end
993+
class B; def m(a, b, c) = [:b, a, b, c]; end
994+
class Fwd
995+
def initialize(t) = @t = t
996+
def m(...) = @t.m(...)
997+
end
998+
fa = Fwd.new(A.new)
999+
fb = Fwd.new(B.new)
1000+
200.times { fa.m(1, 2, 3) }
1001+
[fa.m(1, 2, 3), fb.m(4, 5, 6)]
1002+
"#), @"[[:a, 1, 2, 3], [:b, 4, 5, 6]]");
1003+
}
1004+
1005+
// A `ruby2_keywords` frame splats a flagged Hash into the forwarder, which makes the caller's
1006+
// callinfo carry `VM_CALL_ARGS_SPLAT`. That is unspecializable, and the flag still has to reach
1007+
// the target as keywords.
1008+
#[test]
1009+
fn test_standalone_forwarder_ruby2_keywords() {
1010+
assert_snapshot!(assert_keeps_standalone_forward(r#"
1011+
class Target; def bar(*a, **k) = [a, k]; end
1012+
class Fwd
1013+
def initialize(t) = @t = t
1014+
def fwd(...) = @t.bar(...)
1015+
ruby2_keywords def r2k(*a) = fwd(*a)
1016+
end
1017+
f = Fwd.new(Target.new)
1018+
200.times { f.r2k(1, k: 2) }
1019+
f.r2k(1, k: 2)
1020+
"#, "send_forward_reject_complex_args", |c| c.send_forward_reject_complex_args), @"[[1], {k: 2}]");
1021+
}
1022+
1023+
// The site's own arguments come first in the merged list, and the forwarded ones are read out of
1024+
// the frame extension in call order after them. (`def fwd(x, ...)` is not a forwardable ISEQ at
1025+
// all -- Ruby compiles it to `*rest, **kwrest, &block` -- so the site's own arguments have to be
1026+
// written at the call rather than declared as parameters.)
1027+
#[test]
1028+
fn test_standalone_forwarder_site_writes_its_own_args() {
1029+
assert_snapshot!(assert_expands_standalone_forward(r#"
1030+
class Target; def bar(a, b, c) = [a, b, c]; end
1031+
class Fwd
1032+
def initialize(t) = @t = t
1033+
def fwd(...) = @t.bar(:first, ...)
1034+
end
1035+
f = Fwd.new(Target.new)
1036+
200.times { f.fwd(2, 3) }
1037+
f.fwd(2, 3)
1038+
"#), @"[:first, 2, 3]");
1039+
}
1040+
1041+
// `vm_adjust_stack_forwarding` measures the frame extension from the *local* ISEQ's table, so a
1042+
// forwarder with body locals of its own has to read the arguments from further down.
1043+
#[test]
1044+
fn test_standalone_forwarder_with_extra_locals() {
1045+
assert_snapshot!(assert_expands_standalone_forward(r#"
1046+
class Target; def bar(a, b) = a + b; end
1047+
class Fwd
1048+
def initialize(t) = @t = t
1049+
def fwd(...)
1050+
extra = 10
1051+
spare = 5
1052+
extra + spare + @t.bar(...)
1053+
end
1054+
end
1055+
f = Fwd.new(Target.new)
1056+
200.times { f.fwd(1, 2) }
1057+
f.fwd(1, 2)
1058+
"#), @"18");
1059+
}
1060+
1061+
// The forwarded arguments stay live in the frame extension across a GC, which is what makes
1062+
// reading them back out of it after arbitrary work in the same frame safe.
1063+
#[test]
1064+
fn test_standalone_forwarder_arguments_survive_a_gc() {
1065+
assert_snapshot!(assert_expands_standalone_forward(r#"
1066+
class Target; def bar(a, b) = a + b; end
1067+
class Fwd
1068+
def initialize(t) = @t = t
1069+
def fwd(...)
1070+
GC.start
1071+
@t.bar(...)
1072+
end
1073+
end
1074+
f = Fwd.new(Target.new)
1075+
200.times { f.fwd("x", "y") }
1076+
f.fwd("a", "b")
1077+
"#), @r#""ab""#);
1078+
}
1079+
7921080
#[test]
7931081
fn test_explicit_super_to_forwardable_callee() {
7941082
assert_snapshot!(inspect(r#"

0 commit comments

Comments
 (0)