Commit 2d26d5b
committed
ZJIT: Specialize
`Function::specialize_send_forward` could only expand a `bar(...)` whose
enclosing `def foo(...)` frame had been inlined, because that is what makes
the forwarded callinfo -- the `...` local -- a compile-time value. A
forwardable ISEQ that is *entered* megamorphically is never inlined, so on
lobsters 673K of the 776K dynamic `sendforward`s per run came from 186 sites
with no inlining context at all. Those are the ActiveRecord reflection
delegators, and every one of them was reaching `rb_vm_sendforward`.
Profile the callinfo instead. `vm_caller_setup_fwd_args` reads it as `TOPN(0)`,
and a site that only ever forwards one *packed* callinfo can guard it with a
single comparison: a packed one carries `RUBY_FIXNUM_FLAG`, so it is an
immediate whose bits are the whole of `(mid, flag, argc)` with no keyword table
and nothing for the GC to keep alive. Heap `imemo_callinfo`s -- what a
keyword-carrying caller produces -- would need a root, and are recorded as
`None` so that a site mixing the two reads as polymorphic rather than stable.
Past the guard the merge is the one the inlined path already does. The
difference is where the forwarded arguments come from: they are not values this
compilation produced, they sit in the frame extension below the local table,
which is where `vm_adjust_stack_forwarding` reads them from and where they stay
for the life of the frame. The guarded callinfo fixes their count, so they can
be loaded relative to the frame's local EP.
Two things a standalone frame cannot know statically:
* Whether it was given a block. The expansion asks for the conservative answer
first -- "was given one", which keeps `&nil`-rejecting callees and callees
that would warn about an unused block off the direct send -- and falls back to
guarding that the frame's block handler is `VM_BLOCK_HANDLER_NONE`. That
second form is what makes the common blockless delegator specializable, and it
is not redundant with the callinfo guard: a literal block at the call site
clears `VM_CALL_ARGS_SIMPLE` and so shows up in the callinfo, but a block
arriving through `public_send` does not.
* Nothing, for chained forwarding. `bar(...)` whose target is itself a
`def bar(...)` stays dynamic and now says so with its own counter: the callee's
`...` local has to *receive* a callinfo, and no `rb_callinfo` describes the
merged argument list.
A guard miss exits to the `sendforward` itself with the site's original stack,
which is exactly what `vm_adjust_stack_forwarding` expects, and asks for a
recompile so a site that turns out to be polymorphic stops speculating.
Where the newly expanded sends land is worth knowing, so
`sendforward_target_not_specialized` is split into the three reasons the merged
call can still fall back for. On lobsters:
ccall_rb_vm_sendforward 775,633 -> 495,311 (-36%)
sendforward_not_specialized 673,409 -> 113,662
...target_not_specialized 102,224 -> 0
...recv_type - -> 156,491
...method_type - -> 208,420
...complex_args - -> 16,738
dynamic_send_count 10,678,566 -> 10,429,465
code_region_bytes 29,290,496 -> 29,491,200
280K forwarding calls per run become direct sends. The largest remaining
bucket is a merged call whose target is a C function rather than an ISEQ, which
this pass does not build a `CCall` for yet.
Ported from zjit/all (0fa6d74). Adaptations for this branch:
* `IseqProfile`'s side tables are plain `HashMap` fields here, not
`Option<Box<..>>`, so `forwarded_cis` is one too and `forwarded_cis_mut`
hands out `&mut self.forwarded_cis`. There is no `heap_size()` accounting on
this branch, so the memory-reporting hunk is dropped; the two-bucket
`FORWARDED_CI_DISTRIBUTION_SIZE` is kept, since it is what makes "did this
site ever forward a second callinfo" answerable.
* `FieldName::StackSlot`, which only names the load for HIR dumps, comes from
an excluded commit and is added here.
* `SendDirectData` has no `guard_state` field on this branch.
[reorder port note] Cherry-picked from stack position 49 onto position 12, so
the diff carried context from ~37 commits that are not underneath it here.
Dropped as foreign context: the `super` specialization helpers
(`emit_super_call_guards`, `load_super_lep`, `load_frame_method_entry`,
`load_super_block_handler`, `emit_super_chain`, `emit_specialized_super`) and
`get_super_method_entries`/`super_cme_mut`; the block-handler and
block-fallback profiling tables (`block_handlers`, `block_fallbacks`,
`observe_block_fallback`, `get_block_handlers`, `block_handlers_mut`); the
`send`/`__send__` method-name tables (`send_mid_mut`,
`get_send_method_names`); `splat_lengths_mut` (this tree reaches the field
directly); and the `SplatLengthChanged`/`SplatLastRuby2Keywords` side-exit
reasons and their counters. `FieldName::StackSlot` already exists here, so its
doc comment was kept as the tree had it.
Adapted: `assume_cme_for_send`/`assume_no_singleton_classes_for_send` and
their `ancestor_class` argument do not exist, so the commit step is the
tree's explicit `assume_no_singleton_classes` plus an
`Insn::PatchPoint { Invariant::MethodRedefined { .. } }`, as the commit below
this one does.
Weakened, deliberately and for the same reason the commit below this one
rejects a block-carrying `bar(...)`: `build_send_direct_args` takes four
arguments here (no `block_arg_passthrough`), `SendDirectData` has no
`block_arg` field, and `BlockHandler::BlockArg` is `unreachable!()` in
`gen_send_iseq_direct`, so a run-time block handler cannot be handed to a
`SendDirect`. The `candidates: &[bool]` restructuring is kept, but `true` is
never a candidate: an inlined frame that was given a block gets an empty
candidate list and keeps its `SendForward` (the pre-existing rejection,
now reported as `SendForwardComplexArgs`), and the standalone case only ever
takes the blockless form -- which is the one the commit's own design guards
with `VM_BLOCK_HANDLER_NONE`, so the common blockless delegator still
specializes. Restoring the passthrough arm is a one-line change to
`candidates` once `SendDirectData` grows a `block_arg`.
`codegen_tests.rs` is kept whole; only the comment on
`test_standalone_forwarder_carries_a_block` was rewritten, because on this
branch that case expands and then always misses the block guard instead of
passing the handler through.
[reorder port note 2] On zjit-lobsters4: `FieldName::StackSlot` no longer
exists upstream -- the frame-extension loads use this base's `FieldName::Stack(i)`
with the forwarded argument's index. `build_send_direct_args` calls pass this
base's fifth `block_arg_passthrough: false` argument. profile.rs keeps the
front block's `block_handlers` table alongside the new `forwarded_cis` at every
keep-both site.bar(...) in standalone forwardable ISEQs1 parent 7fbf7f0 commit 2d26d5b
5 files changed
Lines changed: 613 additions & 34 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
789 | 789 | | |
790 | 790 | | |
791 | 791 | | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
| 861 | + | |
| 862 | + | |
| 863 | + | |
| 864 | + | |
| 865 | + | |
| 866 | + | |
| 867 | + | |
| 868 | + | |
| 869 | + | |
| 870 | + | |
| 871 | + | |
| 872 | + | |
| 873 | + | |
| 874 | + | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
792 | 1080 | | |
793 | 1081 | | |
794 | 1082 | | |
| |||
0 commit comments