Segmentation fault on out-of-bounds index into function.locals

[slatyc]

Hello!

I'm trying to decompile some lua 4.0 bytecode with lua4dec, and running into some trouble.
lua4dec starts outputting chunk header, function, and instruction data to stdout, but then segfaults partway through the file.

gdb gives me the following backtrace:

#0  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign (this=this@entry=0x7fffffffcbb0, __str=<error reading variable: Cannot access memory at address 0x8>)
    at /usr/src/debug/gcc-14.3.1-1.fc41.x86_64/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:282
#1  0x00007ffff7cbe98d in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign (this=0x7fffffffcbb0, __str=...)
    at /usr/src/debug/gcc-14.3.1-1.fc41.x86_64/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:1619
#2  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator= (this=0x7fffffffcbb0, __str=...)
    at /usr/src/debug/gcc-14.3.1-1.fc41.x86_64/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.h:819
#3  0x000000000040f6a4 in debug_instruction (state=..., idx=1, instruction=11, function=...) at /home/emmy/ttahack/lua4dec/source/lua/lua.cpp:292
#4  0x000000000040f25d in debug_function (state=..., function=...) at /home/emmy/ttahack/lua4dec/source/lua/lua.cpp:231
#5  0x000000000040f331 in debug_function (state=..., function=...) at /home/emmy/ttahack/lua4dec/source/lua/lua.cpp:241
#6  0x000000000040ee08 in debug_chunk (chunk=...) at /home/emmy/ttahack/lua4dec/source/lua/lua.cpp:187
#7  0x0000000000401678 in main (argc=2, argv=0x7fffffffd408) at /home/emmy/ttahack/lua4dec/source/main.cpp:31

and, looking at lua.cpp line 292, it looks like the problem is indexing into an empty function.locals vector.

(gdb) p function.locals
$5 = std::vector of length 0, capacity 0
(gdb) p index
$6 = 0

I ran across this problem while trying to reverse engineer the gamecube game Lord of the Rings The Third Age. Here is a link to a bytecode file that triggers this segfault when passed in to luadec_32.

Do you have any thoughts on what's going wrong here?

[styinx]

Hey there!

Great to see someone using the project.

---

Looking at the function header, you can see that this function does not define any local variables, yet it tries to retrieve one in the instruction that produces the segfault.

=== Function ===
Name:         "=(none)"
Line:         16
Params:       7
Variadic:     false
Stack:        28
Locals:       0
Globals:      12
   0: "ON"
   1: "HOLD"
   2: "SendMsg"
   3: "info"
   4: "0, "
   5: " 1901 3 = "
   6: " 1901 4 = "
   7: " 1901 5 = "
   8: " 1901 6 = "
   9: " 1901 1 = "
  10: " 1901 2 = "
  11: "DelayedMsg"
Instructions: 64
   0:          68 (0x00000044) | OP:  4 (0x04) (    PUSHNIL) | A:     0 (0x0000000) | B:   1 (0x001) | U:          1 (0x00000001) | S: -33554430 (0xfe000002)
   1:          11 (0x0000000b) | OP: 11 (0x0b) (   GETLOCAL) | A:     0 (0x0000000) | B:   0 (0x000) | U:          0 (0x00000000) | S: -33554431 (0xfe000001)

---

As far as the parsing of the byte code goes, it seems to be correct. I marked the function header in this image:

It shows:

• length of the function name (uint32)
• function name (char[])
• line number (uint32)
• number of parameters (uint32)
• is variadic (bool/uint8)
• max stack size
• number of local variables

It is possible that the game developers made some adjustments to the stock lua4 compiler/interpreter.
Looking at the lines following the function header, one can see the global names (marked blue).
These don't look to me like valid Lua identifiers. Maybe that has something to do with the issue?

---

Unfortunately, I can't help you at the moment, but thank you very much for your feedback.
I'll try to inspect the issue again, when I find more time.