GH-50707: [C++][Gandiva] fix out-of-bounds read in mask utf8proc length - #50709
Conversation
|
|
raulcd
left a comment
There was a problem hiding this comment.
could you rebase, please? The CI failures are fixed on main and should be fixed on your PR.
@dmitry-chirkov-dremio @lriggs @akravchukdremio @xxlaykxx could you take a look at this PR, please?
| EXPECT_EQ(expected, std::string(result, out_len)); | ||
| } | ||
|
|
||
| TEST(TestGdvFnStubs, TestMaskTruncatedUtf8NoOverread) { |
There was a problem hiding this comment.
Might be worth having a simple test for gdv_mask_last_n_utf8_int32 as well
There was a problem hiding this comment.
Added one. last_n's utf8proc_decompose pre-pass rejects the truncated glyph before the iterate loop, so the test just confirms it errors on that input instead of over-reading.
2e9ab83 to
1869ef3
Compare
|
Rebased on main to pick up the CI fixes, and added a test covering gdv_mask_last_n_utf8_int32 on the truncated input per @lriggs. |
|
Looks good |
…f8_int32 Signed-off-by: abdul rawoof <abdulr@bugqore.com>
|
After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit e611f48. There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 11 possible false positives for unstable benchmarks that are known to sometimes produce them. |
Rationale for this change
The Gandiva mask stubs walk a utf8 string calling
utf8proc_iterate(data + bytes_read, data_len, ...), passing the totaldata_lenas the remaining byte count even though the pointer has already advanced bybytes_read.utf8proc_iteratereads up tostrlenbytes, so a value ending in a truncated multi-byte glyph (for example a lead byte0xF0as the final byte of an exactly sized value buffer) makes it read continuation bytes pastdata + data_len. This is reachable frommask(),mask_first_n()andmask_last_n()on untrusted string columns.gdv_mask_last_n_utf8_int32happens to be shielded by itsutf8proc_decomposepre-pass, butgdv_mask_first_n_utf8_int32andmask_utf8_utf8_utf8_utf8have no such guard and over-read.What changes are included in this PR?
Pass
data_len - bytes_read(the real remaining length) at all fourutf8proc_iteratecall sites so utf8proc reports the truncated glyph instead of reading past the buffer.mask_utf8_utf8_utf8_utf8also lacked thechar_len < 0check that the other two paths already have, so I added it there to reject the now-detected invalid input rather than advancing by a negative length.Are these changes tested?
Yes.
TestMaskTruncatedUtf8NoOverreadfeeds a value whose reported length stops one byte short of a complete euro sign; before the change both functions consumed the out-of-range byte and returned a masked result, after it they report the truncated input. The existing mask tests still pass.Are there any user-facing changes?
No change for valid utf8. A value that ends in a truncated multi-byte glyph now surfaces an invalid-utf8 error instead of being silently masked using bytes past its end.
This PR contains a "Critical Fix". The wrong length argument lets
utf8proc_iterateread past the end of an exactly sized input buffer.