🐛 Fix hash_bytes on big-endian and 32-bit targets

[gkumbhat]

🐛 Fix hash_bytes on big-endian and 32-bit targets

The problem

hash_bytes hashes a token through the generic slice Hash impl:

bytes.hash(&mut hasher);

That impl prefixes the element count via Hasher::write_length_prefix →
write_usize → write(&len.to_ne_bytes()). FnvHasher doesn't override those,
so the length reaches FNV in native byte order, at the host's pointer width.
The same token therefore hashes to a different value on a big-endian target or a 32-bit one.

That breaks the two things that assume the mapping is fixed:

• The hash_factor values callers hardcode — see bpe-openai/build.rs, which
  passes 17846336922010275747 for cl100k_base, o200k_base and
  voyage3_base — were found by find_hash_factor_for_dictionary on a 64-bit
  little-endian machine. Once every hash changes, that factor no longer
  guarantees collision freedom, and the assert_eq! in from_dictionary fires. This breaks build on big-endian platforms.
• bytes_hash_to_token is a serialized field, so a dictionary serialized on one
  host can't be deserialized correctly on a host of different endianness or
  pointer width.

The fix

Write the length explicitly, at a fixed width and a fixed byte order:

hasher.write(&(bytes.len() as u64).to_le_bytes());
hasher.write(bytes);

The hash is now a property of the token bytes alone.

Compatibility

No change on 64-bit little-endian. The slice impl was already making exactly
these two write calls; the only difference is to_ne_bytes → to_le_bytes on
a value that is already 8 bytes wide. So existing hardcoded factors and already
serialized dictionaries stay valid.

[aneubeck]

TY!

[aneubeck]

if you need new crates published, please respond here.

[gkumbhat]

@aneubeck yes please. create release would be helpful

Thank you for the quick review