problem when use Float32Array in object's property

[shinyoshiaki]

Problem

Problem happen when use Float32Array in object's property.

indicate minimum jest test below

import { encode, decode } from "@msgpack/msgpack";

test("issue", () => {
  const f32 = new Float32Array([1.1, 1.2, 1.3]);
  const uint8 = new Uint8Array(f32.buffer);
  expect(new Float32Array(uint8.buffer)).toEqual(f32); //success

  const data = encode({ audio: uint8 });
  const { audio } = decode(data) as { audio: Uint8Array };

  expect(uint8).toEqual(audio); //success

  expect(uint8.buffer).toEqual(audio.buffer); //success

  const fail = new Float32Array(audio.buffer);
  expect(fail).toEqual(f32); //fail

  const success = new Float32Array(new Uint8Array(Object.values(audio)).buffer);
  expect(success).toEqual(f32); //success
});

this test fail in

 const fail = new Float32Array(audio.buffer);
 expect(fail).toEqual(f32); //fail

Enviroment

Node.js : 11.15.0
TypeScript : 3.5.2
Test Framework : Jest

[gfx]

This is the expected behavior of typed arrays.

First of all, the way to chang a typed array class is not new Float32Array(u8array.buffer), but you have to provide the range (offset, length) of the buffer:

const u8array: Uint8Array;
const f32array = new Float32Array(u8array.buffer, u8array.byteOffset, u8array.byteLength);

This is because TypedArray's buffer is a pointer to a slice of an ArrayBuffer. Thus, u8array and f32array shares the same buffer: ArrayBuffer.

[crabmusket]

I have been doing some work on an extension that handles TypedArrays natively and also aligns them at encoding time, rather than doing a copy when decoding.

const floatArray = new Float32Array([1, 2, 3, 4, 5]);
const encoded = encode({ floatArray }, { extensionCodec });
assert.deepStrictEqual(decode(encoded, { extensionCodec }), { floatArray });

Here's a readme describing how my extension works. And in this example you can see the encoding/decoding of different TypedArrays with no boilerplate.

However, in order to align during encoding I had to implement my own "plugin" system which works slightly differently to the existing extensions. This happens here. Basically the "plugin" just calls the encoder's private methods, instead of returning a blob which is then encoded as an ext.

This means my code has to write the ext header itself, but it makes the whole process more efficient and straightforward.

If I'd used a regular extension, it would require some tricky logic to "guess" which type of ext the encoder was going to use for the typed array data and adjust the alignment accordingly. It would also involve an intermediate copy (from TypedArray to ExtData, then from ExtData to the encoding buffer).

I'm not sure if this is of interest to you @gfx - I ask because I'm wondering whether something like my plugin system would ever make it into upstream, or whether I should maintain a fork or try harder to work with the existing extension system. I'm going to experiment with using my fork in some upcoming work and see how I feel about it. My use cases involve pushing around large arrays of float data, obviously!

I don't know whether it seems like I'm trying way too hard to optimise things, or if the idea of aligning data within the payload is a portability concern. Obviously this is really focused on JS-to-JS use cases, but other languages could read the memory into typed array primitives too I'm sure, and get the performance boost of alignment-without-copying.

EDIT: I've realised I can probably make the "really low level extension" concept fit more neatly into the existing code. I might give that a go then propose a separate issue/PR.

EDIT 2: updated my fork to have less intrusive extension changes. ExtData now has a write method and my plugin returns a subclass which does the actual encoding. If I were designing this properly, I'd replace the use of ExtData with an interface to avoid mandatory inheritance hierarchies.

[gfx]

@crabmusket Thanks for the interesting report. Can you file an independent issue? Yours is definitely not related to the original issue.

[crabmusket]

@gfx done!