Binaryen as a qemu backend

[tbodt]

(Continuing conversation from WebAssembly/design#796)

I compiled qemu with --enable-profiler, added a few missing profiler logs, set an uninitialized variable to zero, and tested booting a FreeDOS image with 1 cpu (so no multithreading.) Booting to the C:> prompt took 10 seconds. info jit at the monitor prompt gives this:

Translation buffer state:
gen code size       31204435/33549267
TB count            36102
TB avg target size  17 max=152 bytes
TB avg host size    668 bytes (expansion ratio: 37.7)
cross page TB count 35 (0%)
direct jump count   14863 (41%) (2 jumps=10899 30%)
TB hash buckets     4146/8192 (50.61% head buckets used)
TB hash occupancy   17.63% avg chain occ. Histogram: [0,10)%|█▁▅▁ ▂▁▁ ▁|[90,100]%
TB hash avg chain   1.008 buckets. Histogram: 1|█▁|2

Statistics:
TB flush count      13
TB invalidate count 324441
TLB flush count     484
JIT cycles          4737605000 (1.974 s at 2.4 GHz)
translated TBs      337178 (aborted=0 0.0%)
avg ops/TB          93.9 max=362
deleted ops/TB      2.09
avg temps/TB        41.03 max=46
avg host code/TB    1158.3
avg search data/TB  43.2
ops                 31648062
in bytes            10142744
out bytes           390555017
search bytes        14581774
cycles/op           149.7
cycles/in byte      467.1
cycles/out byte     12.1
cycles/search byte     324.9
  gen_interm time   20.5%
  gen_code time     79.5%
optim./code time    23.0%
liveness/code time  9.1%
cpu_restore count   106
  avg cycles        160.4

What's particularly interesting is the cycles/op number: 149.7. That's the total cycles spent doing compiling (measured by rdtsc) divided by the number of qemu IR operations generated. I doubt binaryen would be that fast. 20% of the time here is spent doing compiling, so a slowdown in that area would be noticeable.

@kripken

[kripken]

Binaryen would add to that, yeah. But, being realistic, Binaryen is going to be faster than the WebAssembly VM compiling it to machine code afterwards anyhow. WebAssembly is an intermediate format, so in an environment like qemu slower compilation is unavoidable.

In a workload dominated by compilation WebAssembly just might not be good enough, period. On the other hand, for code that is compiled once and run many times, increasing the compile time even significantly shouldn't be too bad - that's why I'm very interested in this. But, what's your use case?

[tbodt]

My usecase is having a nice development environment running on an iOS device. Qemu + WebAssembly seemed like the most promising way to make this fast.

[kripken]

I see. My guess is that could be fast enough (but we'd need to measure to be sure, of course).

Happy to help try this out - I can help writing out the Binaryen integration bits, if you set up a repo. I just don't know where to get started on the qemu side.

Btw, another option here is to interpret the wasm (using the Binaryen interpreter, or another), to avoid compilation entirely. Then compilation can be done after the function is used enough times, as a JIT optimization, etc. I'm optimistic we can find ways to make this work efficiently.

[davidar]

Did this go anywhere? I'd also be interested in seeing this happen

[tbodt]

@davidar No, I never found the time to work on it. It's still on my list of unfinished projects that I might work on later though.

[davidar]

@atrosinenko You seem to have quite some experience building QEMU with Emscripten, I'm curious if you have any thoughts on this issue?

[atrosinenko]

@davidar Now I'm planning to rewrite my port of QEMU on top of v3.0 implementing WASM JIT instead of Asm.js one that was implemented more as proof-of-concept (and fall back to compiling plain TCI in Asm.js) -- see qemujs-v2 branch. But I fear Binaryen's Apache 2.0 license is not compatible with QEMU that is AFAIK GPLv2 as a whole (different parts have different license options sometimes).

For Asm.js I was compiling basic block after N executions (hooked qemu_tb_exec -- you can see diff of v2.4.1...emscripten branches) and hope it will be much faster when eliminating machine code -> TCI bytecode -> asm.js -> parse JS -> ... -> machine code flow.

[kripken]

Regarding the license, that is something we should fix if, if it's a problem.

@jfbastien, I know the WebAssembly/ projects, like LLVM, were thinking about doing Apache 2 plus exceptions. Looks like those exceptions might help here (QEMU is GPL2). Do you know what the status is there?

[atrosinenko]

Regarding the license, that is something we should fix if, if it's a problem.

But isn't changing a license a big problem when there are already 64 contributors (or do you have copyright transfer)? (I'm not a lawyer, maybe Apache 2.0 is not a problem...)

Do you mean upstreaming Binaryen support for QEMU? If so, it may be worth reading QEMU contributor guidelines beforehand: Contribute/SubmitAPatch, to not fix formal requirements afterwards. But not sure whether they would accept such a port. For example, AFAIK now multithreading in browsers is disabled due to security reasons, so I have some kludges to make QEMU single-threaded at the cost of stability. On the other hand, if the original issue was about native QEMU linked to system JS/WASM-engine, then only CPU thread would be "single-threaded", so why not... Anyways, it would be great to upstream as much as possible.

Meanwhile, is there some possibility in WASM to compile one basic block at a time to WASM modules and then, when QEMU decides to directly link them, somehow link these modules, so they can pass execution from one piece of AOT compiled code to another one efficiently?

[tbodt]

@atrosinenko Multithreading was recently re-enabled (in Chrome at least) because other spectre mitigations were put in place.

[tbodt]

As for compiling one basic block at a time, you can either:

• Put all the basic blocks in a function, run binaryen's relooper on the function, and reload the module
• Wait for tailcall support, make each basic block its own function that ends with a tailcall, and reload the module

Either way, you have to reload the module with all your code in it, which requires the browser to throw out all of its existing compiled machine code and start over. It seems like WASM wasn't designed with this use case in mind.

[jfbastien]

@kripken http://llvm.org/foundation/relicensing/

[binji]

@tbodt you can also compile the function in its own module, export it, and put it in a table. Then the original module can use call_indirect to call this function.

[kripken]

@jfbastien Thanks! And to be sure, the wasm projects are going to be licensed in the exact same way as LLVM?

@atrosinenko

But isn't changing a license a big problem when there are already 64 contributors?

Well, in practice probably 99% of the code in Binaryen was written by less than 10 people. And I'm pretty sure all of them are very excited about enabling use cases like Qemu (I definitely am), so I think we could dual-license it (say, to add MIT) or some other solution. But it sounds like the LLVM license change will apply here too, in a matter of months, so we may not even need to do that.

Overall, I encourage you to experiment on this stuff :) We'll make the licensing stuff work out.

[jfbastien]

@jfbastien Thanks! And to be sure, the wasm projects are going to be licensed in the exact same way as LLVM?

That was discussed a long time ago. Probably worth bringing up again at the CG meeting. @binji ?