Help on a similar project

[jacksongoode]

Hey Jamie,

I've found we've been working on similar example implementations recently and was wondering you would want to collaborate or even have a look at an example I'm working on which is a peer-to-peer audio example where the audio buffer is modified through a worklet in Rust -> WASM. https://github.com/jacksongoode/decay It's super simple where the audio should just get processed as an octave lower on the receiving end.

I was wondering if you've gotten further than me and are actually processing audio through a WASM worker. I've taken the route of using the FreeQueue ring buffer which I feel like abstracts enough away to get working. I found this blog able to explain at a high level what should be happening with allocating memory in and out of two buffers.

But mostly just curious if you'd be able to provide some pointers in the right direction. I'm using wasm-webpack but maybe there's still some core incompatibility with AudioWorklets there.

[JamieBeverley]

Hey @jacksongoode - thanks for reaching out, sounds like a cool project! Keen to help/collaborate if anything I've tinkered on here may be useful.

I've succeeded in creating an AudioWorkletNode and accompanying AudioWorkletProcessor that runs DSP in WASM compiled from Rust.

I first got an example working of just an infinite-looping sound file with some control over playback rate and then got distracted turning it into more of a granular synthesis patch (I plan on rescuing some of the more reusable bits from it at some point). (with lots of help from some things listed here, particularly this).

Thanks for the links - I hadn't come across those before but will take a look (wasm memory is still pretty opaque to me tbh, I'm sure I'll learn more here).

I've been meaning to document some learnings + failed experiments more meaningfully but in lieu of that here are a few stumbling blocks I hit and found work-arounds for.

Hope its helpful - let me know if I can help or share more details!

wasm-pack

• As far as I know, wasm-pack doesn't have a compile target that works in an AudioWorklet context (where browser APIs like fetch, URL, etc... are not available)
  • so any init/initSync glue from wasm-pack wouldn't work
  • This part of this blog series shows an approach that involves slicing some of the wasm-pack generated JS into the AudioWorkletProcessor which is a bit precarious to maintain
  • this example from the wasm-bindgen book seemed promising too but relies on unstable cargo features that I didn't want to introduce
• So I resorted to the same technique used in an earlier part of the blog series above:
  • load the wasm binary from the main JS thread and send it over to the AudioWorkletProcessor via a post message (this async func)
  • AudioWorkletProcessor instantiates the wasm module here

I hadn't seen that blog you shared before (it would have been helpful!). But I think they get around this problem by instructing emscripten to output wasm inline and b64 encoded in just one JS file via -s SINGLE_FILE=1 (which presumably can run fine ootb in an audio worklet context).

Memory

• The next hurdle was getting memory management working to pass audio blocks between JS and WASM
  • this is where the ruffbox example helped a ton
  • I followed a similar pattern:
    • In Rust, a few alloc functions exposed, eg. here
    • And invoked from JS in the AudioWorkletProcessor here
• Note: the README of the ruffbox repo mentions something curious I don't really understand but kind of take at face value for now:

A challenge here was to provide reliable buffer allocation. Somehow the buffers get neutered easily. Observation: the order of allocation seems important. First, allocate the sample buffers, then allocate the output buffers.

I have vague recollection of facing some issue with sound prior to moving all alloc calls into one function with a deterministic order.

I suspect I could be handling memory more intelligently/safely here (if nothing more than just coupling Float32Array's and their rust memory references together) but still pretty new to this@

this repo

Here's a bit more info on this repo in case it helps to make the code a bit more clear. Apologies for all the junk, I can probably clean some of it up this weekend:

• the root of the project is an npm module that is intended to be a portable rust/wasm AudioWorkletNode
  • I wanted it to be a simple npm install for other JS projects to import this module so the wasm + worklet needed to be bundled too
  • presently it exposes a new AudioWorkletNode (GranularNode)
  • public/rust_wasm.wasm and public/worklet.js get bundled as well as they must be loaded statically
    • GranularNode gets reference to their paths via URL(...).href
• rust-wasm - the rust crate
  • built via cargo build --target wasm32-unknown-unknown (not with wasm-pack as noted above)
  • please pardon the gore - I'm pretty new to rust 😅
• demo
• a small vite vanilla-ts page for testing things (currently binds an x-y grid and a midi device to control granular synthesis over a sound file.)
• seems to be missing a dependency in package.json but it should import ../ (the npm module at the root of the repo)

How I set up/run/develop here roughly:

npm link
cd demo
npm link wasm-audio-worklet # the name of the npm package at the root

# copy a sound file for the demo app (hard-coded as `sound-file.wav` for now...)
cp /path/to/some/soundfile.wav ./public/sound-file.wav

cd ../
# the repo root `package.json` shows all the other build steps (nothing fancy here)
npm run build

# the built demo app should now be in `demo/dist`
cd demo/dist
python -m http.server

[JamieBeverley]

@jacksongoode FYI I've done some house keeping on main which should make this a bit more legible:

• added an MIT license
• cleanup of some junk: unused code, deps, comments, unnecessary build deps
• converted the root module to TypeScript
• moved the granular synthesis stuff elsewhere (granular-synth branch) - the Rust/wasm portion now is just doing basic infinitely-looped sample playback (the janky Rust is hopefully a bit less janky)

[jacksongoode]

Thank you so much for the notes. I'm going to go through and attempt to reflect on these!