• Part 1 - Creating sound (this article!)
• Part 2 - Sounding Better

Guitar Sounds Part 1 - Creating Sound

I’d like to create some software to help me practice guitar, specifically playing by ear, something like this:

• Generate a simple riff
• I listen to it and play it by ear
• Repeat

This post breaks down the process of creating this software. I’ll use my favourite combo of rust + wasm to demo my progress in blog form :) see here for how to set this up.

Contents

• Creating Some Noise
• Why Does This Create Sound?
• Visualisation
• Creating More Interesting Noise
  • Wasm Callbacks
  • Pitch Slider
  • Velocity Change Limited Pitch Slider
• Conclusion

Creating Some Noise

I’ll use cpal for this. It’s a low level library for audio I/O in pure rust. There’s also a wasm demo in their examples folder so we can use that as a starting point!

Let’s see (hear?) how it sounds:

Pretty boring but we have sound, so that’s great.

Why Does This Create Sound?

The code that generates this type of noise comes from this snippet:

let mut sample_clock = 0f32;
let mut next_value = move || {
    sample_clock = (sample_clock + 1.0) % sample_rate;
    (sample_clock * 440.0 * 2.0 * 3.141592 / sample_rate).sin()
};

The value of this closure is used to write data to the device’s stream. It looks like a sine wave when plotted on a graph.

This tells your speaker how to vibrate, this causes air molecules to move and vibrate your ear drum, this tells your brain what it sounds like.

However, it’s not super intuitive how changes to this wave cause changes in the sound. It would be great if we could have a way to:

1. Listen to the audio this function creates,
2. Look at the values this creates in a graph, and
3. Be able to change the values on the fly to experiment with what they sound like

So let’s build something that achieves this.

Visualisation

Let’s tackle the visualisation part first, because it’s easier and because it’s nice to see graphs. I’ll use ChartJs for their simple line chart and good instructions for squeezing perf

I’ll add a static global in rust to track the sample values, and a function to get the latest values from this container and clear it. This means every time we call this function we get a dump of the values that we haven’t seen yet:

static mut VALUES: Mutex<Vec<f32>> = Mutex::new(Vec::new());

#[wasm_bindgen]
pub fn get_new_vals() -> Result<Vec<f32>, JsValue> {
    // Aquire the values lock
    let mut values = unsafe { VALUES.lock().expect("failed to lock") };

    // Clone into my own container
    let values_clone = values.clone();

    // Clear the old values
    unsafe { values.set_len(0) };

    // Return the cloned values
    Ok(values_clone)
}

We also need to modify our next_value function to add to push to this global

let mut next_value = move || {
    sample_clock = (sample_clock + 1.0) % sample_rate;
    let val = (sample_clock * 440.0 * 2.0 * 3.141592 / sample_rate).sin();

    {
        let mut values = unsafe { VALUES.lock().expect("failed to lock") };
        values.push(val);
    }

    val
};

Creating More Interesting Noise

Wasm Callbacks

I’d like to use the same piece of code to play the sound and render a graph. We’ll be editing the values in browser land so we have two options to achive this, either we:

• Poke the editable values into rust wasm, or
• Poke the whole sound-creating function into rust wasm

Let’s go for the second option, it’ll make it easier to extend the function later with more complexity.

It would be good to have an API like this in typescript:

import { Handle, beep as wasmBeep, get_config } from "path/to/wasm/bundle";

const beep = useCallback(() => {
  let config = get_config();
  wasmBeep((sampleIndex: number) => {
    return Math.sin(sampleIndex * 440.0 * 2.0 * Math.PI / config.sample_rate);
  }));
}, [setHandle]);

We can pass JS functions to a rust WASM binary using js_sys::Function. So let’s pipe it into the wasm example from cpal and call it from the next_value closure from above:

let mut sample_clock = 0f32;
let mut next_value = move || {
    sample_clock = (sample_clock + 1.0) % sample_rate;
    let this = JsValue::null();
    js_callback.call1(&this, &JsValue::from(sample_clock))
        .expect("failed to call into js")
        .as_f64()
        .expect("failed to convert to f64") as f32
};

However, Rust is not happy with us, the short version is that JsValues aren’t thread safe and to build an audio output stream we need to pass it a closure (next_value) that moves only thread safe values. Our callback is not thread safe.

let stream = device
    .build_output_stream(
        config,
        move |data: &mut [T], _| write_data(data, channels, &mut next_value),
        err_fn,
        None,
    )

error[E0277]: `*mut u8` cannot be shared between threads safely
    --> src\lib.rs:92:13
     |
90   |.build_output_stream(
     | ------------------- required by a bound introduced by this call
91   |     config,
92   |     move |data: &mut [T], _| write_data(data, channels, &mut next_value),
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut u8` cannot be shared between threads safely

pub struct JsValue {
    idx: u32,
    _marker: marker::PhantomData<*mut u8>, // not at all thread safe
}

    fn build_output_stream<T, D, E>(
        &self,
        config: &StreamConfig,
        mut data_callback: D,
        error_callback: E,
        timeout: Option<Duration>,
    ) -> Result<Self::Stream, BuildStreamError>
    where
        T: SizedSample,
        D: FnMut(&mut [T], &OutputCallbackInfo) + Send + 'static,  // <-- here
        E: FnMut(StreamError) + Send + 'static,

So what can we do differently?

• We could generate X seconds worth of audio, place it in a buffer that is thread safe, then pass this into the next_value callback
  • But this will mean we limit how much sound we generate and I’d like to be able to generate an arbritrary amount of sound
  • It also means that changing any values in the sound wave would mean re-generating this buffer each time
• We could move our typescript into a single file and include this in rust with something like js sandbox
  • But again - to make any changes to this function would involve re-importing this into rust land
• We could try to use some sort of thread safe container, e.g. a channel, and write to it from one thread and read from it in another
  • But javascript is single threaded, we’d need to block the main thread while we wait to see if the consumer thread (our audio stream) needs more data
• We could create an abstract-syntax-tree-like structure to describe the waveforms and pass this to WASM on start and whenever it changes

That last option sounds like the rejected “Poke the editable values into rust wasm” idea from above

My earlier idea was to poke individual values (e.g. 440hz for A4) into WASM, but if we can poke the whole function in in a way that is unpackable and runnable by rust then this is a best-of-both-worlds scenario… Just needs a bit of work.

We’ve gone from creating a simple noise in WASM to thinking about implementing an AST to describe the sound in both typescript and rust. This is classic scope creep and usually I would embrace this but let’s try just poking in values first.

Pitch Slider

Why does the audio “click” when changing pitch?

They are parts of the sine wave that change too quickly. You can see this in the wave:

You can see the old pitch end and the new pitch begin, but the jump between the waves is too quick. The gradient of the curve changes too quickly. How can we fix this?

We could wait until the samples from the old wave are similar to the new wave before switching pitches. However, we might want to switch between waves that don’t span the same values

We could take the difference in gradient at T0 and T1 and cap the change at some sort of upper bound. Because our X axis is a constant sample rate we can just take the difference in Y value and cap it. However, this will not work when the wave changes direction from a positive to a negative.

We could take the velocity of the change and interpolate between the old velocity and the new velocity. Let’s see if we can do that in code.

Velocity Change Limited Pitch Slider

Here we interpolate towards the target PITCH capped at a change rate of CHANGE_LIMIT.

let mut current_pitch = PITCH;
let mut angle = 0.0f32;

let mut next_value = move || {
    if (PITCH - current_pitch).abs() > CHANGE_LIMIT {
        current_pitch += (PITCH - current_pitch).signum() * CHANGE_LIMIT;
    } else {
        current_pitch = PITCH;
    }

    angle += current_pitch * 2.0 * 3.141592 / sample_rate as f32;
    return angle.sin();
};

We can be pretty aggressive with our change limit. With low values (<0.04) you can hear the wave ramp up and down in pitch pretty clearly. And with higher values (0.5) there is very little audible interpolation and no popping.

Conclusion

In this post I walk through the creation of some audio with rust, wasm, cpal, and chartjs. We expand our solution to support clickless pitch shifting.

It sounds nothing like a guitar at the moment, so next we’ll make it sound more realistic!