[francescopace]

Fun fact: I’m working on turning ESPectre into a Wi‑Fi Theremin (the musical instrument you play by moving your hands near an antenna).

The idea of “playing” by simply moving around a room sounds a bit ridiculous… but also kind of fun.

The key is the Moving Variance of the spatial turbulence: this value is continuous and stable, making it perfect for mapping directly to pitch/frequency, just like the original Theremin. Other features can be mapped to volume and timbre.

It’s pure signal processing, running entirely on the ESP32. Has anyone here experimented with audio synthesis or sonification using real-time signal processing?

[quinnjh]

Ive worked on some sonification projects that used signals from xbox kinect lidar, piezos, and other sensors. Co-author on paper i wrote developed a "strummable" theremin that divided physical space with invisible "strings" of various tunings. We preferred running synthesis on PC when possible and just outputting midi and OSC, as DSP on ESP32 has limits for what can be achieved in under 5-10ms. If the goal is hardware audio output, you may need to look into dedicated DSP chips and an audio shield for better DAC- but for prototyping can easily bang a square wave through any of esp32 pins

[francescopace]

Thanks for the insights Quinnjh! Would love to hear more about your invisible strings tuning system!

The ESP32-S3 extracts a moving variance signal from spatial turbulence (updates at 20-50 Hz), and I want to map this directly to audio frequency using a passive buzzer + PWM (square wave, 200-2000 Hz range).

Two quick questions:

1. Do you see any pitfalls with updating PWM frequency at 20-50 Hz for responsive theremin-like behavior?

2. Any recommendations on mapping strategies - linear, logarithmic (musical scale), or quantized to specific notes?

[4gotunameagain]

Great project and great idea, thank you for sharing !

I don't know if it's useful but one technique I have used in sonification during the experimentation phase is to skip the real time aspect, capture all the available "channels" and generate all the possible permutations of what is mapped where.

Then you can listen to the outputs, see what sounds good, and then test it in real time to check if the musicality is actually a result of the physical interaction and not an artifact or a product of noise.

[francescopace]

Thank you 4goturnamesagain.

My first step is to 'listen' to the raw channels and features to quickly find which mapping produces the most musically coherent (i.e., clean and physically predictable) output.

If it sounds like white noise, the mapping is bad or the signal is artifact.

If it sounds like a sine wave moving predictably, the physics are sound.

[James-Bennet]

By "GPLv3" do you mean "GPL-3.0-or-later"? https://www.gnu.org/licenses/gpl-faq.html#VersionThreeOrLate...

[cweagans]

I wonder if somebody could make an open hardware version of the Leap Motion with this technique (though I'm not sure how accurate/repeatable the sensing is - Leap Motion could detect with an accuracy of < 0.7mm)

[francescopace]

That's a great thought. The challenge is that Wi-Fi sensing (CSI) is measuring multipath changes across a few meters, not direct motion in a small volume like Leap Motion. I think its accuracy is measured in centimeters, not sub-millimeters.

[reconnecting]

I'm sure the kids will love this! Wi-Fi Theremin sounds great.

[francescopace]

You hit the nail on the head! That's precisely the motivation.

Having two kids myself, I've thought of turning it into a game: blindfolded hide-and-seek where the pitch of the Wi-Fi Theremin tells the seeker how close they are to the 'signal disruption' of the other person. It's essentially a real-time sonar game!