TuneNN: A transformer-based network model for pitch detection

joonatan|2 years ago

Could someone fill me in why would machine learning be necessary for pitch detection? Isn't it something that could just be solved with FFT or it's a much more complicated task?

ks2048|2 years ago

Pitch is a *subjective* property, inherently tied to the complex processing humans use to perceive sounds. “Simple” physical measures like fundamental frequency of a periodic signal are very closely related, but for real-world audio (aren’t really periodic), the relationship is more complicated.

CMLab|2 years ago

A transformer-based network model, pitch tracking for musical instruments.

The timbre of musical notes is the result of various combinations and transformations of harmonic relationships, harmonic strengths and weaknesses, instrument resonant peaks, and structural resonant peaks over time.

It utilizes the transformer-based tuneNN network model for abstract timbre modeling, supporting tuning for 12+ instrument types.

azinman2|2 years ago

This smells like an automated summary.

rrherr|2 years ago

How does the accuracy of this compare to CREPE?

https://github.com/marl/crepe

https://github.com/maxrmorrison/torchcrepe

Does anyone know what the current state of the art is, within the Music Information Retrieval community?

CMLab|2 years ago

CREPE generally has high latency and error rates in instrument pitch recognition, especially for guitar instruments. Our team will release benchmark test data and results later.

bravura|2 years ago

What's the license?

What are your thoughts on PESTO which learns pitch-prediction very well with a small network, and uses a self-supervised objective?

https://arxiv.org/abs/2309.02265

https://github.com/SonyCSLParis/pesto

vessenes|2 years ago

This is cool! The very best software-based tuning tech out there is probably in piano tuning apps; they cost hundreds of dollars+ and are specifically made to report on harmonics and other piano nuances.

Do you have any comparisons against other pitch detection tech? Accuracy? Delay/Responsiveness? I assume it's much more compute work than a handcoded FFT type pitch detector.

I think it's possible this would find utilization in the piano world if the output offers something new / something that can analyze what a piano tuning maestro can hear and make it accessible to a mid-tier tuner.

jansommer|2 years ago

Sounds like you know a thing or two about pitch detection... I've been working on a C implementation of YIN and PYIN (a real GPL minefield for someone wanting to provide the end result as MIT/public domain!), and am wondering if it's a good choice for real time, cpu-bound speech pitch detection, or if there's better ways. May I ask what your thoughts are on this?

CMLab|2 years ago

Based on our current tests, our algorithm shows significantly higher accuracy and robustness compared to traditional digital signal algorithms such as PEF, NCF, YIN, HPS, etc. Our team is working diligently, and we will release benchmark test data and results in the near future.

ks2048|2 years ago

That's interesting. Can you point to one of these piano tuning apps that are $100+?

squidsoup|2 years ago

It might be worth pointing out that the banjo model is for a four string banjo, given a five string banjo is the more common instrument.

filterfiber|2 years ago

Does anyone know where I should look if I want to detect specific sounds? Like a smoke alarm, food bowl dispenser (its very distinct), cat meowing, 3d printer collision, that sort of thing?

UncleEntity|2 years ago

This one seems interesting -> https://paperswithcode.com/paper/joint-audio-and-speech-unde...

unknown|2 years ago

[deleted]

tristanc|2 years ago

Use any model trained on the AudioSet dataset. There is one called EfficientAT i think that I use regularly and is pretty reliable

mistercheph|2 years ago

You would learn how to do this in the first & second chapters of the fast.ai course.

ranting-moth|2 years ago

To the dev: the tuner gives me an incredibly high error window with the following message. It doesn't prompt to access the mic (I think that's related). Ubuntu/KDE/Firefox:

CMLab|2 years ago

Thank you for providing error feedback. We will work hard to address it. Currently, the model-related data is relatively large, which may be related to network speed.

39 comments