AI vs. Mathematical Approaches to Note Detection

Polyphonic note detection — identifying multiple simultaneous notes in real time — is one of the hardest problems in audio signal processing. Two fundamentally different schools of thought have emerged: classical mathematical methods and AI/machine learning approaches.

Both can detect notes. Neither is universally better. The right choice depends entirely on what you’re building and what tradeoffs you can accept.

The Classical Mathematical Approach

Mathematical note detection uses deterministic signal processing algorithms. The input goes in, the math runs, the notes come out. No training data, no neural networks, no black boxes.

Common Techniques

Autocorrelation analyzes how a signal correlates with delayed copies of itself. Periodic signals (musical notes) produce peaks at delays corresponding to their fundamental period. Simple, fast, and effective for monophonic detection. Struggles with polyphonic content because multiple periodicities overlap and confuse the correlation pattern.

Harmonic product spectrum multiplies the frequency spectrum by downsampled copies of itself. Harmonics of the same fundamental align and reinforce, creating clear peaks at fundamental frequencies. Works reasonably for polyphonic content but suffers when harmonics from different notes overlap — which happens constantly on guitar.

Cepstral analysis transforms the spectrum a second time to separate the “shape” of the spectrum (timbre) from the “fine structure” (pitch). Useful for identifying fundamental pitch independent of timbre, but computationally expensive and not inherently polyphonic.

Subharmonic summation scans for fundamental frequencies by checking whether the expected harmonics are present in the spectrum. For each candidate fundamental, it sums the energy at the 2nd, 3rd, 4th harmonics and so on. Strong total energy = likely real note. Can handle polyphony by iterating: find the strongest note, subtract it, find the next.

Strengths

• Deterministic and predictable. Given the same input, you always get the same output. No randomness, no model variance.
• Explainable. You can trace exactly why a note was detected or missed. Every step is auditable.
• No training data required. Works on any instrument from day one. No need for labeled datasets.
• Low computational cost. Most mathematical methods run efficiently on minimal hardware.
• No model size or memory overhead. The algorithm is compact — just code, no weight matrices.

Weaknesses

• Harmonic ambiguity. Is 220Hz the fundamental of A3, or the second harmonic of A2? Mathematical methods must use heuristics to resolve this, and the heuristics are fragile.
• Overlapping harmonics. When two notes share harmonic frequencies (extremely common on guitar), classical methods struggle to separate them cleanly.
• Instrument-specific tuning. A method optimized for piano performs differently on guitar. The parameters — window sizes, harmonic weighting, thresholds — need manual tuning per instrument class.
• Onset detection is a separate problem. Most mathematical pitch detectors work on sustained signals. Detecting when a new note starts (especially during a strum where all strings excite simultaneously) requires separate onset detection, which adds complexity and latency.
• Graceful degradation is hard. When the algorithm fails, it tends to fail completely — phantom notes, octave errors, missed notes. There’s no “confidence” to fall back on.

The AI / Machine Learning Approach

AI-based note detection trains a model on large datasets of labeled audio. The model learns to map audio features to note activations — essentially learning the relationship between “what sound goes in” and “what notes are present.”

Common Architectures

Convolutional Neural Networks (CNNs) treat spectrograms as images and learn to identify patterns that correspond to specific notes. Can handle polyphony naturally since the output is a multi-label classification (multiple notes can be “on” simultaneously).

Recurrent Neural Networks (RNNs/LSTMs) process audio sequentially, maintaining memory of recent context. Good at tracking note onsets and offsets over time, handling the temporal structure of music.

Transformer models apply attention mechanisms to audio features, relating different time frames and frequency regions to each other. State of the art for offline transcription tasks.

Hybrid models combine classical feature extraction (spectrograms, chromagrams) with neural networks that interpret those features. Often the most practical approach for real-time use.

Strengths

• Handles polyphony naturally. Neural networks learn complex relationships between overlapping harmonics. They don’t need explicit harmonic separation — they learn to detect notes despite the overlap.
• Resolves ambiguity with context. Is 220Hz the fundamental or a harmonic? A well-trained model uses surrounding context — other detected frequencies, temporal patterns, timbral cues — to make the right call.
• Adapts to instruments. With diverse training data, models generalize across instruments without manual parameter tuning.
• Confidence scores. Neural networks naturally output probability scores — “85% confident this is an A3” — which enables graceful degradation and thresholding.
• Onset and pitch in one pass. Modern architectures handle onset detection, pitch estimation, and note tracking in a single forward pass.

Weaknesses

• Latency. Neural networks process audio in frames. Larger frames = more context = better accuracy, but more latency. Real-time constraints force a tradeoff between detection quality and responsiveness.
• Computational cost. Even small models require matrix multiplications every audio frame. On a musician’s laptop running a DAW with 30 plugins, every CPU cycle counts.
• Training data dependency. The model is only as good as its training data. Instruments, playing styles, or recording conditions not represented in training data may get poor results.
• Black box behavior. When the model makes an error, it’s difficult to understand why. Debugging is empirical (try different training data, architectures, hyperparameters) rather than analytical.
• Model size. Weight files add to plugin size and memory footprint. A transformer model might need 50-200MB of weights — significant for an audio plugin.
• Generalization gaps. A model trained primarily on piano and guitar may struggle with harp, banjo, or ukulele. Covering every instrument requires enormous datasets.

The Real-Time Constraint

For a recording studio plugin, the detection must happen in real time — ideally under 10ms of latency. This constraint dramatically narrows the field:

Mathematical methods excel here. Autocorrelation and spectral analysis can run in microseconds per audio frame. The algorithms are lightweight and predictable in their CPU usage.

Large AI models struggle. Transformer-based transcription models that achieve state-of-the-art accuracy on benchmarks often require 50-100ms of audio context and significant GPU resources. Not viable in a real-time plugin.

Small, optimized AI models can work. Lightweight architectures specifically designed for real-time inference — small RNNs, quantized CNNs, distilled models — can run within real-time constraints, but with reduced accuracy compared to their larger counterparts.

This is why the most practical real-time systems tend to be hybrids: classical signal processing for the time-critical first pass, with learned components handling the harder disambiguation tasks.

Accuracy Comparison

On standard benchmarks (like the MIREX multi-F0 estimation task), the leaderboards tell a clear story:

• Pure mathematical methods: 60-75% F-measure on polyphonic piano, lower on guitar
• Deep learning models: 80-90%+ F-measure, with state-of-the-art exceeding 90% on some datasets
• Hybrid approaches: Competitive with deep learning while maintaining lower computational cost

But benchmarks don’t tell the whole story. Most benchmark datasets are piano-heavy, recorded in clean conditions, at standard tuning. Real-world guitar — alternate tunings, heavy distortion, palm muting, harmonics, string noise — is harder than any benchmark.

What Matters for a Stereo Imaging Plugin

For a plugin like TONIQ, note detection serves a specific purpose: routing audio to the correct stereo position and effects zone. This creates a unique set of requirements:

Speed over perfect accuracy. A 5ms detection is more valuable than a 50ms detection that catches one more note. The stereo image needs to track the player’s performance in real time. A slightly imperfect detection that’s fast feels musical. A perfect detection that’s slow feels laggy.

Robustness over coverage. It’s better to confidently detect 5 out of 6 notes in a chord than to attempt all 6 and occasionally hallucinate phantom notes. A phantom note in the wrong stereo position is more distracting than a missed note that stays centered.

Adaptability matters. Guitarists play everything from clean jazz arpeggios to high-gain metal riffs. The detection must handle the full range without manual configuration. A system that needs the user to select “clean” or “distorted” mode is a bad user experience.

Graceful failure is essential. When the detection can’t confidently identify a note — during fast strumming, heavy distortion, or unusual techniques — the system should default to something safe rather than making a wild guess. In stereo imaging, “safe” means keeping uncertain content near the center rather than panning it to an extreme position.

The Trend

The audio industry is moving toward hybrid approaches — and for good reason. Pure mathematical methods hit a ceiling on polyphonic accuracy. Pure AI methods hit a wall on computational cost and latency.

The most compelling real-time systems combine the strengths of both: fast, efficient signal processing for the foundational analysis, with learned components adding musical intelligence where classical methods struggle.

This mirrors a broader trend in audio software. The era of choosing “DSP or AI” is ending. The future is knowing when to use each — and building systems that leverage both seamlessly.