What it is
A web app where you upload a short clip of yourself playing guitar, and it tells you which effect is on it. The clip will go to a FastAPI backend, which runs it through a convolutional network trained from scratch and returns a prediction. The feature pipeline and the model are built; the training loop and the app itself are still to come.
It classifies into one of twelve: feedback delay, slapback delay, reverb, chorus, flanger, phaser, tremolo, vibrato, distortion, overdrive, EQ, or no effect.
What I built
The dataset. IDMT-SMT-AUDIO-EFFECTS, a public research set from Fraunhofer IDMT of guitar and bass recordings with effects applied. I use the guitar half and leave the bass, which comes to 34,452 clips across the twelve classes.
The splits. 24,090 training, 5,181 validation, 5,181 evaluation, split by source recording rather than by file.
The filenames encode instrument, playing technique, pitch, string and fret, so the same physical note appears once under every effect. Splitting at random would put one note in training and that same note in evaluation under a different effect, and the model could score well by recognising the note instead of the effect. Grouping on the note keeps every effect variant of one recording on the same side, and a check asserts that no recording appears in two splits.
Features. Each clip becomes a mel spectrogram: 22.05 kHz, 1024-sample frames, 256 hop, 128 mel bands, converted to decibels with an 80 dB floor. Normalisation statistics are computed from the training clips only, so nothing about validation or evaluation leaks into the inputs.
The model. A baseline CNN built from four convolutional blocks, 16 up to 128 channels, each with batch normalisation and max pooling, then global average pooling into a single linear layer over the twelve classes.
It comes to 98,940 parameters. Small on purpose: it is the baseline a fine-tuned pretrained model has to beat, and if it loses badly that comparison is the result worth reporting.
Different effect spectrograms:
What I learned
torchaudio reads the last dimension of a tensor as time. My stereo files are samples by channels, so a two-second clip arrived as two samples with 88,200 channels. It padded them out and handed back a spectrogram of nothing. Nothing raised an error. The normalisation statistics caught me the same way: measure them after normalising and they describe data that is already normalised.
The checks that have caught anything so far have all checked a shape or a count, never a result.
What it does not do yet
- The training loop. Everything feeding it exists; the loop itself is the piece still to write
- The web app. Planned, not started
- Need to test using real pedals, rather than just simulated ones
- Every clip is two seconds of one note or one chord, so nothing in training looks like playing