Write-up
Melody Diffuser
A gesture-conditioned discrete diffusion transformer for symbolic melody.
- parameters
- 32.2M
- parameters
- training excerpts
- 10M+
- training excerpts
- diffusion steps
- 64
- diffusion steps
- gesture classes
- 8
- gesture classes
Problem
Text-to-music systems produce convincing style and timbre but give little direct control over melodic contour and rhythm. Melody Diffuser makes the contour itself the input: the user traces the shape of a melody with a finger, and the model generates notes that follow it.
Data
The training set is over 10 million four-measure monophonic excerpts, quantised to sixteenth notes, so each melody is a sequence of 64 tokens. The vocabulary has 130 tokens: MIDI pitches 0–127, plus 128 to hold the previous note and 129 for a rest.
No dataset pairs melodies with gestures, so the labels are derived from the music. The pitch movement at each step is mapped to one of eight gesture classes. At inference time the same classes are computed from fingertip movement tracked by MediaPipe, so the model is conditioned on the same kind of signal it was trained on.
| Token | Gesture | Meaning |
|---|---|---|
| 0 | Hold | sustain the current note |
| 1 | Repeat | same pitch again |
| 2 · 3 · 4 | Up | small, medium or large step up |
| 5 · 6 · 7 | Down | small, medium or large step down |
Model
A transformer predicts the clean melody from a noised one. Every block attends to the gesture sequence through cross-attention, so the condition shapes each layer rather than only the input.
| Component | Configuration |
|---|---|
| Width / depth / heads | 512 / 6 blocks / 8 |
| Block | RMSNorm → self-attention → RMSNorm → cross-attention to gestures → RMSNorm → SwiGLU (inner 2048) |
| Gesture input | learned embedding plus learned position embedding |
| Timestep | sinusoidal encoding → two-layer MLP, added to every token |
| Output | logits over the 130-token vocabulary, tied to the input embedding |
| Parameters | 32.2M |
# in every transformer block cond = gesture_emb + gesture_pos_emb attn, _ = self.cross_attn(h, cond, cond) x = x + attn
Diffusion
The forward process corrupts tokens by uniform replacement. Over T = 64 steps with a linear β schedule from 10⁻⁴ to 0.05, each token is replaced by a random token with probability 1 − ∏(1 − βₜ). The network is trained with cross-entropy to recover the original sequence at a random step. The gesture condition is dropped for 15% of training examples, which enables classifier-free guidance at generation time: a scale that trades off following the gesture against producing a natural melody.
Training
| Setting | Value |
|---|---|
| Optimiser | AdamW, learning rate 2·10⁻⁴, weight decay 0.01 |
| Schedule | linear warm-up for 2 epochs, cosine decay |
| Batch / epochs | 2,048 / 10 |
| Gradient clipping | 1.0 |
| Validation | 100,000 held-out excerpts, every 2,000 steps |
| Hardware | NVIDIA A100 |
Serving
The demo captures gestures in the browser and sends the token sequence to the model, which runs on a cloud GPU (NVIDIA T4) through Replicate. The returned tokens are decoded to notes, drawn as a piano roll and played back in the browser with Tone.js.
Next
Polyphony: a variational autoencoder over four-part chorales, so diffusion can run in a latent space of harmony rather than a single line.