\n", "\n", "

\n", "In Chapter 8 of [Müller, FMP, Springer 2015] on audio decomposition, we present a challenging research direction that is closely related to source separation. Within this wide research area, we consider three subproblems: harmonic–percussive separation, main melody extraction, and score-informed audio decomposition. Within these scenarios, we discuss a number of key techniques including instantaneous frequency estimation, fundamental frequency (F0) estimation, spectrogram inversion, and nonnegative matrix factorization (NMF). Furthermore, we encounter a number of acoustic and musical properties of audio recordings that have been introduced and discussed in previous chapters.\n", "

\n", "\n", "\n",
"8.1 Harmonic–Percussive Separation

\n",
"8.2 Melody Extraction

\n",
"8.3 NMF-Based Audio Decomposition

\n",
"8.4 Further Notes\n",
"

Topic | Relation to [Müller, FMP, Springer 2015] & Description | HTML | IPYNB |

Harmonic–Percussive Separation (HPS) | [Section 8.1.1] Harmonic sound; percussive sound; median filter; binary mask; soft mask; signal reconstruction; HPR experiments; Violin–Castanets example; audio examples (diverse) | [html] | [ipynb] |

Harmonic–Residual–Percussive Separation (HRPS) | [Section 8.1.1, Exercise 8.5] Separation factor; residual component; binary mask; cascaded HRPS; Violin–Applause–Castanets example; Bornemark example (Stop Messing With Me) | [html] | [ipynb] |

Signal Reconstruction | [Section 8.1.2] Inverse DFT; inverse STFT; modified STFT; overlap–add procedure; Griffin–Lim optimization problem | [html] | [ipynb] |

Applications of HPS and HPRS | [Section 8.1.3] Feature enhancement; chroma feature; onset detection; time-scale modification; Violin–Castanets example | [html] | [ipynb] |

Instantaneous Frequency Estimation | [Section 8.2.1] Phase wrapping; principle argument; exponential function; phase prediction; instantaneous frequency (IF); polar coordinates; bin offset; visualization of IF values; dependency on hop size; C4 piano example | [html] | [ipynb] |

Salience Representation | [Section 8.2.2] Log-frequency spectrogram; instantaneous frequency; binning; harmonic summation; salience; Weber example (Freischütz) | [html] | [ipynb] |

Fundamental Frequency Tracking | [Section 8.2.3] Frequency trajectory; sonification; salience representation; continuity constraint; dynamic programming; score-informed constraint; constraint region; Weber example (Freischütz); Bornemark example (Stop Messing With Me) | [html] | [ipynb] |

Melody Extraction and Separation | [Section 8.2, Section 8.2.3.3] Melody; salience representation; predominant frequency; F0-trajectory; separation; binary mask; harmonics; frequency-dependent tolerance; signal reconstruction; Weber example (Freischütz); Bornemark example (Stop Messing With Me) | [html] | [ipynb] |

Nonnegative Matrix Factorization (NMF) | [Section 8.3.1] Matrix factorization; nonnegative matrix; rank; template vector; activation vector; gradient descent; multiplicative update rule; magnitude spectrogram; Chopin example (Op. 28, No. 4); C-major scale example | [html] | [ipynb] |

NMF-Based Spectrogram Factorization | [Section 8.3.2] Spectrogram factorization; score-informed NMF; initialization; template constraints; pitch information; activation constraints; score information; onset model; Chopin example (Op. 28, No. 4) | [html] | [ipynb] |

NMF-Based Audio Decomposition | [Section 8.3.3] Score-informed NMF; activation matrix; spectral masking; audio decomposition; audio editing; Chopin example (Op. 28, No. 4) | [html] | [ipynb] |