Source Separation and Restoration of Sound Components in Music Recordings (SeReCo)
The goal of the SeReCo2 project is develop matrix decomposition and source separation techniques for decomposing a music recording into musically meaningful (e.g. note-related) sound components. The project is funded by the German Research Foundation. On this website, we summarize the project's main objectives and provide links to project-related resources (data, demonstrators, websites) and publications.
- Principal investigator: Prof. Dr. Meinard Müller
- Programme period: 2021–2024
- Grant No. 328416299 (MU 2686/10-2)
- Project website of first phase (SeReCo)
Project Description
Source Separation and Restoration of Sound Components in Music Recordings (SeReCo)
This is a follow-up project, which continues the previous DFG-funded project "Source Separation and Restoration of Drum Sound Components in Music Recordings" [MU 2686/10-1] aiming at the development of techniques for separating and restoring sound events as occurring in complex music recordings. In the first phase ([MU 2686/10-1]), we focused on percussive sound sources, where we decomposed a drum recording into individual drum sound events. Using Non-Negative Matrix Factor Deconvolution (NMFD) as our central methodology, we studied how to generate and integrate audio- and score-based side information to guide the decomposition. We tested our approaches within concrete application scenarios, including audio remixing (redrumming) and swing ratio analysis of jazz music. In the second phase of the project ([MU 2686/10-2]), our goals are significantly extended. First, we want to go beyond the drum scenario by considering other challenging music scenarios, including piano music (e.g., Beethoven Sonatas, Chopin Mazurkas), piano songs (e.g., Klavierlieder by Schubert), and string music (e.g., Beethoven String Quartets). In these scenarios, our goal is to decompose a music recording into individual note-related sound events. As our central methodology, we develop a unifying audio decomposition framework that combines classical signal processing and machine learning with recent deep learning (DL) approaches. Furthermore, we adopt generative DL techniques for improving the perceptual quality of restored sound events. As a general goal, we investigate how prior knowledge, such as score information can be integrated into DL-based learning to improve the interpretability of the trained models.
Projektbeschreibung
Quellentrennung und Wiederherstellung von Klangkomponenten in Musikaufnahmen
Dieses Projekt ist eine Fortsetzung von [MU 2686/10-1] mit dem Ziel, Techniken zur Trennung und Wiederherstellung von Klangereignissen, wie sie bei komplexen Musikaufnahmen auftreten, zu entwickeln. In der ersten Phase ([MU 2686/10-1]) konzentrierten wir uns auf die Separation von Schlagzeugaufnahmen in individuelle Schlagzeugklangkomponenten. Unter Verwendung von Techniken der nicht-negativen Matrixzerlegung haben wir systematisch untersucht, wie sich Audio- und Notentext-basierte Seiteninformation generieren, integrieren und zur Steuerung der Zerlegung ausnutzen lässt. Unsere Verfahren wurden im Kontext konkreter Anwendungsszenarien wie dem Audio-Remixing (Redrumming) und der Swing-Analyse von Jazzmusik getestet. In der zweiten Projektphase ([MU 2686/10-2]) erweitern wir unsere Ziele erheblich. Zunächst gehen wir über das Schlagzeugszenario hinaus, indem wir andere komplexe Musikszenarien betrachten, einschließlich Klaviermusik (z.B. Beethoven-Sonaten, Chopin-Mazurkas), Klavierlieder (z. B. von Schubert) und Streichmusik (z.B. Beethoven-Streichquartette). In diesen Szenarien besteht unser Ziel darin, eine Musikaufnahme in einzelne notenbezogene Klangereignisse zu zerlegen. Als zentrale Methodik kombinieren wir klassische Techniken der Signalverarbeitung und des maschinellen Lernens mit aktuellen Deep-Learning-Ansätzen (DL). Weiterhin entwickeln wir generative DL-basierte Methoden, um die perzeptuelle Qualität der separierten Klangereignisse zu verbessern. Als ein übergeordnetes Ziel widmen wir uns der Frage, wie sich musikalische Vorkenntnisse in DL-basierte Lernverfahren integrieren lassen, um auf diese Weise die Interpretierbarkeit der trainierten Modelle zu verbessern.
Projected-Related Resources and Demonstrators
The following list provides an overview of the most important publicly accessible sources created in the SeReCo2 project:
-
Source Separation of Piano Concertos with Test-Time Adaptation (ISMIR2022)
-
Sync Toolbox: Python package with reference implementations for efficient, robust, and accurate music synchronization based on dynamic time warping (DTW)
Projected-Related Publications
The following publications reflect the main scientific contributions of the work carried out in the SeReCo2 project.
- Meinard Müller, Yigitcan Özer, Michael Krause, Thomas Prätzlich, and Jonathan Driedger
Sync Toolbox: A Python Package for Efficient, Robust, and Accurate Music Synchronization
Journal of Open Source Software (JOSS), 6(64): 1–4, 2021. PDF Demo DOI@article{MuellerZ21_SyncToolbox_JOSS, author = {Meinard M{\"u}ller and Yigitcan {\"O}zer and Michael Krause and Thomas Pr{\"a}tzlich and Jonathan Driedger}, title = {{S}ync {T}oolbox: {A} {P}ython Package for Efficient, Robust, and Accurate Music Synchronization}, journal = {Journal of Open Source Software ({JOSS})}, volume = {6}, number = {64}, year = {2021}, pages = {3434:1--4}, doi = {10.21105/joss.03434}, url-pdf = {2021_MuellerOKPD_SyncToolbox_JOSS.pdf}, url-demo = {https://github.com/meinardmueller/synctoolbox} } - Yigitcan Özer, Jonathan Hansen, Tim Zunner, and Meinard Müller
Investigating Nonnegative Autoencoders for Efficient Audio Decomposition
In Proceedings of the European Signal Processing Conference (EUSIPCO): 254–258, 2022. Details@inproceedings{OezerHZM22_NAE_EUSIPCO, author = {Yigitcan {\"O}zer and Jonathan Hansen and Tim Zunner and Meinard M{\"u}ller}, title = {Investigating Nonnegative Autoencoders for Efficient Audio Decomposition}, booktitle = {Proceedings of the European Signal Processing Conference ({EUSIPCO})}, year = {2022}, pages = {254--258}, url-details = {https://ieeexplore.ieee.org/document/9909787} } - Yigitcan Özer, Matej Istvanek, Vlora Arifi-Müller, and Meinard Müller
Using Activation Functions for Improving Measure-Level Audio Synchronization
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 749–756, 2022. PDF DOI@inproceedings{OezerIAM22_ActivationMusicSync_ISMIR, author = {Yigitcan {\"O}zer and Matej Istvanek and Vlora Arifi-M{\"u}ller and Meinard M{\"u}ller}, title = {Using Activation Functions for Improving Measure-Level Audio Synchronization}, booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})}, address = {Bengaluru, India}, year = {2022}, pages = {749--756}, doi = {}, url-pdf = {2022_OezerIAM_MusicSync_ISMIR_ePrint.pdf} } - Yigitcan Özer, Michael Krause, and Meinard Müller
Using the Sync Toolbox for an Experiment on High-Resolution Music Alignment
In Demos and Late Breaking News of the International Society for Music Information Retrieval Conference (ISMIR), 2021. PDF@inproceedings{OezerKM21_SyncToolbox_ISMIR-LBD, author = {Yigitcan {\"O}zer and Michael Krause and Meinard M{\"u}ller}, title = {Using the Sync Toolbox for an Experiment on High-Resolution Music Alignment}, booktitle = {Demos and Late Breaking News of the International Society for Music Information Retrieval Conference ({ISMIR})}, address = {Online}, year = {2021}, url-pdf = {2021_OezerKM_SyncToolbox_ISMIR-LBD.pdf} } - Yigitcan Özer and Meinard Müller
Source Separation of Piano Concertos with Test-Time Adaptation
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR): 493–500, 2022. PDF Demo DOI@inproceedings{OezerM22_PianoSepAdapt_ISMIR, author = {Yigitcan {\"O}zer and Meinard M{\"u}ller}, title = {Source Separation of Piano Concertos with Test-Time Adaptation}, booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})}, address = {Bengaluru, India}, year = {2022}, pages = {493--500}, doi = {}, url-demo = {https://www.audiolabs-erlangen.de/resources/MIR/2022-PianoSep/}, url-pdf = {2022_OezerM_PianoSepAdapt_ISMIR_ePrint.pdf} } - Yigitcan Özer and Meinard Müller
A Computational Approach for Creating Orchestra Tracks from Piano Concerto Recordings
In Proceedings of the Deutsche Jahrestagung für Akustik (DAGA): 1370–1373, 2023. PDF@inproceedings{OezerM23_PianoTracks_DAGA, author = {Yigitcan {\"O}zer and Meinard M{\"u}ller}, title = {A Computational Approach for Creating Orchestra Tracks from Piano Concerto Recordings}, booktitle = {Proceedings of the {D}eutsche {J}ahrestagung f{\"u}r {A}kustik ({DAGA})}, address = {Hamburg, Germany}, year = {2023}, pages = {1370--1373}, url-pdf = {2023_OezerM_PCPipeline_DAGA_ePrint.pdf} } - Yigitcan Özer, Simon Schwär, Vlora Arifi-Müller, Jeremy Lawrence, Emre Sen, and Meinard Müller
Piano Concerto Dataset (PCD): A Multitrack Dataset of Piano Concertos
Transaction of the International Society for Music Information Retrieval (TISMIR), Accepted 2023. Demo DOI@article{OezerSAJEM_PCD_TISMIR, author = {Yigitcan {\"O}zer and Simon Schw{\"a}r and Vlora Arifi-M{\"u}ller and Jeremy Lawrence and Emre Sen and Meinard M{\"u}ller}, title = {Piano Concerto Dataset ({PCD}): A Multitrack Dataset of Piano Concertos}, journal = {Transaction of the International Society for Music Information Retrieval ({TISMIR})}, volume = {}, number = {}, pages = {}, year = {Accepted 2023}, doi = {}, url = {}, url-pdf = {}, url-demo = {} } - Nazif Can Tamer, Xavier Serra, Yigitcan Özer, and Meinard Müller
TAPE: A Timbre-Aware Pitch Estimator
In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): 1–5, 2023. DOI@inproceedings{TamerSOM23_TAPE_ICASSP, author = {Nazif Can Tamer and Xavier Serra and Yigitcan {\"O}zer and Meinard M{\"u}ller}, title = {{TAPE}: {A} Timbre-Aware Pitch Estimator}, booktitle = {Proceedings of the {IEEE} International Conference on Acoustics, Speech, and Signal Processing ({ICASSP})}, address = {Rhodes Island, Greece}, year = {2023}, pages = {1--5}, doi = {10.1109/ICASSP49357.2023.10096762} } - Nazif Can Tamer, Yigitcan Özer, Meinard Müller, and Xavier Serra
High-Resolution Violin Transcription Using Weak Labels
In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Accepted 2023.@inproceedings{TamerOMS23_Violin_ISMIR, author = {Nazif Can Tamer and Yigitcan {\"O}zer and Meinard M{\"u}ller and Xavier Serra}, title = {High-Resolution Violin Transcription Using Weak Labels}, booktitle = {Proceedings of the International Society for Music Information Retrieval Conference ({ISMIR})}, address = {Milano, Italy}, year = {Accepted 2023}, pages = {} }