{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Date: Jul 2019<br>Programmer:  Patricio López-Serrano, Yigitcan Özer\n",
    "This demo illustrates decomposition of a loop-based electronic dance music track, following [1].\n",
    "\n",
    "\n",
    "#### The notebook proceeds in the following steps:\n",
    "<br>1. Load audio files for the complete, downmixed track, as well as for the individual loops that the track contains.\n",
    "<br>2. Compute STFTs for all audio data.\n",
    "<br>3. Each loop becomes a fixed template (\"page\") in the tensor W.\n",
    "<br>The track spectrogram is the target to approximate, V.\n",
    "<br>We wish to learn the activation matrix H, which answers the question\n",
    "<br>\"Where was each loop activated throughout % the track?\"\n",
    "<br>4. Visualize results.\n",
    "\n",
    "\n",
    "#### References\n",
    "<br>[1] Patricio Lopez-Serrano, Christian Dittmar, Jonathan Driedger, and\n",
    "<br>Meinard Mueller.\n",
    "<br>Towards modeling and decomposing loop-based electronic music.\n",
    "<br>In Proceedings of the International Conference\n",
    "<br>on Music Information Retrieval (ISMIR), pages 502–508,\n",
    "<br>New York City, USA, August 2016.\n",
    "\n",
    "\n",
    "#### If you use the 'NMF toolbox' please refer to:\n",
    "[2] Patricio López-Serrano, Christian Dittmar, Yiğitcan Özer, and Meinard Müller\n",
    "<br>NMF Toolbox: Music Processing Applications of Nonnegative Matrix Factorization\n",
    "<br>In Proceedings of the International Conference on Digital Audio Effects (DAFx), 2019.\n",
    "\n",
    "#### License:\n",
    "This file is part of 'NMF toolbox'. 'NMF toolbox' is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. 'NMF toolbox' is distributed in the hope that it will be useful, but ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.\n",
    "\n",
    "#### You should have received a copy of the GNU General Public License along<br>with 'NMF toolbox'. If not, see http://www.gnu.org/licenses/."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import scipy.io.wavfile as wav\n",
    "import matplotlib.pyplot as plt\n",
    "import IPython.display as ipd\n",
    "from copy import deepcopy\n",
    "\n",
    "from NMFToolbox.forwardSTFT import forwardSTFT\n",
    "from NMFToolbox.inverseSTFT import inverseSTFT\n",
    "from NMFToolbox.logFreqLogMag import logFreqLogMag\n",
    "from NMFToolbox.initActivations import initActivations\n",
    "from NMFToolbox.NMFD import NMFD\n",
    "from NMFToolbox.NMFdiag import NMFdiag\n",
    "from NMFToolbox.convModel import convModel\n",
    "from NMFToolbox.visualizeComponentsNMF import visualizeComponentsNMF\n",
    "from NMFToolbox.alphaWienerFilter import alphaWienerFilter\n",
    "from NMFToolbox.utils import EPS, make_monaural, pcmInt16ToFloat32Numpy\n",
    "from NMFToolbox.semiFixedComponentConstraintsNMF import semiFixedComponentConstraintsNMF\n",
    "\n",
    "inpPath = '../data/'\n",
    "outPath = 'output/'\n",
    "\n",
    "filename = 'LSDDM_EM_track.wav'\n",
    "\n",
    "filenameFX = 'LSDDM_EM_Effects.wav'\n",
    "filenameBass = 'LSDDM_EM_bass.wav'\n",
    "filenameMelody = 'LSDDM_EM_melody.wav'\n",
    "filenameDrums = 'LSDDM_EM_drums.wav'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Load the audio signal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "fs, xTr = wav.read(os.path.join(inpPath, filename))\n",
    "\n",
    "fsFX, xFX = wav.read(os.path.join(inpPath, filenameFX))\n",
    "fsBass, xBass = wav.read(os.path.join(inpPath, filenameBass))\n",
    "fsMelody, xMelody = wav.read(os.path.join(inpPath, filenameMelody))\n",
    "fsDrums, xDrums = wav.read(os.path.join(inpPath, filenameDrums))\n",
    "\n",
    "# make monaural if necessary\n",
    "xTr = make_monaural(xTr)\n",
    "xFX = make_monaural(xFX)\n",
    "xBass = make_monaural(xBass)\n",
    "xMelody = make_monaural(xMelody)\n",
    "xDrums = make_monaural(xDrums)\n",
    "\n",
    "# int16 -> float32 conversion\n",
    "xTr = pcmInt16ToFloat32Numpy(xTr)\n",
    "xFX = pcmInt16ToFloat32Numpy(xFX)\n",
    "xBass = pcmInt16ToFloat32Numpy(xBass)\n",
    "xMelody = pcmInt16ToFloat32Numpy(xMelody)\n",
    "xDrums = pcmInt16ToFloat32Numpy(xDrums)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Compute STFT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# spectral parameters\n",
    "paramSTFT = dict()\n",
    "paramSTFT['blockSize'] = 4096;\n",
    "paramSTFT['hopSize'] = 2048;\n",
    "paramSTFT['winFunc'] = np.hanning(paramSTFT['blockSize'])\n",
    "paramSTFT['reconstMirror'] = True\n",
    "paramSTFT['appendFrame'] = True\n",
    "paramSTFT['numSamples'] = len(xTr)\n",
    "\n",
    "# STFT computation\n",
    "XTr, ATr, PTr = forwardSTFT(xTr, paramSTFT);\n",
    "\n",
    "# get dimensions and time and freq resolutions\n",
    "numBinsTr, numFramesTr = XTr.shape\n",
    "deltaT = paramSTFT['hopSize'] / fs\n",
    "deltaF = fs / paramSTFT['blockSize']\n",
    "\n",
    "# get logarithmically-spaced frequency axis version for visualization\n",
    "logFreqLogMagA, logFreqAxis = logFreqLogMag(ATr, deltaF)\n",
    "numLogBins = len(logFreqAxis)\n",
    "\n",
    "# repeat for FX loop ---------------------------------------------------\n",
    "paramSTFT['numSamples'] = len(xFX);\n",
    "\n",
    "# STFT computation\n",
    "XFX, AFX, PFX = forwardSTFT(xFX, paramSTFT)\n",
    "\n",
    "# get dimensions and time and freq resolutions\n",
    "numBinsFX, numFramesFX = XFX.shape\n",
    "\n",
    "# get logarithmically-spaced frequency axis version for visualization purposes\n",
    "logFreqLogMagAFX, logFreqAxisFX = logFreqLogMag(AFX, deltaF)\n",
    "numLogBinsFx = len(logFreqAxisFX)\n",
    "\n",
    "# repeat for bass loop ---------------------------------------------------\n",
    "paramSTFT['numSamples'] = len(xBass);\n",
    "\n",
    "# STFT computation\n",
    "XBass, ABass, PBass = forwardSTFT(xBass, paramSTFT)\n",
    "\n",
    "# get dimensions and time and freq resolutions\n",
    "numBinsBass, numFramesBass = XBass.shape\n",
    "\n",
    "# get logarithmically-spaced frequency axis version for visualization purposes\n",
    "logFreqLogMagABass, logFreqAxisBass = logFreqLogMag(ABass, deltaF)\n",
    "numLogBinsBass = len(logFreqAxisBass)\n",
    "\n",
    "# repeat for Melody loop ---------------------------------------------------\n",
    "paramSTFT['numSamples'] = len(xMelody)\n",
    "\n",
    "# STFT computation\n",
    "XMelody, AMelody, PMelody = forwardSTFT(xMelody, paramSTFT)\n",
    "\n",
    "# get dimensions and time and freq resolutions\n",
    "numBinsMelody, numFramesMelody = XMelody.shape\n",
    "\n",
    "# get logarithmically-spaced frequency axis version for visualization purposes\n",
    "logFreqLogMagAMelody, logFreqAxisMelody = logFreqLogMag(AMelody, deltaF)\n",
    "numLogBinsMelody = len(logFreqAxisMelody)\n",
    "\n",
    "# repeat for Drums loop ---------------------------------------------------\n",
    "paramSTFT['numSamples'] = len(xDrums)\n",
    "\n",
    "# STFT computation\n",
    "XDrums, ADrums, PDrums = forwardSTFT(xDrums, paramSTFT)\n",
    "\n",
    "# get dimensions and time and freq resolutions\n",
    "numBinsDrums, numFramesDrums = XDrums.shape\n",
    "\n",
    "# get logarithmically-spaced frequency axis version for visualization purposes\n",
    "logFreqLogMagADrums, logFreqAxisDrums = logFreqLogMag(ADrums, deltaF)\n",
    "numLogBinsDrums = len(logFreqAxisDrums)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Apply NMF variants to STFT magnitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set common parameters\n",
    "numComp = 4\n",
    "numIter = 30\n",
    "numTemplateFrames = deepcopy(numFramesBass)\n",
    "\n",
    "initW = list()\n",
    "initW.append(ADrums)\n",
    "initW.append(AMelody)\n",
    "initW.append(ABass)\n",
    "initW.append(AFX)\n",
    "\n",
    "# generate initial activations\n",
    "paramActivations = dict()\n",
    "paramActivations['numComp'] = numComp\n",
    "paramActivations['numFrames'] = numFramesTr\n",
    "initH = initActivations(paramActivations, 'uniform')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 213880 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# NMFD parameters\n",
    "paramNMFD = dict()\n",
    "paramNMFD['numComp'] = numComp\n",
    "paramNMFD['numFrames'] = numFramesTr\n",
    "paramNMFD['numIter'] = numIter\n",
    "paramNMFD['numTemplateFrames'] = numTemplateFrames\n",
    "paramNMFD['initW'] = initW\n",
    "paramNMFD['initH'] = initH\n",
    "paramNMFD['numBins'] = numBinsTr\n",
    "paramNMFD['fixW'] = True\n",
    "\n",
    "# generate initial guess for templates\n",
    "paramTemplates = dict()\n",
    "paramTemplates['numComp'] = numComp\n",
    "paramTemplates['numBins'] = numBinsTr\n",
    "paramTemplates['numTemplateFrames'] = numTemplateFrames\n",
    "\n",
    "# set constraint parameters\n",
    "parameterConstr = dict()\n",
    "parameterConstr['initW'] = initW\n",
    "parameterConstr['funcPointerPostProcess'] = semiFixedComponentConstraintsNMF\n",
    "parameterConstr['adaptDegree'] = np.zeros(numComp)\n",
    "parameterConstr['adaptTiming'] = np.ones(numComp)\n",
    "\n",
    "# visualization parameters\n",
    "paramVis = dict()\n",
    "paramVis['deltaF'] = deltaF\n",
    "paramVis['deltaT'] = deltaT\n",
    "paramVis['logComp'] = 1e6\n",
    "paramVis['fontSize'] = 24\n",
    "\n",
    "divKL = np.full(2 * numIter, np.nan)\n",
    "\n",
    "tensorW = np.zeros((initW[0].shape[0], numComp, initW[0].shape[1]))\n",
    "                             \n",
    "                             \n",
    "for r in range(numComp):\n",
    "    tensorW[:,r,:] = initW[r]\n",
    "\n",
    "W = list()\n",
    "nmfdV = list()\n",
    "\n",
    "for r in range(numComp):\n",
    "    W.append(tensorW[:,r,:])\n",
    "    nmfdV.append(convModel(np.expand_dims(tensorW[:,r,:], axis=1), initH[r,:].reshape(1, -1)))\n",
    "\n",
    "H = initH\n",
    "                             \n",
    "fh1, _ = visualizeComponentsNMF(ATr, initW, initH, nmfdV, paramVis);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ozr/workspace/mir/audiolabs/NMF-Toolbox/python/NMFToolbox/visualizeComponentsNMF.py:112: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).\n",
      "  fh = plt.figure(constrained_layout=False, figsize=(20, 20))\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 420884 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 462420 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 484960 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 496812 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 503288 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 507976 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 507572 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 505540 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 503968 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 502808 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 501640 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 500772 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 499780 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 499204 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 499020 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498724 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498792 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498588 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498856 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 499116 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498860 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498520 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498564 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498584 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498400 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498444 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498436 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498492 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498448 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 498356 characters]\n",
      "text/plain": [
       "<Figure size 1440x1440 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i_iter in range(numIter):\n",
    "\n",
    "    paramNMFD['numIter'] = 1\n",
    "\n",
    "    # NMFD core method\n",
    "    nmfdW, nmfdH, nmfdV, _, _ = NMFD(ATr, paramNMFD, parameterConstr)\n",
    "\n",
    "    # alpha-Wiener filtering\n",
    "    nmfdA, _ = alphaWienerFilter(ATr,nmfdV,1);\n",
    "\n",
    "    paramNMFD['initH'] = nmfdH\n",
    "    parameterConstr['initW'] = nmfdW\n",
    "\n",
    "    # stack along second axis\n",
    "    B = np.stack(nmfdA, axis=2)\n",
    "    \n",
    "    # sum along second axis\n",
    "    ATr = np.sum(B, 2)\n",
    "\n",
    "    fh, _ = visualizeComponentsNMF(ATr, nmfdW, nmfdH, nmfdV, paramVis)\n",
    "    fh.savefig(os.path.join(outPath, 'demoEDMDecompositionFourComp_{}.png'.format(i_iter)))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
