{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\"FMP\"\n", "\"AudioLabs\"\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\"C4\"\n", "

Evaluation

\n", "
\n", "\n", "\n", "

\n", "Following Section 4.5 of [Müller, FMP, Springer 2015], we discuss some general evaluation metrics that are the basis for evaluating music processing tasks. A comprehensive Python library that provides implementations for measures to evaluate Music Information Retrieval systems is provided by the mir_eval toolbox:\n", "\n", "

\n", "

" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "We have described various procedures for extracting structural information from a given music recording. However, we have not yet discussed the issue of measuring **how well** a given procedure performs the task at hand. The problem of of (automatically) evaluating music processing algorithms goes far beyond structure analysis and itself constitutes a nontrivial task. A general evaluation approach is to compare an **estimated result** obtained by some automated procedure against some **reference result**. To realize such a general approach, one needs to find answers to the following questions: \n", "\n", "* How is a given analysis result actually modeled?\n", "* How should the estimated result be compared against the reference result? \n", "* Where does the reference result come from and is it reliable?\n", "\n", "In particular the last question easily leads to philosophical considerations on the nature and meaning of musical descriptions. For example, music structure analysis is an ill-posed problem that depends on many different factors, not to mention the musical and acoustic variations that occur in real-world music recordings. Since analysis results largely depend on the musical context and the considered temporal level, even two human experts may disagree in their analysis of a given piece of music. In the case of our Brahms example, one expert may annotate the structure on a larger scale resulting in the musical structure $A_1A_2B_1B_2CA_3B_3B_4D$, while another expert may consider a smaller scale, where the parts are further subdivided. In the following code cell, we give examples of annotations on three different scales.\n", "\n", "\n", "\n", "In practice, one typically assumes that a valid reference structure annotation has been provided by a human expert, even though this is a simplistic and sometimes problematic assumption. Such an annotation is also often referred to as **ground truth**. The objective of the automated procedure is to estimate an annotation that is as close to the reference as possible. " ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:36.522445Z", "iopub.status.busy": "2024-02-15T08:54:36.522164Z", "iopub.status.idle": "2024-02-15T08:54:39.387976Z", "shell.execute_reply": "2024-02-15T08:54:39.387442Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAABACAYAAABP0uObAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAMvUlEQVR4nO3dfXQU9bnA8e+TJURIIwQlvCWALwlLSs8RMSCJYIBaQDEI0jboxUTqAWm5RrjaUtqDYM/lxYIY32J9AYIaqIBQCmqlUaGJXq6iFJA02JsSQlBCCoGQBAjJr3/sJk28edkNgZlJns85OZmdzMw+zz67v2dnZncixhiUUkopuwmwOgCllFKqIdqglFJK2ZI2KKWUUrakDUoppZQtaYNSSillSx38Wdjlcpnq6urLFYtSzRBAP3XaVomAfqi4TSs2xnT3ZwW/GlR1dTXm1Fb/QlKqlUhoAlu36gjWViUkiI4vbZiEJuT7u44e4lNKKWVL2qCUUkrZkjYopZRStqQNSimllC1pg1JKKWVL2qCUUkrZkjYopZRStqQNSimllC1pg1JKKWVL2qBsZE1GJhKaQPyE+VaHomwgI2MhCQnCM88kWx2Kskh7HxP8utRRW7b7s1yWpW4ie3cOp0rKuKZbCIMG9mXW9PFMvjv2isQQPSCClIfv5sbrel2R+1Mtl5u7m02blpGTk01Z2SlCQq6hb99BjB8/i9jYya1yHwMG3Mrdd6cQFTW0Vban/KNjgvXEn3/5LiKmLV4ra8OWLKY+tJyqqmrcUeHEDnVTeraC3Z8dIubmSDamz7M6RJ9VVl4kMLBtvu+wy7X4srI2sHz5VKqrqwgPd+N2x1JRUcqhQ7uJjIxh3ryNVofol4sXK+nQIdDqMGx1LT4dE1qfhCbsMcbc4s861kdtsfLy8zw8N42qqmoSJ4/g9d/NpUMHFwBVVVXkflXY6LoSmgDAsoVJpK16l1MlZcyfO4XhMW4eSnme40UlPJA4imeXzQBgx4df8PMn1pB3+DjlFefp3bMbyfeNYdEv7wM8u/MP/iyV2+MG8dG2xQDsyj7Ar//7DfZ9mU/nzkGMuu17/PbJZHr3uqZeDCsX/4TUl/6IiJC395XL82Apzp8vJy3tYaqrqxgxIpG5c1/H5fK8jKqqqigszG103YQEASApaRnvvptGWdkppkyZj9s9nOeff4iSkuOMGvUAM2Y8C3gO8a1fv4jRo5N49NE1ZGauITX1QQYOjCMyMobMzNUEBQWTnPwU8fH3A3DuXBnr1i3kk0/epqTkOL163cg99/wXo0ZNq7fN2Nh7EQng00//yKxZaYwZk3wZHzVn0THBPtr9Oajs3Qc5eaoUgCd+MbX2iQjgcrmIdvdtdhsrXtjC8Bg3p8+UMW/RWqYkL+XWW6I4f6GS517exp8/2gtA4dcnufaaq0mcPIJpP46n9GwFTz61nvWbdjW43X0H/sH3Jy0g639yGDfmZvpFdCdj407G3ruQysqL9Zad/5vXGRn7XX4wanALHwnli4MHsyktPQnA1KlP1DYn8Dxf+vaNbnYbW7aswO0eTlnZadauncfSpVOIirqVysrzbNv2HHv3/rnJ9XNyssnJySYycignTx7jxRdnUl5+BoDU1AfZvHk5AQEu4uJ+xLFjX7Fy5QPs3Lmu3jY+/ngT33zzf8THT6Nr157+Pgxtmo4J9tHuG1TRidO10/37hgEwb2E6EppQ+9OcFb+ZTsarj9EvIgxjDEmJo0lPm8OddwwB4It9eQA8kDiKObMmEhF+LV2uDuaG6zwDwwd/2dfgdl9a/R6VlRdJmjqa9aseZ9f2JYR178KBnHw+/Mv+ess+/9RM0tPm8NLKn/r/ICifnT5dVDsdFtYfgPT0eSQkSO1Pc6ZPX8Fjj2UQFtYPYwyjRycxZ046Q4bcCUBe3hdNrh8S0o0lS3axYMF2AgJcnDtXRmHhIUpKisjO3gDAk0/uICVlFdOmed51b9/+XL1t9Ox5PcuX72b27JcZMmScz/m3Bzom2Ee7P8QX1r1L7XRBYTGRN/TmtlujKSgsJmPjTp+2MXBABABduwSTXwADIvsAEPKdTgCUlZ8DYNbcNF5O/9P/W/9E8ZkGt3v4iGcwHBgVDkBgYAeu79+TohOnyS8oqrds3LCBPsWqLk2XLmG108XFBfTuHUl09G0UFxewc2eGT9uIiPDUKji4K5BPnz4DAOjUKQTwHKZrSnj4QDp2vAqAq64Kprz8DOfOnaWo6DAAHTt2Iiysn3dZNwBFRfX/FU9k5NB6e3/q33RMsI92vwcVNyyabqGegWHJyg0YY5gwLobHH5nk8zZcroBv3XY1uNzvN2cBsObFFKr+uYVZ08cD0NgHVWrevf3tq6OA52Rn3uFvAOgXEVZv2aAg609ytwfR0XGEhHQDYMOGJRhjiImZwKRJj/u8jYCA+s+Pxp4vjanbWET+vcdWs0d34UIFJ04cAag9J1bTsGoEBgb5dZ/tiY4J9tHu30J17hzEC7+dyf0znmb1m5l8/tc8hg2J4sjRE61+Xz3CunL6TBnP/m4b73+wl83bP2ly+RnJY3ll7fukr/uQiooL5BecoOjEab7r7kv8bYNaPT7VvKCgzsyc+QJPP30/mZmrycv7nKioYbUNwUpdu4YRGzuFjz/eyIIFd+B2x5Gd/RYAd9012+LonEPHBPto93tQAIn3jmTntsVMGBtDQWExqzMy2Z+Tz9gxg3kltfVe2K+mzsYdFc7B3AJKz1YwM7npY/83fe963n97EcNjBvDOjj3848hxEieP4L2NC+nY0fnvjpxq5MhEFi/eSUzMBIqLC8jMXE1+/n4GDx7L7NnWflrqkUdWMXHiHCorL5CV9Xt69LielJTV3H77fZbG5TQ6JtiDfg/KRtJee4efPvYSE+8cxpY3f2V1OLZjl+9BqcvDTt+Dsou2NCa05HtQugdlE29tziJt1bsADBsSZXE0Simr6ZigDco2XnztHfLyj/PDe+L4zxkTrA5HKWUxHRP0QxK2UfMtcaWUAh0TQPeglFJK2ZQ2KKWUUrakDUoppZQtaYNSSillS9qglFJK2ZI2KKWUUrakDUoppZQtaYNSSillS9qglFJK2ZK/F4stBXIvXzhXzLVAsdVBtALNw140D3vRPOxlgDEmxJ8V/L3UUa6/V6O1IxH5TPOwD83DXjQPe2lLefi7jh7iU0opZUvaoJRSStmSvw3q5csSxZWnediL5mEvmoe9tNs8/PqQhFJKKXWl6CE+pZRStqQNSimllC351KBEZJyI5IrI30Vk3uUO6nIRkcMisl9E9rbkI49WEpFVIlIkIgfqzOsmIjtE5Cvv71ArY/RFI3ksFJFCb132isidVsbYHBGJEJEPRSRHRL4UkRTvfEfVo4k8nFaPq0Tkf0Xkr948FnnnO6oe0GQujqoJgIi4ROQLEdnmve13PZo9ByUiLuAQcAdwFPgUmGqMOXjpKVxZInIYuMUY47gvvYnISOAssNYYM8g77yngpDFmqfeNQ6gx5hdWxtmcRvJYCJw1xiy3MjZfiUgvoJcx5nMRCQH2APcAyTioHk3k8SOcVQ8Bgo0xZ0UkEMgCUoDJOKge0GQu43BQTQBEZC5wC3C1MWZCS8YrX/aghgJ/N8bkGWMuAOuBiZcavPKPMWYXcPJbsycC6d7pdDyDi601koejGGO+NsZ87p0uBXKAPjisHk3k4SjG46z3ZqD3x+CwekCTuTiKiIQDdwGv1pntdz18aVB9gII6t4/iwCexlwHeF5E9IjLD6mBaQQ9jzNfgGWyAMIvjuRSzRWSf9xCg7Q/F1BCR/sBgYDcOrse38gCH1cN7OGkvUATsMMY4th6N5ALOqskzwM+B6jrz/K6HLw1KGpjnuI7uFWeMuRkYD/zMe7hJWS8NuAG4CfgaWGFpND4Ske8Am4BHjTFnrI6npRrIw3H1MMZUGWNuAsKBoSIyyOKQWqyRXBxTExGZABQZY/Zc6rZ8aVBHgYg6t8OBY5d6x1Ywxhzz/i4CNuM5fOlkx73nEWrOJxRZHE+LGGOOe1+U1cArOKAu3vMDm4A3jTFve2c7rh4N5eHEetQwxpQAH+E5Z+O4etRVNxeH1SQOSPCe818PjBaRN2hBPXxpUJ8CkSJynYh0BBKBrS2N3CoiEuw9EYyIBAM/AA40vZbtbQWSvNNJwB8sjKXFap60XpOweV28J7JfA3KMMU/X+ZOj6tFYHg6sR3cR6eqd7gR8H/gbDqsHNJ6Lk2pijPmlMSbcGNMfT7/4wBjzH7SgHs1ezdwYc1FEZgN/AlzAKmPMly0N3kI9gM2e1yQdgAxjzHvWhuQ7EVkHxAPXishR4AlgKfCWiPwEOAL80LoIfdNIHvEichOeQ8eHgZlWxeejOGAasN97rgBgPs6rR2N5THVYPXoB6d5PHAcAbxljtonIJzirHtB4Lq87rCYN8fv1oZc6UkopZUt6JQmllFK2pA1KKaWULWmDUkopZUvaoJRSStmSNiillFK2pA1KKaWULWmDUkopZUv/AuNyIpOcvFYcAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAABACAYAAABP0uObAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAJJElEQVR4nO3df2zcdR3H8ee7ZRTXsh9SNha2OMYWgQwDji3gjBQjP5zzEFTiIogbMBJZxPgHamMiBBMQmBAXEIYjwAYu04HURdEmtjZMkLI53dYVAaVuDLjgBLYSxsa9/aPflq7edXcdu+/7i69HsvTuu93d6533fe69+36v3zN3R0REJJqatAOIiIgUowElIiIhaUCJiEhIGlAiIhKSBpSIiIR0RCX/uLa21guFwuHKcsgMCP2ZxOgBo+eLH1CK+MB0LXohKeWrqamhzLnwmrsfW8l9VzSgCoUC3tJSyU2qynK58PlaPG6+nGUgX0vkVwgpJpez0OuyXFq/pR+3nF9XMrOeSu9bu/hERCQkDSgREQlJA0pERELSgBIRkZA0oEREJCQNKBERCUkDSkREQtKAEhGRkDSgREQkJA0oEREJqaJTHb0f3J0TrrqKnnwegK477+TkKVOqHaOkyPmunHol+Z6+XDW1NYxpHMNJZ53EoqWLOG7acSmni58voq6u9axdezPd3X/i7bd7OeaY45k1ax6LFi1l1Kgj044XXuT1WkykNRIpSylVfwfVsXXrwJMJYGVbW7UjDCt6PoDZ82cz7xvzqB9bz1O/eoplVy5LO9IBoueLoqNjNc3NZ9PZuY7Gximcc85lTJw4jccfv5u9e99KO14mZGG9FhNpjUTKMlTVB9Sq9nYATp82DYCHOzrKOtFgtUTPB3DuFeey+CeLWXjbQgBe6n4p5UQHip4vgr173+Kee66hUHiXpqZLuf32jSxZci833tjKXXd1U1c3Ou2ImZCF9VpMpDUSKctQVR1Qe/ft45fr1wOwdNEixjc00JPP07F1azVjlBQ9X7/WFa0s/+Zy7r/ufgDO+uJZ6QYaInq+CLq61rN79y4ALrnk+9TUvLcUJ006Ubv3ypCV9VpMpDUSKctQVT0Gta6zk9d7e5kwdixnz5zJ/NmzWdnWxqr2ds6eObOaUTKZr1/nus6By6PqRjF91vQU0/yv6PkieOON93ZLTZjwkRSTZFdW1msxkdZIpCxDVfUdVP/b8c/PmUNNTQ0XnXkmAL9Yv569+/ZVM0pR0fP1a360mccKj3HrU7fiBWfZFct45Z+vpB1rQPR8EYwdO2Hgcj5f8dfkCNlZr8VEWiORsgxVtQH1nz17+M0zzwCworUVy+W4+KabAHijt5dfP/10taIUFT3fUGbG9FnTqauvo1Ao8MoLMZ5Q/aLnS9vJJ3+ChobxAKxZ88MDvpE0n+9h//7YL7Bpy9p6LSbSGomUZbCq7eJb88QTvLN/P2NGj+acU08d2N61fTvP7dzJyrY2vjR3brXiZC7fYK0rWtnctpkXNr5A7+u91I2uY+rHpqYda0D0fBEcdVQ9ixcv4447vkZ7+yp6ejYzY8Ycdu3ayaZNrTz44Ks0NIxLO2ZYWVqvxURaI5GyDFW1AfVQ8nb86vPP55aFCwe2/3HLFpqam/ntxo38+803OWbMmGpFylS+wfr3GdePq+eUT57CghsWMG7CuHRDDRI9XxRNTV+lsXEKjzzyI7q7n2T79m00Nk7mvPOu0qf4DiJL67WYSGskUpahrJKPZJqZe0v1v/O+XJbLET1fi8fNl7MM5GuJ/xFiOVAuZ6HXZbm0fks/bjlzxMw2uPsZldy3TnUkIiIhaUCJiEhIGlAiIhKSBpSIiISkASUiIiFpQImISEgaUCIiEpIGlIiIhKQBJSIiIWlAiYhISBpQIiISkgaUiIiEVOnJYncDzx6+OFXTCLyWdoj3geqIRXXEojpi+ai7H13JDSr9uo1nKz0bbURm9ozqiEN1xKI6Yvkg1VHpbbSLT0REQtKAEhGRkCodUMsPS4rqUx2xqI5YVEcs/7d1VPQhCRERkWrRLj4REQlJA0pEREIqa0CZ2QVm9qyZPW9m3z3coQ4XM3vRzDab2aaRfOQxTWZ2n5nlzWzLoG0fNrNWM3su+Tk+zYzlKFHH9Wb2UtKXTWY2L82MB2NmU8yszcy2mdlWM7s22Z6pfgxTR9b6cZSZPW1mf03quCHZnql+wLC1ZKonAGZWa2Z/MbN1yfWK+3HQY1BmVgv8HTgX2AF0AgvcvevQS6guM3sROMPdM/dLb2b2KWAP8KC7z0y23QLscvebk/84jHf376SZ82BK1HE9sMfdb0szW7nMbBIwyd03mtnRwAbgC8DXyVA/hqnjErLVDwPq3X2PmY0CngCuBS4mQ/2AYWu5gAz1BMDMvg2cAYxx9/kjeb0q5x3UHOB5d/+Hu78DrAYuPNTwUhl37wB2Ddl8IfBAcvkB+l5cQitRR6a4+8vuvjG5vBvYBhxPxvoxTB2Z4n32JFdHJX+cjPUDhq0lU8xsMvA54GeDNlfcj3IG1PHA9kHXd5DBJ3HCgd+b2QYzW5x2mPfBRHd/GfpebIAJKec5FEvM7G/JLsDwu2L6mdlU4HTgz2S4H0PqgIz1I9mdtAnIA63untl+lKgFstWTO4DrgMKgbRX3o5wBZUW2ZW6iJ+a6+8eBzwLXJLubJH0/BU4ETgNeBpammqZMZtYArAW+5e5vpp1npIrUkbl+uPu77n4aMBmYY2YzU440YiVqyUxPzGw+kHf3DYd6X+UMqB3AlEHXJwM7D/WB0+DuO5OfeeBR+nZfZtmryXGE/uMJ+ZTzjIi7v5osygJwLxnoS3J8YC3wkLs/kmzOXD+K1ZHFfvRz99eBdvqO2WSuH4MNriVjPZkL5JJj/quBT5vZKkbQj3IGVCcww8xOMLMjga8ALSNNnhYzq08OBGNm9cB5wJbhbxVeC3B5cvly4LEUs4xY/5M2cRHB+5IcyF4BbHP3Hw/6q0z1o1QdGezHsWY2Lrn8IeAzQDcZ6weUriVLPXH377n7ZHefSt+8+IO7X8oI+nHQs5m7+34zWwL8DqgF7nP3rSMNn6KJwKN9a5IjgIfd/fF0I5XPzH4ONAGNZrYD+AFwM7DGzK4A/gV8Ob2E5SlRR5OZnUbfruMXgavTylemucBlwObkWAFAM9nrR6k6FmSsH5OAB5JPHNcAa9x9nZk9Sbb6AaVrWZmxnhRT8frQqY5ERCQknUlCRERC0oASEZGQNKBERCQkDSgREQlJA0pERELSgBIRkZA0oEREJKT/AiBE9DAHETI8AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAABACAYAAABP0uObAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAJ/klEQVR4nO3de2zV9RnH8fdDaQGh3C9yrwi4RVBmzoiLzGGjRpGVOZyZZl4yJ0RGpmyJyCCDTRMYE+OM0QiTTNnFuDizCtm8L+DihSIoMK5C5VKk5dKbYgv22R89uM7Rcs7p73C+P/m8EsI5v3Ke7/PwnG+f9vc7PTV3R0REJDQdcp2AiIjIqWhAiYhIkDSgREQkSBpQIiISJA0oEREJUsd0/nFeXp43NTVFtrgBUb6GMPR4kctVgsH/x6Qq9GfM2RUv7Oyy4Euyfzt06ECKc+GQu/dLJ3ZaA6qpqQkvLU3nIW2ykpKzLl6Zl0UWL2EJHi2dFlm8mSVLU8ovK+uWnfmdmkhYpOsmEsYD06J7vsxbWnLWxdP+zVwu928qP65kZh+mG1un+EREJEgaUCIiEiQNKBERCZIGlIiIBEkDSkREgqQBJSIiQdKAEhGRIGlAiYhIkDSgREQkSBpQgUhYgoQlqCivyHUqQeWSigULbieRMJYsuSfWa0h8hbRnQsqlvTSgREQkSBpQIiISpLTeLLYtNz/4IKs3b6aqpoaC/Hy+PnIkv73zTsYWFSleGt566S1W/GYFR6uOUjy1mHsfvZfOXTpnFOvooXpeWLGWHZsOUFd9jD4DCrl11kSGj+qf1Vzau+7pbNjwBgsX3sX+/bsoLp7K8eONkcTN5hrV9VW8+NbvKf9oM8dPNDC43ygmfeMOBvQerngpisMe1v6NVmTfQX1YWcm3xozhR1dfzSUjRvD6xo3cuHix4qXpiV88wbhvjiO/IJ/S5aU8Pu/xjOI0NpzgkXmreOf1HeQX5DH+ilGc060TNYc/yWouUazblrq6ambN+jYffLCJsWMvpbq6ildf/UsksbO1RuOJBpavnMumXW9wbp8izh98MeUHNvHkyrl8/Gmt4qUoDntY+zdakQ2oZ2fP5tILLqCwSxcuSn4FsnXfPioOH1a8NMx5Yg7zl89n7rK5AKx6elVGcTaX7aGqoobuvc/hvoencvPMy/nZ4ilcmBiW1VyiWLcta9aspK6umqFDR/LYY6/wyCN/Z/TocZHEztYa2/es5UjtRxR27U3fHoPp3rUPPbr15ZNPa9m861+Kl6I47GHt32hFcopvR0UFl8yaRf2xY//3saraWgb16aN4KTrvq+cBUPSVIgCqD1XT2NBIQaeCtOIcPlgHwKDhvSno9N8253VM/WuS1nLJ9rptqazcD8DQoaMwMwCGDRvN1q3vRhI/G2scrasEoPbjw7y56YX/+djh2gOKl4K47GHt32hFMqBWrV1L/bFjjB0+nDWLFtFw/DgDbr0VIKVfZHW2x2tp95bdDB89nPKt5QD07Nsz7Sc3QJ8BhQBUfHiExoYTnz/ZPvusiby81J5smeQSxbpt6d9/MAB79+7A3TEz9uzZ3u642VyjV2HzuftBfUdy1/VLPh96xxrqFS9FcdnD2r/RimRADejZE4AdBw5w97JlbNi9W/EytHD6QlaXrmbNC2sAmHTLpIziXJgYRr9BPaiqqOHX9zzHyDEDObivhuIpY7no0qKs5RLFum2ZMOE6unXrwd69O5kx40ry8wvYtm19u+Nmc43RQxP0KjyXikM7WVo6m3N7F1FTX8Xuio3ccu18Rgwaq3inEZc9rP0brUhG4o0TJnDHVVeRn5fHK++9x5wbblC8DE3/1XTWr15PY0Mjk2+bzIwHZmQUp6BTR35y/3WMv2IUjQ0nePu1HdTVHKNHn3OymksU67ale/dePPRQKSNGXMj7779J1649KC6eGknsbK1RkN+ZH06+n4vOv5ya+irWb3+Nqur9XDxqIv16Dla8FMRlD2v/RsvS+XbWzNxLS6NbvKSEsy1emZdFFi9hCR4tnRZZvJklS1PKLyvrlrXv1GgmEgmLdN1EwnhgWnTPl3lLS866eNq/mcvl/k1ljpjZOndPpBNbP6grIiJB0oASEZEgaUCJiEiQNKBERCRIGlAiIhIkDSgREQmSBpSIiARJA0pERIKkASUiIkHSgBIRkSBpQImISJA0oEREJEjpvllsHbAte+mcMX2BQ7lOIgKqIyyqIyyqIywXuHthOg9I9/dBbUv33WhDZGZlqiMcqiMsqiMsX6Y60n2MTvGJiEiQNKBERCRI6Q6opVnJ4sxTHWFRHWFRHWE5a+tI60USIiIiZ4pO8YmISJA0oEREJEgpDSgzu8bMtpnZTjO7L9tJZYuZlZvZRjPbkMlLHnPJzJabWaWZbWpxrLeZvWxmO5J/98pljqlopY4FZrY/2ZcNZjYplzmejpkNNbPXzWyLmW02s7uTx2PVjzbqiFs/OpvZO2b2XrKOXyaPx6of0GYtseoJgJnlmdl6M1uZvJ92P057DcrM8oDtwFXAPmAtcJO7/7v9JZxZZlYOJNw9dj/0ZmaXA/XA0+4+JnlsMXDE3Rclv3Do5e6zc5nn6bRSxwKg3t0fzGVuqTKzgcBAd3/XzAqBdcB3gNuJUT/aqONG4tUPA7q6e72Z5QNvAHcD3yVG/YA2a7mGGPUEwMx+CiSA7u4+OZPPV6l8BzUe2Onuu9y9EXgGmNLe5CU97r4aOPKFw1OAp5K3n6L5k0vQWqkjVtz9gLu/m7xdB2wBBhOzfrRRR6x4s/rk3fzkHydm/YA2a4kVMxsCXAf8rsXhtPuRyoAaDOxtcX8fMXwSJznwkpmtM7NpuU4mAgPc/QA0f7IB+uc4n/aYaWbvJ08BBn8q5iQzKwK+BrxNjPvxhTogZv1Ink7aAFQCL7t7bPvRSi0Qr548DNwLNLU4lnY/UhlQdopjsZvoSZe5+yXAtcCPk6ebJPceB84HxgEHgCU5zSZFZtYNeA64x91rc51Ppk5RR+z64e6fufs4YAgw3szG5DiljLVSS2x6YmaTgUp3X9feWKkMqH3A0Bb3hwAV7V04F9y9Ivl3JfA8zacv4+xg8jrCyesJlTnOJyPufjC5KZuAZcSgL8nrA88Bf3T3vyYPx64fp6ojjv04yd2rgX/SfM0mdv1oqWUtMevJZUBJ8pr/M0Cxmf2BDPqRyoBaC4wys/PMrAD4PlCaaea5YmZdkxeCMbOuwNXAprYfFbxS4Lbk7duAv+Uwl4ydfNImXU/gfUleyH4S2OLuD7X4UKz60VodMexHPzPrmbzdBbgS2ErM+gGt1xKnnrj7HHcf4u5FNM+L19z9B2TQj9O+m7m7nzCzmcCLQB6w3N03Z5p8Dg0Anm/ek3QE/uTu/8htSqkzsz8DE4G+ZrYPmA8sAp41szuAPcD3cpdhalqpY6KZjaP51HE5MD1X+aXoMuAWYGPyWgHAz4lfP1qr46aY9WMg8FTyFccdgGfdfaWZvUm8+gGt17IiZj05lbT3h97qSEREgqR3khARkSBpQImISJA0oEREJEgaUCIiEiQNKBERCZIGlIiIBEkDSkREgvQfqPnNb/vKjagAAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import os, sys, librosa\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from matplotlib.colors import ListedColormap\n", "sys.path.append('..')\n", "import libfmp.b\n", "from libfmp.b import FloatingBox\n", "import libfmp.c4\n", "%matplotlib inline\n", "\n", "ann_Brahms = {}\n", "ann_Brahms[0] = [[0, 16, 'G major'], [16, 28, 'G minor'], [28, 40, 'G major']]\n", "\n", "ann_Brahms[1] = [[0, 4, 'A'], [4, 8, 'A'], [8, 12, 'B'], [12, 16, 'B'],\n", " [16, 27, 'C'], [27, 32, 'A'], [32, 36, 'B'], [36, 39, 'B'], [39, 40, '']]\n", "\n", "ann_Brahms[2] = [[0, 2, 'a'], [2, 4, 'a'], [4, 6, 'a'], [6, 8, 'a'],\n", " [8, 10, 'b'], [10, 12, 'c'], [12, 13, 'b'], [13, 15, 'c'],\n", " [15, 18, 'd'], [18, 20, 'd'], \n", " [20, 22, 'e'], [22, 24, 'e'], [24, 26, 'e'], [26, 28, 'e'],\n", " [28, 30, 'a'], [30, 32, 'a'], \n", " [32,34, 'b'], [34, 36, 'c'], [36, 37, 'b'], [37, 39, 'c'], [39, 40, '']]\n", "\n", "color_ann_Brahms = {'G major': [1, 0.8, 0, 0.3], 'G minor': [0, 0, 1, 0.3],\n", " 'A': [1, 0, 0, 0.3], 'B': [0, 1, 0, 0.3], 'C': [0, 0, 1, 0.3], '': [1, 1, 1, 0.3],\n", " 'a': [1, 0, 0, 0.3], 'b': [0, 1, 0, 0.2], 'c': [0.3, 0.5, 0, 0.6], \n", " 'd': [0, 0, 1, 0.2], 'e': [0.2, 0, 0.5, 0.6], '': [1, 1, 1, 1]} \n", "\n", "figsize = (6,1)\n", "for k in range(3):\n", " libfmp.b.plot_segments(ann_Brahms[k], figsize = figsize, colors=color_ann_Brahms); " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Precision, Recall, F-Measure\n", "\n", "Many evaluation measures are based on some notion of precision, recall, and F-measure—a concept that has been borrowed from the fields of information retrieval and pattern recognition. We now introduce this general concept in the context of binary classification. First, let $\\mathcal{I}$ be a finite set of so-called **items**. For this set, one has a reference annotation that is the result of a binary classification. Each item $i\\in \\mathcal{I}$ is assigned either a label '$+$' (item is **positive** or **relevant**) or a label '$-$' (item is **negative** or **not relevant**). Let $\\mathcal{I}^\\mathrm{Ref}_+$ be the set of positive items, and $\\mathcal{I}^\\mathrm{Ref}_-$ be the set of negative items. Furthermore, one has an automated procedure that estimates the annotation for each item. Let $\\mathcal{I}^\\mathrm{Est}_+$ be the set of items being estimated as positive, and $\\mathcal{I}^\\mathrm{Est}_-$ be the set of items being estimated as negative. An item $i\\in\\mathcal{I}^\\mathrm{Est}_+$ estimated as positive is called a **true positive** (TP) if it belongs to $\\mathcal{I}^\\mathrm{Ref}_+$, i.e., if $i\\in\\mathcal{I}^\\mathrm{Est}_+\\cap\\mathcal{I}^\\mathrm{Ref}_+$. Otherwise, if $i\\in\\mathcal{I}^\\mathrm{Est}_+\\cap\\mathcal{I}^\\mathrm{Ref}_-$, it is called a **false positive** (FP). Similarly, an item $i\\in\\mathcal{I}^\\mathrm{Est}_-$ estimated as negative is called a **false negative** (FN) if it belongs to $\\mathcal{I}^\\mathrm{Ref}_+$, and **true negative** (TN) otherwise.\n", "\n", "The **precision** $\\mathrm{P}$ of the estimation is defined as the number of true positives divided by the total number of items estimated as positive:\n", "\n", "\\begin{equation}\n", " \\mathrm{P} = \\frac{|\\mathcal{I}^\\mathrm{Est}_+\\cap\\mathcal{I}^\\mathrm{Ref}_+|}{|\\mathcal{I}^\\mathrm{Est}_+|} = \\frac{\\#\\mathrm{TP}}{\\#\\mathrm{TP}+\\#\\mathrm{FP}}. \n", "\\end{equation}\n", "\n", "In contrast, the **recall** $\\mathrm{R}$ is defined as the number of true positives divided by the total number of positive items:\n", "\n", "\\begin{equation}\n", " \\mathrm{R} = \\frac{|\\mathcal{I}^\\mathrm{Est}_+\\cap\\mathcal{I}^\\mathrm{Ref}_+|}{|\\mathcal{I}^\\mathrm{Ref}_+|} = \\frac{\\#\\mathrm{TP}}{\\#\\mathrm{TP}+\\#\\mathrm{FN}}. \n", "\\end{equation}\n", "\n", "Note that both precision and recall have values in the interval $[0,1]$. A perfect precision $\\mathrm{P}=1$ means that every item estimated as positive is indeed positive. In this case, there is no false positive, but there may exist some false negatives. In contrast, a perfect recall $\\mathrm{R}=1$ means that every positive item was also estimated as positive. In this case, there is no false negative, but there may exist some false positives. Only in the case $\\mathrm{P}=1$ and $\\mathrm{R}=1$ does the estimated annotation coincide with the reference annotation. Precision and recall are often combined by taking their **harmonic mean** to form a single measure, often referred to as the **F-measure**:\n", "\n", "\\begin{equation}\n", " \\mathrm{F} = \\frac{2\\cdot \\mathrm{P}\\cdot \\mathrm{R}}{\\mathrm{P} + \\mathrm{R}}. \n", "\\end{equation}\n", "\n", "One main property is that $\\mathrm{F}\\in[0,1]$ with $\\mathrm{F}=1$ if and only if $\\mathrm{P}=1$ and $\\mathrm{R}=1$. The following figure illustrates the definitions:\n", "\n", "\"FMP_C4_F29\"\n", "\n", "\n", "Given the sets $\\mathcal{I}$, $\\mathcal{I}^\\mathrm{Ref}_+$, and $\\mathcal{I}^\\mathrm{Est}_+$, we provide in the following code cell a function for computing the various measures and give some examples. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:39.390679Z", "iopub.status.busy": "2024-02-15T08:54:39.390436Z", "iopub.status.idle": "2024-02-15T08:54:39.397519Z", "shell.execute_reply": "2024-02-15T08:54:39.396863Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example 1\n", "TP = {0, 1, 2} ; FN = {3} ; FP = {4, 5}\n", "P = 0.600; R = 0.750; F = 0.667\n" ] } ], "source": [ "def measure_prf(num_TP, num_FN, num_FP):\n", " \"\"\"Compute P, R, and F from size of TP, FN, and FP [FMP, Section 4.5.1]\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " num_TP (int): True positives\n", " num_FN (int): False negative\n", " num_FP (int): False positives\n", "\n", " Returns:\n", " P (float): Precision\n", " R (float): Recall\n", " F (float): F-measure\n", " \"\"\"\n", " P = num_TP / (num_TP + num_FP)\n", " R = num_TP / (num_TP + num_FN)\n", " if (P + R) > 0:\n", " F = 2 * P * R / (P + R)\n", " else:\n", " F = 0\n", " return P, R, F\n", " \n", "\n", "def measure_prf_sets(I, I_ref_pos, I_est_pos, details=False):\n", " \"\"\"Compute P, R, and F from sets I, I_ref_pos, I_est_pos [FMP, Section 4.5.1]\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " I: Set of items\n", " I_ref_pos: Reference set of positive items\n", " I_est_pos: Set of items being estimated as positive\n", " details: Print details (Default value = False)\n", "\n", " Returns:\n", " P (float): Precision\n", " R (float): Recall\n", " F (float): F-measure\n", " \"\"\"\n", " I_ref_neg = I.difference(I_ref_pos)\n", " I_est_neg = I.difference(I_est_pos)\n", " TP = I_est_pos.intersection(I_ref_pos)\n", " FN = I_est_neg.intersection(I_ref_pos)\n", " FP = I_est_pos.intersection(I_ref_neg)\n", " P, R, F = measure_prf(len(TP), len(FN), len(FP))\n", " if details:\n", " print('TP = ', TP, '; FN = ', FN, '; FP = ', FP)\n", " print('P = %0.3f; R = %0.3f; F = %0.3f' % (P, R, F))\n", " return P, R, F\n", "\n", "print('Example 1')\n", "I = {0,1,2,3,4,5,6,7,8,9}\n", "I_ref_pos = {0,1,2,3}\n", "I_est_pos = {0,1,2,4,5}\n", "P, R, F = measure_prf_sets(I, I_ref_pos, I_est_pos, details=True)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:39.400457Z", "iopub.status.busy": "2024-02-15T08:54:39.400212Z", "iopub.status.idle": "2024-02-15T08:54:39.404123Z", "shell.execute_reply": "2024-02-15T08:54:39.403481Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example 2\n", "TP = {0, 1, 2} ; FN = {3, 4, 5, 6} ; FP = {8, 9, 7}\n", "P = 0.500; R = 0.429; F = 0.462\n" ] } ], "source": [ "print('Example 2')\n", "I = {0,1,2,3,4,5,6,7,8,9}\n", "I_ref_pos = {0,1,2,3,4,5,6}\n", "I_est_pos = {0,1,2,7,8,9}\n", "P, R, F = measure_prf_sets(I, I_ref_pos, I_est_pos, details=True)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:39.406734Z", "iopub.status.busy": "2024-02-15T08:54:39.406503Z", "iopub.status.idle": "2024-02-15T08:54:39.410061Z", "shell.execute_reply": "2024-02-15T08:54:39.409540Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Example 3\n", "TP = {0, 1, 2} ; FN = {3, 4, 5, 6, 7, 8, 9} ; FP = set()\n", "P = 1.000; R = 0.300; F = 0.462\n" ] } ], "source": [ "print('Example 3')\n", "I = {0,1,2,3,4,5,6,7,8,9}\n", "I_ref_pos = I\n", "I_est_pos = {0,1,2}\n", "P, R, F = measure_prf_sets(I, I_ref_pos, I_est_pos, details=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Labeling Evaluation\n", "\n", "In the following scenario, we consider a frame-level music analysis task. Let $\\varphi:[1:N]$ be the time axis given in frames and let $\\Lambda$ a set of possible labels. We define a **label function** to be a function $\\varphi:[1:N] \\to \\Lambda$ that assigns for each frame index $n\\in[1:N]$ a label $\\varphi(n)\\in\\Lambda$. We now introduce some frame-based evaluation measures, which are referred to as **pairwise precision**, **pairwise recall**, and **pairwise F-measure**. \n", "\n", "Let $\\varphi^\\mathrm{Ref}$ and $\\varphi^\\mathrm{Est}$ be the label functions for the reference and estimated structure annotation, respectively. In order to become independent of the actual label naming, the main idea is to not directly look at the labels, but to look for label co-occurrences. To this end, we consider pairs of frames that are assigned to the same label. More precisely, we define the set\n", "\n", "\\begin{equation}\n", " \\mathcal{I}=\\{(n,m)\\in[1:N]\\times[1:N]\\mid m-->" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:39.412784Z", "iopub.status.busy": "2024-02-15T08:54:39.412590Z", "iopub.status.idle": "2024-02-15T08:54:39.735544Z", "shell.execute_reply": "2024-02-15T08:54:39.734881Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Segment-based structure annotation:\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACICAYAAADZATEXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAN3klEQVR4nO3de5BU9ZnG8e/DICBEBURcAkRxgyhBjdiF9xuwVaa8kFVYtcqEymKFNQaBWCAmoqVxDbsR42UtyggKq4hRdBdExBiigpo1DmgSEVhdBRydiKwixNrlYr/7R7eIVjYzDcy8M/TzqZrq7sOc7qd+9O+Z06e7z1FEYGZmza9NdgAzs2rlAjYzS+ICNjNL4gI2M0viAjYzS9K2kl+uqamJYrHYVFmsVRHgT9C0BP6faDE2RMRBlaxQUQEXi0WitraySLZXUqFAba2nfUtQKMjzsgVQobC20nW8C8LMLIkL2MwsiQvYzCyJC9jMLIkL2MwsiQvYzCyJC9jMLIkL2MwsiQvYzCyJC9jMLIkL2MwsiQvYzCyJC9jMLElFR0OrZhFBn/POY219PQCvPfwwR/bpk5zKKnXuuYdSX186aFVNTQ2dO3fjqKNOZNy4qfTqdVhyuuriOeUt4EZbsnz5jicKwH0LFyamsd116qnnMHz49+jU6QCeeebfufHGS7MjVR3PKRdwo93/xBMAHNuvHwAPLFpEhI+H21oNGzaKCRNuZ9y4mwFYs2ZVcqLq4znlAm6ULVu3MnfxYgCmjh9Pl/33Z219PUuWL09OZrtq3rwZ/PSnV3D77RMBGDz4guRE1cVzqsQF3AgLli5l4+bNdO/aldMHDuScU04BPvsLbq3P0qUL+MUv7mDNmlW0a9eeI488LjtSVfGcKnEBN8KnT4pzTz2VNm3a8LdnngnAw7/6FVu2bs2MZrvo5pv/jZdeKjJz5n9QLBb58Y9H8c47b2XHqhqeUyUu4AZ8uGkTC59/HoAZ8+ahQoHzJ0wA4KM//YnHlizJjGe7QRJHHHEc++7biWKxSF3df2VHqgqeU5/xx9Aa8NBTT7F12zb279SJMwuFHctfe+stXl+3jvsWLmT40KGJCW1XzJs3g9rap1m1ajmbN2+kQ4eO9O17dHasquA59RkXcANmL1oEwOjzz+efx47dsfzZZcs4Y/RonnjhBf5740YO7Nw5KaHtiqVLFwCw336d+frXT2H06Ovp2rV7cqrq4Dn1GVXysQ9J4dNfG/i09C2JT0vfMqhQWBYRhYZ/8zPeB2xmlsQFbGaWxAVsZpbEBWxmlsQFbGaWxAVsZpbEBWxmlsQFbGaWxAVsZpbEBWxmlsQFbGaWxAVsZpbEBWxmlsQFbGaWpNLDUW4GVjddnFahG7AhO0QL4HEo8TiUeBygX0TsV8kKlR6QfXWlx7vc20iqrfYxAI/DpzwOJR6H0hhUuo53QZiZJXEBm5klqbSAf94kKVoXj0GJx6HE41DicdiFMajoTTgzM9tzvAvCzCyJC9jMLEmjCljSWZJWS3pD0qSmDtUSSeot6WlJKyWtkDQ2O1MWSTWSXpa0IDtLFkmdJc2VtKr8nDgxO1MGSePL8+FVSXMkdcjO1Bwk3SNpvaRXd1rWVdJTkl4vX3Zp6H4aLGBJNcCdwDeA/sDFkvrvTvhWajtwZUQcCZwAXF6l4wAwFliZHSLZbcCiiDgCOIYqHA9JPYErgEJEDABqgItyUzWbmcBZX1g2CVgcEX2BxeXbf1FjtoAHAW9ExJsRsRV4EBhWWdbWLyLqI2J5+fpmShOuZ26q5iepF3A2MD07SxZJ+wOnATMAImJrRGxMDZWnLbCvpLZAR+Dd5DzNIiKWAB98YfEwYFb5+izgmw3dT2MKuCfw9k6366jC4tmZpEOBY4EXk6NkuBWYCBSTc2Q6DHgfuLe8K2a6pE7ZoZpbRLwD3AysA+qBjyLil7mpUh0cEfVQ2mADuje0QmMKWH9mWdV+dk3Sl4BHgHERsSk7T3OSdA6wPiKWZWdJ1hYYCEyLiGOBj2nEy829TXkf5zCgD/BloJOkS3JTtS6NKeA6oPdOt3tRJS8zvkjSPpTKd3ZEPJqdJ8HJwHmS1lDaFTVY0v25kVLUAXUR8ekroLmUCrnaDAXeioj3I2Ib8ChwUnKmTO9J6gFQvlzf0AqNKeCXgL6S+khqR2kn+/zditkKSRKlfX4rI+KW7DwZIuLqiOgVEYdSeh78OiKqbosnIv4IvC2pX3nREOC1xEhZ1gEnSOpYnh9DqMI3I3cyHxhZvj4SmNfQCg0eDS0itkv6PvAkpXc574mIFbuTspU6GfgW8AdJr5SX/TAiFuZFskRjgNnljZI3ge8k52l2EfGipLnAckqfEnqZKvlKsqQ5wBlAN0l1wHXAFOAhSaMo/XEa0eD9+KvIZmY5/E04M7MkLmAzsyQuYDOzJC5gM7MkLmAzsyQuYDOzJC5gM7MkLmAzsyQuYDOzJC5gM7MkLmAzsyQNHoxnZzU1NVEsZh+HW1Tx4YhLPAT2OX5CtJAR2BARB1WyQkUFXCwWidrayiLtYSoUqK1tAUOdqFAQtZH7/2AtR0GeE4WCWkI3ra10He+CMDNL4gI2M0viAjYzS+ICNjNL4gI2M0viAjYzS+ICNjNL4gI2M0viAjYzS+ICNjNL4gI2M0viAjYzS+ICNjNL0qQF/MjixahQoOvgwdRv2ADAJ598wvEjR6JCgfFTpzblw+8REcF3v3s6hYL4wQ+G7Vi+ceMGhgw5kEJBzJ79s8SE1eOxmY9RUOH//bHmsTfMiZbSTU1awBcMGcKIoUP5cNMm/uGmmwD42QMP8NsVK/jrXr34x8svb8qH3yMk8aMf3U27du1ZsmQ+zz47H4DbbpvARx99QP/+BS666IrklNWhT/8+XDz24h0/F465kHbt2wHwtUFfS05XPfaGOdFSuqmi4wHvijuvuopnli1j/pIl3HD33UyZORNJzJg8mY4dOjT1w+8RhxxyOKNGTWbatGuYOnUs7dvvy4IFs6ipacs110ynpqYmO2JVGDBoAAMGDdhxe9rkaWzdspUDuh7AlIemJCarPnvDnGgJ3dTk+4AP6tKFf5k4EYDr7rqL/9myhe8NH87pxx3X1A+9R40cOZGvfvUo3n13DePHn0NE8O1vT+Dww4/JjlaVfvPkb7j3pnuRxPX/ej09DumRHanqtPY50RK6qVnehLtg8GB6du++4/aYCy9sjofdo9q23YdrrplOmzZt2LZtK1/5Sl8uvfTa7FhV6b2695h8yWSKxSIjrxrJKWefkh2pKu0NcyK7m5qlgG+ZPZt31q9HEgBX3nprczzsHjdgwCCOPvokAM4/fzTt27eOXSh7k+3bt3P1hVezccNGBp42kMtuvCw7UlVr7XMiu5uavID/c+1arr3rLiTx0E9+wkFduvD4c89x3+OPN/VDN4k2bWo+d2nN646r7uD3L/yeAw8+kJsevKlV7Gvc27XWOdESuqlJ34QrFov8/Q038L9btnD5iBEMHzqUAP5u0iTGTp3K3xx/PH/VrVtTRrC9yHMLn2P2LbMB6N23N7P+adbn/v3Say/lgK4HZESzVqaldFOTbgHf/uCDPP+733FIjx5MGTMGgBFDh3LB4MF8uGkTl03xO9fWeCt+u2LH9Veee4U5t8353M/Hmz5OTGetSUvpJkU0/nTWkqIFnPrZp+D2aeltJz4tfYs5Lf2yiKjoG0H+KrKZWRIXsJlZEhewmVkSF7CZWRIXsJlZEhewmVkSF7CZWRIXsJlZEhewmVkSF7CZWRIXsJlZEhewmVkSF7CZWRIXsJlZkkoPR7kZWN10cVqFbsCG7BAtgMehxONQ4nGAfhGxXyUrVHpGjNWVHu9ybyOpttrHADwOn/I4lHgcSmNQ6TreBWFmlsQFbGaWpNIC/nmTpGhdPAYlHocSj0OJx2EXxqCiN+HMzGzP8S4IM7MkLmAzsySNKmBJZ0laLekNSZOaOlRLJKm3pKclrZS0QtLY7ExZJNVIelnSguwsWSR1ljRX0qryc+LE7EwZJI0vz4dXJc2R1CE7U3OQdI+k9ZJe3WlZV0lPSXq9fNmloftpsIAl1QB3At8A+gMXS+q/O+Fbqe3AlRFxJHACcHmVjgPAWGBldohktwGLIuII4BiqcDwk9QSuAAoRMQCoAS7KTdVsZgJnfWHZJGBxRPQFFpdv/0WN2QIeBLwREW9GxFbgQWBYZVlbv4ioj4jl5eubKU24nrmpmp+kXsDZwPTsLFkk7Q+cBswAiIitEbExNVSetsC+ktoCHYF3k/M0i4hYAnzwhcXDgFnl67OAbzZ0P40p4J7A2zvdrqMKi2dnkg4FjgVeTI6S4VZgIlBMzpHpMOB94N7yrpjpkjplh2puEfEOcDOwDqgHPoqIX+amSnVwRNRDaYMN6N7QCo0pYP2ZZVX72TVJXwIeAcZFxKbsPM1J0jnA+ohYlp0lWVtgIDAtIo4FPqYRLzf3NuV9nMOAPsCXgU6SLslN1bo0poDrgN473e5FlbzM+CJJ+1Aq39kR8Wh2ngQnA+dJWkNpV9RgSffnRkpRB9RFxKevgOZSKuRqMxR4KyLej4htwKPAScmZMr0nqQdA+XJ9Qys0poBfAvpK6iOpHaWd7PN3K2YrJEmU9vmtjIhbsvNkiIirI6JXRBxK6Xnw64ioui2eiPgj8LakfuVFQ4DXEiNlWQecIKljeX4MoQrfjNzJfGBk+fpIYF5DKzR4NLSI2C7p+8CTlN7lvCciVuxOylbqZOBbwB8kvVJe9sOIWJgXyRKNAWaXN0reBL6TnKfZRcSLkuYCyyl9SuhlquQryZLmAGcA3STVAdcBU4CHJI2i9MdpRIP3468im5nl8DfhzMySuIDNzJK4gM3MkriAzcySuIDNzJK4gM3MkriAzcyS/B8OUsSbe2crEAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Frame-based label sequence:\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAACICAYAAADZATEXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAANEUlEQVR4nO3df2zV9b3H8eebFqd0Oljsoi3M1riAiFvAr/yQxC1Sdl3YxrJFp4hbxJvd7Afjx65O4WqyGG+4u7PZ7uSybKJuGRGwajadrpRtyQiJbBVYQJHcDoHWQlZM5qZztuW898c55WDlRio73/cnPa9H0nDON5DPiw+f8+Kcz/ec8zV3R0RE8jcmOoCISLVSAYuIBFEBi4gEUQGLiARRAYuIBKkdyW+uqanxQqFQqSynxYAU3reRQo4UMhSlkCSFDJBCjvgERSnkSCFDyTF3rx9+cEQFXCgU8M7Of16kd8GyLDxDKjlSyDCUo7MzdplnmYVnSCVHllky6yI6RwoZSjkOneq4tiBERIKogEVEgqiARUSCqIBFRIKogEVEgqiARUSCqIBFRIKogEVEgqiARUSCqIBFRIKogEVEgqiARUSCqIBFRIKogEVEguRWwPdv2oRlGZZl7D94MK9hk8uQSo4UMvT2HiTLjCwzrrxyDC0t9axevYi//e21qsqQUo4U1kUqOfLIkFsBb966lTFjxpy4HSGFDKnkSCHDkMmTp3PPPRu4/PLZtLc/QlvbuqrMkEKOVNZFCjnyyJBLAff29bH9D3/g+pYWGurr2dzRkcewyWVIJUcKGU42YUI9s2a1MGXKFQAcPz5YlRmic6SyLlLIkVeGEV0R4916dOtWCoUC17W0cP748dy/eTMvHDjA1IsvzmP4ZDKkkiOFDCd79tktzJ//AQDq6xtYuPDWqswQnSOVdZFCjrwy5PIMeFNHB2eNHcuUpiZmTZsGkPv/ailkSCVHChlONm3aLNau7WDJktX09fXy6KP/W5UZonOksi5SyJFXhooXcPfRozy7Zw/9AwNcdv313Hz33UDxL5iXFDKkkiOFDMONH38+s2a1sGTJagC2b3+6KjNE5khlXaSQI88MFd+C2Lx1K+7OnbfcwszLLgNg/c9+xlPbtrGnq4vLL7mk0hGSyJBKjhQyDNfX10t7+0Z2794GQENDU1VmiMyRyrpIIUeeGSpfwB0dmBkrFi2ifsIEAPoHBnhq2zY2bdmSz4QmkCGVHClkGG7//l2sXn0jdXXnMWfOv7Bs2X9XZYbIHKmsixRy5JnB3E//Etpm5tGXeE7oMtPhOVLIMJQjhUuxR2dIJYcuS59WhlKO59w9G35cn4QTEQmiAhYRCaICFhEJogIWEQmiAhYRCaICFhEJogIWEQmiAhYRCaICFhEJogIWEQmiAhYRCaICFhEJogIWEQmiAhYRCTLSr6PsAw6dwXjnA8fO4M+PJpqLMs1FmeaibDTNxUXuXj/84IgK+EyZWeepvhOzGmkuyjQXZZqLsmqYC21BiIgEUQGLiATJu4B/mPN4KdNclGkuyjQXZaN+LnLdAxYRkTJtQYiIBFEBi4gEya2AzexaM9tvZl1mdkde46bGzCaZ2W/MbJ+ZPW9my6IzRTOzGjPbZWZPRWeJZGbjzazNzF4srY850ZmimNmK0uNjr5k9YmZnR2eqhFwK2MxqgLXAJ4CpwI1mNjWPsRM0CHzD3S8FZgNfreK5GLIM2BcdIgHfA37p7lOAj1Clc2JmjcDXgczdpwE1wA2xqSojr2fAM4Eudz/g7v3ARmBhTmMnxd2PuPvO0u2/UnyQNcamimNmE4EFwAPRWSKZ2XnA1cB6AHfvd/c/h4aKVQucY2a1wDigNzhPReRVwI1A90n3e6ji0hliZk3AdGBHcJRI3wVuBwrBOaJdDPQBD5W2Yx4ws7roUBHc/WXgO8Bh4AjwqrtviU1VGXkVsJ3iWFW//83M3gs8Bix3979E54lgZp8E/uTuz0VnSUAtMANY5+7TgdeBqjxXYmYTKL5CbgYagDozWxybqjLyKuAeYNJJ9ycySl9SnA4zG0uxfDe4++PReQLNBT5tZgcpbktdY2Y/jY0UpgfocfehV0NtFAu5GrUAL7l7n7sPAI8DVwVnqoi8Cvj3wIfMrNnMzqK4of7znMZOipkZxX2+fe7eGp0nkrvf6e4T3b2J4pr4tbuPymc678TdjwLdZja5dGge8EJgpEiHgdlmNq70eJnHKD0hWZvHIO4+aGZfA9opntF80N2fz2PsBM0Fbgb2mNnu0rFV7v50XCRJxFJgQ+lJygHgluA8Idx9h5m1ATspvmtoF6P0Y8n6KLKISBB9Ek5EJIgKWEQkiApYRCSIClhEJIgKWEQkiApYRCSIClhEJIgKWEQkiApYRCSIClhEJIgKWEQkyIi+jKempsYLhejvzTbS+CrhBHIkEAFII0cKGSCRHEmEIIUc8QlOOObu9cMPjqiAC4UC3tn5z4v0LliW0dkZP6VZZuE5sszo9Nh/D4DMsvAcKWRIJUdmeoycnCG6swAsyw6d6ri2IEREgqiARUSCqIBFRIKogEVEgqiARUSCqIBFRIKogEVEgqiARUSCqIBFRIKogEVEgqiARUSCqIBFRIKogEVEglS8gH/0xBNYlrGytRWA/oEBpnzuc4ybO5eXXn650sMDMDg4wHXXTWXmzBpefHEnAN3dXcyZ8x4WLJjEG2+8XlU5nnz4STLL3vbzpY99KZfxU8mQSo4UMqSyNlPIkWdnVbyA//Uzn+GqD3+Y/9m0id3797Pm4YfZf+gQ/3HrrTQ3NlZ6eABqa8dy++33UygUuO++5QC0tq5kYKCfFStaOeecuqrKMeOjM7j3kXtP/DQ0NQBw9aeuzmX8VDKkkiOFDKmszRRy5NlZI/o+4HfDzPjBqlXMuOkmbr77brp6eri0uZnbvvCFSg/9FldeeQ3z53+ejo5NrFnzFbZte5JZs+bT0nJd1eVobG6ksbm4kJ740RP0Huzlio9dwU0rb6qqDKnkSCEDpLE2U8iRZ2flsgd8+SWXsHzRIvb+8Y/8/c03WXfHHYytrXj3v83Kla3U1Z1LW9s6xo49i9tu+37uGVLK0d3VTeuKVs4dfy7f+sm3MLOqzJBKjhQypLI2o3Pk1Vm5nYQ7cuzYidtHX3klr2Hfor6+gcWL/x2Az37232hqmly1OQYHB7lr8V288fobfHPtN7lg0gVVmSGVHClkgDTWZio58uisXAr4V7/7HRueeYZ5M2fy/ve9j+X33cerr72Wx9Bvc8EFH3zLr1Gic6y/Zz17d+zl4zd8nGsXXVu1GVLJkUKGIdFrM4UceXVWxQv4zf5+vrxmDePOPpv1d93Ft5cu5egrr7Bq7dpKDy3/j7079vLgvQ9SU1vDjI/OoH1jO+0b2/ntk7+tqgyp5Eghg5Tl2VkV34j9z4ce4v8OH+a/li7logsvZMnChfz4F7/gB489xhcXLGDmtGmVjiDDbH9mO8ePHwdgzZfXnDh+4UUX5nbmPYUMqeRIIYOU5dlZ5n76Vy01M4++wqiuijwsg64EnEyGVHLoqshvzRDdWQCWZc+5ezb8uD4JJyISRAUsIhJEBSwiEkQFLCISRAUsIhJEBSwiEkQFLCISRAUsIhJEBSwiEkQFLCISRAUsIhJEBSwiEkQFLCISRAUsIhJEBSwiEmSk3wfcBxw6g/HOB4694++qDpqLMs1FmeaibDTNxUXuXj/84IgK+EyZWeepvpS4GmkuyjQXZZqLsmqYC21BiIgEUQGLiATJu4B/mPN4KdNclGkuyjQXZaN+LnLdAxYRkTJtQYiIBFEBi4gEya2AzexaM9tvZl1mdkde46bGzCaZ2W/MbJ+ZPW9my6IzRTOzGjPbZWZPRWeJZGbjzazNzF4srY850ZmimNmK0uNjr5k9YmZnR2eqhFwK2MxqgLXAJ4CpwI1mNjWPsRM0CHzD3S8FZgNfreK5GLIM2BcdIgHfA37p7lOAj1Clc2JmjcDXgczdpwE1wA2xqSojr2fAM4Eudz/g7v3ARmBhTmMnxd2PuPvO0u2/UnyQNcamimNmE4EFwAPRWSKZ2XnA1cB6AHfvd/c/h4aKVQucY2a1wDigNzhPReRVwI1A90n3e6ji0hliZk3AdGBHcJRI3wVuBwrBOaJdDPQBD5W2Yx4ws7roUBHc/WXgO8Bh4AjwqrtviU1VGXkVsJ3iWFW//83M3gs8Bix3979E54lgZp8E/uTuz0VnSUAtMANY5+7TgdeBqjxXYmYTKL5CbgYagDozWxybqjLyKuAeYNJJ9ycySl9SnA4zG0uxfDe4++PReQLNBT5tZgcpbktdY2Y/jY0UpgfocfehV0NtFAu5GrUAL7l7n7sPAI8DVwVnqoi8Cvj3wIfMrNnMzqK4of7znMZOipkZxX2+fe7eGp0nkrvf6e4T3b2J4pr4tbuPymc678TdjwLdZja5dGge8EJgpEiHgdlmNq70eJnHKD0hWZvHIO4+aGZfA9opntF80N2fz2PsBM0Fbgb2mNnu0rFV7v50XCRJxFJgQ+lJygHgluA8Idx9h5m1ATspvmtoF6P0Y8n6KLKISBB9Ek5EJIgKWEQkiApYRCSIClhEJIgKWEQkiApYRCSIClhEJMg/ALhPQ98k/Y6kAAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "def convert_ann_to_seq_label(ann):\n", " \"\"\"Convert structure annotation with integer time positions (given in indices)\n", " into label sequence\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " ann (list): Annotation (list ``[[s, t, 'label'], ...]``, with ``s``, ``t`` being integers)\n", "\n", " Returns:\n", " X (list): Sequencs of labels\n", " \"\"\"\n", " X = []\n", " for seg in ann:\n", " K = seg[1] - seg[0]\n", " for k in range(K):\n", " X.append(seg[2])\n", " return X\n", "\n", "def plot_seq_label(ax, X, Fs=1, color_label=[], direction='horizontal',\n", " fontsize=10, time_axis=False, print_labels=True):\n", " \"\"\"Plot label sequence in the style of annotations\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " ax: Axis used for plotting\n", " X: Label sequence\n", " Fs: Sampling rate (Default value = 1)\n", " color_label: List of colors for labels (Default value = [])\n", " direction: Parameter used for :func:`libfmp.b.b_plot.plot_segments` (Default value = 'horizontal')\n", " fontsize: Parameter used for :func:`libfmp.b.b_plot.plot_segments` (Default value = 10)\n", " time_axis: Parameter used for :func:`libfmp.b.b_plot.plot_segments` (Default value = False)\n", " print_labels: Parameter used for :func:`libfmp.b.b_plot.plot_segments` (Default value = True)\n", "\n", " Returns:\n", " ann_X: Structure annotation for label sequence\n", " \"\"\"\n", " ann_X = []\n", " for m, cur_x in enumerate(X):\n", " ann_X.append([(m-0.5)/Fs, (m+0.5)/Fs, cur_x])\n", " libfmp.b.plot_segments(ann_X, ax=ax, time_axis=time_axis, fontsize=fontsize,\n", " direction=direction, colors=color_label, print_labels=print_labels)\n", " return ann_X\n", "\n", "color_label = {'A': [1, 0, 0, 0.2], 'B': [0, 0, 1, 0.2], 'C': [0, 1, 0, 0.2],\n", " 'X': [1, 0, 0, 0.2], 'Y': [0, 0, 1, 0.2], 'Z': [0, 1, 0, 0.2]}\n", "\n", "ann_ref = [[0, 4, 'A'], [4, 7, 'B'], [7, 10, 'A']]\n", "ann_est = [[0, 1, 'X'], [1, 3, 'Y'], [3, 7, 'Z'], [7, 9, 'Y'], [9, 10, 'X']]\n", "\n", "X_ref = convert_ann_to_seq_label(ann_ref)\n", "X_est = convert_ann_to_seq_label(ann_est)\n", "\n", "print('Segment-based structure annotation:')\n", "plt.figure(figsize=(5,2))\n", "ax = plt.subplot(211)\n", "libfmp.b.plot_segments(ann_ref, ax=ax, colors=color_label); \n", "#ax.set_xticks([])\n", "ax = plt.subplot(212)\n", "libfmp.b.plot_segments(ann_est, ax=ax, colors=color_label); \n", "#ax.set_xticks([])\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "print('Frame-based label sequence:')\n", "plt.figure(figsize=(5,2))\n", "ax = plt.subplot(211)\n", "plot_seq_label(ax, X_ref, color_label=color_label, time_axis=True);\n", "ax = plt.subplot(212)\n", "plot_seq_label(ax, X_est, color_label=color_label, time_axis=True);\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the first sequence as reference $\\varphi^\\mathrm{Est}$ and the second one as estimation $\\varphi^\\mathrm{Est}$, we now illustrate the pairwise precision, recall, and F-measure. In particular, we show the positive items (indicated by gray entries) of the reference and estimated annotations. Furthermore, we visualize true positive (TP), false negative (FN), and false positive (FP) items. " ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:39.738382Z", "iopub.status.busy": "2024-02-15T08:54:39.738178Z", "iopub.status.idle": "2024-02-15T08:54:40.119564Z", "shell.execute_reply": "2024-02-15T08:54:40.118868Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "#TP = 10 ; #FN = 14 ; #FP = 3\n", "P = 0.769; R = 0.417; F = 0.541\n" ] } ], "source": [ "def compare_pairwise(X):\n", " \"\"\"Compute set of positive items from label sequence [FMP, Section 4.5.3]\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " X (list or np.ndarray): Label sequence\n", "\n", " Returns:\n", " I_pos (np.ndarray): Set of positive items\n", " \"\"\"\n", " N = len(X)\n", " I_pos = np.zeros((N, N))\n", " for n in range(1, N):\n", " for m in range(n):\n", " if X[n] is X[m]:\n", " I_pos[n, m] = 1\n", " return I_pos\n", "\n", "def evaluate_pairwise(I_ref_pos, I_est_pos):\n", " \"\"\"Compute pairwise evaluation measures [FMP, Section 4.5.3]\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " I_ref_pos (np.ndarray): Referenence set of positive items\n", " I_est_pos (np.ndarray): Set of items being estimated as positive\n", "\n", " Returns:\n", " P (float): Precision\n", " R (float): Recall\n", " F (float): F-measure\n", " num_TP (int): Number of true positives\n", " num_FN (int): Number of false negatives\n", " num_FP (int): Number of false positives\n", " I_eval (np.ndarray): Data structure encoding TP, FN, FP\n", " \"\"\"\n", " I_eval = np.zeros(I_ref_pos.shape)\n", " TP = (I_ref_pos + I_est_pos) > 1\n", " FN = (I_ref_pos - I_est_pos) > 0\n", " FP = (I_ref_pos - I_est_pos) < 0\n", " I_eval[TP] = 1\n", " I_eval[FN] = 2\n", " I_eval[FP] = 3\n", " num_TP = np.sum(TP)\n", " num_FN = np.sum(FN)\n", " num_FP = np.sum(FP)\n", " P, R, F = measure_prf(num_TP, num_FN, num_FP)\n", " return P, R, F, num_TP, num_FN, num_FP, I_eval\n", "\n", "def plot_matrix_label(M, X, color_label=None, figsize=(3, 3), cmap='gray_r', fontsize=8, print_labels=True):\n", " \"\"\"Plot matrix and label sequence\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " M: Matrix\n", " X: Label sequence\n", " color_label: List of colors for labels (Default value = None)\n", " figsize: Figure size (Default value = (3, 3))\n", " cmap: Colormap for imshow (Default value = 'gray_r')\n", " fontsize: Font size (Default value = 8)\n", " print_labels: Display labels inside Rectangles (Default value = True)\n", "\n", " Returns:\n", " fig: Handle for figure\n", " ax: Handle for axes\n", " \"\"\"\n", " fig, ax = plt.subplots(2, 3, gridspec_kw={'width_ratios': [0.1, 1, 0.05],\n", " 'wspace': 0.2, 'height_ratios': [1, 0.1]},\n", " figsize=figsize)\n", "\n", " colorList = np.array([[1, 1, 1, 1], [0, 0, 0, 0.7]])\n", " cmap = ListedColormap(colorList)\n", " im = ax[0, 1].imshow(M, aspect='auto', cmap=cmap, origin='lower', interpolation='nearest')\n", " im.set_clim(vmin=-0.5, vmax=1.5)\n", " ax_cb = plt.colorbar(im, cax=ax[0, 2])\n", " ax_cb.set_ticks(np.arange(0, 2, 1))\n", " ax_cb.set_ticklabels(np.arange(0, 2, 1))\n", " ax[0, 1].set_xticks([])\n", " ax[0, 1].set_yticks([])\n", " plot_seq_label(ax[1, 1], X, color_label=color_label, fontsize=fontsize, print_labels=print_labels)\n", " ax[1, 2].axis('off')\n", " ax[1, 0].axis('off')\n", " plot_seq_label(ax[0, 0], X, color_label=color_label, fontsize=fontsize, print_labels=print_labels,\n", " direction='vertical')\n", " return fig, ax\n", "\n", "def plot_matrix_pairwise(I_eval, figsize=(3, 2.5)):\n", " \"\"\"Plot matrix I_eval encoding TP, FN, FP (see :func:`libfmp.c4.c4s5_evaluation.evaluate_pairwise`)\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " I_eval: Data structure encoding TP, FN, FP\n", " figsize: Figure size (Default value = (3, 2.5))\n", "\n", " Returns:\n", " fig: Handle for figure\n", " im: Handle for imshow\n", " \"\"\"\n", " fig = plt.figure(figsize=figsize)\n", " colorList = np.array([[1, 1, 1, 1], [0, 0.7, 0, 1], [1, 0, 0, 1], [1, 0.5, 0.5, 1]])\n", " cmap = ListedColormap(colorList)\n", " im = plt.imshow(I_eval, aspect='auto', cmap=cmap, origin='lower', interpolation='nearest')\n", " im.set_clim(vmin=-0.5, vmax=3.5)\n", " plt.xticks([])\n", " plt.yticks([])\n", " ax_cb = plt.colorbar(im)\n", " ax_cb.set_ticks(np.arange(0, 4, 1))\n", " ax_cb.set_ticklabels(['', 'TP', 'FN', 'FP'])\n", " return fig, im\n", "\n", "float_box = libfmp.b.FloatingBox(align='top')\n", "I_ref_pos = compare_pairwise(X_ref)\n", "fig, ax = plot_matrix_label(I_ref_pos, X_ref, color_label=color_label)\n", "ax[0,1].set_title('Reference')\n", "float_box.add_fig(fig)\n", "\n", "I_est_pos = compare_pairwise(X_est)\n", "fig, ax = plot_matrix_label(I_est_pos, X_est, color_label=color_label)\n", "ax[0,1].set_title('Estimation')\n", "float_box.add_fig(fig)\n", "\n", "P, R, F, num_TP, num_FN, num_FP, I_eval = evaluate_pairwise(I_ref_pos, I_est_pos)\n", "fig, im = plot_matrix_pairwise(I_eval)\n", "plt.title('Pairwise TP, FN, FP')\n", "float_box.add_fig(fig)\n", "float_box.show()\n", "\n", "print('#TP = ', num_TP, '; #FN = ', num_FN, '; #FP = ', num_FP)\n", "print('P = %0.3f; R = %0.3f; F = %0.3f' % (P, R, F))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the first plot, the reference annotations are indicated. Note that $24$ out of the $45$ items are positive (indicated by gray boxes) with regard to this annotation. In the second plot, which indicates the estimated annotations, there are $13$ positive items (indicated by gray boxes). In the third plot, the true positives ($\\#\\mathrm{TP}=10$), false positives ($\\#\\mathrm{FP}=3$), and false negatives ($\\#\\mathrm{FN}=14$) are indicated. From this, one obtains\n", "\n", "\\begin{eqnarray}\n", "\\mathrm{P} &=& \\#\\mathrm{TP}/(\\#\\mathrm{TP}+\\#\\mathrm{FP})=10/13\\approx 0.769,\\\\\n", "\\mathrm{R} &=& \\#\\mathrm{TP}/(\\#\\mathrm{TP}+\\#\\mathrm{FN})=10/24\\approx 0.417,\\\\\n", "\\mathrm{F} &=& 2\\mathrm{P}\\mathrm{R}/(\\mathrm{P} + \\mathrm{R})\\approx 0.541. \n", "\\end{eqnarray}\n", "\n", "In this example, the precision of nearly $77\\%$ is relatively high, whereas the recall of $42\\%$ is relatively low. The F-measure is between these two values with a bias towards the smaller one. As further examples, we now compare the three Brahms annotations, which we introduced at the beginning of this notebook. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:40.122185Z", "iopub.status.busy": "2024-02-15T08:54:40.121987Z", "iopub.status.idle": "2024-02-15T08:54:41.228971Z", "shell.execute_reply": "2024-02-15T08:54:41.228109Z" } }, "outputs": [ { "data": { "text/html": [ "\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "#TP = 226 ; #FN = 218 ; #FP = 12\n", "P = 0.950; R = 0.509; F = 0.663\n" ] }, { "data": { "text/html": [ "\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "#TP = 143 ; #FN = 301 ; #FP = 4\n", "P = 0.973; R = 0.322; F = 0.484\n" ] }, { "data": { "text/html": [ "\n", " \n", "
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "#TP = 136 ; #FN = 102 ; #FP = 11\n", "P = 0.925; R = 0.571; F = 0.706\n" ] } ], "source": [ "X_0 = convert_ann_to_seq_label(ann_Brahms[0])\n", "X_1 = convert_ann_to_seq_label(ann_Brahms[1])\n", "X_2 = convert_ann_to_seq_label(ann_Brahms[2])\n", "\n", "X_set = [X_0, X_1, X_2]\n", "combinations = [(0,1), (0,2), (1,2)]\n", "case_label = ['Coarse', 'Medium', 'Fine']\n", "\n", "for c in combinations:\n", " X_ref = X_set[c[0]]\n", " X_est = X_set[c[1]]\n", " \n", " float_box = libfmp.b.FloatingBox(align='top')\n", " I_ref_pos = compare_pairwise(X_ref)\n", " fig, ax = plot_matrix_label(I_ref_pos, X_ref, color_label=color_ann_Brahms, print_labels=False)\n", " ax[0,1].set_title('Reference: '+case_label[c[0]])\n", " float_box.add_fig(fig)\n", "\n", " I_est_pos = compare_pairwise(X_est)\n", " fig, ax = plot_matrix_label(I_est_pos, X_est, color_label=color_ann_Brahms, print_labels=False)\n", " ax[0,1].set_title('Estimation: '+case_label[c[1]])\n", " float_box.add_fig(fig)\n", "\n", " P, R, F, num_TP, num_FN, num_FP, I_eval = evaluate_pairwise(I_ref_pos, I_est_pos)\n", " fig, im = plot_matrix_pairwise(I_eval)\n", " plt.title('Pairwise TP, FN, FP')\n", " float_box.add_fig(fig)\n", "\n", " float_box.show()\n", "\n", " print('#TP = ', num_TP, '; #FN = ', num_FN, '; #FP = ', num_FP)\n", " print('P = %0.3f; R = %0.3f; F = %0.3f' % (P, R, F))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Boundary Evaluation\n", "\n", "The pairwise precision, recall, and F-measure are solely based on label information, whereas segment boundaries are treated implicitly by the presence of label changes. For other structure analysis tasks such as novelty-based segmentation, the precise detection of boundaries is the focus. To evaluate such procedures, one measures the deviation of the estimated segment boundaries from the boundaries of a reference annotation. To mathematically model this scenario, we introduce the notion of a **boundary annotation**, which is given by a sequence \n", "\n", "$$B=(b_1,b_2,\\ldots,b_K)$$ \n", "\n", "of increasing indices $b_k\\in[1:N]$, $k\\in[1:K]$. For example, such a boundary annotation may be derived from a structure annotation by taking the start and possibly the end indices of the annotated segments. In the following, let $B^\\mathrm{Ref}$ be the reference boundary annotation and $B^\\mathrm{Est}$ the estimated boundary annotation. There are many ways to compare $B^\\mathrm{Est}$ against $B^\\mathrm{Ref}$. For example, using $\\mathcal{I}=[1:N]$ as a set of items, one can define $\\mathcal{I}^\\mathrm{Ref}_+:=B^\\mathrm{Ref}$ and $\\mathcal{I}^\\mathrm{Est}_+:=B^\\mathrm{Est}$. From this, the precision, recall, and F-measure can be computed in the usual way. In this case, an estimated boundary is considered correct only if it agrees with a reference boundary. \n", "\n", "For certain applications small deviations in the boundary positions are acceptable. Therefore, one generalizes the previous measures by introducing a tolerance parameter $\\tau\\geq 0$ for the maximal acceptable deviation. An estimated boundary $b^\\mathrm{Est}\\in B^\\mathrm{Est}$ is then considered **correct** if it lies within the $\\tau$-neighborhood of a reference boundary $b^\\mathrm{Ref}\\in B^\\mathrm{Ref}$:\n", "\n", "\\begin{equation}\n", " |b^\\mathrm{Est}-b^\\mathrm{Ref}|\\leq \\tau.\n", "\\end{equation}\n", "\n", "In this case, the sets $\\mathcal{I}^\\mathrm{Ref}_+$ and $\\mathcal{I}^\\mathrm{Est}_+$ can no longer be used for defining precision and recall. Instead, we generalize the notions of true positives, false positives, and false negatives. The **true positives** (TP) are defined to be the items $b^\\mathrm{Est}\\in B^\\mathrm{Est}$ that are correct, and the **false positives** (FP) are the items $b^\\mathrm{Est}\\in B^\\mathrm{Est}$ that are not correct. Furthermore, the **false negatives** (FN) are defined to be the items $b^\\mathrm{Ref}\\in B^\\mathrm{Ref}$ with no estimated item in a $\\tau$-neighborhood. Based on these definitions, one can compute precision, recall, and F-measure from $\\#\\mathrm{TP}$, $\\#\\mathrm{FP}$, and $\\#\\mathrm{FN}$ as before.\n", "\n", "However, this generalization needs to be taken with care. Because of the tolerance parameter $\\tau$, several estimated boundaries may be contained in the $\\tau$-neighborhood of a single reference boundary. Conversely, a single estimated boundary may be contained in the $\\tau$-neighborhood of several reference boundaries. As a result, one may obtain a perfect F-measure even in the case that the sets $B^\\mathrm{Est}$ and $B^\\mathrm{Ref}$ contain a different number of boundaries. From a semantic point of view, this is not meaningful. To avoid such anomalies, one may introduce an additional assumption in the definition of a boundary annotation by requiring\n", "\n", "\\begin{equation}\n", " |b_{k+1}-b_k| > 2\\tau\n", "\\end{equation}\n", "\n", "for $k\\in[1:N-1]$. This is also a meaningful requirement from a musical point of view: a musical section (determined by two subsequent boundaries) should be much longer than the size of the tolerance parameter. The following figure illustrates the boundary evaluation measures by means of a simple example. \n", "\n", "\n", "\n", "\"FMP_C4_F31_text\"\n", "\n", "* Using the tolerance parameter $\\tau=0$ (see (c)), one obtains $\\#\\mathrm{TP}=1$, $\\#\\mathrm{FP}=3$, and $\\#\\mathrm{FN}=2$. This yields $\\mathrm{P}=1/4$, $\\mathrm{R}=1/3$, and $\\mathrm{F}=2/7$. \n", "\n", "* In the case $\\tau=1$ (see (e)), one obtains $\\#\\mathrm{TP}=2$, $\\#\\mathrm{FP}=2$, and $\\#\\mathrm{FN}=1$, which results in $\\mathrm{P}=1/2$, $\\mathrm{R}=2/3$, and $\\mathrm{F}=4/7$.\n", "\n", "* Finally, when using $\\tau=2$ (see (g)), one obtains a perfect F-measure. However, in this case the minimal distance condition is violated and the meaning of the evaluation measure is questionable." ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "execution": { "iopub.execute_input": "2024-02-15T08:54:41.232122Z", "iopub.status.busy": "2024-02-15T08:54:41.231855Z", "iopub.status.idle": "2024-02-15T08:54:41.813022Z", "shell.execute_reply": "2024-02-15T08:54:41.812368Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "====== Evaluation using tau = 0 ======\n", "#TP = 1 ; #FN = 2 ; #FP = 3\n", "P = 0.250; R = 0.333; F = 0.286\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgYAAACcCAYAAAAEXWOsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAe10lEQVR4nO3df5xVVb3/8dcbFAHlogiVk8rcRPtlQQ5mj66mPTRTc9J+GJqapGnd21fzllkWFVbeqG+F2s9rv+iHKZpp/kgwr6L4WyZEQ299SSQUMVDkh6Aofb5/rHVOm8OcmTMzZ5jDzPv5eJwHe++199pr77OH/Tlrr72WIgIzMzMzgEF9XQAzMzNrHA4MzMzMrMyBgZmZmZU5MDAzM7MyBwZmZmZW5sDAzMzMyhwYbKMkvVzS7ZLWSvpWX5enOyTNkPTVBijHIZIe78H2J0q6qZ5lyvmOkfRnSUO7uN1CSYd0kD5H0kd6Wr4qeffoXPYFSWdJmtbX5TBrFA4M+pCkxyRtkLRO0vJ8o9ypxs3PAFYC/xIRn+rFYlonIuLSiDi8F7L+LPCziHi+i+V5fUTMAZA0VdKvuluAbfFG3w2XACdJellfF8SsETgw6HutEbETMAF4E3BejduNBR6ObvRQJWm7rm5j7eutcylpB+AUoNs39W1NX12XOfC6EfhQX+zfrNE4MGgQEbEcmE0KEACQ9BZJd0l6VtKCUvWwpBmkm8a5ubbhMEmDJH1W0l8lPS3pCkmj8vrNkkLSaZL+BtySl58q6RFJqyTNljS2sO+Q9DFJ/y+nf0+SCumn523XSnpY0n55eZOkqyStkLRY0lmdHPpoSX/I+dxWUYa3Srpf0ur871sLaY9JOqwwX/5lXDjeUyT9TdJKSZ8vrDss186skvQwsH+xQIXzWDq29xTSJku6U9J0Sc8AU/OyOwrrvCYf0zP5UcAHCmlH5TzXSnpC0jlVzssBwLMR8Xje7u2SHirkc7Ok+wrzd0g6tnhuJB0BfA6YlK+TBYX8x+bjWCvpJkmjKwsgaUfSDbMpb78uf787SLpQ0rL8uTAHMlvo6HrI39lvJP1K0hpgsqQ3S7o7X/NPSvqupCGFbXrrupwDvKvKd2E2sESEP330AR4DDsvTuwMPARfl+VcCTwNHkQK4d+T5MTl9BvDVQl5nA/fkfHYA/hu4LKc1AwH8AtgRGAYcCywCXgtsB0wB7irkF8D1wM7AnsAK4IicdhzwBOmGKmAcqQZjENAGfBEYArwKeBR4Z5XjnwGsBd6Wy3wRcEdOGwWsAk7O5Tshz+9aee7y/FTgVxXH+6N8rOOBF4DX5vRpwNy8jz2APwGPF/I6DmjKxzMJeA7YLadNBl4CzszlGpaXlcq9I7AU+HBO34/0yOf1Of1J4KA8vQuwX5Vz83HghsL8UGADMDrnuxxYBozIZdjQ3rkpnpdCXnOAvwL75G3nANOqlOOQ4rnJy75MutZeBowB7gK+Url+Z9dDLtuLpGtxUC5LC/CWfIzNwCPA2b19Xebv6Zm+/j/BH38a4dPnBRjIn/wf+DrSzTGA/wF2zmmfAX5Zsf5s4JQ8PYPNA4NHgEML87vl/3RL/8EG8KpC+o3AaYX5QcB6YGyeD+DAQvoVwGcL5fhEO8dzAPC3imXnkZ6Tt3f8M4DLC/M7AZtIN+uTgfsq1r8bmFw4d50FBrsX0u8Djs/Tj5ZuJnn+DCpufhX7fQA4Jk9PbucYJ/PPwGASMLci/b+BL+XpvwEfJbUN6eja+Hzx3ORlc4H3km6cN+Xv5Ajg7cCDFddVZ4HBlML8fwCzqpTjkMpzQwoqjirMvxN4rHL9zq6HXLbbOzkPZwNXF+Z75boE9gY2dVQWf/wZKB8/a+57x0bEzZIOBn5N+kX4LOmXznGSWgvrbg/cWiWfscDVkv5RWLYJeHlhfmnF+hdp8zcaRKqpWJLnlxfS1pNu3JBu3H+tUoYmSc8Wlg0m3dCqKZcpItbl6vmm/FlSse6SXL5aVSt/E5ufi832I+lDwCdJAQZ5u2JVe3HbSmOBAyrOwXbAL/P0+0i1M9MkPUi6qd3dTj6rSLUBRbeRb7x5ehVwMKk25LYOytSeauemFpXfzZK8rFIt18Nm51LSPsC3gYnAcNK5a6ux7D25LkcAq9vZ1vq5lpaWWLNmTdX0RYsWzY6II7ZikfqcA4MGERG3KbUd+CapanUpqcbg9BqzWAqcGhF3ViZIai7tpmL9CyLi0m4UdymwV5XliyNi7y7ktUdpQumNjFGkKvJlpP/Qi/YEZuXp50g3jpJXdGGfT+b9LizkWyrDWNIjiEOBuyNik6QHSEFTSUcNPpcCt0XEO9pLjIj7gWMkbQ/8H9Iv3j3aWfVB4D8rlt0GfItU6zCNFBj8iBQYfK9KeXo6fGp725e+m+L5W9bOerVcD5X5/wCYD5wQEWslnQ28v8ay9uS6fC2woIN066fWrFnD9OnTq6a3trZu0f6mv3Pjw8ZyIfAOSRNIrdFbJb1T0mBJQ5VeHdu9yrY/BC4oNd5Tegf+mA729UPgPEmvz+uPlHRcjeX8MXCOpBYl4/J+7wPWSPpMbuA3WNK+kvbvIK+jJB2YG5h9Bbg3IpYCvwf2kfRBSdtJmgS8jvR8GVL1/vGStpc0kdpvHpBuxudJ2iWfzzMLaTuSblYrACR9GNi3C3lfn8t9ci7b9pL2l/RaSUOU+jwYGREvAmtItTrtuQ/YWVKxhuQu4NXAm0mPWRaSayiA26vk8xTQLKm7f+tPAbtKGllYdhkwJV9jo0nP7tt7e6I718MI0nlZJ+k1wL93oaw9uS4PJj1eMxvwHBg0kIhYQWog+IV8czyG1Kp8BelXz6ep/p1dBFwL3CRpLalx2AEd7Otq4OvA5blF+J+AI2ss55XABaRHH2uBa4BREbEJaCW9WbGY1Ojux8DIdjNKfg18CXiG1PDsxLyPp4GjgU+RGl2eCxwdESvzdl8g/TpcBZyf86nV+aTq78WkZ/Wlan4i4mHSr/K7STfFNwBb1MJUExFrgcOB40m/opeTznOp1f7JwGP5nH8MOKlKPhtJbTBOKix7DvgjsDCnk8u5JCL+XqVIV+Z/n5b0x1qPo7DP/yUFAo/mNwWagK8C80i1Gg/lMm3RUVU3r4dzgA+SrqsfATO7UNZuXZdKHUgdBfy81n2ZAUjaVdID+bNc6U2j0vyQinXn5B8xDU8RPa1pNLPeIGkM6Tn4myJiQ1+Xp7+SdCawR0Sc29dlsa1v7733jk4eJbRFRKc3dElTgXUR8c0q6XOAcyJiXi3lkjQ4B7VbnWsMzBpURKyIiNc4KOhdEfEdBwVWL5IOlTRf0kOSfqp2+viQdLhSfx1/lHRlbl9V6oPki0r9ohyn1C/H/Ur92FwlaXheb4aki5X6uXlU0vsLeZ+b971AuatvSXtJmiWpTdLc/JiuKgcGZmZm9TGU9AhwUkS8gdTAf7N2MrldzhTSK8X7kR7LfbKwyvMRcWBEXA78NiL2j4jxpFfSTyustxtwIOmRaykAOJLUeP2AvM038rqXAGdGRAvpcd33OzoIv5VgZmZW3WhJxer/SyLikirrDia9AfOXPP9zUmdlFxbWeQupIfWdSp12DiG1FSoptqvZV2mguZ1Jr+XOLqRdExH/AB6WVHot/TBS/xzrASLimVwb8VbgSv2zk9B2eyotcWBgZmZW3cpa2hhkz9WwjoA/RMQJNeQxg9TXzQJJk0n9mJS8UJFn6d/KhoODSN2rT6ihbOUNzMzMrOeGkl4PHpfnT2bLzsfuAf6ttI6k4Uode7VnBPCkUr8nJ9aw/5uAUwttEUZFxBpgcel19Pwq7/iOMulSjcHo0aOjubm5K5ts09raKjtc65mWlpa65meNw9dK/+Xvtmfa2tpWRsSYvi7HVvI8aZyUK5VGC72f1GdMWUSsyL/+Lys0TJwC/IUtfQG4l/R69UNs2RvqZiJillI/OPMkbST1B/M5UlDxA0lTSD3oXk4HHXp16XXFiRMnxrx5Nb1p0S8UnsfUhV8N7b98rfRf/m57RlJNr/v1lXq9rtif+FGCmZmZlTkwMDMzszK/lWBmZgPWokWLaG1t7XzFAcQ1BmZmZlbmwMDMzKybJG3Kgyb9KXdvPLzzrTbbvknSb/L0BElHFdLeLemz9S5zZxwYmJmZdd+GiJgQEfsCG0mjptYsIpZFRGmsgwmkkT5LaddGxLS6lbRGDgzMzMzqYy4wTtIoSddIelDSPZLeCCDp4MKwzPMljZDUnGsbhgBfBibl9EmSJkv6rqSReYClQTmf4ZKWStq+qwMk1cKBgZmZWQ/lDo2OJHVEdD4wPyLeSOpg6Bd5tXOAj+fuiQ8CyiOnRsRG4IvAzFwDMbOQtprUIdHBeVErMDsiXqSLAyTVwm8lmJmZVdfZIErDJD2Qp+cCPyH1Vvg+gIi4RdKukkYCdwLflnQpaeTEx7vQgdZMYBJwK3A88P3uDJBUCwcGZmZm1XU2iNKGygGK1P7dPiJimqQbSO0I7pF0GKkb5VpcC3xN0iigBbgF2JEuDpBUCz9KMDMzq6/byYMeSTqEFFyskbRXRDwUEV8H5gGV7QHWUmU8hIhYB9wHXARcHxGbujNAUi0cGJiZmdXXVGCipAeBacApefnZuaHhAlL7ghsrtrsVeF2p8WE7+c4ETsr/lpwInJbzXAgc09PCexClDnjwFKuVr5X+y99tzzT6IEqSOvtCGrr8vcE1BmZmZlbmwMDMzMzKHBiYmdmA1dLSQkRU/QxEDgzMzMyszIGBmZmZlXWpg6O2traVkpb0VmH6u3q3brb+y9dK/zUAv9uxfV0A65ouBQYRMaa3CmJmZmZ9z48SzMzMrMxjJZiZmVXX2SBK/Y4DAzMzs+o6G0Sp3/GjBDMzMytzYGBmZmZlDgzMzMyszIGBmZmZlTkwMDMzszIN1EEizMzMJM0CRnewysqIOGJrlacRODAwMzOzMj9KsIYh6SBJf+7rcrRH0iGSHu8gPSSN25plqlKOGZK+2oPtb5R0Sj3LZGbbFgcG1mOSHpO0QdK6wue7NWy32c00IuZGxKt7qYw9umEOFBFxZET8vK/LYWZ9xz0fWr20RsTNfV0I6x6lIf8UEf/o67KYWd9yjYH1KknjJN0mabWklZJm5uW351UW5BqGSZXV9bkm4tOSHpT0nKSfSHp5ru5eK+lmSbsU1r9S0vK8r9slvT4vPwM4ETg37+u6vLxJ0lWSVkhaLOmsQl7Dci3DKkkPA/vXcLhHSXo0H+f/lTQo5zVI0hRJSyT9XdIvJI3MaVs8osjHfVienirpirzNWkkLJU0srPsmSX/MaTOBoYW0XSRdn49vVZ7evZA+R9IFku4E1gOvyss+UljnVEmP5O1nSxqbl0vS9Hw8q/N3tG8N58jMGpwDA+ttXwFuAnYBdge+AxARb8vp4yNip4iYWWX79wHvAPYBWoEbgc+RWhEPAs4qrHsjsDfwMuCPwKV5X5fk6W/kfbXmm/Z1wALglcChwNmS3pnz+hKwV/68E6jluft7gInAfsAxwKl5+eT8eTvwKmAnoNNHLQXvBi4HdgauLW0raQhwDfBLYBRwJel8lQwCfgaMBfYENrSz35OBM4ARwJJigqRjSef6vcAYYC5wWU4+HHgb6XvZGZgEPN2FYzKzBuXAwOrlGknPFj6n5+Uvkm5MTRHxfETc0cV8vxMRT0XEE6Qb070RMT8iXgCuBt5UWjEifhoRa3PaVGB86Zd5O/YHxkTElyNiY0Q8CvwIOD6nfwC4ICKeiYilwMU1lPXref2/ARcCJ+TlJwLfjohHI2IdcB5wvKRaH+XdERG/j4hNpCBgfF7+FmB74MKIeDEifgPcX9ooIp6OiKsiYn1ErAUuAA6uyHtGRCyMiJci4sWKtI8CX4uIRyLiJeC/gAm51uBFUjDxGtIjiEci4skaj8fMGpjbGFi9HFuljcG5pFqD+yStAr4VET/tQr5PFaY3tDO/E4CkwaQb33GkX7elZ+WjgdXt5DsWaJL0bGHZYFLwAdAELC2kbfZruorK9ZsKeS2pSNsOeHkNeQIsL0yvB4bmoKIJeCI2f+e4vB9Jw4HpwBGkGhuAEZIG5yCjssyVxgIXSfpWYZmAV0bELbmB6feAPSVdDZwTEWtqPCazhtDS0hJr1lS/bBctWjR7oPVj4MDAelVELAdOB5B0IHCzpNsjYlGdd/VBUvX9YcBjwEhgFelGBlDZYcdSYHFE7F0lvyeBPYCFeX7PGspQuf6yPL2MdJOlkPYSKchpAoaXEnKAM6aGfZXK+EpJKgQHewJ/zdOfAl4NHBARyyVNAObzz3MCW56XoqWkWpNL20uMiIuBiyW9DLgC+DTwhRrLbtYQ1qxZw/Tp06umt7a2dtT5Ub/kRwnWqyQdV2jwtop0Iyr9Wn2K9My9HkYAL5Cecw8nVXsXVe7rPmCNpM/khoaDJe0rqdTI8ArgvNyAb3fgzBrK8Om8/h7AJ4BSu4nLgP+U9K+Sdsplm5mr5/9CqgF4l6TtgSnADjUe892kAOMsSdtJei/w5kL6CFKtyrOSRpHaTXTFD0nnoNSIc6Sk4/L0/pIOyGV+Dnief36vZrYNc2Bg9XKdNu/H4Oq8fH/gXknrSA3nPhERi3PaVODnuU3CB3q4/1+QqtGfAB4G7qlI/wnwuryva3JVeiswAVgMrAR+TKppADg/57eY1HjylzWU4XdAG/AAcEPeJ8BP8/a35/yeJwcaEbEa+I+87ydIN9mqHSkVRcRGUsPAyaSgaxLw28IqFwLD8rHdA8yqJd9C/lcDXwcul7QG+BNwZE7+F1KbjFWk8/Q08M2u5G+2rZO0q6QH8me5pCcK80Mq1p1TfKOokblLZDMzG7D23nvv6ORRQltEdHpDlzQVWBcR7QbIkuaQ2uHMq6VcFW2BtirXGJiZmdWJpEMlzZf0kKSfStri0aCkwyXdnfsguTI/Yiz1YfJFSXcAx0k6XdL9khYo9bkyPK83Q9LFku5S6jvl/YW8z837XiBpWl62l6RZktokzZX0mo6OwYGBmZlZfQwFZgCTIuINpAb+/15cQdJoUluiwyJiP2Ae8MnCKs9HxIERcTnw24jYPyLGA48ApxXW2w04EDgaKAUARwLHkhocjwe+kde9BDgzIlqAc4Dvd3QQfivBzMysutGSitX/l+RO09ozmPS201/y/M+Bj5Pa+5S8BXgdcKckgCGkhsQlxc7e9lUa42Vn0qvZswtp1+QuzB+WVHr1+TDgZxGxHiAinsm1EW8Frsz7g04aODswMDMzq25lLW0MsudqWEfAHyLihCrpxTxmkPqIWSBpMnBIIe2FijxL/1Y2HBwEPBsRE2ooW3kDMzMz67mhQLP+OWrsycBtFevcA/xbaR1JwyXtUyW/EcCT+bXgE2vY/03AqYW2CKNyp2OLC68aS9L4jjLpUo3B6NGjo7m5uSubWEFbW1vd8mppaalbXtZz9fxuYeB9v/7b6L5Gv/ba2tpWRkStnXZt654HPkyqtt+O1EX5D4srRMSK/Ov/skLDxCmkPk0qfQG4l/RK8EOkQKGqiJiVOzKbJ2kj8HvSeCcnAj+QNIXUjfrlpHFi2tWl1xUnTpwY8+bV9KaFtaPwfKfH/JppY6nndwsD7/v130b3Nfq1J6mm1/36Sr1eV+xP/CjBzMzMyhwYmJmZWZnfSjAzswFr0aJFtLa29nUxGoprDMzMzKzMgYGZmVk3SdqUB036U+7eeHjnW222fZOk3+TpCZKOKqS9W9Jn613mzjgwMDMz674NETEhIvYFNgIf68rGEbEsIkpjHUwAjiqkXRsR0+pW0ho5MDAzM6uPucA4SaMkXSPpQUn3SHojgKSDC8Myz5c0QlJzrm0YAnwZmJTTJ0maLOm7kkbmAZYG5XyGS1oqafuuDpBUCwcGZmZmPZQ7NDqS1BHR+cD8iHgjqYOhX+TVzgE+nrsnPgjYUNo+IjYCXwRm5hqImYW01aQOiQ7Oi1qB2RHxIl0cIKkWfivBzMysus4GURom6YE8PRf4Cam3wvcBRMQtknaVNBK4E/i2pEtJIyc+3oUOqmYCk4BbgeOB73dngKRaODAwMzOrrrNBlDZUDlCk9u/2ERHTJN1Aakdwj6TDSN0o1+Ja4GuSRgEtwC3AjnRxgKRa+FGCmZlZfd1OHvRI0iGk4GKNpL0i4qGI+DowD6hsD7CWKuMhRMQ64D7gIuD6iNjUnQGSauHAwMzMrL6mAhMlPQhMA07Jy8/ODQ0XkNoX3Fix3a3A60qND9vJdyZwUv635ETgtJznQuCYnhbegyhtRR4opv9q9IFsGp3/Nrqv0a+9Rh9ESVJnB9zQ5e8NrjEwMzOzMgcGZmZmVubAwMzMBqyWlhYioupnIHJgYGZmZmUODMzMzKysSx0ctbW1rZS0pLcKY7Wrd0tkayz+frvP565neuH8ja13hta7uhQYRMSY3iqImZmZ9T0/SjAzM7Myj5VgZmZWXWeDKPU7DgzMzMyq62wQpX7HjxLMzMyszIGBmZmZlTkwMDMzszIHBmZmZlbmwMDMzMzKNFAHiTAzM5M0CxjdwSorI+KIrVWeRuDAwMzMzMr8KMGsA5LmSPpIL+X9OUk/7o28zcy6y4GB9RuSHpO0QdK6wue7fV0uAEmHSHq8uCwi/isieiXoMDPrLvd8aP1Na0Tc3NeFMDPbVrnGwPo1STtIelbSvoVlY3LNwssk7SLpekkrJK3K07tXyWuqpF8V5pslhaTt8vyHJT0iaa2kRyV9NC/fEbgRaCrUZDS1k9+7JS3M5Z0j6bWFtMcknSPpQUmrJc2UNLT+Z8zMBjoHBtavRcQLwG+BEwqLPwDcFhF/J/0N/Iw0ZvyewAagu48f/g4cDfwL8GFguqT9IuI54EhgWUTslD/LihtK2ge4DDgbGAP8HrhO0pCKch8B/CvwRmByN8tpZlaVAwPrb67Jv7hLn9OBX7N5YPDBvIyIeDoiroqI9RGxFrgAOLg7O46IGyLir5HcBtwEHFTj5pOAGyLiDxHxIvBNYBjw1sI6F0fEsoh4BrgOmNCdcpqZdcRtDKy/ObayjYGkQcAwSQcAy0k31Ktz2nBgOumX+C55kxGSBkfEpq7sWNKRwJeAfUhB93DgoRo3bwKWlGYi4h+SlgKvLKyzvDC9Pm9jZj1wxLhxsXL9+qrpbU8+OXug9WPgwMD6vXyTvYJUa/AUcH2uHQD4FPBq4ICIWC5pAjAfUDtZPUe62Ze8ojQhaQfgKuBDwO8i4kVJ1xTy6azDkGXAGwr5CdgDeKKWYzSz7lm5fj3zPvrRqumaOrWjzo+QtInNfwAcCzQDvwMeBYYCl0fE+T0t69biRwk2UPyaVF1/Yp4uGUFqV/CspFGkX/zVPAC8TdKekkYC5xXShgA7ACuAl3LtweGF9KeAXfN27bkCeJekQyVtTwpYXgDuqvH4zKxvbIiICYXPY3n53Ih4EzAROElSS98VsWscGFh/c11FPwZXA0TEvaRf/E2kNwRKLiQ9y18J3APMqpZxRPwBmAk8CLQB1xfS1gJnkW7wq0jtGK4tpP8vqXHho7ntw2aPASLiz8BJwHdyWVpJr15u7MY5MLMGkRsftwF79XVZauUukc3MbMCa2NQUnTxKaIuIiVXTN3+UsDgi3iPpEOCciDha0q6kwOBdEbGwfiXvPW5jYGZmVt1oSfMK85dExCWF+Q0RMaGd7Q6SNB/4BzBtWwkKwIGBmZlZR1Z2VGPQgbkRcXTdS7MVuI2BmZmZlTkwMDMzszI/SjAzM+umiNipnWVzgDlbvTB10qW3EkaPHh3Nzc29V5pG09ZW3/x2261+eTW507tG0ra6vtdKy8ht5pXn+li2rPN1alXvv416/z/QUufvtp7nDup+/tra2lZGxJi6ZlpHE6WY10G6oMO3EvqjLtUYNDc3M29eR6ewn1F7nd/1wBln1C+vqVPrl5f1mK6r77Uyr3UA/Z1Bfa/nev9t1Pv/gXr/H1rv461zfpKWdL7WtquDng9vBd4dEdfl9a4HvplrExqaHyWYmZl13xavK0pqBh4HPk8a8Gyb4saHZmZm9bcAWC3pHX1dkK5yYGBmZtZ9wyQ9kD9XV6R9FZjSF4XqCT9KMDMzq667PR8SEXMlIemgXi1hnTkwMDMzq667PR+WXEBqa/BSncrT6/wowczMrJdExE3ALsD4vi5LrRwYmJmZ9a4LgN37uhC18qMEMzOzbqql58OIuBaoc4cYvceBgZmZDVht40DTO1ihdasVpWE4MDAzM+sGSbsC/5NnXwFsAlbk+fGkvgy2Ax4BTomI9Vu9kN3gNgZmZmbdEBFPR8SE/LriD4Hphfnn8vS+wEbgY31Y1C5xYGBmZta75gLj+roQtXJgYGZm1kskbQccyeYDLTU0tzEwMzOrrrOeD6sZJumBPD0X+EndS9ZLHBiYmZlV192eD6t2ldzo/CjBzMzMyhwYmJmZWZkfJZiZmfVQREytmN+iR8RthSKi9pWlFcCS3iuOmZn1M2MjYkxfF6KaiRMnxrx586qmS2rr4eiK25wu1Rg08pdrZmZmPec2BmZmZlbmwMDMzMzKHBiYmZlZmd9KMDMzq667PR9usxwYmJmZVdfdng+3WX6UYGZmZmUODMzMzKzMgYGZmZmVdannQzMzs/5E0ixgdAerrIyII7ZWeRqBAwMzMzMr86MEMzMzK3NgYGZmZmUODMzMzKzMgYGZmZmVOTAwMzOzsv8PzVUenTCBty0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "====== Evaluation using tau = 1 ======\n", "#TP = 2 ; #FN = 1 ; #FP = 2\n", "P = 0.500; R = 0.667; F = 0.571\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgYAAACcCAYAAAAEXWOsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfA0lEQVR4nO3df5xVVb3/8dcbFAUlFKEfk8rcRPtFQQ5mj66mPTRTc9R+GJqapFnd21fzllmWFVbeqG+F2s9rv+iHKZppoPkjr6L425kQTb11SSQUMVDkh6Aofe4fa53T5jBn5syZM8ww834+HufB3nvtvfba+2xmf85aa6+tiMDMzMwMYEhfF8DMzMz6DwcGZmZmVubAwMzMzMocGJiZmVmZAwMzMzMrc2BgZmZmZQ4MtlKSXibpVklrJH2rr8tTD0kzJX21H5TjQEmP9WD74yXd0Mgy5XzHSvqzpO27ud2Dkg7sJH2upA/3tHxV8u7RuewLkk6XNL2vy2HWXzgw6EOSHpW0XtJaScvyjXLHGjf/CLACeElEfKoXi2ldiIiLI+KQXsj6s8DPIuK5bpbn9RExF0DSNEm/qrcAW+ONvg4XASdIemlfF8SsP3Bg0PdaI2JHYBLwJuDsGrcbBzwUdYxQJWmb7m5jHeutcylpO+AkoO6b+tamr67LHHhdC3ywL/Zv1t84MOgnImIZcD0pQABA0lsk3SHpGUkLStXDkmaSbhpn5dqGgyUNkfRZSX+V9JSkyySNzus3SwpJp0j6G3BTXn6ypIclrZR0vaRxhX2HpI9J+t+c/j1JKqSfmrddI+khSXvn5U2SrpC0XNIiSad3cehjJP0h53NLRRneKuleSavyv28tpD0q6eDCfPmXceF4T5L0N0krJH2+sO7wXDuzUtJDwD7FAhXOY+nY3l1ImyrpdkkzJD0NTMvLbius85p8TE/npoD3F9IOz3mukfS4pDOrnJd9gWci4rG83dslPVDI50ZJ9xTmb5N0dPHcSDoU+BwwJV8nCwr5j8vHsUbSDZLGVBZA0g6kG2ZT3n5t/n63k3S+pKX5c34OZDbT2fWQv7PfSPqVpNXAVElvlnRnvuafkPRdScMK2/TWdTkXeFeV78JscIkIf/roAzwKHJyndwUeAC7I868EngIOJwVw78jzY3P6TOCrhbzOAO7K+WwH/BdwSU5rBgL4BbADMBw4GlgIvBbYBjgHuKOQXwBXAzsBuwPLgUNz2jHA46QbqoDxpBqMIUA78EVgGPAq4BHgnVWOfyawBnhbLvMFwG05bTSwEjgxl++4PL9L5bnL89OAX1Uc74/ysU4Engdem9OnA/PyPnYD/gQ8VsjrGKApH88U4FngFTltKvAicFou1/C8rFTuHYAlwIdy+t6kJp/X5/QngP3z9M7A3lXOzceBawrz2wPrgTE532XAUmBkLsP6js5N8bwU8poL/BXYK287F5hepRwHFs9NXvZl0rX2UmAscAfwlcr1u7oectleIF2LQ3JZWoC35GNsBh4Gzujt6zJ/T0/39d8Ef/zpD58+L8Bg/uQ/4GtJN8cA/hvYKad9BvhlxfrXAyfl6ZlsGhg8DBxUmH9F/qNb+gMbwKsK6dcCpxTmhwDrgHF5PoD9CumXAZ8tlOMTHRzPvsDfKpadTWon7+j4ZwKXFuZ3BDaSbtYnAvdUrH8nMLVw7roKDHYtpN8DHJunHyndTPL8R6i4+VXs9z7gqDw9tYNjnMo/A4MpwLyK9P8CvpSn/wZ8lNQ3pLNr4/PFc5OXzQPeQ7px3pC/k0OBtwP3V1xXXQUG5xTm/x24rko5Dqw8N6Sg4vDC/DuBRyvX7+p6yGW7tYvzcAZwZWG+V65LYE9gY2dl8cefwfJxW3PfOzoibpR0APBr0i/CZ0i/dI6R1FpYd1vg5ir5jAOulPSPwrKNwMsK80sq1r9Amz7RIFJNxeI8v6yQto5044Z04/5rlTI0SXqmsGwo6YZWTblMEbE2V8835c/iinUX5/LVqlr5m9j0XGyyH0kfBD5JCjDI2xWr2ovbVhoH7FtxDrYBfpmn30uqnZku6X7STe3ODvJZSaoNKLqFfOPN0yuBA0i1Ibd0UqaOVDs3taj8bhbnZZVquR42OZeS9gK+DUwGRpDOXXuNZe/JdTkSWNXBtjbAtbS0xOrVq6umL1y48PqIOHQLFqnPOTDoJyLiFqW+A98kVa0uIdUYnFpjFkuAkyPi9soESc2l3VSsf15EXFxHcZcAe1RZvigi9uxGXruVJpSeyBhNqiJfSvqDXrQ7cF2efpZ04yh5eTf2+UTe74OFfEtlGEdqgjgIuDMiNkq6jxQ0lXTW4XMJcEtEvKOjxIi4FzhK0rbA/yP94t2tg1XvB/6jYtktwLdItQ7TSYHBj0iBwfeqlKenr0/taPvSd1M8f0s7WK+W66Ey/x8A84HjImKNpDOA99VY1p5cl68FFnSSbgPU6tWrmTFjRtX01tbWzfrfDHTufNi/nA+8Q9IkUm/0VknvlDRU0vZKj47tWmXbHwLnlTrvKT0Df1Qn+/ohcLak1+f1R0k6psZy/hg4U1KLkvF5v/cAqyV9JnfwGyppgqR9OsnrcEn75Q5mXwHujoglwO+BvSR9QNI2kqYAryO1L0Oq3j9W0raSJlP7zQPSzfhsSTvn83laIW0H0s1qOYCkDwETupH31bncJ+aybStpH0mvlTRMacyDURHxArCaVKvTkXuAnSQVa0juAF4NvJnUzPIguYYCuLVKPk8CzZLq/b/+JLCLpFGFZZcA5+RrbAyp7b6jpyfquR5Gks7LWkmvAf6tG2XtyXV5AKl5zWzQc2DQj0TEclIHwS/km+NRpF7ly0m/ej5N9e/sAmA2cIOkNaTOYft2sq8rga8Dl+Ye4X8CDquxnJcD55GaPtYAVwGjI2Ij0Ep6smIRqdPdj4FRHWaU/Br4EvA0qePZ8XkfTwFHAJ8idbo8CzgiIlbk7b5A+nW4Ejg351Orc0nV34tIbfWlan4i4iHSr/I7STfFNwCb1cJUExFrgEOAY0m/opeRznOp1/6JwKP5nH8MOKFKPhtIfTBOKCx7Fvgj8GBOJ5dzcUT8vUqRLs//PiXpj7UeR2Gf/0MKBB7JTwo0AV8F2ki1Gg/kMm02UFWd18OZwAdI19WPgFndKGtd16XSAFKHAz+vdV9mAJJ2kXRf/ixTetKoND+sYt25+UdMv6eIntY0mllvkDSW1A7+pohY39flGagknQbsFhFn9XVZbMvbc889o4umhPaI6PKGLmkasDYivlklfS5wZkS01VIuSUNzULvFucbArJ+KiOUR8RoHBb0rIr7joMAaRdJBkuZLekDST9XBGB+SDlEar+OPki7P/atKY5B8UWlclGOUxuW4V2kcmyskjcjrzZR0odI4N49Iel8h77PyvhcoD/UtaQ9J10lqlzQvN9NV5cDAzMysMbYnNQFOiYg3kDr4b9JPJvfLOYf0SPHepGa5TxZWeS4i9ouIS4HfRsQ+ETGR9Ej6KYX1XgHsR2pyLQUAh5E6r++bt/lGXvci4LSIaCE1132/s4PwUwlmZmbVjZFUrP6/KCIuqrLuUNITMH/J8z8nDVZ2fmGdt5A6Ut+uNGjnMFJfoZJiv5oJSi+a24n0WO71hbSrIuIfwEOSSo+lH0wan2MdQEQ8nWsj3gpcrn8OEtrhSKUlDgzMzMyqW1FLH4Ps2RrWEfCHiDiuhjxmksa6WSBpKmkck5LnK/Is/VvZcXAIaXj1STWUrbyBmZmZ9dz2pMeDx+f5E9l88LG7gH8trSNphNLAXh0ZCTyhNO7J8TXs/wbg5EJfhNERsRpYVHocPT/KO7GzTLpVYzBmzJhobm7uziZb1KpVjR24bOHChQ3Nb/z48V2vVKNRozp74su6MpiuFRhc14u/2/6lvb19RUSM7etybCHPkd6TcrnS20LvJY0ZUxYRy/Ov/0sKHRPPAf7C5r4A3E16vPoBNh8NdRMRcZ3SODhtkjaQxoP5HCmo+IGkc0gj6F5KJwN6detxxcmTJ0dbW01PWvSJOXPmNDS/I488sqH5zZ49u2F5tba2dr2SVTWYrhUYXNeLv9v+RVJNj/v1lUY9rjiQuCnBzMzMyhwYmJmZWZmfSjAzs0Fr4cKF/b45ZktzjYGZmZmVOTAwMzOrk6SN+aVJf8rDG4/oeqtNtm+S9Js8PUnS4YW0IyV9ttFl7ooDAzMzs/qtj4hJETEB2EB6a2rNImJpRJTedTCJ9KbPUtrsiJjesJLWyIGBmZlZY8wDxksaLekqSfdLukvSGwEkHVB4LfN8SSMlNefahmHAl4EpOX2KpKmSvitpVH7B0pCczwhJSyRt290XJNXCgYGZmVkP5QGNDiMNRHQuMD8i3kgaYOgXebUzgY/n4Yn3B8pvTo2IDcAXgVm5BmJWIW0VaUCiA/KiVuD6iHiBbr4gqRZ+KsHMzKy6rl6iNFzSfXl6HvAT0miF7wWIiJsk7SJpFHA78G1JF5PenPhY4cVGXZkFTAFuBo4Fvl/PC5Jq4cDAzMysuq5eorS+8gVF6vhuHxExXdI1pH4Ed0k6mDSMci1mA1+TNBpoAW4CdqCbL0iqhZsSzMzMGutW8kuPJB1ICi5WS9ojIh6IiK8DbUBlf4A1VHkfQkSsBe4BLgCujoiN9bwgqRYODMzMzBprGjBZ0v3AdOCkvPyM3NFwAal/wbUV290MvK7U+bCDfGcBJ+R/S44HTsl5Pggc1dPCuynBzMysThGxYwfLnqaDG3REnNZBFo8CEwrb7VORPrOw/W+ATZopImIRcGg3i90p1xiYmZlZmQMDMzMzK3NgYGZmg1ZLSwsRUfUzGDkwMDMzszIHBmZmZlam7lSVSFoOLO694piZ2QAzLiLG9nUhqpk8eXK0tbVVTZfU3sUARwNOtx5X7M9frpmZmfWcmxLMzMyszAMcmZmZVdfVS5QGHAcGZmZm1XX1EqUBx00JZmZmVubAwMzMzMocGJiZmVmZAwMzMzMrc2BgZmZmZd0a+dDMzGwgkXQdMKaTVVZExKFbqjz9gQMDMzMzK3NTgvUbkvaX9Oe+LkdHJB0o6bFO0kPS+C1ZpirlmCnpqz3Y/lpJJzWyTGa2dXFgYD0m6VFJ6yWtLXy+W8N2m9xMI2JeRLy6l8rYoxvmYBERh0XEz/u6HGbWdzzyoTVKa0Tc2NeFsPpIEqlp8R99XRYz61uuMbBeJWm8pFskrZK0QtKsvPzWvMqCXMMwpbK6PtdEfFrS/ZKelfQTSS/L1d1rJN0oaefC+pdLWpb3dauk1+flHwGOB87K+5qTlzdJukLSckmLJJ1eyGt4rmVYKekhYJ8aDvdwSY/k4/z/kobkvIZIOkfSYkl/l/QLSaNy2mZNFPm4D87T0yRdlrdZI+lBSZML675J0h9z2ixg+0LazpKuzse3Mk/vWkifK+k8SbcD64BX5WUfLqxzsqSH8/bXSxqXl0vSjHw8q/J3NKGGc2Rm/ZwDA+ttXwFuAHYGdgW+AxARb8vpEyNix4iYVWX79wLvAPYCWoFrgc+RehEPAU4vrHstsCfwUuCPwMV5Xxfl6W/kfbXmm/YcYAHwSuAg4AxJ78x5fQnYI3/eCdTS7v5uYDKwN3AUcHJePjV/3g68CtgR6LKppeBI4FJgJ2B2aVtJw4CrgF8Co4HLSeerZAjwM2AcsDuwvoP9ngh8BBgJLC4mSDqadK7fA4wF5gGX5ORDgLeRvpedgCnAU904JjPrpxwYWKNcJemZwufUvPwF0o2pKSKei4jbupnvdyLiyYh4nHRjujsi5kfE88CVwJtKK0bETyNiTU6bBkws/TLvwD7A2Ij4ckRsiIhHgB8Bx+b09wPnRcTTEbEEuLCGsn49r/834HzguLz8eODbEfFIRKwFzgaOlVRrU95tEfH7iNhICgIm5uVvAbYFzo+IFyLiN8C9pY0i4qmIuCIi1kXEGuA84ICKvGdGxIMR8WJEvFCR9lHgaxHxcES8CPwnMCnXGrxACiZeQ2qCeDginqjxeMysH3MfA2uUo6v0MTiLVGtwj6SVwLci4qfdyPfJwvT6DuZ3BJA0lHTjO4b067bUVj4GWNVBvuOAJknPFJYNJQUfAE3AkkLaJr+mq6hcv6mQ1+KKtG2Al9WQJ8CywvQ6YPscVDQBj8emzxyX9yNpBDADOJRUYwMwUtLQHGRUlrnSOOACSd8qLBPwyoi4KXcw/R6wu6QrgTMjYnWNx2TWL7S0tMTq1dUv24ULF14/2MYxcGBgvSoilgGnAkjaD7hR0q0RsbDBu/oAqfr+YOBRYBSwknQjA6gcsGMJsCgi9qyS3xPAbsCDeX73GspQuf7SPL2UdJOlkPYiKchpAkaUEnKAM7aGfZXK+EpJKgQHuwN/zdOfAl4N7BsRyyRNAubzz3MCm5+XoiWkWpOLO0qMiAuBCyW9FLgM+DTwhRrLbtYvrF69mhkzZlRNb21t7WzwowHJTQnWqyQdU+jwtpJ0Iyr9Wn2S1ObeCCOB50nt3CNI1d5Flfu6B1gt6TO5o+FQSRMklToZXgacnTvw7QqcVkMZPp3X3w34BFDqN3EJ8B+S/kXSjrlss3L1/F9INQDvkrQtcA6wXY3HfCcpwDhd0jaS3gO8uZA+klSr8oyk0aR+E93xQ9I5KHXiHCXpmDy9j6R9c5mfBZ7jn9+rmW3FHBhYo8zRpuMYXJmX7wPcLWktqePcJyJiUU6bBvw890l4fw/3/wtSNfrjwEPAXRXpPwFel/d1Va5KbwUmAYuAFcCPSTUNAOfm/BaROk/+soYy/A5oB+4Drsn7BPhp3v7WnN9z5EAjIlYB/573/TjpJlt1IKWiiNhA6hg4lRR0TQF+W1jlfGB4Pra7gOtqybeQ/5XA14FLJa0G/gQclpNfQuqTsZJ0np4Cvtmd/M22dpJ2kXRf/iyT9HhhfljFunOLTxT1Zx4S2czMBq0999wzumhKaI+ILm/okqYBayOiwwBZ0lxSP5y2WspV0Rdoi3KNgZmZWYNIOkjSfEkPSPqppM2aBiUdIunOPAbJ5bmJsTSGyRcl3QYcI+lUSfdKWqA05sqIvN5MSRdKukNp7JT3FfI+K+97gaTpedkekq6T1C5pnqTXdHYMDgzMzMwaY3tgJjAlIt5A6uD/b8UVJI0h9SU6OCL2BtqATxZWeS4i9ouIS4HfRsQ+ETEReBg4pbDeK4D9gCOAUgBwGHA0qcPxROAbed2LgNMiogU4E/h+ZwfhpxLMzMyqGyOpWP1/UR40rSNDSU87/SXP/xz4OKm/T8lbgNcBt0sCGEbqSFxSHOxtgtI7XnYiPZp9fSHtqjyE+UOSSo8+Hwz8LCLWAUTE07k24q3A5Xl/0EUHZwcGZmZm1a2opY9B9mwN6wj4Q0QcVyW9mMdM0hgxCyRNBQ4spD1fkWfp38qOg0OAZyJiUg1lK29gZmZmPbc90Kx/vjX2ROCWinXuAv61tI6kEZL2qpLfSOCJ/Fjw8TXs/wbg5EJfhNF50LFFhUeNJWliZ5l0q8ZgzJgx0dzc3J1NrKC9vb1hebW0tDQsL+u5Rn63MPi+X//fqF9/v/ba29tXREStg3Zt7Z4DPkSqtt+GNET5D4srRMTy/Ov/kkLHxHNIY5pU+gJwN+mR4AdIgUJVEXFdHsisTdIG4Pek950cD/xA0jmkYdQvJb0npkPdelxx8uTJ0dZW05MW1oFC+06P+THT/qWR3y0Mvu/X/zfq19+vPUk1Pe7XVxr1uOJA4qYEMzMzK3NgYGZmZmV+KsHMzAathQsX0tra2tfF6FdcY2BmZmZlDgzMzMzqJGljfmnSn/LwxiO63mqT7Zsk/SZPT5J0eCHtSEmfbXSZu+LAwMzMrH7rI2JSREwANgAf687GEbE0IkrvOpgEHF5Imx0R0xtW0ho5MDAzM2uMecB4SaMlXSXpfkl3SXojgKQDCq9lni9ppKTmXNswDPgyMCWnT5E0VdJ3JY3KL1gakvMZIWmJpG27+4KkWjgwMDMz66E8oNFhpIGIzgXmR8QbSQMM/SKvdibw8Tw88f7A+tL2EbEB+CIwK9dAzCqkrSINSHRAXtQKXB8RL9DNFyTVwk8lmJmZVdfVS5SGS7ovT88DfkIarfC9ABFxk6RdJI0Cbge+Leli0psTH+vGAFWzgCnAzcCxwPfreUFSLRwYmJmZVdfVS5TWV76gSB3f7SMipku6htSP4C5JB5OGUa7FbOBrkkYDLcBNwA508wVJtXBTgpmZWWPdSn7pkaQDScHFakl7RMQDEfF1oA2o7A+whirvQ4iItcA9wAXA1RGxsZ4XJNXCgYGZmVljTQMmS7ofmA6clJefkTsaLiD1L7i2YrubgdeVOh92kO8s4IT8b8nxwCk5zweBo3paeL9EaQvyi2IGrv7+Ipv+zv836tffr73+/hIlSV0dcL8uf29wjYGZmZmVOTAwMzOzMgcGZmY2aLW0tBARVT+DkQMDMzMzK3NgYGZmZmXdGuCovb19haTFvVUYq12jeyJb/+Lvt34+dz3TC+dvXKMztN7VrcAgIsb2VkHMzMys77kpwczMzMr8rgQzM7PqunqJ0oDjwMDMzKy6rl6iNOC4KcHMzMzKHBiYmZlZmQMDMzMzK3NgYGZmZmUODMzMzKxMg/UlEWZmZpKuA8Z0ssqKiDh0S5WnP3BgYGZmZmVuSjDrhKS5kj7cS3l/TtKPeyNvM7N6OTCwAUPSo5LWS1pb+Hy3r8sFIOlASY8Vl0XEf0ZErwQdZmb18siHNtC0RsSNfV0IM7OtlWsMbECTtJ2kZyRNKCwbm2sWXippZ0lXS1ouaWWe3rVKXtMk/aow3ywpJG2T5z8k6WFJayQ9IumjefkOwLVAU6Emo6mD/I6U9GAu71xJry2kPSrpTEn3S1olaZak7Rt/xsxssHNgYANaRDwP/BY4rrD4/cAtEfF30v+Bn5HeGb87sB6ot/nh78ARwEuADwEzJO0dEc8ChwFLI2LH/Fla3FDSXsAlwBnAWOD3wBxJwyrKfSjwL8Abgal1ltPMrCoHBjbQXJV/cZc+pwK/ZtPA4AN5GRHxVERcERHrImINcB5wQD07johrIuKvkdwC3ADsX+PmU4BrIuIPEfEC8E1gOPDWwjoXRsTSiHgamANMqqecZmadcR8DG2iOruxjIGkIMFzSvsAy0g31ypw2AphB+iW+c95kpKShEbGxOzuWdBjwJWAvUtA9Anigxs2bgMWlmYj4h6QlwCsL6ywrTK/L25hZDxw6fnysWLeuanr7E09cP9jGMXBgYANevsleRqo1eBK4OtcOAHwKeDWwb0QskzQJmA+og6yeJd3sS15empC0HXAF8EHgdxHxgqSrCvl0NWDIUuANhfwE7AY8Xssxmll9VqxbR9tHP1o1XdOmdTb4EZI2sukPgKOBZuB3wCPA9sClEXFuT8u6pbgpwQaLX5Oq64/P0yUjSf0KnpE0mvSLv5r7gLdJ2l3SKODsQtowYDtgOfBirj04pJD+JLBL3q4jlwHvknSQpG1JAcvzwB01Hp+Z9Y31ETGp8Hk0L58XEW8CJgMnSGrpuyJ2jwMDG2jmVIxjcCVARNxN+sXfRHpCoOR8Ulv+CuAu4LpqGUfEH4BZwP1AO3B1IW0NcDrpBr+S1I9hdiH9f0idCx/JfR82aQaIiD8DJwDfyWVpJT16uaGOc2Bm/UTufNwO7NHXZamVh0Q2M7NBa3JTU3TRlNAeEZOrpm/alLAoIt4t6UDgzIg4QtIupMDgXRHxYONK3nvcx8DMzKy6MZLaCvMXRcRFhfn1ETGpg+32lzQf+AcwfWsJCsCBgZmZWWdWdFZj0Il5EXFEw0uzBbiPgZmZmZU5MDAzM7MyNyWYmZnVKSJ27GDZXGDuFi9Mg3TrqYQxY8ZEc3Nz75VmgGtf1d6wvFpGbTWPxA4KjfxuYRB+v0uXdr1OrZoaPCBke2O/W1oa/N028txBw89fe3v7iogY29BMG2iyFG2dpAs6fSphIOpWjUFzczNtbZ2dQuuM5nQ0mF592lr9PfQnjfxuYRB+v9Om9c+8ANTY75ZG/w1t9PE2OD9Ji7tea+vVyciHNwNHRsScvN7VwDdzbUK/5qYEMzOz+m32uKKkZuAx4POkF55tVdz50MzMrPEWAKskvaOvC9JdDgzMzMzqN1zSfflzZUXaV4Fz+qJQPeGmBDMzs+rqHfmQiJgnCUn792oJG8yBgZmZWXX1jnxYch6pr8GLDSpPr3NTgpmZWS+JiBuAnYGJfV2WWjkwMDMz613nAbv2dSFq5aYEMzOzOtUy8mFEzAYaPCBG73FgYGZmg1b7eNCMTlZo3WJF6TccGJiZmdVB0i7Af+fZlwMbgeV5fiJpLINtgIeBkyJi3RYvZB3cx8DMzKwOEfFUREzKjyv+EJhRmH82T08ANgAf68OidosDAzMzs941Dxjf14WolQMDMzOzXiJpG+AwNn3RUr/mPgZmZmbVdTXyYTXDJd2Xp+cBP2l4yXqJAwMzM7Pq6h35sOpQyf2dmxLMzMyszIGBmZmZlbkpwczMrIciYlrF/GYjIm4tFBG1rywtBxb3XnHMzGyAGRcRY/u6ENVMnjw52traqqZLau/h2xW3Ot2qMejPX66ZmZn1nPsYmJmZWZkDAzMzMytzYGBmZmZlfirBzMysunpHPtxqOTAwMzOrrt6RD7dabkowMzOzMgcGZmZmVubAwMzMzMq6NfKhmZnZQCLpOmBMJ6usiIhDt1R5+gMHBmZmZlbmpgQzMzMrc2BgZmZmZQ4MzMzMrMyBgZmZmZU5MDAzM7Oy/wOsKT1R3y1+cQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "====== Evaluation using tau = 2 ======\n", "#TP = 4 ; #FN = 0 ; #FP = 0\n", "P = 1.000; R = 1.000; F = 1.000\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgYAAACcCAYAAAAEXWOsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAe60lEQVR4nO3de5gcVZ3/8fcnQAiBGAiJlwgkKwFvaCIziI8Lgg+IARnACwYEJIKou/5AVhFFUYPKGv2pAbwu3uIFISCCXCRBFgLhzowhILC6kRADIZhAyIWEW/zuH+d0W+lMz/T09GQ6M5/X8/STqjpVp05V16S+fc6pU4oIzMzMzACG9HcBzMzMrHk4MDAzM7MyBwZmZmZW5sDAzMzMyhwYmJmZWZkDAzMzMytzYLCFkvQySbdIWiPpW/1dnnpIminpq01QjgMlPdqL7Y+TdH0jy5TzHSPpz5KG9XC7ByQd2EX6XEkf7m35quTdq3PZHySdJml6f5fDrFk4MOhHkh6RtF7SWknL8o1yhxo3/wiwAnhJRHyqD4tp3YiIiyLikD7I+rPAzyLi2R6W5/URMRdA0jRJv6q3AFvijb4OFwLHS3ppfxfErBk4MOh/bRGxAzAJeBNwVo3bjQMejDpGqJK0dU+3sc711bmUtC1wIlD3TX1L01/XZQ68rgM+2B/7N2s2DgyaREQsA+aQAgQAJL1F0u2Snpa0oFQ9LGkm6aZxZq5tOFjSEEmflfRXSU9KulTSqLz+eEkh6WRJfwNuzMtPkvSQpJWS5kgaV9h3SPqYpP/N6d+TpEL6KXnbNZIelLR3Xj5W0uWSlktaJOm0bg59tKQ/5HxurijDWyXdI2lV/vethbRHJB1cmC//Mi4c74mS/iZphaTPF9bdLtfOrJT0ILBPsUCF81g6tncX0qZKuk3SDElPAdPyslsL67wmH9NTuSng/YW0w3KeayQ9JumMKudlX+DpiHg0b/d2SfcX8rlB0t2F+VslHVU8N5ImA58DpuTrZEEh/3H5ONZIul7S6MoCSNqedMMcm7dfm7/fbSWdJ2lp/pyXA5lNdHU95O/sN5J+JWk1MFXSmyXdka/5xyV9V9LQwjZ9dV3OBd5V5bswG1wiwp9++gCPAAfn6V2A+4Hz8/wrgSeBw0gB3Dvy/JicPhP4aiGv04E7cz7bAv8FXJzTxgMB/ALYHtgOOApYCLwW2Bo4G7i9kF8A1wA7ArsBy4HJOe1o4DHSDVXABFINxhCgA/giMBR4FfAw8M4qxz8TWAO8LZf5fODWnDYKWAmckMt3bJ7fufLc5flpwK8qjvdH+VgnAs8Br83p04F5eR+7An8CHi3kdTQwNh/PFOAZ4BU5bSrwInBqLtd2eVmp3NsDS4AP5fS9SU0+r8/pjwP75+mdgL2rnJuPA9cW5ocB64HROd9lwFJgRC7D+s7OTfG8FPKaC/wV2DNvOxeYXqUcBxbPTV72ZdK19lJgDHA78JXK9bu7HnLZXiBdi0NyWVqAt+RjHA88BJze19dl/p6e6u//E/zxpxk+/V6AwfzJ/4GvJd0cA/hvYMec9hnglxXrzwFOzNMz2TgweAg4qDD/ivyfbuk/2ABeVUi/Dji5MD8EWAeMy/MB7FdIvxT4bKEcn+jkePYF/lax7CxSO3lnxz8TuKQwvwOwgXSzPgG4u2L9O4CphXPXXWCwSyH9buCYPP1w6WaS5z9Cxc2vYr/3Akfm6amdHONU/hkYTAHmVaT/F/ClPP034KOkviFdXRufL56bvGwe8B7SjfP6/J1MBt4O3FdxXXUXGJxdmP93YHaVchxYeW5IQcVhhfl3Ao9Urt/d9ZDLdks35+F04IrCfJ9cl8AewIauyuKPP4Pl47bm/ndURNwg6QDg16RfhE+TfukcLamtsO42wE1V8hkHXCHpH4VlG4CXFeaXVKx/vjZ+okGkmorFeX5ZIW0d6cYN6cb91yplGCvp6cKyrUg3tGrKZYqItbl6fmz+LK5Yd3EuX62qlX8sG5+LjfYj6YPAJ0kBBnm7YlV7cdtK44B9K87B1sAv8/R7SbUz0yXdR7qp3dFJPitJtQFFN5NvvHl6JXAAqTbk5i7K1Jlq56YWld/N4rysUi3Xw0bnUtKewLeBVmA46dx11Fj23lyXI4BVnWxrA1xLS0usXr26avrChQvnRMTkzVikfufAoElExM1KfQe+SapaXUKqMTilxiyWACdFxG2VCZLGl3ZTsf65EXFRHcVdAuxeZfmiiNijB3ntWppQeiJjFKmKfCnpP/Si3YDZefoZ0o2j5OU92Ofjeb8PFPItlWEcqQniIOCOiNgg6V5S0FTSVYfPJcDNEfGOzhIj4h7gSEnbAP+P9It3105WvQ/4j4plNwPfItU6TCcFBj8iBQbfq1Ke3r4+tbPtS99N8fwt7WS9Wq6Hyvx/AMwHjo2INZJOB95XY1l7c12+FljQRboNUKtXr2bGjBlV09va2jbpfzPQufNhczkPeIekSaTe6G2S3ilpK0nDlB4d26XKtj8Ezi113lN6Bv7ILvb1Q+AsSa/P64+UdHSN5fwxcIakFiUT8n7vBlZL+kzu4LeVpL0k7dNFXodJ2i93MPsKcFdELAF+D+wp6QOStpY0BXgdqX0ZUvX+MZK2kdRK7TcPSDfjsyTtlM/nqYW07Uk3q+UAkj4E7NWDvK/J5T4hl20bSftIeq2koUpjHoyMiBeA1aRanc7cDewoqVhDcjvwauDNpGaWB8g1FMAtVfJ5Ahgvqd6/9SeAnSWNLCy7GDg7X2OjSW33nT09Uc/1MIJ0XtZKeg3wbz0oa2+uywNIzWtmg54DgyYSEctJHQS/kG+OR5J6lS8n/er5NNW/s/OBq4DrJa0hdQ7bt4t9XQF8Hbgk9wj/E3BojeW8DDiX1PSxBrgSGBURG4A20pMVi0id7n4MjOw0o+TXwJeAp0gdz47L+3gSOBz4FKnT5ZnA4RGxIm/3BdKvw5XAOTmfWp1Dqv5eRGqrL1XzExEPkn6V30G6Kb4B2KQWppqIWAMcAhxD+hW9jHSeS732TwAeyef8Y8DxVfJ5ntQH4/jCsmeAPwIP5HRyORdHxN+rFOmy/O+Tkv5Y63EU9vk/pEDg4fykwFjgq0A7qVbj/lymTQaqqvN6OAP4AOm6+hEwqwdlreu6VBpA6jDg57XuywxA0s6S7s2fZUpPGpXmh1asOzf/iGl6iuhtTaOZ9QVJY0jt4G+KiPX9XZ6BStKpwK4RcWZ/l8U2vz322CO6aUroiIhub+iSpgFrI+KbVdLnAmdERHst5ZK0VQ5qNzvXGJg1qYhYHhGvcVDQtyLiOw4KrFEkHSRpvqT7Jf1UnYzxIekQpfE6/ijpsty/qjQGyReVxkU5WmlcjnuUxrG5XNLwvN5MSRcojXPzsKT3FfI+M+97gfJQ35J2lzRbUoekebmZrioHBmZmZo0xjNQEOCUi3kDq4L9RP5ncL+ds0iPFe5Oa5T5ZWOXZiNgvIi4BfhsR+0TERNIj6ScX1nsFsB+pybUUABxK6ry+b97mG3ndC4FTI6KF1Fz3/a4Owk8lmJmZVTdaUrH6/8KIuLDKuluRnoD5S57/OWmwsvMK67yF1JH6NqVBO4eS+gqVFPvV7KX0orkdSY/lzimkXRkR/wAelFR6LP1g0vgc6wAi4qlcG/FW4DL9c5DQTkcqLXFgYGZmVt2KWvoYZM/UsI6AP0TEsTXkMZM01s0CSVNJ45iUPFeRZ+nfyo6DQ0jDq0+qoWzlDczMzKz3hpEeD56Q509g08HH7gT+tbSOpOFKA3t1ZgTwuNK4J8fVsP/rgZMKfRFGRcRqYFHpcfT8KO/ErjLpUY3B6NGjY/z48T3ZpEurVjX3QGMLFy5saH4TJkzofqV+MnJkV0+Q9T9fK82l0ddLM3+//m57p6OjY0VEjGlops3rWdJ7Ui5TelvoPaQxY8oiYnn+9X9xoWPi2cBf2NQXgLtIj1ffz6ajoW4kImYrjYPTLul50ngwnyMFFT+QdDZpBN1L6GJArx49rtja2hrt7TU9aVGTq6++umF59YUjjjiiofldddVVDc2vkdra2rpfqR/5Wmkujb5emvn79XfbO5JqetyvvzTqccWBxE0JZmZmVubAwMzMzMr8VIKZmQ1aCxcubPqm1M3NNQZmZmZW5sDAzMysTpI25Jcm/SkPbzy8+6022n6spN/k6UmSDiukHSHps40uc3ccGJiZmdVvfURMioi9gOdJb02tWUQsjYjSuw4mkd70WUq7KiKmN6ykNXJgYGZm1hjzgAmSRkm6UtJ9ku6U9EYASQcUXss8X9IISeNzbcNQ4MvAlJw+RdJUSd+VNDK/YGlIzme4pCWStunpC5Jq4cDAzMysl/KARoeSBiI6B5gfEW8kDTD0i7zaGcDH8/DE+wPlN6dGxPPAF4FZuQZiViFtFWlAogPyojZgTkS8QA9fkFQLP5VgZmZWXXcvUdpO0r15eh7wE9Johe8FiIgbJe0saSRwG/BtSReR3pz4aOHFRt2ZBUwBbgKOAb5fzwuSauHAwMzMrLruXqK0vvIFRer8bh8RMV3StaR+BHdKOpg0jHItrgK+JmkU0ALcCGxPD1+QVAs3JZiZmTXWLeSXHkk6kBRcrJa0e0TcHxFfB9qByv4Aa6jyPoSIWAvcDZwPXBMRG+p5QVItHBiYmZk11jSgVdJ9wHTgxLz89NzRcAGpf8F1FdvdBLyu1Pmwk3xnAcfnf0uOA07OeT4AHNnbwrspwczMrE4RsUMny56ikxt0RJzaSRaPAHsVttunIn1mYfvfABs1U0TEImByD4vdJdcYmJmZWZkDAzMzMytzYGBmZoNWS0sLEVH1Mxg5MDAzM7MyBwZmZmZWpp5UlUhaDizuu+KYmdkAMy4ixvR3IappbW2N9vb2qumSOroZ4GjA6dHjis385ZqZmVnvuSnBzMzMyjzAkZmZWXXdvURpwHFgYGZmVl13L1EacNyUYGZmZmUODMzMzKzMgYGZmZmVOTAwMzOzMgcGZmZmVtajkQ/NzMwGEkmzgdFdrLIiIiZvrvI0AwcGZmZmVuamBGsakvaX9Of+LkdnJB0o6dEu0kPShM1ZpirlmCnpq73Y/jpJJzayTGa2ZXFgYL0m6RFJ6yWtLXy+W8N2G91MI2JeRLy6j8rYqxvmYBERh0bEz/u7HGbWfzzyoTVKW0Tc0N+FsPpIEqlp8R/9XRYz61+uMbA+JWmCpJslrZK0QtKsvPyWvMqCXMMwpbK6PtdEfFrSfZKekfQTSS/L1d1rJN0gaafC+pdJWpb3dYuk1+flHwGOA87M+7o6Lx8r6XJJyyUtknRaIa/tci3DSkkPAvvUcLiHSXo4H+f/lzQk5zVE0tmSFkv6u6RfSBqZ0zZposjHfXCenibp0rzNGkkPSGotrPsmSX/MabOAYYW0nSRdk49vZZ7epZA+V9K5km4D1gGvyss+XFjnJEkP5e3nSBqXl0vSjHw8q/J3tFcN58jMmpwDA+trXwGuB3YCdgG+AxARb8vpEyNih4iYVWX79wLvAPYE2oDrgM+RehEPAU4rrHsdsAfwUuCPwEV5Xxfm6W/kfbXlm/bVwALglcBBwOmS3pnz+hKwe/68E6il3f3dQCuwN3AkcFJePjV/3g68CtgB6LappeAI4BJgR+Cq0raShgJXAr8ERgGXkc5XyRDgZ8A4YDdgfSf7PQH4CDACWFxMkHQU6Vy/BxgDzAMuzsmHAG8jfS87AlOAJ3twTGbWpBwYWKNcKenpwueUvPwF0o1pbEQ8GxG39jDf70TEExHxGOnGdFdEzI+I54ArgDeVVoyIn0bEmpw2DZhY+mXeiX2AMRHx5Yh4PiIeBn4EHJPT3w+cGxFPRcQS4IIayvr1vP7fgPOAY/Py44BvR8TDEbEWOAs4RlKtTXm3RsTvI2IDKQiYmJe/BdgGOC8iXoiI3wD3lDaKiCcj4vKIWBcRa4BzgQMq8p4ZEQ9ExIsR8UJF2keBr0XEQxHxIvCfwKRca/ACKZh4DakJ4qGIeLzG4zGzJuY+BtYoR1XpY3AmqdbgbkkrgW9FxE97kO8Then1nczvACBpK9KN72jSr9tSW/loYFUn+Y4Dxkp6urBsK1LwATAWWFJI2+jXdBWV648t5LW4Im1r4GU15AmwrDC9DhiWg4qxwGOx8TPH5f1IGg7MACaTamwARkjaKgcZlWWuNA44X9K3CssEvDIibswdTL8H7CbpCuCMiFhd4zGZNYWWlpZYvbr6Zbtw4cI5g20cAwcG1qciYhlwCoCk/YAbJN0SEQsbvKsPkKrvDwYeAUYCK0k3MoDKATuWAIsiYo8q+T0O7Ao8kOd3q6EMlesvzdNLSTdZCmkvkoKcscDwUkIOcMbUsK9SGV8pSYXgYDfgr3n6U8CrgX0jYpmkScB8/nlOYNPzUrSEVGtyUWeJEXEBcIGklwKXAp8GvlBj2c2awurVq5kxY0bV9La2tq4GPxqQ3JRgfUrS0YUObytJN6LSr9UnSG3ujTACeI7Uzj2cVO1dVLmvu4HVkj6TOxpuJWkvSaVOhpcCZ+UOfLsAp9ZQhk/n9XcFPgGU+k1cDPyHpH+RtEMu26xcPf8XUg3AuyRtA5wNbFvjMd9BCjBOk7S1pPcAby6kjyDVqjwtaRSp30RP/JB0DkqdOEdKOjpP7yNp31zmZ4Bn+ef3amZbMAcG1ihXa+NxDK7Iy/cB7pK0ltRx7hMRsSinTQN+nvskvL+X+/8FqRr9MeBB4M6K9J8Ar8v7ujJXpbcBk4BFwArgx6SaBoBzcn6LSJ0nf1lDGX4HdAD3AtfmfQL8NG9/S87vWXKgERGrgH/P+36MdJOtOpBSUUQ8T+oYOJUUdE0BfltY5Txgu3xsdwKza8m3kP8VwNeBSyStBv4EHJqTX0Lqk7GSdJ6eBL7Zk/zNtnSSdpZ0b/4sk/RYYX5oxbpzi08UNTMPiWxmZoPWHnvsEd00JXRERLc3dEnTgLUR0WmALGkuqR9Oey3lqugLtFm5xsDMzKxBJB0kab6k+yX9VNImTYOSDpF0Rx6D5LLcxFgaw+SLkm4FjpZ0iqR7JC1QGnNleF5vpqQLJN2uNHbK+wp5n5n3vUDS9Lxsd0mzJXVImifpNV0dgwMDMzOzxhgGzASmRMQbSB38/624gqTRpL5EB0fE3kA78MnCKs9GxH4RcQnw24jYJyImAg8BJxfWewWwH3A4UAoADgWOInU4ngh8I697IXBqRLQAZwDf7+og/FSCmZlZdaMlFav/L8yDpnVmK9LTTn/J8z8HPk7q71PyFuB1wG2SAIaSOhKXFAd720vpHS87kh7NnlNIuzIPYf6gpNKjzwcDP4uIdQAR8VSujXgrcFneH3TTwdmBgZmZWXUrauljkD1TwzoC/hARx1ZJL+YxkzRGzAJJU4EDC2nPVeRZ+rey4+AQ4OmImFRD2cobmJmZWe8NA8brn2+NPQG4uWKdO4F/La0jabikPavkNwJ4PD8WfFwN+78eOKnQF2FUHnRsUeFRY0ma2FUmPaoxGD16dIwfP74nm1hBR0dHw/JqaWlpWF7We438bmHwfb/+26hfs197HR0dKyKi1kG7tnTPAh8iVdtvTRqi/IfFFSJief71f3GhY+LZpDFNKn0BuIv0SPD9pEChqoiYnQcya5f0PPB70vtOjgN+IOls0jDql5DeE9OpHj2u2NraGu3tNT1pYZ0otO/0mh8zbS6N/G5h8H2//tuoX7Nfe5JqetyvvzTqccWBxE0JZmZmVubAwMzMzMr8VIKZmQ1aCxcupK2trb+L0VRcY2BmZmZlDgzMzMzqJGlDfmnSn/LwxsO732qj7cdK+k2eniTpsELaEZI+2+gyd8eBgZmZWf3WR8SkiNgLeB74WE82joilEVF618Ek4LBC2lURMb1hJa2RAwMzM7PGmAdMkDRK0pWS7pN0p6Q3Akg6oPBa5vmSRkgan2sbhgJfBqbk9CmSpkr6rqSR+QVLQ3I+wyUtkbRNT1+QVAsHBmZmZr2UBzQ6lDQQ0TnA/Ih4I2mAoV/k1c4APp6HJ94fWF/aPiKeB74IzMo1ELMKaatIAxIdkBe1AXMi4gV6+IKkWvipBDMzs+q6e4nSdpLuzdPzgJ+QRit8L0BE3ChpZ0kjgduAb0u6iPTmxEd7MEDVLGAKcBNwDPD9el6QVAsHBmZmZtV19xKl9ZUvKFLnd/uIiOmSriX1I7hT0sGkYZRrcRXwNUmjgBbgRmB7eviCpFq4KcHMzKyxbiG/9EjSgaTgYrWk3SPi/oj4OtAOVPYHWEOV9yFExFrgbuB84JqI2FDPC5Jq4cDAzMyssaYBrZLuA6YDJ+blp+eOhgtI/Quuq9juJuB1pc6HneQ7Czg+/1tyHHByzvMB4MjeFt4vUdqM/KKYgavZX2TT7Py3Ub9mv/aa/SVKkro74KYuf19wjYGZmZmVOTAwMzOzMgcGZmY2aLW0tBARVT+DkQMDMzMzK3NgYGZmZmU9GuCoo6NjhaTFfVUYq12jeyJbc/H3Wz+fu97pg/M3rtEZWt/qUWAQEWP6qiBmZmbW/9yUYGZmZmV+V4KZmVl13b1EacBxYGBmZlZddy9RGnDclGBmZmZlDgzMzMyszIGBmZmZlTkwMDMzszIHBmZmZlamwfqSCDMzM0mzgdFdrLIiIiZvrvI0AwcGZmZmVuamBLMuSJor6cN9lPfnJP24L/I2M6uXAwMbMCQ9Imm9pLWFz3f7u1wAkg6U9GhxWUT8Z0T0SdBhZlYvj3xoA01bRNzQ34UwM9tSucbABjRJ20p6WtJehWVjcs3CSyXtJOkaScslrczTu1TJa5qkXxXmx0sKSVvn+Q9JekjSGkkPS/poXr49cB0wtlCTMbaT/I6Q9EAu71xJry2kPSLpDEn3SVolaZakYY0/Y2Y22DkwsAEtIp4DfgscW1j8fuDmiPg76W/gZ6R3xu8GrAfqbX74O3A48BLgQ8AMSXtHxDPAocDSiNghf5YWN5S0J3AxcDowBvg9cLWkoRXlngz8C/BGYGqd5TQzq8qBgQ00V+Zf3KXPKcCv2Tgw+EBeRkQ8GRGXR8S6iFgDnAscUM+OI+LaiPhrJDcD1wP717j5FODaiPhDRLwAfBPYDnhrYZ0LImJpRDwFXA1MqqecZmZdcR8DG2iOquxjIGkIsJ2kfYFlpBvqFTltODCD9Et8p7zJCElbRcSGnuxY0qHAl4A9SUH3cOD+GjcfCywuzUTEPyQtAV5ZWGdZYXpd3sbMemHyhAmxYt26qukdjz8+Z7CNY+DAwAa8fJO9lFRr8ARwTa4dAPgU8Gpg34hYJmkSMB9QJ1k9Q7rZl7y8NCFpW+By4IPA7yLiBUlXFvLpbsCQpcAbCvkJ2BV4rJZjNLP6rFi3jvaPfrRquqZN62rwIyRtYOMfAEcB44HfAQ8Dw4BLIuKc3pZ1c3FTgg0WvyZV1x+Xp0tGkPoVPC1pFOkXfzX3Am+TtJukkcBZhbShwLbAcuDFXHtwSCH9CWDnvF1nLgXeJekgSduQApbngNtrPD4z6x/rI2JS4fNIXj4vIt4EtALHS2rpvyL2jAMDG2iurhjH4AqAiLiL9It/LOkJgZLzSG35K4A7gdnVMo6IPwCzgPuADuCaQtoa4DTSDX4lqR/DVYX0/yF1Lnw4933YqBkgIv4MHA98J5eljfTo5fN1nAMzaxK583EHsHt/l6VWHhLZzMwGrdaxY6ObpoSOiGitmr5xU8KiiHi3pAOBMyLicEk7kwKDd0XEA40red9xHwMzM7PqRktqL8xfGBEXFubXR8SkTrbbX9J84B/A9C0lKAAHBmZmZl1Z0VWNQRfmRcThDS/NZuA+BmZmZlbmwMDMzMzK3JRgZmZWp4jYoZNlc4G5m70wDdKjpxJGjx4d48eP77vSDHAdqzoallfLyC3mkdhBoZHfLQy+79d/G/Vr9muvo6NjRUSMaWimDdQqRXsX6YIun0oYiHoUGLS2tkZ7e1en0LqiqzsbTK8+0ebHTJtJI79bGHzfr/826tfs156kpr6x9jYw6GLkw5uAIyLi6rzeNcA3c21CU3NTgpmZWf02eVxR0njgUeDzpBeebVHc+dDMzKzxFgCrJL2jvwvSUw4MzMzM6redpHvz54qKtK8CZ/dHoXrDTQlmZmbV1TvyIRExTxKS9u/TEjaYAwMzM7Pq6h35sORcUl+DFxtUnj7npgQzM7M+EhHXAzsBE/u7LLVyYGBmZta3zgV26e9C1MpNCWZmZnWqZeTDiLgKaOyAE33IgYGZmQ1aHRNAM7pYoW2zFaVpODAwMzOrg6Sdgf/Osy8HNgDL8/xE0lgGWwMPASdGxLrNXsg6uI+BmZlZHSLiyYiYlB9X/CEwozD/TJ7eC3ge+Fg/FrVHHBiYmZn1rXnAhP4uRK0cGJiZmfURSVsDh7Lxi5aamvsYmJmZVdfdyIfVbCfp3jw9D/hJw0vWRxwYmJmZVVfvyIdVh0pudm5KMDMzszIHBmZmZlbmpgQzM7NeiohpFfObjIi4pVBE1L6ytBxY3HfFMTOzAWZcRIzp70JU09raGu3t7VXTJXX08u2KW5we1Rg085drZmZmvec+BmZmZlbmwMDMzMzKHBiYmZlZmZ9KMDMzq67ekQ+3WA4MzMzMqqt35MMtlpsSzMzMrMyBgZmZmZU5MDAzM7OyHo18aGZmNpBImg2M7mKVFRExeXOVpxk4MDAzM7MyNyWYmZlZmQMDMzMzK3NgYGZmZmUODMzMzKzMgYGZmZmV/R8+mUvVFWvkTwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "def evaluate_boundary(B_ref, B_est, tau):\n", " \"\"\"Compute boundary evaluation measures [FMP, Section 4.5.4]\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " B_ref (np.ndarray): Reference boundary annotations\n", " B_est (np.ndarray): Estimated boundary annotations\n", " tau (int): Tolerance parameter.\n", " Note: Condition ``|b_{k+1}-b_k|>2tau`` should be fulfilled [FMP, Eq. 4.58]\n", "\n", " Returns:\n", " P (float): Precision\n", " R (float): Recall\n", " F (float): F-measure\n", " num_TP (int): Number of true positives\n", " num_FN (int): Number of false negatives\n", " num_FP (int): Number of false positives\n", " B_tol (np.ndarray): Data structure encoding B_ref with tolerance\n", " I_eval (np.ndarray): Data structure encoding TP, FN, FP\n", " \"\"\"\n", " N = len(B_ref)\n", " num_TP = 0\n", " num_FN = 0\n", " num_FP = 0\n", " B_tol = np.zeros((np.array([B_ref])).shape)\n", " B_eval = np.zeros((np.array([B_ref])).shape)\n", " for n in range(N):\n", " min_idx = max(0, n - tau)\n", " max_idx = min(N - 1, n + tau)\n", " if B_ref[n] == 1:\n", " B_tol[:, min_idx:max_idx+1] = 2\n", " B_tol[:, n] = 1\n", " temp = sum(B_est[min_idx:max_idx+1])\n", " if temp > 0:\n", " num_TP += temp\n", " else:\n", " num_FN += 1\n", " B_eval[:, n] = 2\n", " if B_est[n] == 1:\n", " if sum(B_ref[min_idx:max_idx+1]) == 0:\n", " num_FP += 1\n", " B_eval[:, n] = 3\n", " else:\n", " B_eval[:, n] = 1\n", " P, R, F = measure_prf(num_TP, num_FN, num_FP)\n", " return P, R, F, num_TP, num_FN, num_FP, B_tol, B_eval\n", "\n", "def plot_boundary_measures(B_ref, B_est, tau, figsize=(8, 2.5)):\n", " \"\"\"Plot B_ref and B_est (see :func:`libfmp.c4.c4s5_evaluation.evaluate_boundary`)\n", "\n", " Notebook: C4/C4S5_Evaluation.ipynb\n", "\n", " Args:\n", " B_ref: Reference boundary annotations\n", " B_est: Estimated boundary annotations\n", " tau: Tolerance parameter\n", " figsize: Figure size (Default value = (8, 2.5))\n", "\n", " Returns:\n", " fig: Handle for figure\n", " ax: Handle for axes\n", " \"\"\"\n", " P, R, F, num_TP, num_FN, num_FP, B_tol, B_eval = evaluate_boundary(B_ref, B_est, tau)\n", "\n", " colorList = np.array([[1., 1., 1., 1.], [0., 0., 0., 1.], [0.7, 0.7, 0.7, 1.]])\n", " cmap_tol = ListedColormap(colorList)\n", " colorList = np.array([[1, 1, 1, 1], [0, 0.7, 0, 1], [1, 0, 0, 1], [1, 0.5, 0.5, 1]])\n", " cmap_measures = ListedColormap(colorList)\n", "\n", " fig, ax = plt.subplots(3, 2, gridspec_kw={'width_ratios': [1, 0.02],\n", " 'wspace': 0.2, 'height_ratios': [1, 1, 1]},\n", " figsize=figsize)\n", "\n", " im = ax[0, 0].imshow(B_tol, cmap=cmap_tol, interpolation='nearest')\n", " ax[0, 0].set_title('Reference boundaries (with tolerance)')\n", " im.set_clim(vmin=-0.5, vmax=2.5)\n", " ax[0, 0].set_xticks([])\n", " ax[0, 0].set_yticks([])\n", " ax_cb = plt.colorbar(im, cax=ax[0, 1])\n", " ax_cb.set_ticks(np.arange(0, 3, 1))\n", " ax_cb.set_ticklabels(['', 'Positive', 'Tolerance'])\n", "\n", " im = ax[1, 0].imshow(np.array([B_est]), cmap=cmap_tol, interpolation='nearest')\n", " ax[1, 0].set_title('Estimated boundaries')\n", " im.set_clim(vmin=-0.5, vmax=2.5)\n", " ax[1, 0].set_xticks([])\n", " ax[1, 0].set_yticks([])\n", " ax_cb = plt.colorbar(im, cax=ax[1, 1])\n", " ax_cb.set_ticks(np.arange(0, 3, 1))\n", " ax_cb.set_ticklabels(['', 'Positive', 'Tolerance'])\n", "\n", " im = ax[2, 0].imshow(B_eval, cmap=cmap_measures, interpolation='nearest')\n", " ax[2, 0].set_title('Evaluation')\n", " im.set_clim(vmin=-0.5, vmax=3.5)\n", " ax[2, 0].set_xticks([])\n", " ax[2, 0].set_yticks([])\n", " ax_cb = plt.colorbar(im, cax=ax[2, 1])\n", " ax_cb.set_ticks(np.arange(0, 4, 1))\n", " ax_cb.set_ticklabels(['', 'TP', 'FN', 'FP'])\n", " plt.show()\n", " return fig, ax\n", "\n", "\n", "B_ref = [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0]\n", "B_est = [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0]\n", "tau_list = [0,1,2]\n", "\n", "for tau in tau_list:\n", " print('====== Evaluation using tau = %d ======'%tau)\n", " P, R, F, num_TP, num_FN, num_FP, B_tol, B_eval = evaluate_boundary(B_ref, B_est, tau) \n", " print('#TP = ', num_TP, '; #FN = ', num_FN, '; #FP = ', num_FP)\n", " print('P = %0.3f; R = %0.3f; F = %0.3f' % (P, R, F))\n", " fig, ax = plot_boundary_measures(B_ref, B_est, tau=tau)\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Further Notes\n", "\n", "Many evaluation measures involve some sort of precision and recall rate. We have seen some typical examples in this notebook. Here are some further links:\n", "\n", "* The [Python library `mir_eval`](https://craffel.github.io/mir_eval/) provides various standardized evaluation measures used in Music Information Retrieval (MIR). Additionally, the library also contains methods for the [visualization](../B/B_Annotations.html) and [sonification](../B/B_Sonification.html) of evaluation results.\n", "\n", "* Systematic evaluations of different structure analysis methods have been performed over recent years within the [Music Information Retrieval Evaluation eXchange (MIREX) campaign](https://www.music-ir.org/mirex/wiki/MIREX_HOME), which provides a framework for evaluating various kinds of music processing algorithms. On the MIREX websites, one finds not only the evaluation results of numerous procedures, but also links to information on datasets, annotations, evaluation measures, and relevant literature. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "Acknowledgment: This notebook was created by Meinard Müller and Tim Zunner.\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\"C0\"\"C1\"\"C2\"\"C3\"\"C4\"\"C5\"\"C6\"\"C7\"\"C8\"
" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.16" } }, "nbformat": 4, "nbformat_minor": 1 }