{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Comparison and assessment of GRN inference methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load GRNs inferences\n", "+ Load the rankings/scores provided by different methods, let $M$ be the number of \"methods\".\n", "+ Concatenate the rankings use the `pandas.concat` function\n", "\n", "__Check the `pandas.concat` example:__" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
score_method1score_method2score_method3
a130100
b220300
c310200
\n", "
" ], "text/plain": [ " score_method1 score_method2 score_method3\n", "a 1 30 100\n", "b 2 20 300\n", "c 3 10 200" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# create a dummy dataframe a\n", "a = pd.DataFrame()\n", "a[\"score_method1\"] = [1,2,3]\n", "a.index = [\"a\",\"b\",\"c\"]\n", "# create a dummy dataframe b\n", "b = pd.DataFrame()\n", "b[\"score_method2\"] = [10,20,30]\n", "b.index = [\"c\",\"b\",\"a\"]\n", "# create a dummy dataframe c\n", "c = pd.DataFrame()\n", "c[\"score_method3\"] = [100,200,300]\n", "c.index = [\"a\",\"c\",\"b\"]\n", "# join the three dummy datasets\n", "join_df = pd.concat([a,b,c], axis=1,sort=True) # axis=1 for rows and axis=0 for columns\n", "join_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ It could be usefull to set the index of each ranking in the following format: \"TFid_TGid\"\n", "\n", "__Check the index modification example:__" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
TFTGscorerank
tfa_tgatfatga0.90
tfb_tgbtfbtgb0.21
tfc_tgctfctgc0.12
\n", "
" ], "text/plain": [ " TF TG score rank\n", "tfa_tga tfa tga 0.9 0\n", "tfb_tgb tfb tgb 0.2 1\n", "tfc_tgc tfc tgc 0.1 2" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Create a dummy ranking dataframe\n", "ranking = pd.DataFrame()\n", "ranking[\"TF\"] = [\"tfa\",\"tfb\",\"tfc\"]\n", "ranking[\"TG\"] = [\"tga\",\"tgb\",\"tgc\"]\n", "ranking[\"score\"] = [0.9,0.2,0.1]\n", "ranking[\"rank\"] = [0,1,2]\n", "# Change the index\n", "ranking.index = ranking[\"TF\"] + \"_\" + ranking[\"TG\"]\n", "ranking" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Load \"High Confidence\" GRN" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let an oriented graph $GRN_{HC} = \\langle V_{HC}, E_{HC} \\rangle$ denote the \"high confidence\" GRN presented in [Poitier et al. 2014](https://www.sciencedirect.com/science/article/pii/S2211124714010043), where $V_{HC}$ denotes the set of nodes (genes) and $E_{HC}$ denotes the set of edges (regulatory relationships).\n", "\n", "+ Load the \"high confidence\" GRN\n", "+ Set the index (as in the previous step).\n", "+ Select only the edges that involve highly expressed genes (those that you have used to infer your GRNs)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Compare methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ Represent the correlation between the different methods using a heatmap " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ Generate a [dendogram](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.cluster.hierarchy.dendrogram.html) or a [clustermap](https://seaborn.pydata.org/generated/seaborn.clustermap.html) using the ranking or the score vectors " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ " + Compare the top $k$ links of each method using the [Jaccard similarity score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_similarity_score.html), represent the results in a matrix" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ For each method, keep only the edges that are reported in the \"high confidence\" GRN.\n", "+ Apply a [PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) to the rankings of the methods for the chosen links, keep 3 dimensions, and the location of each method in this new space (you go from $M$ points in a $|E_{HC}|$-th dimensional space to $M$ points in a 3D space ). \n", "\n", "__Check the following example:__" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dummy ranking:\n", " method0 method1 method2 method3 method4 method5 method6 method7 \\\n", "edge0 327 924 545 53 740 27 755 108 \n", "edge1 247 908 773 376 189 69 191 536 \n", "edge2 185 115 305 833 433 500 747 997 \n", "edge3 504 206 287 403 203 862 221 538 \n", "edge4 323 449 549 438 388 573 85 539 \n", "\n", " method8 method9 \n", "edge0 213 936 \n", "edge1 138 166 \n", "edge2 686 496 \n", "edge3 511 306 \n", "edge4 573 441 \n", "\n", "Explained variance ratio for each PC\n", "[0.16138035 0.1454675 ]\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbUAAAEOCAYAAAAOp4lrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X90VeWZ9//3RdQ8qDjIKMoPrWBDKEkgCSHIUhBEjIADhI7VPqypYKlTB9uvfSoOtE+AscUyRalFO7Y4A6WutqKDhKjMoFjbwswjGCUioqkIGSTSQMuvEBIh5Pr+cXZOD5gIhOScnZPPa629ss+1f5xrHwIX+z73vm9zd0RERJJBp0QnICIi0lpU1EREJGmoqImISNJQURMRkaShoiYiIklDRU1ERJKGipqIiCQNFTUREUkaKmoiIpI0VNRERCRpnJfoBMLisssu82uuuSbRaYiItBtvvvnmn9z98kTnESs0Rc3MrgJ+AVwBOLDE3X9sZt2AFcA1QAXwJXc/YGYG/BgYBxwFprr7W8G57gL+b3Dq77v78tO9/zXXXENpaWnrXpSISBIzs/9JdA6nClPzYz3wbXcfAFwHzDCzAcAs4FV3TwNeDV4DjAXSguUe4EmAoAjOBYYC+cBcM7s0nhciItKUsrIy1qxZE309b948HnnkkRafL/b45557joyMDDp16tSh/4MemqLm7nsa77TcvRp4D+gFTAQa77SWA5OC9YnALzzidaCrmfUACoBX3H2/ux8AXgFujeOliIg06dSi1poyMzN5/vnnGTFiRJucv70ITVGLZWbXADnARuAKd98TbPojkeZJiBS8j2IO2x3EmouLiJyziooK+vfvz9SpU+nXrx9Tpkxh3bp1XH/99aSlpbFp0yZqamq4++67yc/PJycnh9WrV3Ps2DHmzJnDihUryM7OZsWKFQBs27aNkSNH0rdvXxYvXhx9n0WLFpGZmUlmZiaPPfZYND5//nz69evHDTfcQHl5eTT+hS98gfT09Ph9ECEVmu/UGpnZxcBK4H53Pxz56izC3d3MWm0CODO7h0jTJVdffXVrnVZEktz27dt57rnnWLp0KUOGDOFXv/oVGzZsoKSkhIcffpgBAwZw0003sXTpUg4ePEh+fj4333wzDz30EKWlpTzxxBNApPnw/fff57XXXqO6upr09HTuvfdetmzZwrJly9i4cSPuztChQ7nxxhtpaGjgmWeeoaysjPr6enJzcxk8eHCCP41wCVVRM7PziRS0X7r780G4ysx6uPueoHlxbxCvBK6KObx3EKsERp4S/21T7+fuS4AlAHl5eZotVUSaVLy5koVry/n4YC3d/BDde15FVlYWABkZGYwePRozIysri4qKCnbv3k1JSUn0+666ujp27drV5LnHjx9PamoqqampdO/enaqqKjZs2EBhYSEXXXQRAJMnT2b9+vU0NDRQWFjIhRdeCMCECRPicPXtS2iaH4PejP8GvOfui2I2lQB3Bet3Aatj4l+xiOuAQ0Ez5VrgFjO7NOggcksQExE5a8WbK5n9/DtUHqzFgarDdfy5zineXAlAp06dSE1Nja7X19fj7qxcuZKysjLKysrYtWsXX/jCF5o8f+OxACkpKdTX17f5NSWz0BQ14Hrg74CbzKwsWMYBC4AxZvYBcHPwGmANsAPYDjwF/AOAu+8Hvge8ESwPBTERkbO2cG05tcdPnBRzdxauLW/mCCgoKODxxx/HPdIAtHnzZgC6dOlCdXX1ad9z+PDhFBcXc/ToUWpqali1ahXDhw9nxIgRFBcXU1tbS3V1NS+88MI5XFlyCk3zo7tvAKyZzaOb2N+BGc2caymwtPWyE5GO6uODtWcVBygqKuL+++9n4MCBNDQ00KdPH1588UVGjRrFggULyM7OZvbs2c0en5uby9SpU8nPzwdg+vTp5OTkAHDHHXcwaNAgunfvzpAhQ6LHrFq1im984xvs27eP8ePHk52dzdq1Ha+Ryhr/J9HR5eXleUd+tkNEmnb9gt9Q2UQB69W1M/8166YEZBQeZvamu+clOo9YYWp+FBEJnZkF6XQ+P+WkWOfzU5hZoO7zYRSa5kcRkTCalBN5zLWx92PPrp2ZWZAejUu4qKiJiJzGpJxeKmLthJofRUQkaaioiYhI0lBRExGRpKGiJiIiSUNFTUREkoaKmoiIJA0VNRERSRoqaiIikjRU1EREJGmoqImISNJQURMRkaShoiYiIklDRU1ERJKGipqIiCSN0BQ1M1tqZnvNbGtMbJ6ZVZpZWbCMi9k228y2m1m5mRXExG8NYtvNbFa8r0NERBInNEUN+DlwaxPxH7l7drCsATCzAcCdQEZwzL+YWYqZpQA/AcYCA4AvB/uKiEgHEJpJQt3992Z2zRnuPhF4xt0/AXaa2XYgP9i23d13AJjZM8G+21o5XRERCaEw3ak15z4z2xI0T14axHoBH8XsszuINRcXEZEOIOxF7UngWiAb2AM82ponN7N7zKzUzEr37dvXmqcWEZEECHVRc/cqdz/h7g3AU/ylibESuCpm195BrLl4c+df4u557p53+eWXt27yIiISd6EuambWI+ZlIdDYM7IEuNPMUs2sD5AGbALeANLMrI+ZXUCkM0lJPHMWEZHECU1HETP7NTASuMzMdgNzgZFmlg04UAH8PYC7v2tmzxLpAFIPzHD3E8F57gPWAinAUnd/N86XIiIiCWLunugcQiEvL89LS0sTnYaISLthZm+6e16i84gV6uZHERGRs6GiJiIiSUNFTUREkoaKmoiIJA0VNRERSRoqaiIikjRU1EREJGmoqImISNJQURMRkaShoiYiIklDRU1ERJKGipqIiCQNFTUREUkaKmoiIpI0VNRERCRpqKiJiEjSUFETEZGkEaqiZmZLzWyvmW2NiXUzs1fM7IPg56VB3MxssZltN7MtZpYbc8xdwf4fmNldibiWs1FWVsaaNWuir+fNm8cjjzzS4vPFHl9WVsZ1111HdnY2eXl5bNq06ZzzFREJq1AVNeDnwK2nxGYBr7p7GvBq8BpgLJAWLPcAT0KkCAJzgaFAPjC3sRCG1alFrTU9+OCDzJ07l7KyMh566CEefPDBNnkfEZEwCFVRc/ffA/tPCU8Elgfry4FJMfFfeMTrQFcz6wEUAK+4+353PwC8wqcLZaurqKigf//+TJ06lX79+jFlyhTWrVvH9ddfT1paGps2baKmpoa7776b/Px8cnJyWL16NceOHWPOnDmsWLGC7OxsVqxYAcC2bdsYOXIkffv2ZfHixdH3WbRoEZmZmWRmZvLYY49F4/Pnz6dfv37ccMMNlJeXR+NmxuHDhwE4dOgQPXv2bOuPQkQkcdw9VAtwDbA15vXBmHVrfA28CNwQs+1VIA94APi/MfEi4IHTve/gwYP9XOzcudNTUlJ8y5YtfuLECc/NzfVp06Z5Q0ODFxcX+8SJE3327Nn+9NNPu7v7gQMHPC0tzY8cOeLLli3zGTNmRM81d+5cHzZsmNfV1fm+ffu8W7dufuzYMS8tLfXMzEw/cuSIV1dX+4ABA/ytt96KxmtqavzQoUN+7bXX+sKFC93dfdu2bX7VVVd57969vWfPnl5RUXFO1yki0ggo9RDUjdjlvLhW0HPk7m5m3lrnM7N7iDRdcvXVV5/z+fr06UNWVhYAGRkZjB49GjMjKyuLiooKdu/eTUlJSfT7rrq6Onbt2tXkucaPH09qaiqpqal0796dqqoqNmzYQGFhIRdddBEAkydPZv369TQ0NFBYWMiFF14IwIQJE6LnefLJJ/nRj37EF7/4RZ599lm++tWvsm7dunO+VhGRMGoPRa3KzHq4+56geXFvEK8ErorZr3cQqwRGnhL/bVMndvclwBKAvLy8sy6WxZsrWbi2nI8P1tLND/GJp0S3derUidTU1Oh6fX09KSkprFy5kvT09JPOs3Hjxk+du/FYgJSUFOrr6882PQCWL1/Oj3/8YwBuv/12pk+f3qLziIi0B6H6Tq0ZJUBjD8a7gNUx8a8EvSCvAw65+x5gLXCLmV0adBC5JYi1quLNlcx+/h0qD9biQNXhOqoO11G8ubLZYwoKCnj88ccbm0XZvHkzAF26dKG6uvq07zl8+HCKi4s5evQoNTU1rFq1iuHDhzNixAiKi4upra2lurqaF154IXpMz549+d3vfgfAb37zG9LS0j7zPdqyJ+b+/fsZM2YMaWlpjBkzhgMHDrT4vCIiTQlVUTOzXwP/D0g3s91m9lVgATDGzD4Abg5eA6wBdgDbgaeAfwBw9/3A94A3guWhINaqFq4tp/b4iZNi7s7CteXNHAFFRUUcP36cgQMHkpGRQVFREQCjRo1i27ZtJ3UUaUpubi5Tp04lPz+foUOHMn36dHJycsjNzeWOO+5g0KBBjB07liFDhkSPeeqpp/j2t7/NoEGD+M53vsOSJUs+87rasifmggULGD16NB988AGjR49mwYIFpz9IROQsWONdQ0eXl5fnpaWlZ7x/n1kv0dQnZ8DOBeNbLa+WqKio4NZbb+W6667jv//7vxkyZAjTpk1j7ty57N27l1/+8pdkZGTwjW98g61bt3L8+HHmzZvH2LFj+fznP09tbS29evVi9uzZvPfee+zatYsdO3awa9cu7r//fr75zW8CkZ6YS5cuBWD69Oncf//9QKQn5vLly+nevTtXXXUVgwcP5oEHHiA9PZ3f/va39OjRgz179jBy5MiTemqKSPtiZm+6e16i84jVHr5TC6WeXTtTebC2yXgYbN++neeee46lS5cyZMgQfvWrX7FhwwZKSkp4+OGHGTBgADfddBNLly7l4MGD5Ofnc/PNN/PQQw9RWlrKE088AUSaD99//31ee+01qqurSU9P595772XLli0sW7aMjRs34u4MHTqUG2+8kYaGBp555hnKysqor68nNzeXwYMHA1BVVUWPHj0AuPLKK6mqqkrY5yMiyUlFrYVmFqQz+/l3TmqC7Hx+CjML0j/jqPgJY0/MWGaGmbX2ZYtIB6ei1kKTcnoBRHs/9uzamZkF6dF4vLWHnphXXHEFe/bsiTY/du/evUXnERFpTqg6irQ3k3J68V+zbmLngvH816ybElrQ2kNPzAkTJrB8eWRwmOXLlzNx4sRzuGoRkU9TUUsC7aUn5qxZs3jllVdIS0tj3bp1zJo1q9nzi4i0hHo/Bs6292OYhLknpogkrzD2ftSdWhJorsdlWHpiiojEi4paEphZkE7n81NOioWpJ6aISLyo92MSCFtPTBGRRFFRSxKTcnqpiIlIh6fmRxERSRoqaiIikjRU1EREJGmoqImISNJQURMRkaShoiYiIklDRU1ERJKGipqIiCSNdlPUzKzCzN4xszIzKw1i3czsFTP7IPh5aRA3M1tsZtvNbIuZ5SY2exERiYd2U9QCo9w9O2ZU6FnAq+6eBrwavAYYC6QFyz3Ak3HPVERE4q69FbVTTQSWB+vLgUkx8V94xOtAVzPrkYgERUQkftpTUXPgZTN708zuCWJXuPueYP2PwBXBei/go5hjdwcxERFJYu2pqN3g7rlEmhZnmNmI2I0eme30rGY8NbN7zKzUzEr37dvXiqm2XFlZGWvWrIm+njdvHo888kiLz3fq8Y8//jj9+/cnIyODBx988JxyFREJm3YzSr+7VwY/95rZKiAfqDKzHu6+J2he3BvsXglcFXN47yB26jmXAEsgMvN1W+Z/psrKyigtLWXcuHGtfu7XXnuN1atX8/bbb5OamsrevXtPf5CISDvSLu7UzOwiM+vSuA7cAmwFSoC7gt3uAlYH6yXAV4JekNcBh2KaKdtcRUUF/fv3Z+rUqfTr148pU6awbt06rr/+etLS0ti0aRM1NTXcfffd5Ofnk5OTw+rVqzl27Bhz5sxhxYoVZGdns2LFCgC2bdvGyJEj6du3L4sXL46+z6JFi8jMzCQzM5PHHnssGp8/fz79+vXjhhtuoLy8PBp/8sknmTVrFqmpqQB07949Tp+IiEicuHvoF6Av8HawvAt8N4j/NZFejx8A64BuQdyAnwAfAu8Aead7j8GDB3tr2blzp6ekpPiWLVv8xIkTnpub69OmTfOGhgYvLi72iRMn+uzZs/3pp592d/cDBw54WlqaHzlyxJctW+YzZsyInmvu3Lk+bNgwr6ur83379nm3bt382LFjXlpa6pmZmX7kyBGvrq72AQMG+FtvvRWN19TU+KFDh/zaa6/1hQsXurv7oEGDfM6cOZ6fn+8jRozwTZs2tdo1i0jHA5R6CGpE7NIumh/dfQcwqIn4n4HRTcQdmBGH1E5SvLmShWvL+Z//qeD8rlfyYX03sjp1IiMjg9GjR2NmZGVlUVFRwe7duykpKYl+31VXV8euXbuaPO/48eNJTU0lNTWV7t27U1VVxYYNGygsLOSiiy4CYPLkyaxfv56GhgYKCwu58MILAZgwYUL0PPX19ezfv5/XX3+dN954gy996Uvs2LEDM2vjT0ZEJD7aRVFrD4o3VzL7+XeoPX4CgBOWwuzn3wGgU6dO0Sa/Tp06UV9fT0pKCitXriQ9Pf2k82zcuPFT5248FiAlJYX6+voW5di7d28mT56MmZGfn0+nTp3405/+xOWXX96i84mIhE27+E6tPVi4tjxa0BrVHj/BwrXlTe5fUFDA448/3ti8yubNmwHo0qUL1dXVp32/4cOHU1xczNGjR6mpqWHVqlUMHz6cESNGUFxcTG1tLdXV1bzwwgvRYyZNmsRrr70GwB/+8AeOHTvGZZdd1qLrFREJIxW1VvLxwdqzihcVFXH8+HEGDhxIRkYGRUVFAIwaNYpt27ad1FGkKbm5uUydOpX8/HyGDh3K9OnTycnJITc3lzvuuINBgwYxduxYhgwZEj3m7rvvZseOHWRmZnLnnXeyfPlyNT2KSFKxxjuFji4vL89LS0tbfPz1C35DZRMFrFfXzvzXrJvOJTURkVAyszf9L8MWhoLu1FrJzIJ0Op+fclKs8/kpzCxIb+YIERFpbeoo0kom5URG4Vq4tpyPD9bSs2tnZhakR+MiItL2VNRa0aScXipiIiIJpOZHERFJGipqIiKSNFTUREQkaaioiYhI0jjnomZm/9gaiYiIiJyrsy5qZvZszPIcML0N8hL5lLacQPXtt99m2LBhZGVl8Td/8zccPnz4nPMVkfhryZ3aYXf/UrDcTmTKF5E2d2pRa03Tp09nwYIFvPPOOxQWFrJw4cI2eR8RaVstKWrzT3n93dZIRDqGsE6g+oc//IERI0YAMGbMGFauXBmnT0REWtXpJlwDioAZiZ74ra2X1pwkVJoX1glUhw0b5qtWrXJ390cffdQvvvji+H84Iu0M7XSS0C8BQ04Nmtl04HJ3/0GrVllJen369CErKwsgNBOoLl26lG9+85t873vfY8KECVxwwQVt+RGISBs5k6J23N3rmog/DbwFhLKomdmtwI+BFOBf3X1BglPqsBpnBP/4YC3d/BCf+F8Gfg7LBKr9+/fn5ZdfBiJNkS+99FKLziMiiXUm36kdM7Mepwbd/RPgeOundO7MLAX4CTAWGAB82cwGJDarjqlxRvDKg7U4UHW4jqrDdRRvrmz2mERMoLp3714AGhoa+P73v8/Xv/71c7hqEUmUMylqjwKrzexzsUEz6w6EdTK2fGC7u+9w92PAM8DEBOfUITU1I7i7NzsjOCRmAtVf//rX9OvXj/79+9OzZ0+mTZt2jlcuIolwRpOEmtldRIrb60AZkWJ4OzDP3X/Zphm2gJn9LXCru08PXv8dMNTd72vumHOdJFSa1mfWS03+z8eAnQvGxzsdEWlF7XaSUHdfDvQBVgDnA3XA/w5jQTsbZnaPmZWaWem+ffsSnU5S6tm181nFRUTOxWmLmpndZWZ/Av4HGAN8390fcvc32jy7lqsErop53TuIncTdl7h7nrvnXX755XFLriPRjOAiEk9ncqdWRKSY9SdS2B5u04xaxxtAmpn1MbMLgDuBkgTn1CFNyunFDyZn0atrZwzo1bUzP5icpclURaRNnEmX/sPuvjlYLzKzT/erDhl3rzez+4C1RLr0L3X3dxOcVoelGcFFJF7OpKj1MLN7gPeB94h8pxZ67r4GaJuBAkVEJJTOpPlxLpAFfA8oBzLNbI2Z/cDMvtym2YmEWFvOGlBUVMTAgQPJzs7mlltu4eOPPz7nfEU6gtMWtaAzxTfc/UZ37wb0BR4HDgLj2jpBkbBqy1kDZs6cyZYtWygrK+O2227joYceapP3EUk2Zz1Kv7vvdvf/cPd/dve/a4ukROIlrLMGXHLJJdH1mpoazCwOn4ZIEkj0iMphWTRKf8cU1lkD3N2/853veO/evT0jI8P37t0b989G5HQI4Sj9LZlPTSSpNM4a0KlTpyZnDXj55ZdZsGAB2dnZjBw58oxmDbjsssuanDXg4osvjs4asH79+uisAZdccslJswZA5C7uo48+YsqUKTzxxBPx+ChEzpqZZZvZuJjX88zsgXM4X/R4M1thZmXBUmFmZac7/kx6P4oklfYwa0CsKVOmMG7cOP7pn/7pnM8l0gaygTzaoLe5u9/RuG5mjwKHTneM7tSkQ2kvswZ88MEH0fXVq1fTv3//Fl6xyOm19LtlIsO4PgTcEdxNNRahAWb2WzPbYWbfbHwfM/s/ZrY1WO6PiX/XzP5gZhuATw03ZJEvlb8E/Pp016I7NelQPmvWgOYeEC8qKuL+++9n4MCBNDQ00KdPH1588UVGjRoVbZacPXt2s+8ZO2sAEJ01AIjOGtC9e/eTZg2YNWsW5eXldOrUic997nP89Kc/PddLF/lM27dv57nnnmPp0qUMGTKEX/3qV2zYsIGSkhIefvhhBgwYwE033cTSpUs5ePBg4++zAXOAPA8GjDezeURGoBoFdAHKzexJYCAwDRgaHLfRzH5H5ObqTiJ3fOcRmafzzVPSGw5UufsHnMYZjdLfEWiU/sQqKyvj448/Zty4SNP8vHnzuPjii3nggZY1zTd1/KOPPsoDDzxA72/8kpQL/+qk/TVrgHRkFRUVjBkzJtpC8JWvfIWCggKmTJnCjh07mDx5Mueddx51dXWcd17kXmj//v189NFH7wKP8Omidtzd5wev3yMy1OIXgb929zlB/HvAPiJFrVtMfBHwsbtHH/oMiuJ2d3/0dNeiOzUJhbKyMkpLS6NFrbV99NFHvPzyy1zQtXuT2zVrgHQ0rfHdspnVNXP6T2LWT3AOtcbMzgMmA4PPZH99pyatJqzPfAF861vf4oc//CF/1fkC/pdmDZAOrjW/WwaqiTQzns56YJKZXWhmFwGFQez3QbyzmXUB/uaU424G3nf33WdybbpTk1bVknb5m2++mYceeojS0tJo1/V58+bx/vvv89prr1FdXU16ejr33nsvW7ZsYdmyZWzcuBF3Z+jQodx44400NDTwzDPPUFZWRn19Pbm5uQweHPmP3erVq+nVqxeDBg3iwgtSKBo/gJ9t2sfHB2vp2bUzMwvSNeCydCit9d1y4DVgVtDd/gfNvae7v2VmPwc2BaF/9WCwfDNbAbwN7CUyy0qsOzmDDiKNVNSkVTU+8wU0+czX7t27KSkpiY5xeCbPfKWmpjb5zBcQfearoaEh+swXEH3m6+jRozz88MO8/PLL0fOOG9iDr9yU1WafgUjYfXyw9qTX5/3VFfT86r9E4z//+c+j26655hq2bt0KwM9+9rOTjjMz3H0/MIRmuHtmzPoiYFET+8wH5jdz/NTPvpqTqajJOQn7M18ffvghO3fuZNCgQQDs3r2b3NxcNm3axJVXXnnW5xNJBj27dqbylMLWGG/v9J2atFh7eOYrKyuLvXv3UlFRQUVFBb179+att95SQZMOLZlnpFdRkxb7rHb55hQVFXH8+HEGDhxIRkYGRUVFAIwaNYpt27ad1FGkKbHPfA0dOjT6zFdubm70ma+xY8ee9MyXiJwsmWek13NqAT2ndvb6zHqJpn579MyXSMdgZm+6e16i84ilOzVpseba35OhXV5E2qdQF7VgtObKmFGaY0eCnm1m282s3MwKYuK3BrHtZjYrMZl3DMncLi8i7VN76P34o9jhUgDMbACRZxcygJ7AOjPrF2z+CZEhWXYDb5hZibtvi2fCHUVj+3tj70c98yUiidYeilpTJgLPuPsnwE4z2w7kB9u2u/sOADN7JthXRa2NTMrppSImIqER6ubHwH1mtsXMlprZpUGsF/BRzD67g1hzcRER6QASXtTMbF3M/Dqxy0TgSeBaIlMS7AFOO0LzWb73PWZWamal+/bta81Ti4hIAiS8+dHdbz6T/czsKeDF4GUlcFXM5t5BjM+IN/XeS4AlEOnSf4Ypi4hISCX8Tu2zmFmPmJeFwNZgvQS408xSzawPkEZkkMw3gDQz62NmFxDpTFISz5xFRCRxEn6ndho/NLNswIEK4O8B3P1dM3uWSAeQemCGu58AMLP7gLVACrDU3d9NROIiIhJ/GlEkoBFFRETOjkYUERERaUMqaiIikjRU1EREJGmoqImISNJQURMRkaShoiYiIklDRU1ERJKGipqIiCQNFTUREUkaKmoiIpI0VNRERCRpqKiJiEjSUFETEZGkoaImIiJJQ0VNRESShoqaiIgkDRU1ERFJGqEoamZ2u5m9a2YNZpZ3yrbZZrbdzMrNrCAmfmsQ225ms2LifcxsYxBfYWYXxPNaREQkcUJR1ICtwGTg97FBMxsA3AlkALcC/2JmKWaWAvwEGAsMAL4c7Avwz8CP3P3zwAHgq/G5BBERSbRQFDV3f8/dy5vYNBF4xt0/cfedwHYgP1i2u/sOdz8GPANMNDMDbgL+PTh+OTCp7a9ARETCIBRF7TP0Aj6Keb07iDUX/2vgoLvXnxIXEZEO4Lx4vZGZrQOubGLTd919dbzyiGVm9wD3AFx99dWJSEFERFpR3Iqau9/cgsMqgatiXvcOYjQT/zPQ1czOC+7WYvdvKqclwBKAvLw8b0F+IiISImFvfiwB7jSzVDPrA6QBm4A3gLSgp+MFRDqTlLi7A68BfxscfxeQkLtAERGJv1AUNTMrNLPdwDDgJTNbC+Du7wLPAtuA/wRmuPuJ4C7sPmAt8B7wbLAvwD8C/8fMthMQ5D1eAAAP50lEQVT5ju3f4ns1IiKSKBa5uZG8vDwvLS1NdBoiIu2Gmb3p7nmn3zN+QnGnJiJ/UVZWxpo1a6Kv582bxyOPPNLi88UeP3PmTPr378/AgQMpLCzk4MGD55yvSJioqImEzKlFrTWNGTOGrVu3smXLFvr168cPfvCDNnkfkURRURNpAxUVFfTv35+pU6fSr18/pkyZwrp167j++utJS0tj06ZN1NTUcPfdd5Ofn09OTg6rV6/m2LFjzJkzhxUrVpCdnc2KFSsA2LZtGyNHjqRv374sXrw4+j6LFi0iMzOTzMxMHnvssWh8/vz59OvXjxtuuIHy8r+Ma3DLLbdw3nmRTs/XXXcdu3fvjtMnIhIn7q7FncGDB7tIa9m5c6enpKT4li1b/MSJE56bm+vTpk3zhoYGLy4u9okTJ/rs2bP96aefdnf3AwcOeFpamh85csSXLVvmM2bMiJ5r7ty5PmzYMK+rq/N9+/Z5t27d/NixY15aWuqZmZl+5MgRr66u9gEDBvhbb70VjdfU1PihQ4f82muv9YULF34qx9tuuy36/iItAZR6CP79jl3i9pyaSEfTp08fsrKyAMjIyGD06NGYGVlZWVRUVLB7925KSkqi33fV1dWxa9euJs81fvx4UlNTSU1NpXv37lRVVbFhwwYKCwu56KKLAJg8eTLr16+noaGBwsJCLrzwQgAmTJjwqfPNnz+f8847jylTprTFpYskjIqaSCsp3lzJwrXlfHywlm5+iE88JbqtU6dOpKamRtfr6+tJSUlh5cqVpKenn3SejRs3furcjccCpKSkUF9f/6l9ztTPf/5zXnzxRV599VUiw6WKJA99pybSCoo3VzL7+XeoPFiLA1WH66g6XEfx5mYHtKGgoIDHH38cDx6r2bx5MwBdunShurr6tO85fPhwiouLOXr0KDU1NaxatYrhw4czYsQIiouLqa2tpbq6mhdeeCF6zH/+53/ywx/+kJKSkuidnEgyUVETaQUL15ZTe/zESTF3Z+HapiafiCgqKuL48eMMHDiQjIwMioqKABg1ahTbtm07qaNIU3Jzc5k6dSr5+fkMHTqU6dOnk5OTQ25uLnfccQeDBg1i7NixDBkyJHrMfffdR3V1NWPGjCE7O5uvf/3r53jlIuGih68DevhazkWfWS/R1N8kA3YuGB/vdETiQg9fiySpnl07n1VcRNqGippIK5hZkE7n81NOinU+P4WZBenNHCEibUG9H0VawaScyFy0jb0fe3btzMyC9GhcROJDRU2klUzK6aUiJpJgan4UEZGkoaImIiJJQ0VNRESShoqaiIgkDRU1ERFJGqEoamZ2u5m9a2YNZpYXE7/GzGrNrCxYfhqzbbCZvWNm281ssQUjs5pZNzN7xcw+CH5emohrEhGR+AtFUQO2ApOB3zex7UN3zw6W2IHqngS+BqQFy61BfBbwqrunAa8Gr0VEpAMIRVFz9/fcvfmRX09hZj2AS9z99WCiul8Ak4LNE4HlwfrymLiIiCS5UBS10+hjZpvN7HdmNjyI9QJi56HfHcQArnD3PcH6H4Er4pSniIgkWNxGFDGzdcCVTWz6rruvbuawPcDV7v5nMxsMFJtZxpm+p7u7mTU7DYGZ3QPcA3D11Vef6WlFRCSk4lbU3P3mFhzzCfBJsP6mmX0I9AMqgd4xu/YOYgBVZtbD3fcEzZR7P+P8S4AlEJl65mzzExGRcAl186OZXW5mKcF6XyIdQnYEzYuHzey6oNfjV4DGu70S4K5g/a6YuIiIJLlQFDUzKzSz3cAw4CUzWxtsGgFsMbMy4N+Br7v7/mDbPwD/CmwHPgT+I4gvAMaY2QfAzcFrERHpADTzdUAzX4uInB3NfC0iItKGVNRERCRpqKiJiEjSUFETOUtlZWWsWbMm+nrevHk88sgjLT5f7PHz5s2jV69eZGdnk52dfdL7iMjpxe05NZFkUVZWRmlpKePGjWuT83/rW9/igQceaJNziyQ73alJh1RRUUH//v2ZOnUq/fr1Y8qUKaxbt47rr7+etLQ0Nm3aRE1NDXfffTf5+fnk5OSwevVqjh07xpw5c1ixYgXZ2dmsWLECgG3btjFy5Ej69u3L4sWLo++zaNEiMjMzyczM5LHHHovG58+fT79+/bjhhhsoLz/jYU9F5HTcXYs7gwcPduk4du7c6SkpKb5lyxY/ceKE5+bm+rRp07yhocGLi4t94sSJPnv2bH/66afd3f3AgQOelpbmR44c8WXLlvmMGTOi55o7d64PGzbM6+rqfN++fd6tWzc/duyYl5aWemZmph85csSrq6t9wIAB/tZbb0XjNTU1fujQIb/22mt94cKF0XN97nOf86ysLJ82bZrv378/IZ+PyJkASj0E/37HLrpTkw6rT58+ZGVl0alTJzIyMhg9ejRmRlZWFhUVFbz88sssWLCA7OxsRo4cSV1dHbt27WryXOPHjyc1NZXLLruM7t27U1VVxYYNGygsLOSiiy7i4osvZvLkyaxfv57169dTWFjIhRdeyCWXXMKECROi57n33nv58MMPKSsro0ePHnz729+O18chkhT0nZp0GMWbK1m4tpyPD9bSzQ/xiadEt3Xq1InU1NToen19PSkpKaxcuZL09PSTzrNx48ZPnbvxWICUlBTq6+tblOMVV/xlUomvfe1r3HbbbS06j0hHpTs16RCKN1cy+/l3qDxYiwNVh+uoOlxH8ebKZo8pKCjg8ccfx4NRdzZv3gxAly5dqK6uPu17Dh8+nOLiYo4ePUpNTQ2rVq1i+PDhjBgxguLiYmpra6muruaFF16IHrNnz57o+qpVq8jMzGzhFYt0TLpTkw5h4dpyao+fOCnm7ixcW86knF5NHlNUVMT999/PwIEDaWhooE+fPrz44ouMGjUq2iw5e/bsZt8zNzeXqVOnkp+fD8D06dPJyckB4I477mDQoEF0796dIUOGRI958MEHKSsrw8y45ppr+NnPfnauly7SoWjsx4DGfkxufWa9RFO/6QbsXDA+3umIJAWN/SiSID27dj6ruIi0Typq0iHMLEin8/kpJ8U6n5/CzIL0Zo4QkfZI36lJh9D4vVlj78eeXTszsyC92e/TRKR9UlGTDmNSTi8VMZEkp+ZHERFJGqEoama20MzeN7MtZrbKzLrGbJttZtvNrNzMCmLitwax7WY2Kybex8w2BvEVZnZBvK9HREQSIxRFDXgFyHT3gcAfgNkAZjYAuBPIAG4F/sXMUswsBfgJMBYYAHw52Bfgn4EfufvngQPAV+N6JSIikjChKGru/rK7N44r9DrQO1ifCDzj7p+4+05gO5AfLNvdfYe7HwOeASaamQE3Af8eHL8cmBSv6xARkcQKRVE7xd3AfwTrvYCPYrbtDmLNxf8aOBhTIBvjIiLSAcSt96OZrQOubGLTd919dbDPd4F64Jdxyuke4J7g5REzC8vEVpcBf0p0EmdB+bYt5du22lu+EJ6cP5foBE4Vt6Lm7jd/1nYzmwrcBoz2v4zdVQlcFbNb7yBGM/E/A13N7Lzgbi12/6ZyWgIsOYvLiAszKw3b0DOfRfm2LeXbttpbvtA+c46XUDQ/mtmtwIPABHc/GrOpBLjTzFLNrA+QBmwC3gDSgp6OFxDpTFISFMPXgL8Njr8LWB2v6xARkcQKy8PXTwCpwCuRvh687u5fd/d3zexZYBuRZskZ7n4CwMzuA9YCKcBSd383ONc/As+Y2feBzcC/xfdSREQkUUJR1ILu981tmw/MbyK+BljTRHwHkd6R7VnomkRPQ/m2LeXbttpbvtA+c44LTT0jIiJJIxTfqYmIiLQGFbU4M7PbzexdM2sws7xTtoV+SDAzm2dmlWZWFizjWpp/IoQpl1hmVmFm7wSfaWkQ62Zmr5jZB8HPS4O4mdni4Bq2mFluHPJbamZ7zWxrTOys8zOzu4L9PzCzu+Kcb2h/d83sKjN7zcy2Bf8+/H9BPLSfcWi5u5Y4LsAXgHTgt0BeTHwA8DaRDjN9gA+JdIJJCdb7AhcE+wwIjnkWuDNY/ylwbxzynwc80ET8rPNPwGcfmlyayK0CuOyU2A+BWcH6LOCfg/VxRAYoMOA6YGMc8hsB5AJbW5of0A3YEfy8NFi/NI75hvZ3F+gB5AbrXYgMFzggzJ9xWBfdqcWZu7/n7k095N3ehwQ7q/wTlGOYcjkTE4n8ucLJf74TgV94xOtEns3s0ZaJuPvvgf3nmF8B8Iq773f3A0TGfL01jvk2J+G/u+6+x93fCtargfeIjIYU2s84rFTUwqM9DQl2X9DksbSxOeQz8mwunghhyuVUDrxsZm9aZKQbgCvcfU+w/kfgimA9LNdxtvmFIe/Q/+6a2TVADrCR9vkZJ5SKWhsws3VmtrWJJcx3BVGnyf9J4FogG9gDPJrQZJPHDe6eS2TmiRlmNiJ2o0falkLbVTns+QVC/7trZhcDK4H73f1w7LZ28hknXCieU0s2fpohwZrRpkOCnY0zzd/MngJeDF6ebf6J8Fk5JpS7VwY/95rZKiJNX1Vm1sPd9wRNS3uD3cNyHWebXyUw8pT4b+OQJwDuXtW4HsbfXTM7n0hB+6W7Px+E29VnHAa6UwuPdjEk2Cnf3RQCjb3Lzir/ts6zGWHKJcrMLjKzLo3rwC1EPtcSIn+ucPKfbwnwlaAH3HXAoZgmqng62/zWAreY2aVB098tQSwuwvy7G3xH/m/Ae+6+KGZTu/qMQyHRPVU62kLkL9Nu4BOgClgbs+27RHpblQNjY+LjiPSG+pDIrAaN8b5E/vJtB54DUuOQ/9PAO8AWIn+xerQ0/wR9/qHJ5ZQ/x7eD5d3GvIh8b/oq8AGwDugWxI3IJLkfBn8WeXHI8ddEmuyOB7+/X21JfkSmltoeLNPinG9of3eBG4g0LW4ByoJlXJg/47AuGlFERESShpofRUQkaaioiYhI0lBRExGRpKGiJiIiSUNFTUREkoaKmoiIJA0VNRERSRoqaiIhYGZ/b2Z/DOb52mFmU2O23W6RefPKgrm25iYwVZFQU1ETCYcsYJ67ZxMZ+uxRiEz4CPwj8MVg2xDOfEoVkQ5HI4qIhICZ/R6Y4+6/NbPuRIZmuhrYCQxx9x0JTVCkndCdmkg4ZAHvBQPbfpPICPKTiMxorIImcoZU1EQSzMyuAi4mMpr6JuBSYAaQSWRg26aOucjMlpvZU2Y2JW7JioSc5lMTSbws4FV3vzU2aGY1QOdmjpkM/Lu7v2BmK4BftnGOIu2C7tREEm8gkWlnTvUfwO1mdgVAMN/X14JtvYGPgvUTbZ+iSPugoiaSeFlE5tE6ibtvAuYBa82scZ6t7sHm3UQKG+jvsUiUej+KtEPBDNlPAHXABndX86MIKmoiIpJE1GwhIiJJQ0VNRESShoqaiIgkDRU1ERFJGipqIiKSNFTUREQkaaioiYhI0lBRExGRpKGiJiIiSeP/B3uk1JifgwH+AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# create a dummy join ranking of 10 methods and 100 edges\n", "rankings = pd.DataFrame(np.random.randint(0,1000,size=(100,10)))\n", "rankings.columns = [\"method\"+str(i) for i in rankings.columns]\n", "rankings.index = [\"edge\"+str(i) for i in rankings.index]\n", "print(\"Dummy ranking:\")\n", "print(rankings.head())\n", "# transpose the ranking matrix\n", "rankingsT = rankings.T\n", "# import PCA\n", "from sklearn.decomposition import PCA\n", "# create PCA object with k principal axis\n", "k = 2\n", "pca = PCA(n_components=k)\n", "# Apply the PCA\n", "rankingsT_pca = pca.fit_transform(rankingsT)\n", "# Explained variance ratio for each principal axis\n", "print(\"\\nExplained variance ratio for each PC\")\n", "print(pca.explained_variance_ratio_)\n", "# plot the coordinates of each method along the Principal Axis\n", "plt.plot(rankingsT_pca[:,0],rankingsT_pca[:,1],\"o\")\n", "plt.xlabel(\"$PC_0$\")\n", "plt.ylabel(\"$PC_1$\")\n", "for i,method in enumerate(rankingsT.index):\n", " plt.text(x=rankingsT_pca[i,0], y=rankingsT_pca[i,1], s=method)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Answer the following questions:\n", "+ How different are the distinct methods between them?\n", "+ Which methods are more similar between them?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Assess the quality of the inferred GRNs\n", "In this section we are going to use the following quality measures to evaluate the inferred GRNs with respect to the \"High Confidence\" GRN:\n", "\n", "+ [Precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html)\n", "+ [Recall](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html)\n", "+ [Accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)\n", "\n", "These measures are defined using the __confusion matrix__\n", "$$ConfMat=\\begin{pmatrix}& T_{real} & F_{real}\\\\T_{pred}& TP & FP \\\\F_{pred} & FN & TN \\end{pmatrix}$$\n", "\n", "__Where__:\n", "+ $TP$: True positive (correctly predicted as True)\n", "+ $FP$: False positive (wrongly predicted as True)\n", "+ $FN$: False negative (wrongly predicted as False)\n", "+ $TN$: True negative (correctly predicted as False)\n", "\n", "__Metrics__:\n", "+ $Recall = \\frac{TP}{TP + FN}$\n", "+ $Precision = \\frac{TP}{TP + FP}$\n", "+ $Accuracy = \\frac{TP+TN}{TP+TN+FP+FN}$\n", "\n", "__Bonus__:\n", "You can also use the follwing evaluation criteria:\n", "+ [ROC curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html)\n", "+ [AUC curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score)\n", "+ [Average Precision Score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ Select the to $k = |E_{HC}|$ best edges for each method\n", "+ Evaluate the different results with respect to the \"High Confidence\" GRN using the previous measures\n", "+ Which are the best methods, w.r.t. this dataset? Explain" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Build a meta learner\n", "In [Marbach et al. 2012](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3512113/), the authors have suggest that \"no single inference method performs optimally across all datasets. In contrast, integration of predictions from multiple inference methods shows robust and high performance across diverse datasets\"\n", "\n", "+ Compute a new GRN by averaging the ranks of the different methods. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ Compute the similarity of this new GRN w.r.t. the previous ones" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ Compute the quality of this new GRN" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "+ Let $X^{rank}$ be a matrix with $E$ rows (one for each edge) and $M$ columns (one for each method). The value $X^{rank}_{i,j}$ is the rank that method $j$ has attributed to edge $i$. \n", "+ Let $y$ be a binary label vector with $E$ rows (one for each edge), s.t. $y_e=1$ if $e \\in E_{HC}$ and $y_e=0$ otherwise\n", "+ Train a Random Forest classifier to predict $y$, from the values $X_{rank}$\n", "+ Extract the feature importance from this classifier to infer which are the better methods\n", "\n", "__Check the following example:__" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Dummy ranking:\n", " method0 method1 method2 method3 method4 method5 method6 method7 \\\n", "edge0 7 320 25 672 333 413 487 657 \n", "edge1 86 444 726 745 502 224 11 610 \n", "edge2 569 849 788 281 921 207 329 700 \n", "edge3 190 531 193 827 744 45 160 813 \n", "edge4 712 344 347 893 817 409 785 823 \n", "\n", " method8 method9 \n", "edge0 953 692 \n", "edge1 847 820 \n", "edge2 57 614 \n", "edge3 648 480 \n", "edge4 899 420 \n", "\n", "Dummy labels:\n", " labels\n", "edge0 1\n", "edge1 1\n", "edge2 0\n", "edge3 0\n", "edge4 1\n" ] }, { "data": { "text/plain": [ "array([0.08026446, 0.10601871, 0.08706498, 0.09726857, 0.1003492 ,\n", " 0.09425878, 0.15110857, 0.0748779 , 0.08805547, 0.12073336])" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# create a dummy join ranking of 10 methods and 100 edges\n", "rankings = pd.DataFrame(np.random.randint(0,1000,size=(100,10)))\n", "rankings.columns = [\"method\"+str(i) for i in rankings.columns]\n", "rankings.index = [\"edge\"+str(i) for i in rankings.index]\n", "print(\"Dummy ranking:\")\n", "print(rankings.head())\n", "# create dummy $y$ vector (binary labels)\n", "y = pd.DataFrame()\n", "y[\"labels\"] = np.random.binomial(p=0.5,n=1,size=100)\n", "y.index = rankings.index\n", "print(\"\\nDummy labels:\")\n", "print(y.head())\n", "# Train the RF classifier\n", "from sklearn.ensemble import RandomForestClassifier\n", "rf = RandomForestClassifier(n_estimators=100)\n", "rf.fit(rankings,y[\"labels\"])\n", "rf.feature_importances_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "__Question__: Since this method belongs to a supervised learning cathegory, how could you test its performance?\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" } }, "nbformat": 4, "nbformat_minor": 2 }