From ed8711a2557703548d77d70ca519b1bc1304fb3e Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Wed, 14 Feb 2018 23:47:32 +0530 Subject: [PATCH 1/8] added overview for AdaBoost --- learning.ipynb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/learning.ipynb b/learning.ipynb index 16bb4bd6b..89aba0c0c 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -1778,6 +1778,35 @@ "source": [ "The Perceptron didn't fare very well mainly because the dataset is not linearly separated. On simpler datasets the algorithm performs much better, but unfortunately such datasets are rare in real life scenarios." ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## AdaBoost\n", + "\n", + "### Overview\n", + "\n", + "**AdaBoost** is an algorithm which uses **ensemble learning**. In ensemble learning the hypotheses in the collection, or ensemble, vote for what the output should be and the output with the majority votes is selected as the final answer.\n", + "\n", + "AdaBoost algorithm, as mentioned in the book, works with a **weighted training set** and **weak learners** (classifiers that have about 50%+epsilon accuracy i.e slightly better than random guessing). It manipulates the weights attached to the the examples that are showed to it. Importance is given to the examples with higher weights.\n", + "\n", + "All the examples start with equal weights and a hypothesis is generated using these examples. \n", + "Examples which are incorrectly classified, their weights are increased so that they can be classified correctly by the next hypothesis. The examples that are correctly classified, their weights are reduced. This process is repeated *k* times (here *k* is an input to the algorithm) and hence, *k* hypotheses are generated.\n", + "\n", + "These *k* hypotheses are also assigned weights according to their performance on the weighted training set. The final ensemble hypothesis is the weighted-majority combination of these *k* hypotheses.\n", + "\n", + "The especiality of AdaBoost is that by using weak learners and sufficiently large *k*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [] } ], "metadata": { From c9596b045830c1b0ab11acf84223e706d7bd93a5 Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Wed, 14 Feb 2018 23:49:30 +0530 Subject: [PATCH 2/8] added implementation for AdaBoost --- learning.ipynb | 172 ++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 171 insertions(+), 1 deletion(-) diff --git a/learning.ipynb b/learning.ipynb index 89aba0c0c..118eb3875 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -11,7 +11,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "metadata": { "collapsed": true }, @@ -1799,6 +1799,176 @@ "The especiality of AdaBoost is that by using weak learners and sufficiently large *k*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Implementation\n", + "\n", + "As seen in the previous section, the perceptron doesnot perform that well on the iris dataset as it is a linear classifier. We'll use perceptron as the learner in AdaBoost algorithm and try to increase the accuracy. \n", + "\n", + "Let's first see what the AdaBoost algorithm is exactly:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + "\n", + " \n", + " \n", + " \n", + "\n", + "\n", + "

\n", + "\n", + "
def AdaBoost(L, K):\n",
+       "    """[Figure 18.34]"""\n",
+       "    def train(dataset):\n",
+       "        examples, target = dataset.examples, dataset.target\n",
+       "        N = len(examples)\n",
+       "        epsilon = 1. / (2 * N)\n",
+       "        w = [1. / N] * N\n",
+       "        h, z = [], []\n",
+       "        for k in range(K):\n",
+       "            h_k = L(dataset, w)\n",
+       "            h.append(h_k)\n",
+       "            error = sum(weight for example, weight in zip(examples, w)\n",
+       "                        if example[target] != h_k(example))\n",
+       "            # Avoid divide-by-0 from either 0% or 100% error rates:\n",
+       "            error = clip(error, epsilon, 1 - epsilon)\n",
+       "            for j, example in enumerate(examples):\n",
+       "                if example[target] == h_k(example):\n",
+       "                    w[j] *= error / (1. - error)\n",
+       "            w = normalize(w)\n",
+       "            z.append(math.log((1. - error) / error))\n",
+       "        return WeightedMajority(h, z)\n",
+       "    return train\n",
+       "
\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "psource(AdaBoost)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here we can see that the learner **L** takes in as inputs: a dataset and the weights associated with the examples. But the `PerceptronLearner` doesnot handle weights and only takes a dataset as its input. \n", + "To remedy that we will give as input to the PerceptronLearner a modified dataset in which the examples will be repeated according to the weights associated to them. Intuitively what this will do is force the learner to repeatedly learn the same example again and again until it can classify it correctly. \n", + "We would have to pass the `PerceptronLearner` through the **`WeightedLearner`** function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "psource(WeightedLearner)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `WeightedLearner` function will then call the `PerceptronLearner`, in each iteration, with the modified dataset which contains the examples according to the weights associated with them." + ] + }, { "cell_type": "code", "execution_count": null, From 370e2e5b334dbacc57aadecb19a2c9061b845cba Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Wed, 14 Feb 2018 23:51:35 +0530 Subject: [PATCH 3/8] added example for AdaBoost --- learning.ipynb | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/learning.ipynb b/learning.ipynb index 118eb3875..12fcab33d 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -1969,6 +1969,76 @@ "The `WeightedLearner` function will then call the `PerceptronLearner`, in each iteration, with the modified dataset which contains the examples according to the weights associated with them." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example\n", + "\n", + "First we will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *k* equals to 5." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "WeightedPerceptron = WeightedLearner(PerceptronLearner)\n", + "AdaboostLearner = AdaBoost(WeightedPerceptron, 5)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "iris2 = DataSet(name=\"iris\")\n", + "iris2.classes_to_numbers()\n", + "\n", + "adaboost = AdaboostLearner(iris2)\n", + "\n", + "adaboost([5, 3, 1, 0.1])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Yet again it gave the correct answer. Let's check the error rate of adaboost with perceptron " + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Error ratio for adaboost: 0.046666666666666634\n" + ] + } + ], + "source": [ + "print(\"Error ratio for adaboost: \", err_ratio(adaboost, iris2))" + ] + }, { "cell_type": "code", "execution_count": null, From 6b1313a9936b7c5b61933b4f34d2363599885556 Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Thu, 15 Feb 2018 00:48:15 +0530 Subject: [PATCH 4/8] added tests for AdaBoost --- learning.ipynb | 8 ++++---- tests/test_learning.py | 15 +++++++++++++++ 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/learning.ipynb b/learning.ipynb index 12fcab33d..985b8e9c1 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -2040,13 +2040,13 @@ ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": { "collapsed": true }, - "outputs": [], - "source": [] + "source": [ + "It reduced the error rate considerably. Unlike the `PerceptronLearner`, `AdaBoost` was able to learn the complexity in the iris dataset." + ] } ], "metadata": { diff --git a/tests/test_learning.py b/tests/test_learning.py index 8a21d6462..3f9312c71 100644 --- a/tests/test_learning.py +++ b/tests/test_learning.py @@ -218,3 +218,18 @@ def test_random_weights(): assert len(test_weights) == num_weights for weight in test_weights: assert weight >= min_value and weight <= max_value + + +def test_adaboost(): + iris = DataSet(name="iris") + iris.classes_to_numbers() + WeightedPerceptron = WeightedLearner(PerceptronLearner) + AdaboostLearner = AdaBoost(WeightedPerceptron, 5) + adaboost = AdaboostLearner(iris) + assert adaboost([5, 3, 1, 0.1]) == 0 + assert adaboost([5, 3.5, 1, 0]) == 0 + assert adaboost([6, 3, 4, 1.1]) == 1 + assert adaboost([6, 2, 3.5, 1]) == 1 + assert adaboost([7.5, 4, 6, 2]) == 2 + assert adaboost([7, 3, 6, 2.5]) == 2 + assert err_ratio(adaboost, iris) < 0.05 From 85c639bdde44929478d75bcd2bcfa949db46770b Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Thu, 15 Feb 2018 01:03:36 +0530 Subject: [PATCH 5/8] rephrased sentences --- learning.ipynb | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/learning.ipynb b/learning.ipynb index 985b8e9c1..7917de9f9 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -1791,10 +1791,9 @@ "\n", "AdaBoost algorithm, as mentioned in the book, works with a **weighted training set** and **weak learners** (classifiers that have about 50%+epsilon accuracy i.e slightly better than random guessing). It manipulates the weights attached to the the examples that are showed to it. Importance is given to the examples with higher weights.\n", "\n", - "All the examples start with equal weights and a hypothesis is generated using these examples. \n", - "Examples which are incorrectly classified, their weights are increased so that they can be classified correctly by the next hypothesis. The examples that are correctly classified, their weights are reduced. This process is repeated *k* times (here *k* is an input to the algorithm) and hence, *k* hypotheses are generated.\n", + "All the examples start with equal weights and a hypothesis is generated using these examples. Examples which are incorrectly classified, their weights are increased so that they can be classified correctly by the next hypothesis. The examples that are correctly classified, their weights are reduced. This process is repeated *K* times (here *K* is an input to the algorithm) and hence, *K* hypotheses are generated.\n", "\n", - "These *k* hypotheses are also assigned weights according to their performance on the weighted training set. The final ensemble hypothesis is the weighted-majority combination of these *k* hypotheses.\n", + "These *k* hypotheses are also assigned weights according to their performance on the weighted training set. The final ensemble hypothesis is the weighted-majority combination of these *K* hypotheses.\n", "\n", "The especiality of AdaBoost is that by using weak learners and sufficiently large *k*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." ] @@ -1805,9 +1804,9 @@ "source": [ "### Implementation\n", "\n", - "As seen in the previous section, the perceptron doesnot perform that well on the iris dataset as it is a linear classifier. We'll use perceptron as the learner in AdaBoost algorithm and try to increase the accuracy. \n", + "As seen in the previous section, the `PerceptronLearner` doesnot perform that well on the iris dataset. We'll use perceptron as the learner for the AdaBoost algorithm and try to increase the accuracy. \n", "\n", - "Let's first see what the AdaBoost algorithm is exactly:" + "Let's first see what AdaBoost is exactly:" ] }, { @@ -1946,9 +1945,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Here we can see that the learner **L** takes in as inputs: a dataset and the weights associated with the examples. But the `PerceptronLearner` doesnot handle weights and only takes a dataset as its input. \n", - "To remedy that we will give as input to the PerceptronLearner a modified dataset in which the examples will be repeated according to the weights associated to them. Intuitively what this will do is force the learner to repeatedly learn the same example again and again until it can classify it correctly. \n", - "We would have to pass the `PerceptronLearner` through the **`WeightedLearner`** function." + "AdaBoost takes as inputs: **L** and *K* where **L** is the learner and *K* is the number of hypotheses to be generated. The learner **L** takes in as inputs: a dataset and the weights associated with the examples. But the `PerceptronLearner` doesnot handle weights and only takes a dataset as its input. \n", + "To remedy that we will give as input to the PerceptronLearner a modified dataset in which the examples will be repeated according to the weights associated to them. Intuitively what this will do is force the learner to repeatedly learn the same example again and again until it can classify it correctly. \n", + "TO convert `PerceptronLearner` so that it can take weights as input too, we will have to pass it through the **`WeightedLearner`** function." ] }, { @@ -1975,7 +1974,7 @@ "source": [ "### Example\n", "\n", - "First we will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *k* equals to 5." + "We will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *k* equals to 5." ] }, { From ca87d152f0d7ac711a491f1215d31262f5c86049 Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Thu, 15 Feb 2018 01:11:29 +0530 Subject: [PATCH 6/8] final changes to AdaBoost --- learning.ipynb | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/learning.ipynb b/learning.ipynb index 7917de9f9..6bf527eb0 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -1795,7 +1795,7 @@ "\n", "These *k* hypotheses are also assigned weights according to their performance on the weighted training set. The final ensemble hypothesis is the weighted-majority combination of these *K* hypotheses.\n", "\n", - "The especiality of AdaBoost is that by using weak learners and sufficiently large *k*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." + "The speciality of AdaBoost is that by using weak learners and a sufficiently large *k*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." ] }, { @@ -1945,9 +1945,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "AdaBoost takes as inputs: **L** and *K* where **L** is the learner and *K* is the number of hypotheses to be generated. The learner **L** takes in as inputs: a dataset and the weights associated with the examples. But the `PerceptronLearner` doesnot handle weights and only takes a dataset as its input. \n", + "AdaBoost takes as inputs: **L** and *K* where **L** is the learner and *K* is the number of hypotheses to be generated. The learner **L** takes in as inputs: a dataset and the weights associated with the examples in the dataset. But the `PerceptronLearner` doesnot handle weights and only takes a dataset as its input. \n", "To remedy that we will give as input to the PerceptronLearner a modified dataset in which the examples will be repeated according to the weights associated to them. Intuitively what this will do is force the learner to repeatedly learn the same example again and again until it can classify it correctly. \n", - "TO convert `PerceptronLearner` so that it can take weights as input too, we will have to pass it through the **`WeightedLearner`** function." + "\n", + "To convert `PerceptronLearner` so that it can take weights as input too, we will have to pass it through the **`WeightedLearner`** function." ] }, { @@ -1965,7 +1966,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The `WeightedLearner` function will then call the `PerceptronLearner`, in each iteration, with the modified dataset which contains the examples according to the weights associated with them." + "The `WeightedLearner` function will then call the `PerceptronLearner`, during each iteration, with the modified dataset which contains the examples according to the weights associated with them." ] }, { @@ -2018,7 +2019,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Yet again it gave the correct answer. Let's check the error rate of adaboost with perceptron " + "Yet again it gave the correct answer! Let's check the error rate of adaboost with perceptron." ] }, { From ba79f82f02e5c201cf8776fa79b264d664e0b661 Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Thu, 15 Feb 2018 10:49:10 +0530 Subject: [PATCH 7/8] changed adaboost tests to use grade_learner --- learning.ipynb | 6 +++--- tests/test_learning.py | 15 ++++++++------- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/learning.ipynb b/learning.ipynb index 6bf527eb0..35e015b28 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -1793,9 +1793,9 @@ "\n", "All the examples start with equal weights and a hypothesis is generated using these examples. Examples which are incorrectly classified, their weights are increased so that they can be classified correctly by the next hypothesis. The examples that are correctly classified, their weights are reduced. This process is repeated *K* times (here *K* is an input to the algorithm) and hence, *K* hypotheses are generated.\n", "\n", - "These *k* hypotheses are also assigned weights according to their performance on the weighted training set. The final ensemble hypothesis is the weighted-majority combination of these *K* hypotheses.\n", + "These *K* hypotheses are also assigned weights according to their performance on the weighted training set. The final ensemble hypothesis is the weighted-majority combination of these *K* hypotheses.\n", "\n", - "The speciality of AdaBoost is that by using weak learners and a sufficiently large *k*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." + "The speciality of AdaBoost is that by using weak learners and a sufficiently large *K*, a highly accurate classifier can be learned irrespective of the complexity of the function being learned or the dullness of the hypothesis space." ] }, { @@ -1975,7 +1975,7 @@ "source": [ "### Example\n", "\n", - "We will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *k* equals to 5." + "We will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *K* equals to 5." ] }, { diff --git a/tests/test_learning.py b/tests/test_learning.py index 3f9312c71..3c6d02d28 100644 --- a/tests/test_learning.py +++ b/tests/test_learning.py @@ -226,10 +226,11 @@ def test_adaboost(): WeightedPerceptron = WeightedLearner(PerceptronLearner) AdaboostLearner = AdaBoost(WeightedPerceptron, 5) adaboost = AdaboostLearner(iris) - assert adaboost([5, 3, 1, 0.1]) == 0 - assert adaboost([5, 3.5, 1, 0]) == 0 - assert adaboost([6, 3, 4, 1.1]) == 1 - assert adaboost([6, 2, 3.5, 1]) == 1 - assert adaboost([7.5, 4, 6, 2]) == 2 - assert adaboost([7, 3, 6, 2.5]) == 2 - assert err_ratio(adaboost, iris) < 0.05 + tests = [([5, 3, 1, 0.1], 0), + ([5, 3.5, 1, 0], 0), + ([6, 3, 4, 1.1], 1), + ([6, 2, 3.5, 1], 1), + ([7.5, 4, 6, 2], 2), + ([7, 3, 6, 2.5], 2)] + assert grade_learner(adaboost, tests) > 5/6 + assert err_ratio(adaboost, iris) < 0.1 From 000c5b88c6a436f9d300a24936b348d1f2b1ea80 Mon Sep 17 00:00:00 2001 From: aswanipranjal Date: Thu, 15 Feb 2018 11:11:29 +0530 Subject: [PATCH 8/8] grammar check --- learning.ipynb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/learning.ipynb b/learning.ipynb index 35e015b28..0e4d97934 100644 --- a/learning.ipynb +++ b/learning.ipynb @@ -1804,7 +1804,7 @@ "source": [ "### Implementation\n", "\n", - "As seen in the previous section, the `PerceptronLearner` doesnot perform that well on the iris dataset. We'll use perceptron as the learner for the AdaBoost algorithm and try to increase the accuracy. \n", + "As seen in the previous section, the `PerceptronLearner` does not perform that well on the iris dataset. We'll use perceptron as the learner for the AdaBoost algorithm and try to increase the accuracy. \n", "\n", "Let's first see what AdaBoost is exactly:" ] @@ -1946,7 +1946,7 @@ "metadata": {}, "source": [ "AdaBoost takes as inputs: **L** and *K* where **L** is the learner and *K* is the number of hypotheses to be generated. The learner **L** takes in as inputs: a dataset and the weights associated with the examples in the dataset. But the `PerceptronLearner` doesnot handle weights and only takes a dataset as its input. \n", - "To remedy that we will give as input to the PerceptronLearner a modified dataset in which the examples will be repeated according to the weights associated to them. Intuitively what this will do is force the learner to repeatedly learn the same example again and again until it can classify it correctly. \n", + "To remedy that we will give as input to the PerceptronLearner a modified dataset in which the examples will be repeated according to the weights associated to them. Intuitively, what this will do is force the learner to repeatedly learn the same example again and again until it can classify it correctly. \n", "\n", "To convert `PerceptronLearner` so that it can take weights as input too, we will have to pass it through the **`WeightedLearner`** function." ] @@ -1975,7 +1975,7 @@ "source": [ "### Example\n", "\n", - "We will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *K* equals to 5." + "We will pass the `PerceptronLearner` through `WeightedLearner` function. Then we will create an `AdaboostLearner` classifier with number of hypotheses or *K* equal to 5." ] }, { @@ -2019,7 +2019,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Yet again it gave the correct answer! Let's check the error rate of adaboost with perceptron." + "That is the correct answer. Let's check the error rate of adaboost with perceptron." ] }, {