[ML] Add a constant to the prediction which minimises the unregularised loss for classification and regression#1192
Merged
tveasey merged 3 commits intoelastic:masterfrom May 5, 2020
Conversation
valeriy42
approved these changes
May 5, 2020
Contributor
valeriy42
left a comment
valeriy42
left a comment
There was a problem hiding this comment.
LGTM. A good idea of estimating a constant prior to fitting a non-linear function.
tveasey
added a commit
to tveasey/ml-cpp-1
that referenced
this pull request
May 5, 2020
…ed loss for classification and regression (elastic#1192)
tveasey
added a commit
to tveasey/ml-cpp-1
that referenced
this pull request
May 5, 2020
…ed loss for classification and regression (elastic#1192)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We add on a constant weight to "centre" the data. (Strictly speaking this isn't centring the data in the conventional sense, it is finding a single weight which when added to the ensemble prediction minimises the loss.)
Currently, we choose to minimise regularised loss with this weight, i.e. respecting the weight shrinkage. First, there is no need to do this, shrinkage is used to impose a "smoothness" bias, but a constant function is flat. Second, the subsequent trees spend effort updating the mean predictions to be unbiased and means there is an unfortunate interplay between the degree of smoothing we can use (since it will create bias in the unregularised loss) and the centre of the data.