Add --deterministic-edivisive variation#154
Conversation
Same as original algorithm but with deterministic Student T significance test. The motivation for this variation follows from fixing the bug explained at the top of #96 The intuition is that the split-merge approach introduced by Datastax is addressing the same problem that the _kappa_ variable does in the original paper. Now that we compute correctly over all values of kappa, the split-merge part should be unnecessary, as the original algorithm with kappa bug fixed, will find the same change points and more. Therefore the conclusion is we want to go back as much as possible to the original and real algorithm from the Matteson & James paper. But even then, we find that Student T as significance test is both much faster but also qualitatively produces better results for the use case we're in at least, that we want to continue using T test and not random permutations for the significance test. TBD: Whether weak change points are still helpful or not. By reading the problem they fix appears unrelated from the split-merge vs **kappa** symptoms. TODO: Incremental e-divisive is not supported in this mode. This was easy to implement on top of the split-merge variation. Not clear what is the correct way here. An easy solution is to rerun from the last change-point, but the problem is the last change point could itself be influenced by the new data appended.
|
@vishnuchalla and @StephanDollberg this intersects with both of your work, you're welcome to review. Stephan:
|
| index=cp_ttest.index, qhat=cp_ttest.qhat, | ||
| time=series.time[cp_ttest.index], metric=metric, stats=cp_ttest.stats | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Arguably this is too much copy paste from above and should be refactored into a separate method. The reason I don't do that is that this transformation from one ChangePoint to another is the more fundamental problem that we need to discuss and fix. I've opened #151 for that discussion. In the mean time I prefer to keep this code ugly and visible so we don't forget to fix it.Trying to make it look better via refactoring but not solving the fundamental issue is IMO counter productive.
| dest="split_edivisive", | ||
| help="use 'hunter' version of this algorithm, from 2023, featuring " | ||
| "split of data into smaller windows, weak change points and Student T test. (Default)", | ||
| ) |
There was a problem hiding this comment.
Could also have used the argparse choices type. Did this for backward compatibility.
Also one could argue that creating different variations like this is the wrong direction and we should instead just expose all of the sub-features as options the user can use to compose their own combination. My argument against this is that most users want one authoritative solution. And half of our users are not capable of understanding what the math is doing anyway, and the other half don't want to understand. (I'm myself in the latter group, if not the former, even :- )
In this commit deterministic_tigerbeetle_test.py is still identical to tigerbeetle_test.py, just modified to use the new compute_change_points_deterministic(). The tests will fail, as it finds more change points than the default split-merge algorithm. The next commit will modify the tests to pass.
The fact that results stay the same for a wide range of p-values and then suddenly a lot more are found, is an argument for weak change points.
Skipped change points are similar to weak change points, except that we don't need to do 2 passes over the data. We simply ignore candidates that have a p > max_pvalue. And continue searching until we find all points where p < max_pvalue. Note that each such true change point still causes the interval to split and a fresh iteration of the algorithm. The asserts in the tigerbeetle and deterministic_tigerbeetle tests provide an overview of how each variant behaves over different parameters.
Sounds good, I can give it a spin today or tomorrow. |
|
For reference, here's the tigerbeetle demo data set, with the hunter split-merge and weak change points. (What is the default in current Otava) Ok, so here's my analysis of 3 different variations against the tigerbeetle demo dataset: The deterministic variant finds much less change points for the same p-values. It is closer to original edivisive in behavior than to the hunter variant. To some extent this is a matter of perspective too: Who's to say we should compare them with constant p-values. Another way to look at it is that p<=0.01 for deterministic variant is closer to p<=0.001 in the hunter variation. The deterministic+skipped change points otoh is similar to the hunter variation, which of course was the intent all along. Even so, for same p-values, the hunter variation produces more change points than either of the deterministic one. Note that with p >= 0.1 they both find dozens of points, more than 10% of the total data sample, so those results are mostly noise and whether one alogrithm finds more noise than the other is perhaps not the most important aspect. Nevertheless, it is clear that Both of the new alternatives flag less change points than the hunter variant. Also worth noting that they find different change points, one is not a superset of the other. The deterministic variant produces the same result set at both extremes of p-values. For 0.00001 <= p <= 0.001 we get [15, 61, 71, 192, 260] or [15, 61, 71, 192, 212, 260]. Similarly for 0.01 <= p <= 0.05 the result set is the same but for one addition. This made me suspicious that I was observing the same phenomenon that lead to the introduction of weak change points in hunter. (Basically that the algorithm stops too early by selecting a change point candidate that has a high p-value, even if it would have found more change points with low p-values if it only continued. The skipped change points is an attempt at doing the same thing more directly inside the alogrithm, rather than doing 2 passes with two different p-values. For convenience, here's the ascii-art from the source code comments: Qualitatively then, in the tigerbeetle demo data set, the benchmark I'd really like an algorithm to find is the following:
When looking at it with the above goals:
Based on the above analysis I feel like deterministic without skipping change points is actually the winner. This rewards the intuition that as the new implementation of the algorithm fixes a bug that caused Otava to miss valid change points, then now is a good time to go back to the by-the-book implementation or at least very close to it. I will add another push after this which puts on top a variant with deterministic student t significant test, but no other changes to orig_edivisive. I will however intentionally leave the commit with the skipped variant too, so that you can either browse the results it produces (by looking at the asserts in deterministic_tigerbeetle.py) or you can try it on your own data set. |
|
Is this PR ready for review @henrikingo ? |
|
Yed it is! Requiring that #151 be fixed first is a possible opinion though. Given Denis' answer I'm likely to do that anyway. Note that I expect this to become our default before we get to 1..0 so you may want to test thoroughly with the real production data. |
|
I have briefly tried First of all, I could not make Second, I have tried also tried analyzing internal Kafka Streams benchmark results. Here are 3 different datasets, let's call them A, B, C. Data
Results (reproducible with otava-playground) A: B: C: 
As you can see, results for |
|
For the sake of discussion, I will assume that we are comparing the following methods:
@henrikingo mentioned hunter vs two deterministic methods, which seems to be a typo. Let me know if I'm mistaken. The items, I feel worth discussing / pointing out:
|
Focusing on the three you listed makes sense, but that is not what I did, and that explains the different results. I did not compare against --orig-edivisive at all. I mean in the text I did, for the purpose of describing high level what ---deterministic-edivisive feels like. But I didn't run any tests on --orig-edivisive. It is obvious to me orig-edivisive it's identical to this PR except for minor variations due to the randomness in the significance test. Your explanation of the theory is again admirably layman understandable. But I would add to that the layman's law of large numbers: in practice natural phenomena can be approximated as being normally distributed, even if the "approximately" sometimes is very rough. But it is of course good that Alex did compare also --orig-edivisive. (Note: With my history, for me to call a variant "orig" it would actually require to reintroduce the bug where Kappa stays constant. :-) I compared against hunter/default behavior and two variations that are both introduced in this PR:
The second variant is already reverted, but still available in this PR. You would have to check out 6228b91 if you want to test it. But I felt it was a wrong direction so wanted to kill it early. (Also not add unnecessary cost to eventual reviews of the code.) |
I thought about this, and we probably should do that, but I consider such granular configurability as "advanced options". For most users I would hope Otava at a given point in time to offer one default and strongly recommended option, and the other --nickname-edivisive options are historical (or in this case, possibly a future) defaults. Ideally also the p-value could be the single tunable anyone ever needs. I increasingly believe the min_threshold has mostly been valuable to workaround bugs, and is less and less needed each time we fix yet another. |
4 participants
Same as original algorithm but with deterministic Student T significance test.
The motivation for this variation follows from fixing the bug explained at the top of #96 The intuition is that the split-merge approach introduced by Datastax is addressing the same problem that the kappa variable does in the original paper. Now that we compute correctly over all values of kappa, the split-merge part should be unnecessary, as the original algorithm with kappa bug fixed, will find the same change points and more. Therefore the conclusion is we want to go back as much as possible to the original and real algorithm from the Matteson & James paper. But even then, we find that Student T as significance test is both much faster but also qualitatively produces better results for the use case we're in at least, that we want to continue using T test and not random permutations for the significance test.
TODO: Incremental e-divisive is not supported in this mode. This was easy to implement on top of the split-merge variation. Not clear what is the correct way here. An easy solution is to rerun from the last change-point, but the problem is the last change point could itself be influenced by the new data appended.