Prevent stackoverflow caused by recursive deconstruction#1436
Conversation
|
How is this PR related to #1431? |
|
Wow! I didn't notice there is already an open PR on this. Such an active community! Skimming through the code, the general idea is the same as this PR. Both use heap-allocated stack to unfold the JSON object. I haven't tested it yet though. How should we proceed now? |
|
@nlohmann Sorry for the long delay. While this PR and #1431 are in the same spirit, I believe this PR is more concise and easier to read. Also, it is accompanied by a basic unit test to prevent regression. Regarding the other PR, I don't quite understand the implementation thus I cannot verify its validity. Specifically, I am not sure why there is a |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
stale
Bot
added
the
state: stale
stale
Bot
removed
the
state: stale
|
@nlohmann Yes, this will probably fix #1835. I will check it with bad_json_parsers to make sure. Thanks for reminding me! |
|
I can confirm it passes bad_json_parsers without crashing on any depths (1 to 5,000,000) provided in their test. |
|
Travis CI stalled on one of the jobs (https://travis-ci.org/nlohmann/json/jobs/609598763?utm_medium=notification&utm_source=github_status). Restarting it might help. |
|
Once again, a timeout issue with CI :-). Travis CI terminates This is caused by creating a very deep JSON object in unit-test, that requires a lot of memory allocations hence imposing a heavy burden on valgrind to check for memory leaks. To mitigate this problem, I moved this test to a separate unit-test ( Meanwhile, breaking down |
|
Thanks for taking care about this. I would have taken a more complex approach and would have used the parser's callback mechanism to produce some output to tame Travis. But your approach is much cleaner. Thanks a lot! Sorry for not have merged this issue earlier in January. I have no idea why it slipped through. But I did remember it the moment I saw the stacktrace in #1835. |
🔖 Release itemThis issue/PR will be part of the next release of the library. This template helps preparing the release notes. Type
Description
|
|
No problem at all! I had forgotten about it too, thanks for reminding me :-)
That's interesting, I didn't know that is possible. BTW, there were more considerations on Travis build time which posted in #1836 but I doubt if that qualify for an issue. |
|
There seems to be a problem with the implementation: Calling vector::reserve in a loop can be big pessimization.It effectively disables exponential resizing, and makes everything O(n^2). See a benchmark here: http://quick-bench.com/KT304ZLCCZpN8sd1tVzrGaPkl4A Another question is: an array is cleared after it is inserted to the stack, but not an object. I suppose it will cause only one level of recursion (because values are now moved-from), so maybe it is fine. |
|
Thanks for the feedback.
Very good point! I will look into it. PRs are welcome too :-)
That's correct. I can't remember why the object wasn't explicitly cleared. While it might not make any difference, it's nice to clear it for the sake of consistency at least. |
|
Update: Refer to #1436 (comment) for actual benchmark on object destruction. While the use of Nevertheless, we still might have to get rid of Benchmark results on Release 3.7.1 (recursive destruction): aacdc6b **Benchmark results on Release 3.7.2 (non-recursive destruction): 56109ea ** |
|
Current parsing benchmarks specifically exclude destruction time from measurements. json/benchmarks/src/benchmarks.cpp Lines 22 to 25 in 411158d |
|
I modified the benchmarks to only measure The performance of recursive deconstruction is significantly higher, but I believe avoiding the stack-overflow worth this performance penalty in destruction. non-recursive with non-recursive without recursive: |
|
I made a benchmark that demonstrates the issue. To avoid this being buried in the comments to a closed PR, I created a separate issue #1837 |
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
This includes the following fixes: nlohmann/json#1436 > For a deeply-nested JSON object, the recursive implementation of json_value::destroy function causes stack overflow. nlohmann/json#1708 nlohmann/json#1722 Stack size nlohmann/json#1693 (comment) Integer Overflow nlohmann/json#1447 UTF8, json dump out of bounds nlohmann/json#1445 Possibly influences #7532
4 participants
This pull request fixes #832, #1419, #1835.
For a deeply-nested JSON object, the recursive implementation of
json_value::destroyfunction causes stack overflow.This is problematic for users that wish to expose a public API since an attacker can crash their systems by sending a deeply nested JSON request (e.g. a request containing 512K bytes of
[).In this pull request:
First, a regression test has been added for this particular issue. We should probably add more regression tests for more complex scenarios.
json_value::destroymethod has been modified to deconstruct children of the JSON value iteratively to prevent triggering a recursive deconstruction. The computational complexity and memory requirement of this approach isO(n)which is equivalent to previous recursive deconstruction. Also,std::movehas been used to minimize the overhead of flattening original JSON object.One question: The iterative deconstruction can be also implemented inside
~basic_json::basic_json. I am not sure which place is more appropriate. I have currently implemented it insidejson_value::destorymerely because I find it easier.Pull request checklist
Read the Contribution Guidelines for detailed information.
include/nlohmanndirectory, runmake amalgamateto create the single-header filesingle_include/nlohmann/json.hpp. The whole process is described here.Please don't
#ifdefs or other means.