fix bug: schedule ratio support prefix-caching

[Tsundoku958]

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

During the _schedule_prefill process, the system evaluates whether the free_ratio is less than 0.1. If this condition is met, it bypasses the prefill phase and proceeds directly to decode. When the service is launched with the --enable-prefix-caching option, a significant portion of the GPU memory blocks may be occupied by the block_trie structure. This leads to repeated invocations of decode operations, gradually depleting the available memory blocks. As a result, the waiting queue remains unscheduled, while the running queue becomes fully exhausted. Ultimately, when _schedule_decoding is called, an assertion error is triggered due to the condition assert len(running) != 0 not being satisfied.

Reproduction:

GPU: 4090

Command:

lmdeploy serve api_server ../vllm_build/qwen/qwen_0.6B/  --backend pytorch --max-batch-size 256 --enable-prefix-caching

python benchmark/profile_restful_api.py --backend lmdeploy  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1000

Traceback:

Modification

evict_block_trie when the free_ratio is less than 0.1 during the _schedule_prefill process

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
4. The documentation has been modified accordingly, like docstring or example tutorials.
   @lvhan028 @grimoire

[RunningLeon]

LGTM