[sgl] view could hold the memory too long and introduced large memory

[bixue2010]

consumption

Motivation

in current maybe_collect_customized_info, v is a batch tensor (B, ...), v[i] takes a view, and when it got put into req.customized_info, req.customized_info will be accumulated until request is finished. since v[i] takes a view, hence the memory can't be released until all the requests in the batch finished. which means this will depend on the longest running request finish to be able to release the memory.

In some situation, when we stored slightly larger info in customized_info, this could accumulate quite large memory, which introduced OOM.

Modifications

suggest to take a copy of v[i] to break the view.

Accuracy Tests

unit test.

Benchmarking and Profiling

N/A

Checklist

• [X ] Format your code according to the Format code with pre-commit.
• [ X] Add unit tests according to the Run and add unit tests.
• [ X] Update documentation according to Write documentations.
• [X ] Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• [ X] Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Qiaolin-Yu]

/tag-and-rerun-ci