Skip to content

Add turbomind metrics#3811

Merged
lvhan028 merged 28 commits intoInternLM:mainfrom
lvhan028:turbomind-metrics
Aug 13, 2025
Merged

Add turbomind metrics#3811
lvhan028 merged 28 commits intoInternLM:mainfrom
lvhan028:turbomind-metrics

Conversation

@lvhan028
Copy link
Copy Markdown
Collaborator

@lvhan028 lvhan028 commented Aug 4, 2025

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Implemented metric collection for scheduler and inference in TurboMind engine ( Resolve #3801)

  • Metric collection now covers both scheduling and inference processes
  • Modified files: csrc/*, bind.cpp, and turbomind.py

Optimized metric polling mechanism

  • Switched from push to pull model for scheduler metrics, as scheduler metrics change less frequently than per-token metrics

Corrected metrics statistics

  • Tokens per second (TPS)
  • Inference time latency (ITL)

Log condition

  • Metrics are logged to console only when requests are received.

@lvhan028 lvhan028 added the enhancement New feature or request label Aug 4, 2025
@lvhan028 lvhan028 requested review from CUHKSZzxy and lzhangzz August 4, 2025 12:11
@CUHKSZzxy CUHKSZzxy mentioned this pull request Aug 5, 2025
9 tasks
@lvhan028 lvhan028 mentioned this pull request Aug 6, 2025
3 tasks
@lvhan028 lvhan028 requested a review from grimoire August 6, 2025 10:52
Copy link
Copy Markdown
Collaborator

@CUHKSZzxy CUHKSZzxy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lvhan028 lvhan028 merged commit c362e00 into InternLM:main Aug 13, 2025
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] metrics for turbomind engine

4 participants