feat: add preStop hook for llamacpp and tgi in the BackendRuntime

[cr7258]

What this PR does / why we need it

Add preStop hook for llamacpp and tgi in the BackendRuntime to ensure graceful termination. I didn't add preStop hook for Ollama and SGLang, because:

• Ollama does not provide metrics for the number of requests being processed, and it currently even lacks a Add metrics endpoint and basic request metrics otel based ollama/ollama#6537. I have created a Provide an API to retrieve the number of requests being processed ollama/ollama#10419 on the Ollama repo.
• SGLang refreshes the Prometheus metrics every 30 seconds, which is too long for us to determine the termination. Additionally, SGLang natively supports graceful termination, see logs below.

[2025-04-26 10:13:01] SIGTERM received. signum=None frame=None. Draining requests and shutting down...
[2025-04-26 10:13:04] Gracefully exiting... remaining number of requests 3
[2025-04-26 10:13:09] Gracefully exiting... remaining number of requests 2
[2025-04-26 10:13:14] Gracefully exiting... remaining number of requests 2
2025-04-26 10:13:18,881 - INFO - flashinfer.jit: Finished loading JIT ops: cascade
[2025-04-26 10:13:18 TP0] Decode batch. #running-req: 1, #token: 11, token usage: 0.00, gen throughput (token/s): 0.85, #queue-req: 0, 
[2025-04-26 10:13:19] Gracefully exiting... remaining number of requests 2
[2025-04-26 10:13:19 TP0] Decode batch. #running-req: 1, #token: 51, token usage: 0.00, gen throughput (token/s): 342.30, #queue-req: 0, 
[2025-04-26 10:13:19 TP0] Decode batch. #running-req: 1, #token: 91, token usage: 0.00, gen throughput (token/s): 381.66, #queue-req: 0, 
[2025-04-26 10:13:19] INFO:     127.0.0.1:50206 - "POST /v1/completions HTTP/1.1" 200 OK
[2025-04-26 10:13:21 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-04-26 10:13:24] Gracefully exiting... remaining number of requests 0

In this PR, I also increase the terminationGracePeriodSeconds from default 30s to 130. Generally, the termination grace period needs to last longer than the slowest request we expect to serve plus any extra time spent waiting for load balancers to take the model server out of rotation. For the detailed explanation, please see here.

Which issue(s) this PR fixes

Fixes #320

Special notes for your reviewer

llamacpp
related doc
output logs:

Terminating: Running: 1, Waiting: 0
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /metrics 127.0.0.1 200
srv  log_server_r: request: GET /metrics 127.0.0.1 200
Terminating: Running: 1, Waiting: 0
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  log_server_r: request: GET /metrics 127.0.0.1 200
srv  log_server_r: request: GET /metrics 127.0.0.1 200
Terminating: Running: 1, Waiting: 0
srv  log_server_r: request: GET /health 240.243.170.78 200
srv  cancel_tasks: cancel task, id_task = 0
slot      release: id  0 | task 0 | stop processing: n_past = 2810, truncated = 0
srv  update_slots: all slots are idle
srv  update_slots: all slots are idle
srv  log_server_r: request: GET /metrics 127.0.0.1 200
srv  update_slots: all slots are idle
srv  log_server_r: request: GET /metrics 127.0.0.1 200
Terminating: No active or waiting requests, safe to terminate
srv    operator(): operator(): cleaning up before exit...

tgi:

related doc
output logs:

Terminating: Running: 1, Waiting: 0
2025-04-27T05:28:12.853024Z  INFO completions{total_time="4.198707127s" validation_time="168.004µs" queue_time="64.762µs" inference_time="4.198474621s" time_per_token="4.198474ms" seed="None"}: text_generation_router::server: router/src/server.rs:402: Success
2025-04-27T05:28:14.820936Z  INFO text_generation_router_v3::radix: backends/v3/src/radix.rs:108: Prefix 0 - Suffix 1008
Terminating: Running: 1, Waiting: 0
2025-04-27T05:28:18.999206Z  INFO completions{total_time="4.178477298s" validation_time="177.684µs" queue_time="74.152µs" inference_time="4.178225622s" time_per_token="4.178225ms" seed="None"}: text_generation_router::server: router/src/server.rs:402: Success
Terminating: No active or waiting requests, safe to terminate

Does this PR introduce a user-facing change?

add preStop hook for llamacpp and tgi in the BackendRuntime

[kerthcet]

Additionally, SGLang natively supports graceful termination, see logs below.

Thanks @cr7258 I didn't know this before, I was wondering why inference engine don't support this, which they should.

[kerthcet]

Only one comment.

[kerthcet]

/lgtm
/approve
/kind feature

Thanks!

[kerthcet]

/lgtm
/approve
/kind feature

Thanks!

[kerthcet]

/lgtm
/approve
/kind feature

Thanks!

[kerthcet]

/triage accepted

[kerthcet]

/lgtm