diff --git a/examples/models/core/deepseek_v3/README.md b/examples/models/core/deepseek_v3/README.md index ab274191d45b..0d7ee2a001df 100644 --- a/examples/models/core/deepseek_v3/README.md +++ b/examples/models/core/deepseek_v3/README.md @@ -28,6 +28,11 @@ Please refer to [this guide](https://nvidia.github.io/TensorRT-LLM/installation/ - [Evaluation](#evaluation) - [Serving](#serving) - [trtllm-serve](#trtllm-serve) + - [B200 FP4 min-latency config](#b200-fp4-min-latency-config) + - [B200 FP4 max-throughput config](#b200-fp4-max-throughput-config) + - [B200 FP8 min-latency config](#b200-fp8-min-latency-config) + - [B200 FP8 max-throughput config](#b200-fp8-max-throughput-config) + - [Launch trtllm-serve OpenAI-compatible API server](#launch-trtllm-serve-openai-compatible-api-server) - [Disaggregated Serving](#disaggregated-serving) - [Dynamo](#dynamo) - [tensorrtllm\_backend for triton inference server (Prototype)](#tensorrtllm_backend-for-triton-inference-server-prototype) @@ -228,56 +233,111 @@ trtllm-eval --model \ ## Serving ### trtllm-serve -Take max-throughput scenario on B200 as an example, the settings are extracted from the [blog](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md#b200-max-throughput). **For users' own models and cases, the specific settings could be different to get best performance.** +Below are example B200 serving configurations for both min-latency and max-throughput in FP4 and FP8. If you want to explore configurations, see the [blog](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md). **Treat these as starting points—tune for your model and workload to achieve the best performance.** To serve the model using `trtllm-serve`: +#### B200 FP4 min-latency config +```bash +cat >./extra-llm-api-config.yml <./extra-llm-api-config.yml <./extra-llm-api-config.yml <./extra-llm-api-config.yml <