NVIDIA · suyoggupta · Feb 5, 2026 · Feb 6, 2026 · Feb 6, 2026 · Feb 6, 2026
@@ -0,0 +1,348 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "# Deploying GLM-4.7-Flash with TensorRT-LLM\n",
+        "\n",
+        "This notebook walks you through deploying the `zai-org/GLM-4.7-Flash` model using TensorRT-LLM.\n",
+        "\n",
+        "[TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/) is NVIDIA's open-source library for accelerating and optimizing LLM inference on NVIDIA GPUs. Support for GLM-4.7-Flash is enabled through the AutoDeploy workflow. More details about AutoDeploy can be found [here](https://nvidia.github.io/TensorRT-LLM/torch/auto_deploy/auto-deploy.html).\n",
+        "\n",
+        "**Model Resources:**\n",
+        "- [HuggingFace Model Card](https://huggingface.co/zai-org/GLM-4.7-Flash)\n",
+        "- [Technical Blog](https://z.ai/blog/glm-4.7)\n",
+        "- [Technical Report (GLM-4.5)](https://arxiv.org/abs/2508.06471)\n",
+        "- [Z.ai API Platform](https://docs.z.ai/guides/llm/glm-4.7)\n",
+        "\n",
+        "**Model Highlights:**\n",
+        "- 30B-A3B Mixture of Experts (MoE) architecture\n",
+        "- 131,072 token context length\n",
+        "- Tool calling support\n",
+        "- MIT License\n",
+        "\n",
+        "**Prerequisites:**\n",
+        "- NVIDIA GPU with recent drivers (≥ 64 GB VRAM for BF16) and CUDA 12.x\n",
+        "- Python 3.10+\n",
+        "- TensorRT-LLM ([container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release) or pip install)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Prerequisites & Environment\n",
+        "\n",
+        "Set up a containerized environment for TensorRT-LLM by running the following command in a terminal:\n",
+        "\n",
+        "```shell\n",
+        "docker run --rm -it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --gpus=all -p 8000:8000 nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc1\n",
+        "```\n",
+        "\n",
+        "You now have TensorRT-LLM set up!"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "# If pip not found\n",
+        "!python -m ensurepip --default-pip"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "%pip install torch openai"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Verify GPU\n",
+        "\n",
+        "Check that CUDA is available and the GPU is detected correctly."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 1,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Python: 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0]\n",
+            "CUDA available: True\n",
+            "Num GPUs: 8\n",
+            "GPU[0]: NVIDIA H100 80GB HBM3\n",
+            "GPU[1]: NVIDIA H100 80GB HBM3\n",
+            "GPU[2]: NVIDIA H100 80GB HBM3\n",
+            "GPU[3]: NVIDIA H100 80GB HBM3\n",
+            "GPU[4]: NVIDIA H100 80GB HBM3\n",
+            "GPU[5]: NVIDIA H100 80GB HBM3\n",
+            "GPU[6]: NVIDIA H100 80GB HBM3\n",
+            "GPU[7]: NVIDIA H100 80GB HBM3\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Environment check\n",
+        "import sys\n",
+        "\n",
+        "import torch\n",
+        "\n",
+        "print(f\"Python: {sys.version}\")\n",
+        "print(f\"CUDA available: {torch.cuda.is_available()}\")\n",
+        "print(f\"Num GPUs: {torch.cuda.device_count()}\")\n",
+        "\n",
+        "if torch.cuda.is_available():\n",
+        "    for i in range(torch.cuda.device_count()):\n",
+        "        print(f\"GPU[{i}]: {torch.cuda.get_device_name(i)}\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## OpenAI-Compatible Server\n",
+        "\n",
+        "Start a local OpenAI-compatible server with TensorRT-LLM via the terminal, within the running docker container.\n",
+        "\n",
+        "Ensure that the following commands are executed from the docker terminal.\n",
+        "\n",
+        "Start with the GLM 4.7 Flash Yaml here: `examples/auto_deploy/model_registry/configs/glm-4.7-flash.yaml`"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Load the Model\n",
+        "\n",
+        "Launch the TensorRT-LLM server with GLM-4.7-Flash:\n",
+        "\n",
+        "```shell\n",
+        "trtllm-serve \"zai-org/GLM-4.7-Flash\" \\\n",
+        "  --host 0.0.0.0 \\\n",
+        "  --port 8000 \\\n",
+        "  --backend _autodeploy \\\n",
+        "  --trust_remote_code \\\n",
+        "  --extra_llm_api_options examples/auto_deploy/model_registry/configs/glm-4.7-flash.yaml\n",
+        "```"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Your server is now running!"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Use the API\n",
+        "\n",
+        "Use the OpenAI-compatible client to send requests to the TensorRT-LLM server."
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": 11,
+      "metadata": {},
+      "outputs": [],
+      "source": [
+        "from openai import OpenAI\n",
+        "\n",
+        "# Setup client\n",
+        "BASE_URL = \"http://0.0.0.0:8000/v1\"\n",
+        "API_KEY = \"null\"\n",
+        "client = OpenAI(base_url=BASE_URL, api_key=API_KEY)\n",
+        "\n",
+        "MODEL_ID = \"zai-org/GLM-4.7-Flash\""
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Chat Completion Example\n",
+            "==================================================\n",
+            "Response:\n",
+            "1.  **Analyze the Request:** The user wants to know 15% of 85 and wants to see the reasoning behind the calculation.\n",
+            "\n",
+            "2.  **Identify the Core Task:** Calculate $15\\% \\times 85$.\n",
+            "\n",
+            "3.  **Determine the Mathematical Approach:** There are several ways to solve this:\n",
+            "    *   *Method 1: Fraction multiplication.* Convert 15% to a fraction ($\\frac{15}{100}$), then multiply by 85.\n",
+            "    *   *Method 2: Decimal multiplication.* Convert 15% to a decimal ($0.15$), then multiply by 85.\n",
+            "    *   *Method 3: Decomposition (Breaking it down).* $15\\% = 10\\% + 5\\%$.\n",
+            "        *   $10\\%$ of $85 = 8.5$\n",
+            "        *   $5\\%$ of $85 = \\frac{8.5}{2} = 4.25$\n",
+            "        *   Sum: $8.5 + 4.25 = 12.75$\n",
+            "\n",
+            "4.  **Select the Best Approach for Explanation:** Method 3 is often easiest for a general audience to follow step-by-step because it avoids dealing with decimals until the end or simplifies large multiplications. Method 2 is the most direct standard school method. I will use Method 3 (Splitting 15% into 10% and 5%) as the primary reasoning because it is intuitive, but I might briefly mention the standard formula ($\\frac{\\text{percent}}{100} \\times \\text{number}$).\n",
+            "\n",
+            "5.  **Execute the Calculation (Method 3):**\n",
+            "    *   Step 1: Find 10% of 85.\n",
+            "        *   Moving the decimal point one place to the left: 8.5.\n",
+            "    *   Step 2: Find 5% of 85.\n",
+            "        *   Since 5% is half of 10%, take half of 8.5.\n",
+            "        *   $8.5 / 2 = 4.25$.\n",
+            "    *   Step 3: Add them together.\n",
+            "        *   $8.5 + 4.25$.\n",
+            "        *   $8.5 + 4 = 12.5$.\n",
+            "        *   $12.5 + 0\n"
+          ]
+        }
+      ],
+      "source": [
+        "# Basic chat completion\n",
+        "print(\"Chat Completion Example\")\n",
+        "print(\"=\" * 50)\n",
+        "\n",
+        "response = client.chat.completions.create(\n",
+        "    model=MODEL_ID,\n",
+        "    messages=[\n",
+        "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+        "        {\"role\": \"user\", \"content\": \"What is 15% of 85? Show your reasoning.\"},\n",
+        "    ],\n",
+        "    temperature=1,\n",
+        "    top_p=0.95,\n",
+        "    max_tokens=512,\n",
+        ")\n",
+        "\n",
+        "print(\"Response:\")\n",
+        "print(response.choices[0].message.content)"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {},
+      "outputs": [
+        {
+          "name": "stdout",
+          "output_type": "stream",
+          "text": [
+            "Streaming response:\n",
+            "==================================================\n",
+            "1.  **Analyze the Request:** The user is asking for the \"first 5 prime numbers\".\n",
+            "\n",
+            "2.  **Define \"Prime Number\":** A prime number is a natural number greater than 1 that is not a product of two smaller natural numbers. In other words, it has exactly two distinct positive divisors: 1 and itself.\n",
+            "\n",
+            "3.  **Identify the First Numbers:**\n",
+            "    *   Start checking from 1 (exclusive).\n",
+            "    *   Check 2: Divisors are 1 and 2. Prime. (1st)\n",
+            "    *   Check 3: Divisors are 1 and 3. Prime. (2nd)\n",
+            "    *   Check 4: Divisors are 1, 2, 4. Not prime (2 * 2).\n",
+            "    *   Check 5: Divisors are 1 and 5. Prime. (3rd)\n",
+            "    *   Check 6: Divisors are 1, 2, 3, 6. Not prime.\n",
+            "    *   Check 7: Divisors are 1 and 7. Prime. (4th)\n",
+            "    *   Check 8: Divisors are 1, 2, 4, 8. Not prime.\n",
+            "    *   Check 9: Divisors are 1, 3, 9. Not prime.\n",
+            "    *   Check 10: Divisors are 1, 2, 5, 10. Not prime.\n",
+            "    *   Check 11: Divisors are 1 and 11. Prime. (5th)\n",
+            "\n",
+            "4.  **Compile the List:** 2, 3, 5, 7, 11.\n",
+            "\n",
+            "5.  **Formulate the Output:** Present the list clearly.\n",
+            "\n",
+            "6.  **Final Review:** Does this answer the user's prompt accurately? Yes.</think>The first 5 prime numbers are **2, 3, 5, 7, and 11**."
+          ]
+        }
+      ],
+      "source": [
+        "# Streaming chat completion\n",
+        "print(\"Streaming response:\")\n",
+        "print(\"=\" * 50)\n",
+        "\n",
+        "stream = client.chat.completions.create(\n",
+        "    model=MODEL_ID,\n",
+        "    messages=[\n",
+        "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
+        "        {\"role\": \"user\", \"content\": \"What are the first 5 prime numbers?\"},\n",
+        "    ],\n",
+        "    temperature=0.7,\n",
+        "    max_tokens=1024,\n",
+        "    stream=True,\n",
+        ")\n",
+        "\n",
+        "for chunk in stream:\n",
+        "    if chunk.choices[0].delta.content:\n",
+        "        print(chunk.choices[0].delta.content, end=\"\", flush=True)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Evaluation Parameters\n",
+        "\n",
+        "For optimal results, use the following parameters based on your task:\n",
+        "\n",
+        "**Default Settings (Most Tasks)**\n",
+        "- `temperature`: 1.0\n",
+        "- `top_p`: 0.95\n",
+        "- `max_tokens`: 131072\n",
+        "\n",
+        "**Agentic Tasks (SWE-bench, Terminal Bench)**\n",
+        "- `temperature`: 0.7\n",
+        "- `top_p`: 1.0\n",
+        "- `max_tokens`: 16384\n",
+        "\n",
+        "**Deterministic Tasks**\n",
+        "- `temperature`: 0\n",
+        "- `max_tokens`: 16384"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Additional Resources\n",
+        "\n",
+        "- [TensorRT-LLM Documentation](https://nvidia.github.io/TensorRT-LLM/)\n",
+        "- [AutoDeploy Guide](https://nvidia.github.io/TensorRT-LLM/torch/auto_deploy/auto-deploy.html)\n",
+        "- [GLM-4.7-Flash on HuggingFace](https://huggingface.co/zai-org/GLM-4.7-Flash)\n",
+        "- [Z.ai Discord Community](https://discord.gg/QR7SARHRxK)"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.12.3"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 4
+}
@@ -0,0 +1,12 @@
+compile_backend: torch-cudagraph
+max_batch_size: 64
+max_seq_len: 4096
+enable_chunked_prefill: true
+cuda_graph_batch_sizes: [1, 2, 4, 8, 16, 32, 64]
+transforms:
+  multi_stream_moe:
+    stage: compile
+    enabled: true
+  multi_stream_mla_attn:
+    stage: compile
+    enabled: true
@@ -221,3 +221,5 @@ models:
   yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'multimodal.yaml', 'llama4_maverick_lite.yaml']
 - name: nvidia/NVIDIA-Nemotron-3-Super-120B-BF16-BF16KV-010726
   yaml_extra: ['dashboard_default.yaml', 'world_size_4.yaml','super_v3.yaml']
+- name: zai-org/GLM-4.7-Flash
+  yaml_extra: ['glm-4.7-flash.yaml']