Include a working Windows wheel link..

[oscarbg]

Hi,
just opening to get notified when the Windows wheel is released as noted here (but not working):

TensorRT-LLM is supported on bare-metal Windows for single-GPU inference. We provide a release wheel for Windows which can be downloaded from https://developer.nvidia.com/.

[jdemouth]

Hi @oscarbg , thanks for raising that question. Our colleagues, who are building the Windows wheel ran into a few issues, the wheel is almost ready and will be published this Friday or early next week. We will update the README - including the links - as soon as it’s ready. Sorry for the inconvenience. Thanks! Julien

[oscarbg]

thanks for the info! waiting patiently.. :-)

[jdemouth-nvidia]

Hi @oscarbg , the wheels for Windows are now available and the Windows README has been updated with latest info. Please, let us know if you face any issue. Thanks! Julien

[oscarbg]

Thanks!! will try..

[oscarbg]

Hi @jdemouth-nvidia,
just tested on Windows and seems to work..
on a RTX 4070 I get on bloom sample the following output.. seems OK, performance wise?
EDIT:

python benchmark.py -m gpt_350m --mode plugin --batch_size "1;8;64" --input_output_len "60,20;128,20"
(env) C:\Users\oscarbar\inference\TensorRT-LLM\benchmarks\python>python benchmark.py -m gpt_350m --mode plugin --batch_size "1;8;64" --input_output_len "60,20;128,20"
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 1 input_length 60 output_length 20 gpu_peak_mem(gb) 6.65 build_time(s) 10.67 tokens_per_sec 419.85 percentile95(ms) 50.004 percentile99(ms) 51.425 latency(ms) 47.636 compute_cap sm89
C:\Users\oscarbar\inference\TensorRT-LLM\env\lib\site-packages\tensorrt_llm\runtime\generation.py:658: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. (Triggered internally at ..\aten\src\ATen\NestedTensorImpl.cpp:180.)
  torch.nested.nested_tensor(split_ids_list,
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 8 input_length 60 output_length 20 gpu_peak_mem(gb) 6.64 build_time(s) 10.67 tokens_per_sec 2679.8 percentile95(ms) 62.921 percentile99(ms) 65.853 latency(ms) 59.706 compute_cap sm89
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 64 input_length 60 output_length 20 gpu_peak_mem(gb) 7.16 build_time(s) 10.67 tokens_per_sec 10406.25 percentile95(ms) 129.032 percentile99(ms) 131.58 latency(ms) 123.003 compute_cap sm89
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 1 input_length 128 output_length 20 gpu_peak_mem(gb) 6.69 build_time(s) 10.67 tokens_per_sec 413.86 percentile95(ms) 51.18 percentile99(ms) 55.813 latency(ms) 48.326 compute_cap sm89
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 8 input_length 128 output_length 20 gpu_peak_mem(gb) 6.71 build_time(s) 10.67 tokens_per_sec 2408.91 percentile95(ms) 70.509 percentile99(ms) 72.737 latency(ms) 66.42 compute_cap sm89
[BENCHMARK] model_name gpt_350m world_size 1 num_heads 16 num_kv_heads 16 num_layers 24 hidden_size 1024 vocab_size 51200 precision float16 batch_size 64 input_length 128 output_length 20 gpu_peak_mem(gb) 7.57 build_time(s) 10.67 tokens_per_sec 6368.92 percentile95(ms) 207.738 percentile99(ms) 212.592 latency(ms) 200.976 compute_cap sm89

(env) C:\Users\oscarbar\inference\TensorRT-LLM\examples\bloom>python summarize.py --test_trt_llm --hf_model_location ./bloom/560M/ --data_type fp16 --engine_dir ./bloom/560M/trt_engines/fp16/1-gpu/
[10/24/2023-12:21:07] [TRT-LLM] [I] Load tokenizer takes: 0.257509708404541 sec
[10/24/2023-12:21:09] Reusing dataset cnn_dailymail (C:\Users\oscarbar\.cache\huggingface\datasets\ccdv___cnn_dailymail\3.0.0\3.0.0\0107f7388b5c6fae455a5661bcd134fc22da53ea75852027040d8d1e997f101f)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 49.57it/s]
[10/24/2023-12:21:09] [TRT] [I] Loaded engine size: 1560 MiB
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 24154, GPU 2751 (MiB)
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +5, GPU +8, now: CPU 24159, GPU 2759 (MiB)
[10/24/2023-12:21:09] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.8.1
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +1556, now: CPU 0, GPU 1556 (MiB)
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 24163, GPU 3761 (MiB)
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +4, GPU +8, now: CPU 24167, GPU 3769 (MiB)
[10/24/2023-12:21:09] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.8.1
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1556 (MiB)
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 24196, GPU 3783 (MiB)
[10/24/2023-12:21:09] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 24196, GPU 3791 (MiB)
[10/24/2023-12:21:09] [TRT] [W] TensorRT was linked against cuDNN 8.9.4 but loaded cuDNN 8.8.1
[10/24/2023-12:21:10] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 1556 (MiB)
[10/24/2023-12:21:10] [TRT-LLM] [I] Load engine takes: 1.0273706912994385 sec
C:\Users\oscarbar\inference\TensorRT-LLM\examples\bloom\summarize.py:160: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  [torch.tensor(line_encoded[i], dtype=torch.int32), pad],
[10/24/2023-12:21:10] [TRT-LLM] [I] ---------------------------------------------------------
[10/24/2023-12:21:10] [TRT-LLM] [I] TensorRT-LLM Generated :
[10/24/2023-12:21:10] [TRT-LLM] [I]  Article : ['(CNN)James Best, best known for his portrayal of bumbling sheriff Rosco P. Coltrane on TV\'s "The Dukes of Hazzard," died Monday after a brief illness. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in theater and in Hollywood, Best didn\'t become famous until 1979, when "The Dukes of Hazzard\'s" cornpone charms began beaming into millions of American homes almost every Friday night. For seven seasons, Best\'s Rosco P. Coltrane chased the moonshine-running Duke boys back and forth across the back roads of fictitious Hazzard County, Georgia, although his "hot pursuit" usually ended with him crashing his patrol car. Although Rosco was slow-witted and corrupt, Best gave him a childlike enthusiasm that got laughs and made him endearing. His character became known for his distinctive "kew-kew-kew" chuckle and for goofy catchphrases such as "cuff \'em and stuff \'em!" upon making an arrest. Among the most popular shows on TV in the early \'80s, "The Dukes of Hazzard" ran until 1985 and spawned TV movies, an animated series and video games. Several of Best\'s "Hazzard" co-stars paid tribute to the late actor on social media. "I laughed and learned more from Jimmie in one hour than from anyone else in a whole year," co-star John Schneider, who played Bo Duke, said on Twitter. "Give Uncle Jesse my love when you see him dear friend." "Jimmy Best was the most constantly creative person I have ever known," said Ben Jones, who played mechanic Cooter on the show, in a Facebook post. "Every minute of his long life was spent acting, writing, producing, painting, teaching, fishing, or involved in another of his life\'s many passions." Born Jewel Guy on July 26, 1926, in Powderly, Kentucky, Best was orphaned at 3 and adopted by Armen and Essa Best, who renamed him James and raised him in rural Indiana. Best served in the Army during World War II before launching his acting career. In the 1950s and 1960s, he accumulated scores of credits, playing a range of colorful supporting characters in such TV shows as "The Twilight Zone," "Bonanza," "The Andy Griffith Show" and "Gunsmoke." He later appeared in a handful of Burt Reynolds\' movies, including "Hooper" and "The End." But Best will always be best known for his "Hazzard" role, which lives on in reruns. "Jimmie was my teacher, mentor, close friend and collaborator for 26 years," Latshaw said. "I directed two of his feature films, including the recent \'Return of the Killer Shrews,\' a sequel he co-wrote and was quite proud of as he had made the first one more than 50 years earlier." People we\'ve lost in 2015 . CNN\'s Stella Chan contributed to this story.']
[10/24/2023-12:21:10] [TRT-LLM] [I]
 Highlights : ['James Best, who played the sheriff on "The Dukes of Hazzard," died Monday at 88 .\n"Hazzard" ran from 1979 to 1985 and was among the most popular shows on TV .']
[10/24/2023-12:21:10] [TRT-LLM] [I]
 Summary : [[' The actor who played Rosco Coltrane in the hit TV series "Hazzard" died on Monday. He was 88. Best died in hospice in Hickory, North Carolina, of complications from pneumonia, said Steve Latshaw, a longtime friend and Hollywood colleague. Although he\'d been a busy actor for decades in theater and in Hollywood, Best didn\'t become famous until 1979, when "The Dukes of Hazzard\'s" cornpone charms began beaming into millions of']]
[10/24/2023-12:21:10] [TRT-LLM] [I] ---------------------------------------------------------
[10/24/2023-12:21:17] [TRT-LLM] [I] TensorRT-LLM (total latency: 5.655469179153442 sec)
[10/24/2023-12:21:17] [TRT-LLM] [I] TensorRT-LLM beam 0 result
[10/24/2023-12:21:17] [TRT-LLM] [I]   rouge1 : 18.340180779834185
[10/24/2023-12:21:17] [TRT-LLM] [I]   rouge2 : 4.443731125076953
[10/24/2023-12:21:17] [TRT-LLM] [I]   rougeL : 13.684856504816143
[10/24/2023-12:21:17] [TRT-LLM] [I]   rougeLsum : 15.527187235590233