[AutoDeploy][Feature]: Draft-Target Speculative Decoding

[govind-ramnarayan]

🚀 The feature, motivation and pitch

A basic implementation of two-model speculative decoding, with draft and target models that run in separate engines. The target model should run in AutoDeploy.

• Perf not important
• TP=1

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[suyoggupta]

Step1:
Any draft + any target (with compatible tokenizers) working with AD using 2 model paradigm. ETA 11/24