idea: Production Level Queue System

[dan-menlo]

Objective

• Do we need a queue system that scales to thousands of requests

Motivation

Nullpointer Errors?

• Currently, inference requests are handled FIFO
• We are adopting an OpenAI API, which means that we will receive requests across Chat, Audio, Vision etc
• Given that users are on laptops with limited RAM and VRAM, we are likely to have to switch models

Preparing for Cloud Native

• Our long-term future is likely as an enterprise OpenAI-alternative, which will be multi-user and have a queue system
• Should we bake in this abstraction, and use a local file-based queue (which is later swapped out for a more sophisticated queue?)

[Van-QA]

Quoted from users from janhq/jan#2704:

Problem
When a generation is ongoing, entering a new prompt causes the generation to be interrupted. It would be nice if subsequent prompts to the same model be queued instead of resulting in interruptions.

Success Criteria
It would be nice if we could queue up a couple of prompts and then get back to the responses after a while.

[louis-jan]

Scoped for refactoring the Cortex Backend.