- Speech-to-text using faster-whisper to capture voice input to text
- Text-to-Speech using the Orpheus TTS model to let the AI talk back
- Fast response times: default settings are optimized to get audio output as fast as possible
- Running on a RTX4080 I'm seeing about 0.2-0.4s delay
- Voice interruptions: talk over the LLM at any time, the voice output will be cut off.
- The response prompt will be approximately cut before storing, so asking the LLM for the last thing they said should work naturally
- Tools: some simple tool call functionality for doing basic operations like controlling home assistant
- Database: simple SQL database to store your conversations.
- New messages get weighted based on semantic similarity, this makes it so you always have some historic and some relevant context
- By default the LLM is set to generate quick responses for natural conversation flow (see system prompt)
Tested on windows and linux with python 3.12 with LM studio 0.3.33
-
Install LM Studio
-
Install dependencies:
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt -
Open LM Studio app
-
Set your username in
utilities/settings.pyif you want to use tools (more info below), otherwise disableENABLE_TOOLS -
Run the script:
python main.py -
The app will download and load all required models from there
See utilities/settings.py for global toggles:
# Various input methods
ENABLE_KEYBOARD:bool = True
ENABLE_MIC:bool = True
# Enable output voice generation
ENABLE_AUDIO:bool = True
# Load and store in database
ENABLE_DATABASE:bool = True
# Prints stats like ttft in output
ENABLE_STATS:bool = False
# Prints debug statements
ENABLE_DEBUG:bool = False
# Loads all tools
ENABLE_TOOLS:bool = TrueMore settings like what models to use are in there. If you change models, depending on what features you want consider the following:
- Instruct models: tool use requires models trained for instructions
- Vision language models: image analysis
By default it will load Qwen3 8B VL and Orpheus 3B quantized at 4 bits. This will take around 9GB VRAM. It's easy to switch to Qwen3 4B VL which reduces it down to about 6GB VRAM total.
While the app works fine without tools for conversing with the LLM, some interesting features are added by adding tools.
In order to not cut down the input context of the model, all these tools are added based on some keywords included in the last n user prompts.
The tools are initialized in main.py where you have to set some directories for ComfyUI or other folders.
Here is a brief description of the tools included can do:
Basic commands like list, copy, move of files in your OS.
Addition to the file tool, you can specify directories so the LLM can quickly look them up. I've only added my downloads folder.
Basic image processing, sending the model with a local path of url to a jpg/png image will have VL models describe the contents.
Addition to the image tool, this tool gets passed a directory and will add new images as prompts to the model. For example specifying your downloads folder here will have it respond by describing any new image you downloads.
Addition to the image tool, we can make a screenshot and directly load it into context for the LLM to describe. You will probably have to define the input screen as bounding box to capture it correctly
Include some home assistant functionality for the LLM
I've really kept it to the basics: turning on/off lights and ACs.
This requires API call to be enabled on your HA installation (enabled by default on new installations)
To enable API access, create passwords.json in the directory with an entry for "homeassistant" and a long lived token.
An experimental tool so the bot can run workflows through ComfyUI. This requires some additional setup in ComfyUI itself as well, you will need the following custom node installed: https://github.com/SethRobinson/comfyui-workflow-to-api-converter-endpoint
With this we can convert the live user workflow to an API version, which we can send to ComfyUI as a post request. I've set it up so it parses a node named "mainprompt" in my workflow (which is a text field). I let the bot change the prompt text on that and run the rest of the workflow.
Combining this with the ImageWatcher tool on the comfyui output folder we get an interesting possibility where the AI can now generate an image and then respond to it. I've used it to have the AI rate their own images and try to improve on them with mild success.
This project wouldn't be possible without the following awesome projects: