Skip to content

Commit 2ca9bce

Browse files
trollkarlentrollkarlen
committed
add jwt and model dependent queue
add JSON Web Token as a user login service. This makes it more secure and easier to scale out the proxy because it does not need any user text file. Users can also be added without restarting the proxy server. Extracted the queue part of the code to a subset of classes so different queues can easily be implemented, tested and used. As long as they conform to an API specified in the BaseQueue class. Add a model depended queue, it prioritize the servers which have a certain model already loaded. It also can have black list and white list of which model it can handle due to cpu/ram/gpu. It conforms to the shortest queue principle of the simple queue. Add logging instead of prints to the stderr output. Add github-ci workflows Signed-off-by: Robert Marklund <robbelibobban@gmail.com>
1 parent 6b63597 commit 2ca9bce

28 files changed

+3729
-139
lines changed

‎.github/workflows/github-actions-build-wheel.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
name: build wheel
3+
4+
"on":
5+
push:
6+
branches:
7+
- main
8+
pull_request:
9+
branches:
10+
- main
11+
12+
defaults:
13+
run:
14+
shell: bash
15+
16+
jobs:
17+
18+
build-wheel:
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Checkout
23+
uses: actions/checkout@v4
24+
25+
- name: Set up Python 3.13
26+
uses: actions/setup-python@v5
27+
with:
28+
python-version: 3.13
29+
30+
- name: Set up the cache
31+
uses: actions/cache@v4
32+
with:
33+
path: .venv
34+
key: cache-python-packages
35+
36+
- name: Set up the project
37+
run: |
38+
pip install poetry
39+
# poetry config virtualenvs.in-project true
40+
41+
# build for all versions of python ?
42+
- name: build
43+
run: poetry build
44+
45+
- name: Archive build and wheel
46+
uses: actions/upload-artifact@v4
47+
with:
48+
name: build-and-wheel
49+
path: dist

‎.github/workflows/github-actions-docker.yml‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
---
2+
name: build docker images
3+
4+
"on":
5+
push:
6+
branches:
7+
- main
8+
pull_request:
9+
branches:
10+
- main
11+
12+
defaults:
13+
run:
14+
shell: bash
15+
16+
jobs:
17+
18+
docker:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- name: Checkout
22+
uses: actions/checkout@v4
23+
24+
# - name: Login to Docker Hub
25+
# uses: docker/login-action@v3
26+
# with:
27+
# username: ${{ vars.DOCKERHUB_USERNAME }}
28+
# password: ${{ secrets.DOCKERHUB_TOKEN }}
29+
30+
- name: Set up QEMU
31+
uses: docker/setup-qemu-action@v3
32+
33+
- name: Set up Docker Buildx
34+
uses: docker/setup-buildx-action@v3
35+
36+
- name: Build and push
37+
uses: docker/build-push-action@v6
38+
with:
39+
context: .
40+
push: false
41+
tags: ollama_proxy_server:latest

‎.github/workflows/github-actions-quality.yml‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
name: run quality tests
3+
4+
"on":
5+
push:
6+
branches:
7+
- main
8+
pull_request:
9+
branches:
10+
- main
11+
12+
defaults:
13+
run:
14+
shell: bash
15+
16+
jobs:
17+
18+
quality:
19+
runs-on: ubuntu-latest
20+
21+
steps:
22+
- name: Checkout
23+
uses: actions/checkout@v4
24+
25+
- name: Set up Python 3.13
26+
uses: actions/setup-python@v5
27+
with:
28+
python-version: 3.13
29+
30+
- name: Set up the cache
31+
uses: actions/cache@v4
32+
with:
33+
path: .venv
34+
key: cache-python-packages
35+
36+
- name: Set up the project
37+
run: |
38+
pip install poetry safety tox
39+
# poetry config virtualenvs.in-project true
40+
41+
- name: Run ruff lint
42+
run: tox -e lint
43+
44+
- name: Run pylint
45+
run: tox -e pylint
46+
47+
code-saftey:
48+
runs-on: ubuntu-latest
49+
steps:
50+
- name: Check out master
51+
uses: actions/checkout@v4
52+
53+
- name: Assign variable
54+
id: safety_api_key
55+
run: echo '::set-output name=secret::${{secrets.SAFETY_API_KEY}}'
56+
57+
- name: Run Safety CLI to check for vulnerabilities
58+
if: steps.secret.outputs.safety_api_key
59+
uses: pyupio/safety-action@v1
60+
with:
61+
# api-key: ${{ secrets.SAFETY_API_KEY }}
62+
# To always see detailed output from this action
63+
args: --detailed-output
64+
env: # Or as an environment variable
65+
SAFETY_API_KEY: ${{ secrets.SAFETY_API_KEY }}
66+
67+
shellcheck:
68+
runs-on: ubuntu-latest
69+
steps:
70+
- name: Check out master
71+
uses: actions/checkout@v4
72+
73+
- uses: ConorMacBride/install-package@v1
74+
with:
75+
apt: shellcheck
76+
77+
# - name: install shellcheck
78+
# run: apt install shellcheck
79+
80+
- name: Run shellcheck
81+
run: |
82+
./scripts/run-shellcheck.sh

‎.github/workflows/github-actions-tests.yml‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
name: run tox test
3+
4+
"on":
5+
push:
6+
branches:
7+
- main
8+
pull_request:
9+
branches:
10+
- main
11+
12+
defaults:
13+
run:
14+
shell: bash
15+
16+
jobs:
17+
tests:
18+
strategy:
19+
max-parallel: 6
20+
matrix:
21+
os: [ubuntu-latest, macos-latest]
22+
python-version: ["3.10", "3.11", "3.12", "3.13"] # , "3.14"]
23+
24+
runs-on: ${{ matrix.os }}
25+
26+
steps:
27+
- name: Checkout
28+
uses: actions/checkout@v4
29+
30+
- name: Set up Python ${{ matrix.python-version }}
31+
uses: actions/setup-python@v5
32+
with:
33+
python-version: ${{ matrix.python-version }}
34+
35+
- name: Set up the cache
36+
uses: actions/cache@v4
37+
env:
38+
cache-name: cache-python-packages
39+
with:
40+
path: .venv
41+
key: ${{ matrix.os }}-${{ matrix.python-version }}-${{ env.cache-name }}
42+
restore-keys: |
43+
${{ matrix.os }}-${{ matrix.python-version }}-
44+
${{ matrix.os }}-
45+
46+
- name: Set up the project
47+
run: |
48+
pip install poetry tox
49+
# poetry config virtualenvs.in-project true
50+
51+
- name: Run the test suite
52+
run: tox -e py${{ matrix.python-version }}-test
53+
54+
- name: Archive coverage report
55+
uses: actions/upload-artifact@v4
56+
with:
57+
name: code-coverage-report-${{ matrix.os }}-${{ matrix.python-version }}
58+
path: coverage.xml
59+
60+
windows-tests:
61+
strategy:
62+
max-parallel: 6
63+
matrix:
64+
os: [windows-latest]
65+
# windows have issues with fast timers prior to python 3.13 so ignore the old pythons
66+
python-version: ["3.13"] # , "3.14"]
67+
68+
runs-on: ${{ matrix.os }}
69+
70+
steps:
71+
- name: Checkout
72+
uses: actions/checkout@v4
73+
74+
- name: Set up Python ${{ matrix.python-version }}
75+
uses: actions/setup-python@v5
76+
with:
77+
python-version: ${{ matrix.python-version }}
78+
79+
- name: Set up the cache
80+
uses: actions/cache@v4
81+
env:
82+
cache-name: cache-python-packages
83+
with:
84+
path: .venv
85+
key: ${{ matrix.os }}-${{ matrix.python-version }}-${{ env.cache-name }}
86+
restore-keys: |
87+
${{ matrix.os }}-${{ matrix.python-version }}-
88+
${{ matrix.os }}-
89+
90+
- name: Set up the project
91+
run: |
92+
pip install poetry tox
93+
# poetry config virtualenvs.in-project true
94+
95+
- name: Run the test suite
96+
run: tox -e py${{ matrix.python-version }}-test
97+
# run: tox -f test -l | tr '\n' ',' | xargs tox -e
98+
99+
- name: Archive coverage report
100+
uses: actions/upload-artifact@v4
101+
with:
102+
name: code-coverage-report-${{ matrix.os }}-${{ matrix.python-version }}
103+
path: coverage.xml

‎.yamllint.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
extends: default
3+
ignore: ".tox*\n.git*"
4+
rules:
5+
line-length:
6+
max: 120
7+
level: warning

‎Dockerfile‎

Lines changed: 58 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,67 @@
1-
FROM python:3.11
1+
FROM python:3.13 AS python-base
22

3-
# Update packagtes, install necessary tools into the base image, clean up and clone git repository
4-
RUN apt update \
5-
&& apt install -y --no-install-recommends --no-install-suggests git apache2 \
6-
&& apt autoremove -y --purge \
7-
&& apt clean \
8-
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
9-
&& git clone https://github.com/ParisNeo/ollama_proxy_server.git
3+
# https://python-poetry.org/docs#ci-recommendations
4+
ENV POETRY_VERSION=2.1.3
5+
ENV POETRY_HOME=/opt/poetry
6+
ENV POETRY_VENV=/opt/poetry-venv
107

11-
# Change working directory to cloned git repository
12-
WORKDIR ollama_proxy_server
8+
# Tell Poetry where to place its cache and virtual environment
9+
ENV POETRY_CACHE_DIR=/opt/.cache
1310

14-
# Install all needed requirements
15-
RUN pip3 install -e .
11+
# Create stage for Poetry installation
12+
FROM python-base AS poetry-base
1613

17-
# Copy config.ini and authorized_users.txt into project working directory
18-
COPY config.ini .
19-
COPY authorized_users.txt .
14+
# Creating a virtual environment just for poetry and install it with pip
15+
RUN python3 -m venv $POETRY_VENV \
16+
&& $POETRY_VENV/bin/pip install -U pip setuptools \
17+
&& $POETRY_VENV/bin/pip install poetry==${POETRY_VERSION}
18+
19+
# Copy Poetry to app image
20+
# COPY --from=poetry-base ${POETRY_VENV} ${POETRY_VENV}
21+
22+
# Add Poetry to PATH
23+
ENV PATH="${PATH}:${POETRY_VENV}/bin"
24+
25+
# temp dir
26+
WORKDIR /app-tmp
27+
28+
# Copy Dependencies
29+
COPY --chown=worker:worker poetry.lock pyproject.toml ./
30+
31+
# [OPTIONAL] Validate the project is properly configured
32+
RUN poetry check
33+
34+
# Copy Application
35+
COPY --chown=worker:worker . ./
36+
37+
# build and install the app
38+
RUN poetry build
2039

21-
# Start the proxy server as entrypoint
22-
ENTRYPOINT ["ollama_proxy_server"]
40+
# Create a new stage from the base python image
41+
FROM python-base AS ollama-proxy-server
42+
43+
RUN adduser worker
44+
WORKDIR /home/worker
45+
46+
# copy wheel and tgz
47+
COPY --from=poetry-base /app-tmp/dist ./dist
48+
49+
RUN pip install dist/*.whl
50+
51+
RUN rm -fr ./dist
52+
53+
# copy entry point
54+
COPY --from=poetry-base /app-tmp/entry-point.sh /
55+
56+
USER worker
57+
58+
# Copy config.ini and authorized_users.txt into project working directory
59+
COPY --chown=worker:worker config.ini .
60+
COPY --chown=worker:worker authorized_users.txt .
2361

2462
# Do not buffer output, e.g. logs to stdout
2563
ENV PYTHONUNBUFFERED=1
2664

27-
# Set command line parameters
28-
CMD ["--config", "./config.ini", "--users_list", "./authorized_users.txt", "--port", "8080"]
65+
# Run Application
66+
EXPOSE 8080
67+
ENTRYPOINT ["/entry-point.sh"]

‎config.ini‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,11 @@
11
[DefaultServer]
22
url = http://localhost:11434
3+
model_white_list = ["llama3.2:1b"]
4+
model_black_list = ["llama3.2:800b"]
35

46
[SecondaryServer]
57
url = http://localhost:3002
8+
model_black_list = ["llama3.2:200b"]
69

710
# Add more servers as you need.
811

0 commit comments

Comments
 (0)