Skip to content

Support ollama#193

Merged
InftyAI-Agent merged 1 commit into
InftyAI:mainfrom
qinguoyi:support-ollama
Nov 11, 2024
Merged

Support ollama#193
InftyAI-Agent merged 1 commit into
InftyAI:mainfrom
qinguoyi:support-ollama

Conversation

@qinguoyi
Copy link
Copy Markdown
Member

@qinguoyi qinguoyi commented Nov 3, 2024 •
edited
Loading

What this PR does / why we need it

Support ollama

Which issue(s) this PR fixes

#91

Special notes for your reviewer

  1. add OLLAMA protocol, when access modelName, there return model.Address and not inject init container
  2. execute multiple shell commands instead of just ollama run；because needs to start first and make sure the ollama service is started
  3. upgrader parse flags impl, there will be a replacement rather than a complete override

Does this PR introduce a user-facing change?

Support ollama

@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 3, 2024
@qinguoyi qinguoyi mentioned this pull request Nov 3, 2024
Closed
3 tasks
@qinguoyi qinguoyi force-pushed the support-ollama branch 2 times, most recently from db49d51 to 05078e4 Compare November 3, 2024 11:00
qinguoyi
qinguoyi commented Nov 3, 2024
View reviewed changes
Comment thread chart/templates/backends/ollama.yaml
while true;do sleep 60;done"
envs:
- name: OLLAMA_HOST
value: 0.0.0.0:8080
Copy link
Copy Markdown
Member Author

@qinguoyi qinguoyi Nov 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OLLAMA_HOST can expose custom port

Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great

qinguoyi
qinguoyi commented Nov 3, 2024
View reviewed changes
Comment thread pkg/controller_helper/backendruntime.go Outdated
}
// replace
value = strings.Replace(value, match[0], replacement, -1)
}
Copy link
Copy Markdown
Member Author

@qinguoyi qinguoyi Nov 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace instaned of override

kerthcet
kerthcet reviewed Nov 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM, have you tested locally? Maybe we can have a e2e test because ollama can still run with CPUs.

Comment thread chart/templates/backends/ollama.yaml
args:
- name: default
flags:
- "ollama serve &
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to support readiness and liveness next, see #21, but this is ok for now.

Copy link
Copy Markdown
Member Author

@qinguoyi qinguoyi Nov 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's support probe in the future

Comment thread chart/templates/backends/ollama.yaml
while true;do sleep 60;done"
envs:
- name: OLLAMA_HOST
value: 0.0.0.0:8080
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great

Comment thread docs/examples/ollama/model.yaml Outdated
spec:
familyName: qwen2
source:
uri: OLLAMA://qwen2:0.5b No newline at end of file
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use lower case for consistency, just like http, although we'll convert to upper case for comparison in the runtime.

Comment thread pkg/controller_helper/model_source/uri.go
modelPath string
modelName string
protocol string
bucket string
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to refactor this part in the future, divide the URIProvider to more specified ones, but not a hurry.

@qinguoyi
Copy link
Copy Markdown
Member Author

qinguoyi commented Nov 4, 2024

Generally LGTM, have you tested locally? Maybe we can have a e2e test because ollama can still run with CPUs.

Thanks for your review, I've done multiple tests locally before committing.
Another, a complete e2e test is very necessary and I will complete it as soon as possible.

kerthcet
kerthcet reviewed Nov 5, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits.

Comment thread pkg/controller_helper/backendruntime.go Outdated
value := flag
matches := re.FindAllStringSubmatch(flag, -1)

if len(matches) > 0 {
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's refactor like this, to reduce the nested blocks.

if len(matches) == 0 {
   return
}

Comment thread pkg/controller_helper/backendruntime.go Outdated
if len(matches) > 0 {
for _, match := range matches {
if len(match) > 1 {
// get key
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this comment, it's meaningless.

Comment thread pkg/controller_helper/backendruntime.go Outdated
if !exists {
return nil, fmt.Errorf("missing flag or the flag has format error: %s", flag)
}
// replace
Copy link
Copy Markdown
Member

@kerthcet kerthcet Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also remove this comment, I think the code is clear enough.

@kerthcet
Copy link
Copy Markdown
Member

kerthcet commented Nov 6, 2024

If you finished the work, feel free to Ping me, not to rush you, just a friendly reminder. 😄

@qinguoyi
Copy link
Copy Markdown
Member Author

qinguoyi commented Nov 6, 2024

If you finished the work, feel free to Ping me, not to rush you, just a friendly reminder. 😄

Thankd for your kind reply.
I am sorry for late reply. I haven't finished this yet, still doing e2e unit test work. I will finish this work as soon as possible in this week

@qinguoyi
Copy link
Copy Markdown
Member Author

qinguoyi commented Nov 10, 2024

If you finished the work, feel free to Ping me, not to rush you, just a friendly reminder. 😄

I have completed the e2e ollama test, please review the code again @kerthcet

kerthcet
kerthcet reviewed Nov 11, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

Please squash the commits.

@qinguoyi
Copy link
Copy Markdown
Member Author

qinguoyi commented Nov 11, 2024

/approve

Please squash the commits.

kind ping @kerthcet, I've squash the commits

@kerthcet
Copy link
Copy Markdown
Member

kerthcet commented Nov 11, 2024

/lgtm
Thanks!

@kerthcet
Copy link
Copy Markdown
Member

kerthcet commented Nov 11, 2024

/kind feature

@InftyAI-Agent InftyAI-Agent added the lgtm Looks good to me, indicates that a PR is ready to be merged. label Nov 11, 2024
@InftyAI-Agent InftyAI-Agent added feature Categorizes issue or PR as related to a new feature. and removed do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 11, 2024
@kerthcet
Copy link
Copy Markdown
Member

kerthcet commented Nov 11, 2024

/approve

@InftyAI-Agent InftyAI-Agent added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 11, 2024
@InftyAI-Agent InftyAI-Agent merged commit 70f9da0 into InftyAI:main Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kerthcet kerthcet kerthcet left review comments

Assignees

@kerthcet kerthcet

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. feature Categorizes issue or PR as related to a new feature. lgtm Looks good to me, indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qinguoyi @kerthcet @InftyAI-Agent