Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support ollama #193

Merged
merged 1 commit into from
Nov 11, 2024
Merged

Support ollama #193

merged 1 commit into from
Nov 11, 2024

Conversation

qinguoyi
Copy link
Member

@qinguoyi qinguoyi commented Nov 3, 2024 •
edited
Loading

What this PR does / why we need it

Support ollama

Which issue(s) this PR fixes

#91

Special notes for your reviewer

  1. add OLLAMA protocol, when access modelName, there return model.Address and not inject init container
  2. execute multiple shell commands instead of just ollama run；because needs to start first and make sure the ollama service is started
  3. upgrader parse flags impl, there will be a replacement rather than a complete override

Does this PR introduce a user-facing change?

Support ollama

@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 3, 2024
@qinguoyi qinguoyi mentioned this pull request Nov 3, 2024
Closed
3 tasks
@qinguoyi qinguoyi force-pushed the support-ollama branch 2 times, most recently from db49d51 to 05078e4 Compare November 3, 2024 11:00
qinguoyi
qinguoyi commented Nov 3, 2024
View reviewed changes
chart/templates/backends/ollama.yaml
while true;do sleep 60;done"
envs:
- name: OLLAMA_HOST
value: 0.0.0.0:8080
Copy link
Member Author

@qinguoyi qinguoyi Nov 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OLLAMA_HOST can expose custom port

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great

qinguoyi
qinguoyi commented Nov 3, 2024
View reviewed changes
pkg/controller_helper/backendruntime.go Outdated
}
// replace
value = strings.Replace(value, match[0], replacement, -1)
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace instaned of override

kerthcet
kerthcet reviewed Nov 4, 2024
View reviewed changes
Copy link
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM, have you tested locally? Maybe we can have a e2e test because ollama can still run with CPUs.

chart/templates/backends/ollama.yaml
args:
- name: default
flags:
- "ollama serve &
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to support readiness and liveness next, see #21, but this is ok for now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's support probe in the future

chart/templates/backends/ollama.yaml
while true;do sleep 60;done"
envs:
- name: OLLAMA_HOST
value: 0.0.0.0:8080
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great

docs/examples/ollama/model.yaml Outdated
spec:
familyName: qwen2
source:
uri: OLLAMA://qwen2:0.5b
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use lower case for consistency, just like http, although we'll convert to upper case for comparison in the runtime.

pkg/controller_helper/model_source/uri.go
modelPath string
modelName string
protocol string
bucket string
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need to refactor this part in the future, divide the URIProvider to more specified ones, but not a hurry.

@qinguoyi
Copy link
Member Author

qinguoyi commented Nov 4, 2024

Generally LGTM, have you tested locally? Maybe we can have a e2e test because ollama can still run with CPUs.

Thanks for your review, I've done multiple tests locally before committing.
Another, a complete e2e test is very necessary and I will complete it as soon as possible.

kerthcet
kerthcet reviewed Nov 5, 2024
View reviewed changes
Copy link
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nits.

pkg/controller_helper/backendruntime.go Outdated
value := flag
matches := re.FindAllStringSubmatch(flag, -1)

if len(matches) > 0 {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's refactor like this, to reduce the nested blocks.

if len(matches) == 0 {
   return
}

pkg/controller_helper/backendruntime.go Outdated
if len(matches) > 0 {
for _, match := range matches {
if len(match) > 1 {
// get key
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this comment, it's meaningless.

pkg/controller_helper/backendruntime.go Outdated
if !exists {
return nil, fmt.Errorf("missing flag or the flag has format error: %s", flag)
}
// replace
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also remove this comment, I think the code is clear enough.

@kerthcet
Copy link
Member

kerthcet commented Nov 6, 2024

If you finished the work, feel free to Ping me, not to rush you, just a friendly reminder. 😄

@qinguoyi
Copy link
Member Author

qinguoyi commented Nov 6, 2024

If you finished the work, feel free to Ping me, not to rush you, just a friendly reminder. 😄

Thankd for your kind reply.
I am sorry for late reply. I haven't finished this yet, still doing e2e unit test work. I will finish this work as soon as possible in this week

@qinguoyi
Copy link
Member Author

If you finished the work, feel free to Ping me, not to rush you, just a friendly reminder. 😄

I have completed the e2e ollama test, please review the code again @kerthcet

kerthcet
kerthcet reviewed Nov 11, 2024
View reviewed changes
Copy link
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

Please squash the commits.

@qinguoyi
Copy link
Member Author

/approve

Please squash the commits.

kind ping @kerthcet, I've squash the commits

@kerthcet
Copy link
Member

/lgtm
Thanks!

@kerthcet
Copy link
Member

/kind feature

@InftyAI-Agent InftyAI-Agent added the lgtm Looks good to me, indicates that a PR is ready to be merged. label Nov 11, 2024
@InftyAI-Agent InftyAI-Agent added feature Categorizes issue or PR as related to a new feature. and removed do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Nov 11, 2024
@kerthcet
Copy link
Member

/approve

@InftyAI-Agent InftyAI-Agent added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 11, 2024
@InftyAI-Agent InftyAI-Agent merged commit 70f9da0 into InftyAI:main Nov 11, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kerthcet kerthcet kerthcet left review comments

Assignees

@kerthcet kerthcet

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. feature Categorizes issue or PR as related to a new feature. lgtm Looks good to me, indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qinguoyi @kerthcet @InftyAI-Agent