Implement conversation context and streaming with OllamaSharp #310

kborowinski · 2024-11-21T15:47:56Z

PR Summary

This PR refactors the AIShell.Ollama.Agent to use the OllamaSharp library, enhancing functionality and maintainability. Key updates include support for conversation context, streaming, improved settings management, and better error handling.

Important:
If you choose this PR, then close Implement settings file for Ollama agent

PR Context

This update improves the AIShell.Ollama.Agent by:

Switching to OllamaSharp:
- Simplifies API calls with OllamaApiClient.
- Supports both streaming and non-streaming responses.
Introducing Context Support:
- Enables the agent to remember previous responses, improving conversational flow.
Enhancing Configuration Management:
- Manages settings via ollama.config.json with real-time updates using FileSystemWatcher.
- Provides a sample configuration file to simplify setup.
Improving Chat Interaction:
- Adds robust error handling, especially when prerequisites (like the Ollama server) are not met.
- Validates settings (e.g., Model and Endpoint) to ensure proper functionality.

This PR also aligns with prerequisites outlined in issue #155 and improves the user experience with clearer feedback and enhanced features.

…ue to Ollama misconfiguration

…nd port; Precise warning message

…sible null exception

…t streaming

daxian-dbw

@kborowinski Thanks for making improvements to the ollama agent!

daxian-dbw · 2024-11-22T21:50:10Z

shell/agents/AIShell.Ollama.Agent/AIShell.Ollama.Agent.csproj

+	<SuppressNETCoreSdkPreviewMessage>true</SuppressNETCoreSdkPreviewMessage>
+	<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>


Suggested change

<SuppressNETCoreSdkPreviewMessage>true</SuppressNETCoreSdkPreviewMessage>

<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>

<SuppressNETCoreSdkPreviewMessage>true</SuppressNETCoreSdkPreviewMessage>

<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>

daxian-dbw · 2024-11-22T21:50:39Z

shell/agents/AIShell.Ollama.Agent/AIShell.Ollama.Agent.csproj

+    <PackageReference Include="OllamaSharp" Version="4.0.7" />
+  </ItemGroup>
+
+	<ItemGroup>


Suggested change

<ItemGroup>

<ItemGroup>

daxian-dbw · 2024-11-22T21:51:24Z

shell/agents/AIShell.Ollama.Agent/OllamaAgent.cs

        if (Process.GetProcessesByName("ollama").Length is 0)
        {
-            host.RenderFullResponse("Please be sure the Ollama is installed and server is running. Check all the prerequisites in the README of this agent are met.");
+            host.MarkupWarningLine($"[[{Name}]]: Please be sure the Ollama is installed and server is running. Check all the prerequisites in the README of this agent are met.");


Suggested change

host.MarkupWarningLine($"[[{Name}]]: Please be sure the Ollama is installed and server is running. Check all the prerequisites in the README of this agent are met.");

host.WriteErrorLine($"[{Name}]: Please be sure the Ollama is installed and server is running. Check all the prerequisites in the README of this agent are met.");

Also, given that the prompt will be @ollama when the agent is in use, maybe we don't need to include the [{Name}] part in the error message.

daxian-dbw · 2024-11-22T21:52:39Z

shell/nuget.config

 <configuration>
  <packageSources>
    <clear />
    <add key="PowerShell_PublicPackages" value="https://pkgs.dev.azure.com/powershell/PowerShell/_packaging/powershell/nuget/v3/index.json" />
+    <add key="NuGet" value="https://api.nuget.org/v3/index.json" />


I have saved the OllamaSharp package to the MS feed (we're required to use it only), so you can revert the changes in this file.

daxian-dbw · 2024-11-22T21:55:12Z

shell/agents/AIShell.Ollama.Agent/OllamaAgent.cs

+        string SampleContent = $$"""
+        {
+            // To use Ollama API service:
+            // 1. Install Ollama:
+            //      winget install Ollama.Ollama
+            // 2. Start Ollama API server:
+            //      ollama serve
+            // 3. Install Ollama model:
+            //      ollama pull phi3
+
+            // Declare Ollama model
+            "Model": "phi3",
+            // Declare Ollama endpoint
+            "Endpoint": "http://localhost:11434",
+            // Enable Ollama streaming
+            "Stream": false
+        }
+        """;


Suggested change

string SampleContent = $$"""

{

// To use Ollama API service:

// 1. Install Ollama:

// winget install Ollama.Ollama

// 2. Start Ollama API server:

// ollama serve

// 3. Install Ollama model:

// ollama pull phi3

// Declare Ollama model

"Model": "phi3",

// Declare Ollama endpoint

"Endpoint": "http://localhost:11434",

// Enable Ollama streaming

"Stream": false

}

""";

string SampleContent = """

{

// To use Ollama API service:

// 1. Install Ollama: `winget install Ollama.Ollama`

// 2. Start Ollama API server: `ollama serve`

// 3. Install Ollama model: `ollama pull phi3`

// Declare Ollama model

"Model": "phi3",

// Declare Ollama endpoint

"Endpoint": "http://localhost:11434",

// Enable Ollama streaming

"Stream": false

}

""";

daxian-dbw · 2024-11-22T22:35:01Z

shell/agents/AIShell.Ollama.Agent/OllamaAgent.cs

+            host.WriteErrorLine($"[{Name}]: {e.Message}");
+            host.WriteErrorLine($"[{Name}]: Selected Model    : \"{_settings.Model}\"");
+            host.WriteErrorLine($"[{Name}]: Selected Endpoint : \"{_settings.Endpoint}\"");
+            host.WriteErrorLine($"[{Name}]: Configuration File: \"{SettingFile}\"");


Nitpick: when using the ollama agent, the prompt will be @ollama, so maybe we don't need to add the [{Name}]: part in the error message.

daxian-dbw · 2024-11-22T22:46:36Z

shell/agents/AIShell.Ollama.Agent/OllamaAgent.cs

+        _request = new GenerateRequest()
+        {
+            Model = _settings.Model,
+            Stream = _settings.Stream
+        };


Given that we will assign Model and Stream per query, maybe we just do the following

Suggested change

_request = new GenerateRequest()

{

Model = _settings.Model,

Stream = _settings.Stream

};

_request = new GenerateRequest();

daxian-dbw · 2024-11-22T22:51:25Z

shell/agents/AIShell.Ollama.Agent/Settings.cs

+    private string _model;
+    private string _endpoint;
+    private bool _stream;


You can consider removing those fields. You can make the properties read-only (e.g. public string Model { get; }), and initialize them in the constructor.

daxian-dbw · 2024-11-22T23:03:39Z

shell/agents/AIShell.Ollama.Agent/OllamaAgent.cs

+                GenerateDoneResponseStream ollamaLastStream = null;
+
+                // Directly process the stream when no spinner is needed
+                await foreach (var ollamaStream in _client.GenerateAsync(_request, token))


In my experiments, there is several-seconds delay before the streaming starts, so we should still use the spinner for the call _client.GenerateAsync(_request, token) to make it more user friendly.

daxian-dbw · 2024-11-22T23:23:03Z

shell/agents/AIShell.Ollama.Agent/OllamaAgent.cs

+                }
+
+                // Update request context
+                _request.Context = ollamaLastStream.Context;


When pressing Ctrl+c while streaming is on, it's possible that the await foreach stops without throwing an exception, and then execution moves to this line, which will fail because ollamaLastStream is null.

kborowinski and others added 11 commits November 19, 2024 12:14

Implement settings file for Ollama agent

24b9e88

Merge branch 'main' into ollama-settings

f5c0ee4

Update readme and links to the documentation; Handle http exception d…

cdcc98e

…ue to Ollama misconfiguration

Implement config file watcher to reload settings on change

c0f8389

Do not force user to specify complete endpoint URL, just IP address a…

0899ddb

…nd port; Precise warning message

Change json to config so comments are not rendered on GitHub in red

bcc5d1e

Revert to json and update the comments in the config file

9c358b8

Merge branch 'PowerShell:main' into ollama-settings

336a3c3

Add model and endpoint self check; Change warnings to errors; Fix pos…

9b57754

…sible null exception

Switch to OllamaSharp nuget package to simplify agent code and suppor…

d9d6fac

…t streaming

Add context support

464d633

kborowinski changed the title ~~Implement settings file for Ollama agent~~ Implement conversation context and streaming with OllamaSharp Nov 21, 2024

kborowinski mentioned this pull request Nov 21, 2024

Implement settings file for Ollama agent #307

Closed

Reset context on refresh

fe68d94

daxian-dbw reviewed Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement conversation context and streaming with OllamaSharp #310

Implement conversation context and streaming with OllamaSharp #310

kborowinski commented Nov 21, 2024 •

edited

Loading

daxian-dbw left a comment

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

daxian-dbw Nov 22, 2024

		<SuppressNETCoreSdkPreviewMessage>true</SuppressNETCoreSdkPreviewMessage>
		<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>

	host.MarkupWarningLine($"[[{Name}]]: Please be sure the Ollama is installed and server is running. Check all the prerequisites in the README of this agent are met.");
	host.WriteErrorLine($"[{Name}]: Please be sure the Ollama is installed and server is running. Check all the prerequisites in the README of this agent are met.");

Implement conversation context and streaming with OllamaSharp #310

Are you sure you want to change the base?

Implement conversation context and streaming with OllamaSharp #310

Conversation

kborowinski commented Nov 21, 2024 • edited Loading

PR Summary

PR Context

daxian-dbw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kborowinski commented Nov 21, 2024 •

edited

Loading