diff --git a/docs/system/Metering.md b/docs/system/Metering.md new file mode 100644 index 0000000..9bc6440 --- /dev/null +++ b/docs/system/Metering.md @@ -0,0 +1,15 @@ +# Metering + +## Purpose + +LLMs and Embedding models often are billed resources, so we want to ensure usage is limited and capped as desired. + +## Interface + +Creates a global llm usage collection + +## Key Details + +- Ensure accurate and real-time metrics collection. +- Provide actionable insights based on collected metrics. +- Maintain data privacy and security. diff --git a/docs/system/chat/ConversationalUI.md b/docs/system/chat/ConversationalUI.md new file mode 100644 index 0000000..e03db00 --- /dev/null +++ b/docs/system/chat/ConversationalUI.md @@ -0,0 +1,13 @@ +# Conversational UI + +## Purpose + +Provide a conversational chat UI with message history and non-text display which can be embedded or accessed on a full page. + +## Interface + +- Access via FAB +- Dedicated Page + - Include conversation history +- Messages should support both markdown and cards + diff --git a/docs/system/chat/LLMs.md b/docs/system/chat/LLMs.md new file mode 100644 index 0000000..dd2f7bf --- /dev/null +++ b/docs/system/chat/LLMs.md @@ -0,0 +1,15 @@ +# Large Language Models (LLMs) + +## Purpose + +Provide a reusable set of LLMs that can be used by the system easily for the right tasks. It should be easy to provide custom LLMs to the interface, and use local models in development. + +## Access + +LLMs should be accessible under custom for usage, subject to [Metering](../Metering.md) limits + + +## Built in models + +- ollama +- openai \ No newline at end of file diff --git a/docs/system/chat/MessageCollection.md b/docs/system/chat/MessageCollection.md new file mode 100644 index 0000000..a140712 --- /dev/null +++ b/docs/system/chat/MessageCollection.md @@ -0,0 +1,36 @@ +# Chat Message Collection + +## Purpose + +Stores each message in a conversation, making them available to use as context and display to the user + +## Interface + +```mermaid +erDiagram + USER{ + string id + string name +} +MESSAGE { + string id + string content + string role + string conversationID + relation user + relationships references + timestamp sent + } + + + MESSAGE ||--o{ USER : user + +``` + +## Key Details + +- Collection Will be added to the payload configuration during plugin initialization. +- Messages Will be readable only by the user who created it +- Messages Supports roles: 'assistant' and 'user' and 'tools +- References field can store links to relevant documents used +- Timestamp allows for chronological ordering of messages diff --git a/docs/system/chat/VectorSearchIntegration.md b/docs/system/chat/VectorSearchIntegration.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/system/semantic-search/EmbeddingModel.md b/docs/system/semantic-search/EmbeddingModel.md new file mode 100644 index 0000000..f56bf46 --- /dev/null +++ b/docs/system/semantic-search/EmbeddingModel.md @@ -0,0 +1,32 @@ +# Embedding Model + +## Purpose + +Convert chunks of text into comparable vectors for indexing and semantic serach. + +## Integration + +Integrated with the VectorStore to automatically embed queries/index documents as they are added. Provided in the form of a class + +```ts + +class CustomEmbeddingModel extends EmbeddingModel { + + private model = new ModelClient(); + + generateEmbeddings(texts: string[]): number[] { + return this.model.embed(text); + } + + numberOfDimensions() { + return 768; + } + +} + +``` + +## Included Integrations + +- Ollama +- OpenAI diff --git a/docs/system/semantic-search/IndexableFields.md b/docs/system/semantic-search/IndexableFields.md new file mode 100644 index 0000000..ccf0261 --- /dev/null +++ b/docs/system/semantic-search/IndexableFields.md @@ -0,0 +1,40 @@ +# Indexable Fields + +## Purpose + +Easily define what to index and how to query it using Payload Fields. This enables using existing fields or creating hidden virtual ones. Indexes are partitioned for performance and automatically integrate via collection hooks. + +## Interface + +```mermaid +flowchart + + subgraph Collection + direction TB + create[Create] --> change[After Change Hook ] + update[Update] --> change + delete[Delete] --> change + + end + + search[Search] + + subgraph VectorStore + direction TB + embedding[Embedding Model] + persist[Persistance] + + embedding <--> persist + end + search --> embedding + change --> embedding + + +``` + +## Key Details + +- Configure any field by name `collection.field` +- Backfills existing fields +- Updates only when field changes + diff --git a/docs/system/semantic-search/RelatedDocumentsField.md b/docs/system/semantic-search/RelatedDocumentsField.md new file mode 100644 index 0000000..fd72559 --- /dev/null +++ b/docs/system/semantic-search/RelatedDocumentsField.md @@ -0,0 +1,9 @@ +# Related Documents Field + +## Purpose + +Provide a Custom Field type that automatically suggests and links related documents based on semantic similarity. + +## Interface + +Configurable based on top X closest, or within X distance diff --git a/docs/system/semantic-search/SearchInterface.md b/docs/system/semantic-search/SearchInterface.md new file mode 100644 index 0000000..46f3b6f --- /dev/null +++ b/docs/system/semantic-search/SearchInterface.md @@ -0,0 +1,6 @@ +# Search Interface + +## Purpose + +Provide a search UI Element that hooks into semantic search. + diff --git a/docs/system/semantic-search/VectorStore.md b/docs/system/semantic-search/VectorStore.md new file mode 100644 index 0000000..e9f7775 --- /dev/null +++ b/docs/system/semantic-search/VectorStore.md @@ -0,0 +1,37 @@ +# Vector Store + +## Purpose + +Efficiently store, index, and query vector representations of content for semantic search and similarity comparisons. + + +## Identifier + +```ts + +type Identifier = { + documentID: string + collection: string + field: string +} + +``` + +## Class Integration + +```ts + +class MyVectorStore extends VectorDB{ + + upsert:(id: Identifier, value:string) => Promise + delete:(id: Identifier) => Promise + search:(filter: Partial, query: string) => Promise + +} + +``` + +## Included Implementation + +LanceDB + diff --git a/docs/use-cases/documentation-chatbot.md b/docs/use-cases/documentation-chatbot.md new file mode 100644 index 0000000..c37e973 --- /dev/null +++ b/docs/use-cases/documentation-chatbot.md @@ -0,0 +1,5 @@ +# Documentation chatbot + +Give your users a friendly chatbot that's integrated with you CMS managed documentation, always up to date. + +## Analyze questions users ask diff --git a/docs/use-cases/draft-new-documents.md b/docs/use-cases/draft-new-documents.md new file mode 100644 index 0000000..e931c26 --- /dev/null +++ b/docs/use-cases/draft-new-documents.md @@ -0,0 +1,9 @@ +# Draft new documents + +## Prefill fields on create from prompt + +- Use previous documents as few shot training + +## Backfill new fields on existing documents + +## Compare to other docs and alert on thigns diff --git a/docs/use-cases/improve-seo.md b/docs/use-cases/improve-seo.md new file mode 100644 index 0000000..af846a5 --- /dev/null +++ b/docs/use-cases/improve-seo.md @@ -0,0 +1,7 @@ +# IImprove SEO + +Find gaps and opportunities for additional content, or where to merge content. + +## Visualize Posts + +## Compare with AHREFs diff --git a/docs/use-cases/improved-search.md b/docs/use-cases/improved-search.md new file mode 100644 index 0000000..f9ddbed --- /dev/null +++ b/docs/use-cases/improved-search.md @@ -0,0 +1 @@ +# Improve Search diff --git a/docs/use-cases/personalized-email-graphics.md b/docs/use-cases/personalized-email-graphics.md new file mode 100644 index 0000000..677bfe1 --- /dev/null +++ b/docs/use-cases/personalized-email-graphics.md @@ -0,0 +1 @@ +# Personalized email graphics diff --git a/docs/use-cases/personalized-recommendations.md b/docs/use-cases/personalized-recommendations.md new file mode 100644 index 0000000..62ce390 --- /dev/null +++ b/docs/use-cases/personalized-recommendations.md @@ -0,0 +1,5 @@ +# Personalized Recommendations + +## Based on whats in your cart + +## Based on questions asked diff --git a/docs/use-cases/related-posts.md b/docs/use-cases/related-posts.md new file mode 100644 index 0000000..87ea23f --- /dev/null +++ b/docs/use-cases/related-posts.md @@ -0,0 +1,3 @@ +# Related Posts + +Use embeddings to recommend related posts, products, etc.