diff --git a/INTERPRETER_MODULES_DOCUMENTATION.md b/INTERPRETER_MODULES_DOCUMENTATION.md new file mode 100644 index 00000000..67524003 --- /dev/null +++ b/INTERPRETER_MODULES_DOCUMENTATION.md @@ -0,0 +1,16 @@ +### Common Characteristics + +All interpreter modules share the following characteristics: +1. They implement the `Interpreter` interface, ensuring a consistent API across different language interpreters. +2. They support the addition of predefined variables, allowing for context to be passed into the executed code. +3. They provide methods for running code, validating syntax, and retrieving language-specific information. +4. They include error handling mechanisms to catch and report execution or compilation errors. + +### Integration with SkyeNet + +These interpreter modules play a crucial role in SkyeNet's multi-language support feature. They allow the AI-powered system to: +1. Execute code snippets in different languages as part of task processing. +2. Validate code syntax before execution, enhancing error handling and user feedback. +3. Integrate language-specific features and libraries into the SkyeNet workflow. + +By providing a unified interface through the `Interpreter` interface, SkyeNet can seamlessly work with multiple programming languages, expanding its capabilities and flexibility in handling diverse coding tasks and applications. \ No newline at end of file diff --git a/README.md b/README.md deleted file mode 100644 index 06d49e17..00000000 --- a/README.md +++ /dev/null @@ -1,166 +0,0 @@ -# SkyeNet - A Helpful Pup! ๐Ÿพโšก - -[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0) - - - -* [SkyeNet - A Helpful Pup! ๐Ÿพโšก](#skyenet---a-helpful-pup-) - * [๐Ÿงช The SkyeNet Experiment](#-the-skyenet-experiment) - * [๐Ÿ—๏ธ The Building Blocks of SkyeNet](#-the-building-blocks-of-skyenet) - * [๐ŸŽจ Customizing Your SkyeNet Experience](#-customizing-your-skyenet-experience) - * [๐Ÿ’ก Example Usage](#-example-usage) - * [๐Ÿ“ฆ To Import](#-to-import) - * [๐ŸŒŸ To Use](#-to-use) - * [๐Ÿš€ Unleashing SkyeNet's Potential](#-unleashing-skyenets-potential) - - - -![SkyeNet](skyenet.svg) - -Greetings, intrepid explorer! You have stumbled upon the delightful realm of SkyeNet, a general-purpose AI assistant -that combines the linguistic prowess of OpenAI ChatGPT with Java's JSR223 scripting engine and Kotlin compiler. SkyeNet -is a unique proof-of-concept designed to bring an interactive, engaging AI experience to life. With the added -capabilities of OpenAI Whisper and Google Cloud Platform, SkyeNet lends its voice recognition and text-to-speech talents -to users brave enough to embark on this journey. - -## ๐Ÿงช The SkyeNet Experiment - -At its core, SkyeNet seeks to interpret natural language commands and execute them as JavaScript code. The assistant is -capable of understanding a wide variety of commands, making it a versatile and powerful tool for users seeking an -interactive AI companion. SkyeNet's components can be customized to meet the needs of different applications or to -create a unique and engaging user experience. - -![UI Screenshots](ui_screenshot.png) - -## ๐Ÿ—๏ธ The Building Blocks of SkyeNet - -SkyeNet's components work together harmoniously to create a cohesive and lively AI assistant: - -* **[Brain](src/main/kotlin/com/simiacryptus/skyenet/Brain.kt)**: The nucleus of SkyeNet's intellect, the brain - interfaces with the OpenAI API and uses ChatGPT to perform coding tasks, giving life to our endearing AI assistant. -* **[Body](src/main/kotlin/com/simiacryptus/skyenet/Body.kt)**: Acting as the AI's vessel, the body connects the heart - and brain, providing a framework for action and allowing SkyeNet to interact with its environment. -* **[Ears](src/main/kotlin/com/simiacryptus/skyenet/Ears.kt)**: These auditory organs capture sounds and transform them - into actionable data, granting SkyeNet the gift of hearing. -* **[Mouth](src/main/kotlin/com/simiacryptus/skyenet/Mouth.kt)**: Bestowed with the ability to speak, SkyeNet's mouth - interfaces with the Google Text-to-Speech API, enabling it to communicate with users through speech. -* **[Head](src/main/kotlin/com/simiacryptus/skyenet/Head.kt)**: The command center of the AI assistant, the head is - responsible for capturing audio through the ears, interacting with the user via the face, and relaying commands to the - body. -* **[Face](src/main/kotlin/com/simiacryptus/skyenet/Face.kt)**: The face is the graphical user interface, allowing - SkyeNet to communicate with users visually and provide input and output in a user-friendly manner. -* **[Heart](src/main/kotlin/com/simiacryptus/skyenet/Heart.kt)**: The heart is the interface to the script evaluation - engines that power SkyeNet, pumping life into the body and allowing the AI assistant to perform its various functions. - -## ๐ŸŽจ Customizing Your SkyeNet Experience - -SkyeNet's ability to interpret natural language commands and execute them as JavaScript code allows you to -extend its functionality and create custom commands specific to your needs. This level of customization opens up a world -of possibilities for users to explore, making SkyeNet a versatile and powerful tool that can be adapted to a wide -variety of situations and tasks. - -Key to this customization is the ability to define custom tools that can be used in commands. These tools can be -anything from simple functions to complex classes, and can be used to perform a wide variety of tasks. For example, a -tool could be used to perform a calculation, retrieve data from an API, or even call back into GPT via a JoePenai proxy. - -## ๐Ÿ’ก Example Usage - -### ๐Ÿ“ฆ To Import - -https://mvnrepository.com/artifact/com.simiacryptus/skyenet - -Maven: - -```xml - - - com.simiacryptus - skyenet-webui - 1.1.7 - -``` - -Gradle: - -```groovy -implementation group: 'com.simiacryptus', name: 'skyenet', version: '1.1.7' -``` - -```kotlin -implementation("com.simiacryptus:skyenet:1.1.7") -``` - -### ๐ŸŒŸ To Use - -```kotlin -// Define OpenAI client -val body = Body( - api = OpenAIClient(apiKey), - apiObjects = mapOf( - "toolObj" to TestTools(googleSpeechKey) - ) -) -val head = Head(body = body, ears = Ears(api = OpenAIClient(apiKey))) -val jFrame = head.start(client = OpenAIClient(apiKey)) -while (jFrame.isVisible) { - Thread.sleep(100) -} - -class TestTools(keyfile: String) { - // Private details will not be exported - private val mouth = Mouth(keyfile) - - // Export methods to be called from the script - @Export - fun speak(text: String) = mouth.speak(text) -} -``` - -## ๐Ÿš€ Unleashing SkyeNet's Potential - -SkyeNet's unique combination of natural language understanding, voice recognition, text-to-speech, and JavaScript -execution capabilities opens up endless possibilities for users to harness the power of AI in creative and innovative -ways. From automating mundane tasks to developing complex applications, SkyeNet's versatility makes it an invaluable -assistant for both novice and experienced developers alike. - -The possibilities are virtually endless, limited only by your imagination and ingenuity. As you embark on your SkyeNet -adventure, you'll find that the AI assistant is not only a versatile and powerful tool but also a delightful and -engaging companion that can help you push the boundaries of what's possible. - -So, brave adventurer, are you ready to unleash the power of SkyeNet and embark on an extraordinary journey into the -world of AI? With its anthropomorphic components, humorous naming, and lively personality, SkyeNet is sure to capture -your heart and inspire your creativity. Don your lab coat, channel your inner Dr. Frankenstein, and prepare to awaken -the adorably powerful AI assistant that is SkyeNet! - -# Disclaimer - -This project is **_NOT_** affiliated with: - -- OpenAI -- Google -- The Terminator Movie Franchise -- Paw Patrol -- Nickelodeon -- Paramount Pictures -- https://skyenet.tech/ -- https://www.skyenet.co.uk/ -- Skyenet wireless communications -- https://www.skye-nets.com/ -- https://skyenet.social/ -- https://www.facebook.com/Skyenetpro/ -- https://eagleskyenet.com/ -- https://www.instagram.com/skyenet/ -- https://www.pinterest.com/Skyenetblog/ -- SKYENET LIMITED UK Company number 12770700 -- Skyenet Internet Solutions in Youngsville PA -- Skyenet - Email Format & Email Checker Service -- SKYENET INVESTMENTS PTY LTD -- Skynet Worldwide Express -- Skyenet Computer Services -- SkyeNet Solutions, LLC -- SKYEnet.live -- E Skyenet Ltd, Rotherham -- SkyeNet Investment Nig. Ltd -- SKYENET GLOBAL, LLC in Sugar Land, TX -- Any similarly named personas on any other social network -- etc diff --git a/core/build.gradle.kts b/core/build.gradle.kts index b3c4d640..44df008a 100644 --- a/core/build.gradle.kts +++ b/core/build.gradle.kts @@ -33,7 +33,7 @@ val hsqldb_version = "2.7.2" dependencies { - implementation(group = "com.simiacryptus", name = "jo-penai", version = "1.1.11") + implementation(group = "com.simiacryptus", name = "jo-penai", version = "1.1.12") implementation(group = "org.hsqldb", name = "hsqldb", version = hsqldb_version) implementation("org.apache.commons:commons-text:1.11.0") diff --git a/core/src/main/dev_documentation.md b/core/src/main/dev_documentation.md deleted file mode 100644 index f708f1e2..00000000 --- a/core/src/main/dev_documentation.md +++ /dev/null @@ -1,3600 +0,0 @@ -# kotlin\com\simiacryptus\skyenet\core\actors\BaseActor.kt - -## Developer Documentation for BaseActor Class - -The `BaseActor` class serves as an abstract base for creating actors that interact with OpenAI's API to generate responses based on input. This class is designed to be extended for -various types of actors that require different input types (`I`) and return types (`R`). - -### Class Overview - -```mermaid -classDiagram - class BaseActor { - +String prompt - +String? name - +ChatModels model - +Double temperature - +respond(input: I, api: API, messages: ApiModel.ChatMessage) R - +response(input: ApiModel.ChatMessage, model: OpenAIModel, api: API) ChatResponse - +answer(input: I, api: API) R - +chatMessages(questions: I) Array - +withModel(model: ChatModels) BaseActor - } -``` - -#### Properties - -- `prompt`: The initial prompt or context used for generating responses. -- `name`: An optional name for the actor, which can be used for identification or logging purposes. -- `model`: The OpenAI model to be used for generating responses. This is specified using the `ChatModels` enum. -- `temperature`: Controls the randomness of the response generation. A lower value makes the model more deterministic. - -#### Methods - -##### `abstract fun respond(input: I, api: API, vararg messages: ApiModel.ChatMessage): R` - -This abstract method must be implemented by subclasses to define how the actor responds to the given input and messages. It uses the provided API instance to interact with OpenAI's -services. - -##### `open fun response(vararg input: ApiModel.ChatMessage, model: OpenAIModel = this.model, api: API): ChatResponse` - -Generates a response using the OpenAI API based on the provided chat messages. It allows overriding the model used for this specific response. - -##### `open fun answer(input: I, api: API): R` - -A high-level method that generates a response based on the input. It internally calls `chatMessages` to convert the input into chat messages and then `respond` to generate the -final response. - -##### `abstract fun chatMessages(questions: I): Array` - -Converts the input into an array of `ApiModel.ChatMessage` instances. This method must be implemented by subclasses to define how input is transformed into a format suitable for -the OpenAI API. - -##### `abstract fun withModel(model: ChatModels): BaseActor` - -Creates a new instance of the actor with the specified model. This method allows changing the model used for response generation. - -### Usage - -To use the `BaseActor` class, you must extend it and implement the abstract methods. Here's a simplified example: - -```kotlin -class TextActor(prompt: String, model: ChatModels) : BaseActor(prompt, model = model) { - override fun respond(input: String, api: API, vararg messages: ApiModel.ChatMessage): String { - // Implementation for generating a response based on the input and messages - } - - override fun chatMessages(questions: String): Array { - // Convert the input string into chat messages - } - - override fun withModel(model: ChatModels): BaseActor { - // Return a new instance of TextActor with the specified model - } -} -``` - -This class is designed to be flexible and extensible for various types of interactions with OpenAI's API, allowing developers to create sophisticated actors for different -applications. - -# kotlin\com\simiacryptus\skyenet\core\actors\ActorSystem.kt - -## ActorSystem Class Documentation - -The `ActorSystem` class is a core component of the application that manages the lifecycle and interactions of various actors within the system. It is designed to be generic, -supporting different types of actors identified by an enum (`T`). This class is responsible for initializing actors, managing their state, and providing a mechanism for actor -interaction through interceptors. - -### Overview - -The `ActorSystem` class is structured to facilitate the management of actors in a session-based context, with support for user-specific data storage and session management. It -leverages a pool of resources and a set of interceptors to enhance or modify the behavior of actors at runtime. - -### Class Diagram - -```mermaid -classDiagram - class ActorSystem { - -actors: Map - -dataStorage: StorageInterface - -user: User? - -session: Session - -sessionDir: File - -pool: ResourcePool - -actorMap: MutableMap - -wrapperMap: MutableMap - +getActor(T): BaseActor - -getWrapper(String): FunctionWrapper - } - ActorSystem --> "1" BaseActor: manages - ActorSystem --> "1" StorageInterface: uses - ActorSystem --> "1" User: optional - ActorSystem --> "1" Session: uses - ActorSystem --> "1" FunctionWrapper: creates -``` - -### Key Components - -- **actors**: A map of enum keys to `BaseActor` instances. This map defines the available actors within the system. -- **dataStorage**: An interface to the storage system, allowing actors to persist and retrieve data. -- **user**: An optional `User` instance representing the current user. This can be `null` if the actor system is used in a context without a specific user. -- **session**: The current `Session` instance, providing context for the actor operations. -- **sessionDir**: A `File` instance pointing to the directory where session-specific data is stored. -- **pool**: A lazy-initialized resource pool used by actors for executing tasks. - -### Methods - -#### getActor(actor: T): BaseActor<*, *> - -Retrieves an instance of `BaseActor` for the specified actor enum. If the actor instance does not exist in the `actorMap`, it is created using the corresponding entry in the -`actors` map. The creation process involves wrapping the actor with an appropriate interceptor based on its type (e.g., `SimpleActorInterceptor`, `ParsedActorInterceptor`). - -#### getWrapper(name: String): FunctionWrapper - -Retrieves or creates a `FunctionWrapper` instance for the specified actor name. This wrapper is used to intercept function calls within the actor, allowing for additional -processing or logging. The wrapper is associated with a `JsonFunctionRecorder` that persists function call data to a file within the session directory. - -### Usage Example - -```kotlin -// Initialize the actor system with a predefined set of actors, storage interface, user, and session -val actorSystem = ActorSystem(actorsMap, storageInterface, user, session) - -// Retrieve an actor from the system -val myActor = actorSystem.getActor(MyActorEnum.SOME_ACTOR) - -// Use the actor for performing operations -myActor.performAction() -``` - -### Conclusion - -The `ActorSystem` class provides a flexible and extensible framework for managing actors within an application. By abstracting actor initialization and providing support for -function interception, it allows for sophisticated runtime behavior modifications and enhances the application's capabilities to handle complex workflows and data processing tasks. - -# java\com\simiacryptus\skyenet\core\OutputInterceptor.java - -## OutputInterceptor Documentation - -The `OutputInterceptor` class is a utility designed to intercept and redirect standard output (`System.out`) and error output (`System.err`) streams in Java applications. This -functionality is particularly useful for capturing and analyzing console output for logging, debugging, or testing purposes. - -### Overview - -The class provides mechanisms to: - -- Intercept and reroute output from `System.out` and `System.err` to custom `ByteArrayOutputStream` instances. -- Retrieve and clear the captured output on both a per-thread basis and globally across all threads. - -### Setup - -To start intercepting output, call the `setupInterceptor` method. This method reroutes the standard and error output streams to internal mechanisms that capture the output. - -```java -OutputInterceptor.setupInterceptor(); -``` - -### Usage - -#### Capturing Output - -- **Thread-specific Output**: Output written by the current thread can be retrieved and cleared using `getThreadOutput` and `clearThreadOutput` methods, respectively. -- **Global Output**: Output written by all threads can be accessed and cleared using `getGlobalOutput` and `clearGlobalOutput` methods, respectively. - -#### Retrieving Captured Output - -- To get the output captured from the current thread: - - ```java - String threadOutput = OutputInterceptor.getThreadOutput(); - ``` - -- To get the output captured globally across all threads: - - ```java - String globalOutput = OutputInterceptor.getGlobalOutput(); - ``` - -#### Clearing Captured Output - -- To clear the output captured from the current thread: - - ```java - OutputInterceptor.clearThreadOutput(); - ``` - -- To clear the output captured globally: - - ```java - OutputInterceptor.clearGlobalOutput(); - ``` - -### Implementation Details - -#### Class Diagram - -```mermaid -classDiagram - class OutputInterceptor { - -PrintStream originalOut - -PrintStream originalErr - -AtomicBoolean isSetup - -Object globalStreamLock - -ByteArrayOutputStream globalStream - -Map threadLocalBuffer - +setupInterceptor() void - +getThreadOutput() String - +clearThreadOutput() void - +getGlobalOutput() String - +clearGlobalOutput() void - } - class OutputStreamRouter { - -PrintStream originalStream - -int maxGlobalBuffer - -int maxThreadBuffer - +write(int b) void - +write(byte[] b, int off, int len) void - } - OutputInterceptor --|> OutputStreamRouter : routes output -``` - -#### Thread Safety - -The `OutputInterceptor` class is designed to be thread-safe. It uses synchronization on a global lock object (`globalStreamLock`) for operations affecting the global output stream, -and a `WeakHashMap` for thread-local output streams to ensure thread safety and avoid memory leaks. - -#### Memory Management - -To prevent excessive memory usage, the `OutputStreamRouter` class imposes limits on the maximum size of both global and thread-local output buffers. When these limits are exceeded, -the buffers are automatically reset. - -### Conclusion - -The `OutputInterceptor` class provides a powerful and flexible way to intercept, capture, and analyze output from Java applications. Its thread-safe design and memory management -features make it suitable for use in a wide range of scenarios, from development and debugging to production logging and monitoring. - -# kotlin\com\simiacryptus\skyenet\core\actors\CodingActor.kt - -## Developer Documentation for `CodingActor` - -The `CodingActor` class is a sophisticated component designed to facilitate the translation of natural language instructions into executable code, leveraging the capabilities of AI -models. It serves as an intermediary that understands both the user's intent expressed in natural language and the technical requirements to fulfill that intent through code -execution. - -### Overview - -`CodingActor` extends `BaseActor` and specializes in handling code-related requests and responses. It integrates with an interpreter to execute the generated code and provides -mechanisms for automatic error correction and code evaluation. - -#### Key Features - -- **Natural Language to Code Translation**: Translates user instructions into executable code. -- **Dynamic Interpreter Integration**: Utilizes a specified interpreter for code execution. -- **Error Handling and Correction**: Attempts to correct errors in generated code automatically. -- **Customizable Code Formatting**: Supports customizable code formatting guidelines. -- **Extensible Type Description**: Uses a `TypeDescriber` for detailed API descriptions. - -### Class Structure - -```mermaid -classDiagram - BaseActor <|-- CodingActor - CodingActor: +interpreterClass KClass~Interpreter~ - CodingActor: +symbols Map~String, Any~ - CodingActor: +describer TypeDescriber - CodingActor: +details String? - CodingActor: +fallbackModel ChatModels - CodingActor: +runtimeSymbols Map~String, Any~ - CodingActor: +evalFormat Boolean - CodingActor: +language String - CodingActor: +prompt String - CodingActor: +apiDescription String - CodingActor: +execute(prefix String, code String) ExecutionResult - CodingActor: +respond(input CodeRequest, api API, messages ChatMessage) CodeResult - CodingActor: +withModel(model ChatModels) CodingActor - class BaseActor { - <> - +prompt String - +name String? - +model ChatModels - +temperature Double - } - class CodeRequest { - +messages List~Pair~ String, Role~~ - +codePrefix String - +autoEvaluate Boolean - +fixIterations Int - +fixRetries Int - } - class CodeResult~interface~ { - <> - +code String - +status Status - +result ExecutionResult - +renderedResponse String? - } - class ExecutionResult { - +resultValue String - +resultOutput String - } -``` - -### Usage - -#### Initialization - -To create an instance of `CodingActor`, you need to provide: - -- `interpreterClass`: The Kotlin class (`KClass`) of the interpreter to use for code execution. -- `symbols`: A map of predefined symbols that can be used within the code. -- `describer`: An instance of `TypeDescriber` for API description. -- Additional optional parameters such as `details`, `model`, `fallbackModel`, `temperature`, and `runtimeSymbols`. - -#### Processing Requests - -To process a coding request, create an instance of `CodeRequest` with: - -- `messages`: A list of pairs containing the message and its role (`Role.system` or `Role.assistant`). -- Optional parameters like `codePrefix`, `autoEvaluate`, `fixIterations`, and `fixRetries`. - -Call the `respond` method with the `CodeRequest` instance and other required parameters. This method returns an instance of `CodeResult`, which contains the generated code, -execution status, and result. - -#### Execution and Error Handling - -The `execute` method runs the generated code using the specified interpreter and handles errors. If `autoEvaluate` is true, `respond` will automatically attempt to execute and -correct the code. - -### Extending `CodingActor` - -To extend `CodingActor`: - -1. Override necessary methods to customize behavior. -2. Use `withModel` to create a new instance with a different AI model. - -### Conclusion - -`CodingActor` is a powerful tool for bridging the gap between natural language instructions and executable code. By leveraging AI models and an extensible architecture, it offers a -flexible and efficient way to automate coding tasks. - -# kotlin\com\simiacryptus\skyenet\core\actors\ImageActor.kt - -## ImageActor Class Documentation - -The `ImageActor` class is a specialized actor designed to transform user requests into image generation prompts and subsequently generate images that align with the user's -preferences. This class extends the `BaseActor` class and operates within the context of interacting with OpenAI's API for image generation, specifically targeting models like -DALL-E 2. - -### Overview - -The `ImageActor` class is structured to first interpret a textual prompt through a chat model and then use the interpreted prompt to generate an image using an image model. This -process involves several key components and steps, which are detailed below. - -### Class Structure - -#### Constructor Parameters - -- `prompt`: The initial prompt used to guide the text generation process. -- `name`: An optional name for the actor. -- `textModel`: The chat model used for interpreting the user's request. -- `imageModel`: The image model used for generating images. Defaults to `ImageModels.DallE2`. -- `temperature`: Controls the randomness of the output. Lower values make the output more deterministic. -- `width`: The width of the generated image. -- `height`: The height of the generated image. - -#### Methods - -- `chatMessages(questions: List)`: Prepares chat messages for interaction with the text model. -- `render(text: String, api: API)`: Generates an image based on the provided text prompt. -- `respond(input: List, api: API, vararg messages: ChatMessage)`: Generates an `ImageResponse` based on the input questions and chat messages. -- `withModel(model: ChatModels)`: Returns a new instance of `ImageActor` with the specified chat model. - -#### Inner Classes - -- `ImageResponseImpl`: An implementation of the `ImageResponse` interface, encapsulating the text prompt and the generated image. - -### Interfaces - -- `ImageResponse`: An interface representing the response from the `ImageActor`, containing the text prompt and the generated image. - -### Usage Flow - -```mermaid -sequenceDiagram - participant User - participant ImageActor - participant OpenAIClient - participant ImageModel - User ->> ImageActor: Request Image - ImageActor ->> OpenAIClient: Send Text Prompt - OpenAIClient ->> ImageActor: Return Interpreted Prompt - ImageActor ->> ImageModel: Send Interpreted Prompt - ImageModel ->> ImageActor: Return Image URL - ImageActor ->> User: Return Image Response -``` - -### Example Usage - -```kotlin -val imageActor = ImageActor( - prompt = "Transform the user request into an image generation prompt that the user will like", - textModel = OpenAIModels.GPT3_5_Turbo, - imageModel = ImageModels.DallE2, - temperature = 0.3, - width = 1024, - height = 1024 -) - -val api = OpenAIClient("your_api_key") - -val questions = listOf("A futuristic cityscape at sunset") -val imageResponse = imageActor.respond(questions, api) - -// Display or process the imageResponse.image as needed -``` - -### Conclusion - -The `ImageActor` class provides a powerful abstraction for generating images based on textual prompts. By leveraging OpenAI's API and models, it simplifies the process of creating -visually appealing images that meet user specifications. - -# kotlin\com\simiacryptus\skyenet\core\actors\opt\Expectation.kt - -## Developer Documentation: Expectation Module - -The `Expectation` module within the `com.simiacryptus.skyenet.core.actors.opt` package provides a framework for evaluating responses based on predefined criteria. This module is -designed to work with the OpenAI API client, facilitating the embedding and comparison of text responses against specified expectations. It consists of an abstract class -`Expectation` and two concrete implementations: `VectorMatch` and `ContainsMatch`. - -### Overview - -The `Expectation` class serves as the foundation, defining the structure and expectations for its subclasses. Each subclass implements specific logic to match and score responses -based on different criteria: - -- `VectorMatch`: Evaluates responses by comparing the semantic similarity between the example text and the response text. -- `ContainsMatch`: Checks if the response contains a specified pattern, supporting both critical and non-critical matches. - -#### Class Diagram - -```mermaid -classDiagram - class Expectation { - <> - +matches(api: OpenAIClient, response: String): Boolean - +score(api: OpenAIClient, response: String): Double - } - class VectorMatch { - -example: String - -metric: DistanceType - +matches(api: OpenAIClient, response: String): Boolean - +score(api: OpenAIClient, response: String): Double - -createEmbedding(api: OpenAIClient, str: String): Embedding - } - class ContainsMatch { - -pattern: Regex - -critical: Boolean - +matches(api: OpenAIClient, response: String): Boolean - +score(api: OpenAIClient, response: String): Double - -_matches(response: String?): Boolean - } - - Expectation <|-- VectorMatch - Expectation <|-- ContainsMatch -``` - -### Usage - -#### VectorMatch - -`VectorMatch` compares the semantic similarity between an example text and a response text. It uses embeddings to calculate the distance (similarity) between texts, supporting -different distance metrics. - -##### Example Usage - -```kotlin -val vectorMatch = VectorMatch("Example text") -val similarityScore = vectorMatch.score(apiClient, "Response text") -``` - -#### ContainsMatch - -`ContainsMatch` checks if the response contains a specified regex pattern. It can be configured to treat matches as critical or non-critical. - -##### Example Usage - -```kotlin -val containsMatch = ContainsMatch(Regex("pattern"), critical = true) -val matchResult = containsMatch.matches(apiClient, "Response text") -``` - -### Methods - -#### Abstract Methods - -- `matches(api: OpenAIClient, response: String): Boolean`: Determines if the response matches the expectation. -- `score(api: OpenAIClient, response: String): Double`: Calculates a score based on how well the response matches the expectation. - -#### VectorMatch Specific - -- `createEmbedding(api: OpenAIClient, str: String)`: Generates an embedding for the given string using the OpenAI API. - -#### ContainsMatch Specific - -- `_matches(response: String?): Boolean`: Helper method to check for regex pattern matches within the response. - -### Logging - -The module utilizes SLF4J for logging information, such as distance calculations in `VectorMatch` and pattern match failures in `ContainsMatch`. - ---- - -This documentation provides an overview of the `Expectation` module's capabilities and usage within the context of evaluating text responses using the OpenAI API. - -# kotlin\com\simiacryptus\skyenet\core\actors\opt\ActorOptimization.kt - -## Developer Documentation for ActorOptimization - -The `ActorOptimization` class is designed to optimize actor responses using genetic algorithms. It leverages the OpenAI API to generate, mutate, and recombine prompts to improve -the quality of responses based on defined test cases. - -### Overview - -The optimization process involves generating a population of prompts, evaluating them against a set of test cases, and iteratively selecting, mutating, and recombining the -best-performing prompts to produce a new generation of prompts. This process is repeated for a specified number of generations, with the goal of evolving prompts that yield the -best responses from the actors. - -### Key Components - -- **OpenAIClient**: The client used to interact with the OpenAI API. -- **ChatModels**: The specific OpenAI model(s) to be used for generating, mutating, and recombining prompts. -- **mutationRate**: The probability of mutating a recombined prompt. -- **mutationTypes**: The types of mutations that can be applied to prompts, along with their associated probabilities. - -### Core Methods - -#### runGeneticGenerations - -```kotlin -fun ,T:Any> runGeneticGenerations( - prompts: List, - testCases: List, - actorFactory: (String) -> BaseActor, - resultMapper: (T) -> String, - selectionSize: Int = defaultSelectionSize(prompts), - populationSize: Int = defaultPositionSize(selectionSize, prompts), - generations: Int = 3 -): List -``` - -Generates and evolves a population of prompts over a specified number of generations, evaluating their performance against a set of test cases. - -#### regenerate - -```kotlin -open fun regenerate(progenetors: List, desiredCount: Int): List -``` - -Generates a new population of prompts by selecting, mutating, and recombining the best-performing prompts from the previous generation. - -#### recombine - -```kotlin -open fun recombine(a: String, b: String): String -``` - -Combines two prompts to produce a new prompt, with an optional mutation step. - -#### mutate - -```kotlin -open fun mutate(selected: String): String -``` - -Applies a random mutation to a prompt, altering it based on the specified mutation types. - -### GeneticApi Interface - -Defines the methods for mutating and recombining prompts. - -#### Methods - -- **mutate**: Applies a mutation to a prompt. -- **recombine**: Combines two prompts into a new prompt. - -### Usage Flow - -The following diagram illustrates the basic flow of using the `ActorOptimization` class to optimize actor prompts: - -```mermaid -sequenceDiagram - participant Developer - participant ActorOptimization - participant OpenAIClient - Developer ->> ActorOptimization: Instantiate with OpenAIClient, model - loop for each generation - ActorOptimization ->> ActorOptimization: Generate population - loop for each prompt in population - ActorOptimization ->> OpenAIClient: Evaluate prompt - OpenAIClient ->> ActorOptimization: Return score - end - ActorOptimization ->> ActorOptimization: Select best prompts - ActorOptimization ->> ActorOptimization: Mutate and recombine - end - ActorOptimization ->> Developer: Return optimized prompts -``` - -### Example - -```kotlin -val apiClient = OpenAIClient("your_api_key") -val actorOptimization = ActorOptimization(apiClient, ChatModels.davinci) -val optimizedPrompts = actorOptimization.runGeneticGenerations( - prompts = listOf("Initial prompt"), - testCases = listOf(TestCase(...) -), -actorFactory = { prompt -> YourActor(prompt) }, -resultMapper = { result -> result.toString() } -) -``` - -This example demonstrates how to instantiate the `ActorOptimization` class, define test cases, and run the optimization process to evolve better prompts for your actors. - -# kotlin\com\simiacryptus\skyenet\core\actors\ParsedActor.kt - -## ParsedActor Class Documentation - -The `ParsedActor` class is a specialized actor designed for parsing user messages into JSON objects of a specified type. It extends the -`BaseActor, ParsedResponse>` class, leveraging the OpenAI API to interpret and transform textual input into structured data. - -### Class Overview - -```mermaid -classDiagram - class ParsedActor~T~ { - -Class~T~ resultClass - -T exampleInstance - -ChatModels parsingModel - -int deserializerRetries - -TypeDescriber describer - +ParsedActor(Class~T~, T, String, String?, ChatModels, Double, ChatModels, Int, TypeDescriber) - +getParser(API) Function~String, T~ - +respond(List~String~, API, ApiModel.ChatMessage) ParsedResponse~T~ - +withModel(ChatModels) ParsedActor~T~ - } - ParsedActor~T~ --> "1" BaseActor: extends -``` - -### Constructor Parameters - -- `resultClass`: The class type of the result. -- `exampleInstance`: An instance of the result class, used as an example for parsing. -- `prompt`: The initial prompt to be used for the chat. -- `name`: An optional name for the actor, defaulting to the simple name of the result class. -- `model`: The OpenAI model to be used for generating responses. -- `temperature`: The temperature setting for the OpenAI model, influencing the randomness of responses. -- `parsingModel`: The OpenAI model specifically used for parsing. -- `deserializerRetries`: The number of retries for deserialization attempts. -- `describer`: A `TypeDescriber` instance for describing the result class type. - -### Methods - -#### getParser - -Returns a function that takes a string input and returns an instance of `T` by parsing the input using the OpenAI API. - -**Parameters:** - -- `api`: The `API` instance to be used for making requests to OpenAI. - -#### respond - -Generates a `ParsedResponse` from a list of input strings and additional chat messages. - -**Parameters:** - -- `input`: A list of input strings to be parsed. -- `api`: The `API` instance to be used for making requests to OpenAI. -- `messages`: Additional chat messages to be included in the request. - -#### withModel - -Creates a new instance of `ParsedActor` with the specified OpenAI model. - -**Parameters:** - -- `model`: The OpenAI model to be used. - -### Usage Example - -```kotlin -val actor = ParsedActor( - resultClass = MyClass::class.java, - prompt = "Please parse the following message:", - model = OpenAIModels.GPT_3_5_Turbo, - parsingModel = OpenAIModels.GPT_3_5_Turbo -) - -val api = OpenAIClient("your_api_key") -val parsedResponse = actor.respond(listOf("Your input message"), api) -val result: MyClass = parsedResponse.obj -``` - -This example demonstrates how to instantiate a `ParsedActor` for a hypothetical `MyClass`, use it to parse an input message, and retrieve the parsed object. - -### Note - -The `ParsedActor` class is designed to work with the OpenAI API and requires a valid API key for operation. It is also important to handle exceptions and errors gracefully, -especially considering the potential for parsing failures or API request limits. - -# kotlin\com\simiacryptus\skyenet\core\actors\ParsedResponse.kt - -## Developer Documentation for `ParsedResponse` Class - -The `ParsedResponse` class serves as an abstract base for handling parsed responses in the `com.simiacryptus.skyenet.core.actors` package. It is designed to encapsulate the -response obtained from various sources, providing a structured way to access both the raw text and the parsed object of a specified type. - -### Class Overview - -```kotlin -package com.simiacryptus.skyenet.core.actors - -abstract class ParsedResponse(val clazz: Class) { - abstract val text: String - abstract val obj: T - override fun toString() = text -} -``` - -#### Generics - -- `T`: The type parameter `T` represents the type of the object that the response text will be parsed into. This allows for flexibility in handling different types of parsed data. - -#### Constructor Parameters - -- `clazz: Class`: This parameter is used to specify the class type of the parsed object. It is essential for runtime type checks and casting. - -#### Properties - -- `text: String`: An abstract property that should be overridden to provide the raw text of the response. -- `obj: T`: An abstract property that should be overridden to provide the parsed object derived from the response text. - -#### Methods - -- `toString()`: Overrides the `toString` method to return the raw text of the response. This can be useful for debugging or logging purposes. - -### Usage Diagram - -To illustrate how the `ParsedResponse` class might be used within a system, consider the following mermaid.js diagram: - -```mermaid -classDiagram - class ParsedResponse { - <> - +clazz: Class - +text: String - +obj: T - +toString(): String - } - class SpecificResponse { - +text: String - +obj: T - } - ParsedResponse <|-- SpecificResponse: Inherits - - class Client { - +fetchData(): ParsedResponse - } - Client --> ParsedResponse: Uses - - class DataProcessor { - +processData(ParsedResponse): void - } - DataProcessor --> ParsedResponse: Uses -``` - -#### Explanation - -- `SpecificResponse`: Represents a concrete implementation of `ParsedResponse` tailored to a specific type of data. It provides concrete implementations for the abstract properties - `text` and `obj`. -- `Client`: A class that fetches data and returns a `ParsedResponse` or one of its subclasses, encapsulating the fetched data. -- `DataProcessor`: A class that takes a `ParsedResponse` as input for processing. It can work with any subclass of `ParsedResponse`, making it flexible to changes in the type of - data being processed. - -### Conclusion - -The `ParsedResponse` class provides a robust foundation for handling and encapsulating parsed responses. By using generics and abstract properties, it offers a flexible and -type-safe way to work with different kinds of data across the system. Implementing classes need to provide concrete details for the raw text and the parsed object, allowing for -specific handling of various response types. - -# kotlin\com\simiacryptus\skyenet\core\actors\record\CodingActorInterceptor.kt - -## Developer Documentation for `CodingActorInterceptor` - -The `CodingActorInterceptor` class is designed to act as a decorator for instances of `CodingActor`, providing an interception layer for method calls related to code generation and -execution. This allows for additional processing, such as logging, metrics collection, or modification of inputs and outputs, without modifying the original `CodingActor` -implementation. - -### Overview - -The `CodingActorInterceptor` extends `CodingActor` and requires an instance of `CodingActor` (`inner`) and a `FunctionWrapper` (`functionInterceptor`) upon instantiation. The -`inner` object represents the original actor that is being decorated, while the `functionInterceptor` is used to wrap method calls, allowing for custom logic to be executed before -and after the original method. - -### Class Diagram - -```mermaid -classDiagram - class CodingActor { - <> - +response(input: ChatMessage[], model: OpenAIModel, api: API) - +respond(input: CodeRequest, api: API, messages: ChatMessage[]) - +execute(prefix: String, code: String) - } - class CodingActorInterceptor { - -inner: CodingActor - -functionInterceptor: FunctionWrapper - +response(input: ChatMessage[], model: OpenAIModel, api: API) - +respond(input: CodeRequest, api: API, messages: ChatMessage[]) - +execute(prefix: String, code: String) - } - class FunctionWrapper { - +wrap(args: Any[], callback: Function) - } - CodingActor <|-- CodingActorInterceptor: extends -``` - -### Key Methods - -#### `response` - -Overrides the `response` method from `CodingActor`. It uses the `functionInterceptor` to wrap the original `response` method of the `inner` `CodingActor`, allowing for interception -of the method call. - -**Parameters:** - -- `input`: An array of `ChatMessage` objects representing the input messages. -- `model`: The `OpenAIModel` to use for generating responses. -- `api`: The `API` instance for interacting with external services. - -#### `respond` - -Overrides the `respond` method from `CodingActor`. It directly calls the `respond` method of the `inner` `CodingActor` but wraps the input and output through the -`functionInterceptor`. - -**Parameters:** - -- `input`: A `CodeRequest` object representing the code generation request. -- `api`: The `API` instance for interacting with external services. -- `messages`: An array of `ChatMessage` objects representing additional context. - -#### `execute` - -Overrides the `execute` method from `CodingActor`. It uses the `functionInterceptor` to wrap the execution of code, allowing for interception and potential modification of the -execution process. - -**Parameters:** - -- `prefix`: A `String` representing the prefix to be used in code execution. -- `code`: The actual code `String` to be executed. - -### Usage Example - -To use the `CodingActorInterceptor`, you first need an instance of `CodingActor` and a `FunctionWrapper` implementation. Then, you can create an instance of -`CodingActorInterceptor` and use it in place of the original `CodingActor`. - -```kotlin -val originalActor: CodingActor = ... -val functionWrapper: FunctionWrapper = ... -val interceptor = CodingActorInterceptor(originalActor, functionWrapper) - -// Now you can use `interceptor` in place of `originalActor` -``` - -This setup allows you to intercept and augment the behavior of `CodingActor` methods transparently, enabling advanced use cases like logging, monitoring, or dynamic modification of -inputs/outputs. - -# kotlin\com\simiacryptus\skyenet\core\actors\record\ParsedActorInterceptor.kt - -## ParsedActorInterceptor Class Documentation - -The `ParsedActorInterceptor` class is an extension of the `ParsedActor` class designed to intercept and modify responses from a parsed actor in a flexible manner. It is part of the -`com.simiacryptus.skyenet.core.actors.record` package and is used to enhance or alter the behavior of an existing `ParsedActor` instance by applying custom logic to its responses. - -### Class Overview - -```mermaid -classDiagram - ParsedActor <|-- ParsedActorInterceptor - ParsedActorInterceptor: +inner - ParsedActorInterceptor: +functionInterceptor - ParsedActorInterceptor: -respond(input, api, messages) - ParsedActorInterceptor: -response(input, model, api) -``` - -#### Constructor Parameters - -- `inner`: The `ParsedActor` instance that this interceptor wraps. This is the actor whose responses are to be intercepted. -- `functionInterceptor`: An instance of `FunctionWrapper`, which provides the logic for intercepting and potentially modifying the responses. - -#### Methods - -##### `respond` - -Overrides the `respond` method from the `ParsedActor` class. It takes a list of input strings and additional parameters to generate a response. This method utilizes the -`functionInterceptor` to apply custom logic to the response. - -**Parameters:** - -- `input`: A list of input strings for the actor to respond to. -- `api`: An instance of `API` to interact with external services. -- `messages`: Vararg parameter of `com.simiacryptus.jopenai.ApiModel.ChatMessage`, representing additional context or messages. - -**Returns:** An instance of `ParsedResponse`, which contains the intercepted and potentially modified response. - -##### `response` - -Overrides the `response` method from the `ParsedActor` class. It is designed to handle responses based on chat messages, model, and API. - -**Parameters:** - -- `input`: Vararg parameter of `com.simiacryptus.jopenai.ApiModel.ChatMessage`, representing the input messages. -- `model`: An instance of `OpenAIModel` specifying the model to use for generating responses. -- `api`: An instance of `API` for interacting with external services. - -**Returns:** The result of the intercepted response, as modified by the `functionInterceptor`. - -### Usage Example - -To use the `ParsedActorInterceptor`, you first need an instance of a `ParsedActor` and a `FunctionWrapper` that defines how you want to intercept and modify the responses. Once you -have these, you can create an instance of `ParsedActorInterceptor` and use it in place of the original `ParsedActor`. - -```kotlin -val originalActor = ParsedActor( - // Initialization parameters for your ParsedActor -) - -val interceptor = FunctionWrapper { input, resultClass, function -> - // Your interception logic here -} - -val actorInterceptor = ParsedActorInterceptor( - inner = originalActor, - functionInterceptor = interceptor -) - -// Now you can use actorInterceptor in place of originalActor -``` - -This setup allows you to seamlessly modify the behavior of any `ParsedActor` instance, making it a powerful tool for customizing response handling in your application. - -# kotlin\com\simiacryptus\skyenet\core\actors\record\ImageActorInterceptor.kt - -## Developer Documentation for `ImageActorInterceptor` - -The `ImageActorInterceptor` class is a part of the `com.simiacryptus.skyenet.core.actors.record` package, designed to intercept and potentially modify the behavior of an -`ImageActor` instance. This class is particularly useful for debugging, logging, or applying custom transformations to the inputs or outputs of the `ImageActor` methods. - -### Overview - -`ImageActorInterceptor` extends `ImageActor`, allowing it to seamlessly integrate into places where an `ImageActor` is expected. It wraps around an existing `ImageActor` instance, -intercepting calls to the `response` and `render` methods. This interception is facilitated by a `FunctionWrapper` instance, which provides the mechanism to wrap and potentially -modify the behavior of these methods. - -### Class Diagram - -```mermaid -classDiagram - class ImageActor { - <> - +response(input: Array, model: OpenAIModel, api: API) - +render(text: String, api: API): BufferedImage - } - class ImageActorInterceptor { - -inner: ImageActor - -functionInterceptor: FunctionWrapper - +response(input: Array, model: OpenAIModel, api: API) - +render(text: String, api: API): BufferedImage - } - class FunctionWrapper { - +wrap(input: Any, action: Function): Any - } - ImageActor <|-- ImageActorInterceptor - ImageActorInterceptor --> FunctionWrapper: uses -``` - -### Constructor Parameters - -- `inner: ImageActor`: The `ImageActor` instance that this interceptor wraps around. -- `functionInterceptor: FunctionWrapper`: The `FunctionWrapper` instance used to intercept and potentially modify the behavior of the `inner` `ImageActor`'s methods. - -### Methods - -#### `response` - -Overrides the `response` method from `ImageActor`. It intercepts calls to the `inner` `ImageActor`'s `response` method, allowing for pre-processing of the input messages and/or -post-processing of the output. - -**Parameters:** - -- `input: vararg com.simiacryptus.jopenai.ApiModel.ChatMessage`: The input messages to the chat model. -- `model: OpenAIModel`: The OpenAI model to use for generating responses. -- `api: API`: The API instance for interacting with OpenAI services. - -**Returns:** The result of the `inner` `ImageActor`'s `response` method, potentially modified by the `functionInterceptor`. - -#### `render` - -Overrides the `render` method from `ImageActor`. It intercepts calls to the `inner` `ImageActor`'s `render` method, allowing for pre-processing of the input text and/or -post-processing of the output image. - -**Parameters:** - -- `text: String`: The input text to render into an image. -- `api: API`: The API instance for interacting with OpenAI services. - -**Returns:** A `BufferedImage` generated by the `inner` `ImageActor`'s `render` method, potentially modified by the `functionInterceptor`. - -### Usage Example - -```kotlin -val originalActor = ImageActor(...) -val functionWrapper = FunctionWrapper { input, action -> - // Custom logic before calling the original method - val result = action(input) - // Custom logic after calling the original method - result -} -val interceptor = ImageActorInterceptor(originalActor, functionWrapper) - -// Now, use `interceptor` wherever an `ImageActor` is expected. -``` - -This setup allows developers to inject custom logic before and after the execution of the `response` and `render` methods of an `ImageActor`, without modifying the original -`ImageActor` implementation. - -# kotlin\com\simiacryptus\skyenet\core\actors\record\SimpleActorInterceptor.kt - -## Developer Documentation: SimpleActorInterceptor - -The `SimpleActorInterceptor` class is designed to act as a decorator for instances of `SimpleActor`, allowing developers to intercept and potentially modify the behavior of the -`SimpleActor`'s `response` method. This class is part of the `com.simiacryptus.skyenet.core.actors.record` package and leverages functionality from the `com.simiacryptus.jopenai` -package for OpenAI model interactions. - -### Overview - -The `SimpleActorInterceptor` extends `SimpleActor` and overrides its `response` method. It introduces an interception layer through which all calls to the `response` method are -passed. This interception is facilitated by a `FunctionWrapper`, which can modify the input, output, or behavior of the `response` method dynamically at runtime. - -### Class Diagram - -```mermaid -classDiagram - SimpleActor <|-- SimpleActorInterceptor - SimpleActorInterceptor: +inner - SimpleActorInterceptor: +functionInterceptor - SimpleActorInterceptor: +response(input, model, api) - class SimpleActor { - <> - +prompt - +name - +model - +temperature - +response(input, model, api) - } - class SimpleActorInterceptor { - +inner SimpleActor - +functionInterceptor FunctionWrapper - +response(input, model, api) - } - class FunctionWrapper { - +wrap(input, model, function) - } -``` - -### Constructor Parameters - -- `inner`: The `SimpleActor` instance that this interceptor wraps. -- `functionInterceptor`: An instance of `FunctionWrapper` that defines how to intercept the `response` method calls. - -### Methods - -#### response - -Overrides the `response` method from `SimpleActor`. It uses the `functionInterceptor` to potentially modify the behavior of the `inner` `SimpleActor`'s `response` method. - -**Signature:** - -```kotlin -override fun response( - vararg input: com.simiacryptus.jopenai.ApiModel.ChatMessage, - model: OpenAIModel, - api: API -): ReturnType -``` - -**Parameters:** - -- `input`: Variable number of `ChatMessage` instances representing the input messages. -- `model`: The `OpenAIModel` to use for generating responses. -- `api`: The `API` instance for interacting with the OpenAI API. - -**Return Value:** - -The return type is dependent on the implementation of the `functionInterceptor`'s `wrap` method, which could modify the return value of the `inner` `SimpleActor`'s `response` -method. - -### Usage Example - -```kotlin -val simpleActor = SimpleActor(prompt = "Your prompt", name = "ActorName", model = yourModel, temperature = 0.5) -val interceptor = FunctionWrapper { input, model, function -> - // Modify input or behavior here - function(input, model) -} -val simpleActorInterceptor = SimpleActorInterceptor(simpleActor, interceptor) - -// Now, when calling response, the interceptor's logic will be applied. -simpleActorInterceptor.response(inputMessages, model, api) -``` - -This documentation provides a comprehensive overview of the `SimpleActorInterceptor` class, its purpose, and how it can be utilized to intercept and modify the behavior of -`SimpleActor` instances dynamically. - -# kotlin\com\simiacryptus\skyenet\core\actors\SimpleActor.kt - -## SimpleActor Class Documentation - -The `SimpleActor` class is a specialized actor designed for interacting with OpenAI's GPT models via the `com.simiacryptus.jopenai` API. It extends the `BaseActor` class, providing -a simple interface for sending prompts to the model and receiving responses. - -### Class Overview - -```mermaid -classDiagram - BaseActor <|-- SimpleActor - BaseActor: +prompt - BaseActor: +name - BaseActor: +model - BaseActor: +temperature - BaseActor: +respond(input, api, messages) - BaseActor: +chatMessages(questions) - BaseActor: +withModel(model) - SimpleActor: +respond(input, api, messages) - SimpleActor: +chatMessages(questions) - SimpleActor: +withModel(model) -``` - -### Constructor - -The `SimpleActor` class constructor initializes a new instance with the specified parameters. - -```kotlin -SimpleActor( - prompt: String, - name: String? = null, -model: ChatModels, -temperature: Double = 0.3, -) -``` - -#### Parameters - -- `prompt`: The initial prompt or question to be sent to the model. -- `name`: An optional name for the actor. If not provided, a default name may be used. -- `model`: The GPT model to be used for generating responses. -- `temperature`: Controls the randomness of the response. Lower values make responses more deterministic. - -### Methods - -#### respond - -Generates a response based on the input and the specified messages. - -```kotlin -override fun respond(input: List, api: API, vararg messages: ApiModel.ChatMessage): String -``` - -##### Parameters - -- `input`: A list of strings representing the input for the model. -- `api`: An instance of the `API` class to interact with the OpenAI API. -- `messages`: Variable number of `ApiModel.ChatMessage` objects representing the conversation history. - -##### Returns - -- A `String` representing the model's response. - -##### Throws - -- `RuntimeException` if no response is received. - -#### chatMessages - -Prepares the chat messages to be sent to the model based on the provided questions. - -```kotlin -override fun chatMessages(questions: List): Array -``` - -##### Parameters - -- `questions`: A list of strings representing the questions to be sent to the model. - -##### Returns - -- An array of `ApiModel.ChatMessage` objects representing the formatted chat messages. - -#### withModel - -Creates a new instance of `SimpleActor` with the specified model. - -```kotlin -override fun withModel(model: ChatModels): SimpleActor -``` - -##### Parameters - -- `model`: The GPT model to be used for generating responses. - -##### Returns - -- A new instance of `SimpleActor` configured with the specified model. - -### Usage Example - -```kotlin -val actor = SimpleActor( - prompt = "Hello, how can I assist you today?", - model = ChatModels.gpt3_5_turbo, - temperature = 0.3 -) - -val api = API("your_api_key") -val response = actor.respond( - input = listOf("What's the weather like today?"), - api = api -) - -println(response) -``` - -This example demonstrates how to create an instance of `SimpleActor`, send a prompt to the model, and print the response. - -# kotlin\com\simiacryptus\skyenet\core\actors\record\TextToSpeechActorInterceptor.kt - -## Developer Documentation for `TextToSpeechActorInterceptor` - -The `TextToSpeechActorInterceptor` class is designed to act as a decorator for instances of `TextToSpeechActor`, providing an interception mechanism for the `response` and `render` -methods through a `FunctionWrapper`. This allows for additional processing, logging, or modification of inputs and outputs without altering the original `TextToSpeechActor` -behavior. - -### Class Overview - -```mermaid -classDiagram - class TextToSpeechActorInterceptor { - +TextToSpeechActor inner - +FunctionWrapper functionInterceptor - -response(input: ChatMessage[], model: OpenAIModel, api: API) ByteArray - -render(text: String, api: API) ByteArray - } - TextToSpeechActorInterceptor --|> TextToSpeechActor: Inherits -``` - -#### Constructor Parameters - -- `inner`: The `TextToSpeechActor` instance that this interceptor will wrap. -- `functionInterceptor`: A `FunctionWrapper` instance used to intercept calls to the `response` and `render` methods. - -#### Methods - -##### `response` - -Intercepts calls to the `response` method of the `TextToSpeechActor`. It allows for pre-processing or post-processing of the input messages and the model before delegating the call -to the original `TextToSpeechActor` instance. - -**Parameters:** - -- `input`: An array of `ChatMessage` objects representing the input messages. -- `model`: An `OpenAIModel` instance specifying the model to use for generating responses. -- `api`: An `API` instance for interacting with the OpenAI API. - -**Returns:** A `ByteArray` representing the audio data generated by the `TextToSpeechActor`. - -##### `render` - -Intercepts calls to the `render` method of the `TextToSpeechActor`. It allows for pre-processing or post-processing of the input text before delegating the call to the original -`TextToSpeechActor` instance. - -**Parameters:** - -- `text`: A `String` representing the text to be converted to speech. -- `api`: An `API` instance for interacting with the OpenAI API. - -**Returns:** A `ByteArray` representing the audio data generated by the `TextToSpeechActor`. - -### Usage Example - -```kotlin -// Create an instance of TextToSpeechActor -val originalActor = TextToSpeechActor("actorName", audioModel, "alloy", 1.0, OpenAIModels.GPT35Turbo) - -// Define a function wrapper to intercept calls -val interceptor = FunctionWrapper() - -// Create the TextToSpeechActorInterceptor -val actorInterceptor = TextToSpeechActorInterceptor(originalActor, interceptor) - -// Use the interceptor to process text to speech -val audioData = actorInterceptor.render("Hello, world!", apiInstance) -``` - -This setup allows developers to inject custom logic, such as logging, metrics collection, or input/output modification, into the process of generating text-to-speech responses -without modifying the core functionality of the `TextToSpeechActor`. - -# kotlin\com\simiacryptus\skyenet\core\actors\test\CodingActorTestBase.kt - -## Developer Documentation for `CodingActorTestBase` - -The `CodingActorTestBase` class is an abstract base class designed for testing coding actors in the Skyenet framework. It extends the `ActorTestBase` class, specializing in -handling code requests and results through the `CodingActor` class. This document provides an overview of its structure, usage, and key components. - -### Overview - -`CodingActorTestBase` facilitates the testing of coding actors, which are specialized actors designed to generate code based on given prompts. It leverages an interpreter ( -specified by the subclass) and a model (GPT-3.5 Turbo by default) to process coding requests and generate code snippets as responses. - -### Class Diagram - -```mermaid -classDiagram - ActorTestBase <|-- CodingActorTestBase - CodingActorTestBase: +interpreterClass KClass - CodingActorTestBase: +actorFactory(prompt String) CodingActor - CodingActorTestBase: +getPrompt(actor BaseActor) String - CodingActorTestBase: +resultMapper(result CodeResult) String - class ActorTestBase { - +actorFactory(prompt String) BaseActor - +getPrompt(actor BaseActor) String - +resultMapper(result Any) String - } - class CodingActor { - -interpreterClass KClass - -details String - -model ChatModels - } - class BaseActor { - } - class CodeResult { - +code String - } -``` - -### Key Components - -#### Properties - -- `abstract val interpreterClass: KClass`: Specifies the class of the interpreter to be used by the `CodingActor`. This must be provided by the subclass. - -#### Methods - -- `override fun actorFactory(prompt: String): CodingActor`: Creates an instance of `CodingActor` with the specified `interpreterClass`, `details` (prompt), and the model set to - `OpenAIModels.GPT35Turbo`. -- `override fun getPrompt(actor: BaseActor): String`: Retrieves the prompt details from the given `CodingActor` instance. -- `override fun resultMapper(result: CodeResult): String`: Maps the `CodeResult` to its `code` property, effectively extracting the generated code snippet. - -### Usage - -To use `CodingActorTestBase`, a subclass must be created that provides a concrete implementation for the `interpreterClass` property. This subclass can then instantiate -`CodingActor` instances tailored to specific testing scenarios, leveraging the provided model and interpreter for generating code based on prompts. - -#### Example Subclass - -```kotlin -class MyCodingActorTest : CodingActorTestBase() { - override val interpreterClass = MyCustomInterpreter::class -} -``` - -In this example, `MyCustomInterpreter` would be a class that extends `Interpreter`, tailored to the specific needs of the test scenarios being developed. - -### Conclusion - -`CodingActorTestBase` provides a structured approach to testing coding actors within the Skyenet framework, abstracting away common functionalities such as actor instantiation and -result mapping. By extending this class and providing an interpreter, developers can efficiently create robust tests for their coding actors, ensuring their functionality and -reliability. - -# kotlin\com\simiacryptus\skyenet\core\actors\test\ActorTestBase.kt - -## Developer Documentation for `ActorTestBase` - -The `ActorTestBase` class serves as an abstract base for testing actors within a system designed to interact with OpenAI's GPT models. It provides a structured way to optimize and -test actors based on predefined test cases. - -### Overview - -`ActorTestBase` is an abstract class that requires the implementation of several key components to facilitate the testing and optimization of actors. These components include the -actor itself, a method to generate actors based on prompts, and a way to map the results to a string representation. Additionally, it integrates with the OpenAI API through the -`OpenAIClient`. - -#### Key Components - -- **Actor**: Represents the entity being tested. It is responsible for generating responses based on inputs and prompts. -- **Test Cases**: A collection of scenarios used to evaluate the actor's performance. -- **Actor Factory**: A method to create new instances of the actor based on a given prompt. -- **Result Mapper**: A method to convert the actor's response into a string format for logging or comparison. - -### Workflow - -The testing and optimization process involves several steps, including setting up test cases, running optimizations, and executing tests to evaluate the actor's performance. - -#### Optimization Process - -The optimization process (`opt` method) utilizes a genetic algorithm approach to evolve the actor based on the provided test cases. It involves generating a population of actors, -evaluating their performance, and selecting the best performers for the next generation. - -```mermaid -flowchart TD - A[Start Optimization] --> B[Generate Population] - B --> C{Evaluate Performance} - C --> D[Select Best Performers] - D --> E{End Condition?} - E -->|No| B - E -->|Yes| F[End Optimization] -``` - -#### Testing Process - -The testing process (`testRun` method) iterates through the provided test cases, generating responses from the actor and logging the results. This allows for a manual review of the -actor's performance in specific scenarios. - -```mermaid -flowchart TD - A[Start Testing] --> B[For Each Test Case] - B --> C[Generate Response] - C --> D[Log Result] - D --> E{More Test Cases?} - E -->|Yes| B - E -->|No| F[End Testing] -``` - -### Implementation Requirements - -To utilize the `ActorTestBase` class, the following abstract members must be implemented: - -- `testCases`: A list of test cases for evaluating the actor. -- `actor`: The instance of the actor being tested. -- `actorFactory(prompt: String)`: A method to create new instances of the actor based on a prompt. -- `getPrompt(actor: BaseActor)`: A method to generate a prompt for the actor based on its current state. -- `resultMapper(result: R)`: A method to convert the actor's response into a string format. - -### Usage - -To use the `ActorTestBase`, extend this class with a concrete implementation that provides the necessary components. Then, call `testOptimize` to run the optimization process or -`testRun` to execute the test cases. - -### Conclusion - -The `ActorTestBase` class provides a structured approach to testing and optimizing actors within a system. By implementing the required abstract members and following the outlined -processes, developers can efficiently evaluate and improve their actors' performance. - -# kotlin\com\simiacryptus\skyenet\core\actors\test\ImageActorTestBase.kt - -## Developer Documentation for `ImageActorTestBase` - -The `ImageActorTestBase` class is an abstract class designed for testing image-based actors within the Skyenet framework. It extends the generic `ActorTestBase` class, specifying -`List` as the input type and `ImageResponse` as the output type. This setup is tailored for testing actors that generate images based on textual prompts. - -### Overview - -The purpose of `ImageActorTestBase` is to provide a foundational structure for testing implementations of `ImageActor`. It encapsulates the common logic required for initializing -an `ImageActor` with a specific prompt and a designated text model. The class is abstract, indicating that it is intended to be extended by concrete test classes that implement -specific test scenarios for image generation actors. - -### Class Diagram - -```mermaid -classDiagram - ActorTestBase <|-- ImageActorTestBase - ImageActorTestBase: +actorFactory(prompt: String) ImageActor -ActorTestBase: +testActor(input: List~String~, expectedOutput: ImageResponse) -class ActorTestBase{ -<> -+actorFactory(prompt: String) -+testActor(input, expectedOutput) - } -class ImageActor{ -+prompt: String -+textModel: ChatModels - } -class ImageResponse{ - } -``` - -### Key Components - -#### `ImageActorTestBase` - -- **Type Parameters**: Inherits `, ImageResponse>` from `ActorTestBase`, indicating it takes a list of strings as input and produces an `ImageResponse`. -- **Methods**: - - `actorFactory(prompt: String)`: This abstract method is overridden to return an instance of `ImageActor` configured with the provided `prompt` and the `GPT35Turbo` model from - `ChatModels`. - -#### `ActorTestBase` - -- This is a generic abstract class that `ImageActorTestBase` extends. It defines the structure for testing actors by specifying an `actorFactory` method that must be implemented by - subclasses and a `testActor` method for running tests. - -#### `ImageActor` - -- Represents an actor that generates images based on textual prompts. It is initialized with a `prompt` and a `textModel` for generating responses. - -#### `ImageResponse` - -- A class representing the response from an `ImageActor`. It encapsulates the generated image data. - -### Usage - -To use `ImageActorTestBase`, developers should create a subclass that implements specific tests for `ImageActor` implementations. The subclass must provide concrete logic for how -tests are executed, including defining expected outputs and handling assertions. - -#### Example - -```kotlin -class MyImageActorTest : ImageActorTestBase() { - @Test - fun testImageGeneration() { - val prompt = "A beautiful landscape" - val expectedImageResponse = ImageResponse(/* Expected image data */) - testActor(listOf(prompt), expectedImageResponse) - } -} -``` - -In this example, `MyImageActorTest` extends `ImageActorTestBase` and implements a test method that specifies a prompt and an expected image response. The `testActor` method from -`ActorTestBase` is used to run the test, which internally uses the `actorFactory` method to create an `ImageActor` instance for the test. - -### Conclusion - -`ImageActorTestBase` provides a structured approach to testing image generation actors within the Skyenet framework. By extending this class and implementing specific test -scenarios, developers can efficiently test and validate the functionality of `ImageActor` implementations. - -# kotlin\com\simiacryptus\skyenet\core\actors\TextToSpeechActor.kt - -## TextToSpeechActor Documentation - -The `TextToSpeechActor` class is part of the `com.simiacryptus.skyenet.core.actors` package and is designed to convert text to speech using OpenAI's API. This document provides an -overview of its functionality, usage, and key components. - -### Overview - -`TextToSpeechActor` extends `BaseActor` and is specialized in handling text-to-speech (TTS) conversion. It leverages OpenAI's API to generate speech from text inputs. The class -supports customization of the audio model, voice, and speed of the speech output. - -### Key Components - -#### Constructor Parameters - -- `name`: Optional name for the actor. -- `audioModel`: The audio model to use for TTS. Defaults to `AudioModels.TTS_HD`. -- `voice`: The voice model to use. Defaults to `"alloy"`. -- `speed`: The speed of the speech. Defaults to `1.0`. -- `models`: The chat models to use for generating responses. - -#### Methods - -- `chatMessages(questions: List)`: Converts a list of questions into an array of `ChatMessage` objects. -- `render(text: String, api: API)`: Generates speech from text using the specified API. -- `respond(input: List, api: API, vararg messages: ChatMessage)`: Processes the input text and generates a `SpeechResponse`. - -#### Inner Classes - -- `SpeechResponseImpl`: Implementation of the `SpeechResponse` interface, holding the generated speech data. - -#### Interfaces - -- `SpeechResponse`: Interface defining the structure for speech response objects. - -### Usage - -To use `TextToSpeechActor`, you need to instantiate it with the desired configuration parameters. Then, you can call the `respond` method with a list of input strings and an API -instance to generate speech. - -```kotlin -val ttsActor = TextToSpeechActor( - name = "ExampleTTSActor", - audioModel = AudioModels.TTS_HD, - voice = "alloy", - speed = 1.0, - models = ChatModels.YourModelHere -) - -val api = OpenAIClient("YourAPIKey") -val inputText = listOf("Hello, world!") -val speechResponse = ttsActor.respond(inputText, api) -val mp3Data = speechResponse.mp3data -``` - -### Diagrams - -#### TextToSpeechActor Workflow - -```mermaid -sequenceDiagram - participant User - participant TextToSpeechActor as TTS Actor - participant OpenAIAPI as OpenAI API - User ->> TTS Actor: Instantiate with Config - User ->> TTS Actor: Call respond(input, api) - TTS Actor ->> OpenAIAPI: Generate Speech - OpenAIAPI -->> TTS Actor: Speech Data - TTS Actor -->> User: SpeechResponse -``` - -This diagram illustrates the basic workflow of using `TextToSpeechActor` to convert text to speech. The user instantiates the actor with the desired configuration, calls the -`respond` method with input text, and the actor interacts with OpenAI's API to generate and return the speech data. - -### Conclusion - -The `TextToSpeechActor` class provides a convenient way to integrate text-to-speech functionality into applications using OpenAI's API. By customizing the audio model, voice, and -speed, developers can generate speech that fits their specific needs. - -# kotlin\com\simiacryptus\skyenet\core\platform\AwsPlatform.kt - -## AWS Platform Integration Documentation - -The `AwsPlatform` class provides a seamless integration with AWS services, specifically Amazon S3 for object storage and AWS Key Management Service (KMS) for encryption and -decryption of data. This document outlines the functionalities provided by the `AwsPlatform` class and how developers can utilize them in their projects. - -### Overview - -`AwsPlatform` implements the `CloudPlatformInterface`, providing methods to upload data to S3, and to encrypt and decrypt data using AWS KMS. It is designed to be easily integrated -into projects requiring cloud storage and data encryption services. - -### Initialization - -The class is initialized with optional parameters for the S3 bucket name, the base URL for shared resources, and the AWS region. If not specified, default values are used. - -```kotlin -val awsPlatform = AwsPlatform( - bucket = "your-bucket-name", - shareBase = "https://your-base-url", - region = Region.US_EAST_1 -) -``` - -### Configuration - -#### S3 Client - -The S3 client is lazily initialized with the specified region. It is used for all interactions with Amazon S3. - -#### KMS Client - -Similarly, the KMS client is lazily initialized and is used for encryption and decryption operations. It defaults to the US East (N. Virginia) region. - -### Functionalities - -#### Upload to S3 - -The `upload` function is overloaded to support uploading either byte arrays or strings. It constructs the S3 object key from the provided path, ensuring it is properly formatted. - -```kotlin -val uploadPath = awsPlatform.upload( - path = "path/to/object", - contentType = "application/octet-stream", - bytes = yourByteArray -) -``` - -#### Encrypt and Decrypt Data - -`encrypt` and `decrypt` functions utilize AWS KMS for data encryption and decryption, respectively. The `encrypt` function requires the key ID for the KMS key, while `decrypt` -operates on the encrypted data. - -```kotlin -val encryptedData = awsPlatform.encrypt(fileBytes = yourData, keyId = "your-key-id") -val decryptedData = awsPlatform.decrypt(encryptedData = yourEncryptedData) -``` - -### Error Handling - -The class includes a companion object with a logger and a static `get` method that initializes the `AwsPlatform` instance, logging any errors encountered during initialization. - -### Mermaid Diagrams - -#### Upload Process Flow - -```mermaid -sequenceDiagram - participant Developer - participant AwsPlatform - participant S3 - Developer ->> AwsPlatform: upload(path, contentType, data) - AwsPlatform ->> S3: PutObjectRequest - S3 -->> AwsPlatform: Success/Failure - AwsPlatform -->> Developer: shareBase/path -``` - -#### Encryption/Decryption Flow - -```mermaid -sequenceDiagram - participant Developer - participant AwsPlatform - participant KMS - Developer ->> AwsPlatform: encrypt(data, keyId) - AwsPlatform ->> KMS: EncryptRequest - KMS -->> AwsPlatform: Encrypted Data - AwsPlatform -->> Developer: Base64 Encoded Data - Note over Developer, AwsPlatform: Decryption follows a similar flow -``` - -### Conclusion - -The `AwsPlatform` class provides a convenient way to interact with AWS services for storing and securing data. By encapsulating the complexities of AWS SDK calls, it allows -developers to focus on their application logic. - -# kotlin\com\simiacryptus\skyenet\core\platform\ApplicationServices.kt - -## Skyenet Core Platform Developer Documentation - -The Skyenet Core Platform provides a comprehensive suite of services and interfaces for managing authentication, authorization, user settings, data storage, and cloud platform -interactions. This document outlines the core components and their interactions to aid developers in understanding and extending the platform. - -### Overview - -The platform is designed around a central `ApplicationServices` object, which acts as a registry for various service interfaces and their implementations. These services include: - -- Authentication -- Authorization -- User Settings Management -- Data Storage -- Cloud Platform Interactions - -Additionally, the platform provides interfaces for managing sessions, users, and usage metrics. - -### Core Components - -#### ApplicationServices - -`ApplicationServices` is a singleton object that acts as a central registry for all the service interfaces. It ensures that the services are easily accessible throughout the -application and enforces a locking mechanism to prevent changes after the application is fully initialized. - -##### Key Properties - -- `isLocked`: A flag indicating whether the service registry is locked. -- `authorizationManager`: Manages authorization operations. -- `userSettingsManager`: Manages user settings. -- `authenticationManager`: Manages user authentication. -- `dataStorageFactory`: Factory method for creating data storage instances. -- `dataStorageRoot`: The root directory for data storage. -- `clientManager`: Manages client interactions. -- `cloud`: Interface for cloud platform interactions. -- `seleniumFactory`: Factory method for creating Selenium instances. -- `usageManager`: Manages usage metrics. - -#### Interfaces - -##### AuthenticationInterface - -Manages user authentication, including retrieving and storing user information based on access tokens. - -##### AuthorizationInterface - -Defines operations for checking user authorization for various operations. - -##### StorageInterface - -Provides methods for storing and retrieving data, including session-specific data and user settings. - -##### UserSettingsInterface - -Manages user-specific settings, including API keys and base URLs for different API providers. - -##### UsageInterface - -Tracks and reports usage metrics for different models and API keys. - -#### Data Models - -##### User - -Represents a user with properties such as email, name, ID, and picture. - -##### Session - -Represents a session with a unique session ID. - -#### CloudPlatformInterface - -Defines methods for interacting with cloud platforms, including uploading files and encrypting/decrypting data. - -### Diagrams - -#### ApplicationServices and Interfaces Interaction - -```mermaid -classDiagram - ApplicationServices --|> AuthenticationInterface - ApplicationServices --|> AuthorizationInterface - ApplicationServices --|> UserSettingsInterface - ApplicationServices --|> StorageInterface - ApplicationServices --|> UsageInterface - ApplicationServices --|> CloudPlatformInterface - class ApplicationServices { - +Boolean isLocked - +AuthorizationInterface authorizationManager - +UserSettingsInterface userSettingsManager - +AuthenticationInterface authenticationManager - +Function dataStorageFactory - +File dataStorageRoot - +ClientManager clientManager - +CloudPlatformInterface cloud - +Function seleniumFactory - +UsageInterface usageManager - } - class AuthenticationInterface { - +getUser(String) - +putUser(String, User) - +logout(String, User) - } - class AuthorizationInterface { - +isAuthorized(Class, User, OperationType) - } - class UserSettingsInterface { - +getUserSettings(User) - +updateUserSettings(User, UserSettings) - } - class StorageInterface { - +getJson(User, Session, String, Class) - +getMessages(User, Session) - +getSessionDir(User, Session) - +listSessions(User) - +setJson(User, Session, String, Any) - +updateMessage(User, Session, String, String) - } - class UsageInterface { - +incrementUsage(Session, User, OpenAIModel, ApiModel.Usage) - +getUserUsageSummary(User) - +getSessionUsageSummary(Session) - +clear() - } - class CloudPlatformInterface { - +upload(String, String, ByteArray) - +encrypt(ByteArray, String) - +decrypt(ByteArray) - } -``` - -This diagram illustrates the relationship between the `ApplicationServices` singleton and the various service interfaces it manages. It also shows the key methods provided by each -interface. - -### Conclusion - -The Skyenet Core Platform provides a robust foundation for managing authentication, authorization, user settings, data storage, and cloud interactions. By understanding the core -components and their interactions, developers can effectively extend and customize the platform to meet their specific needs. - -# kotlin\com\simiacryptus\skyenet\core\platform\file\AuthenticationManager.kt - -## AuthenticationManager Class Documentation - -The `AuthenticationManager` class is a part of the `com.simiacryptus.skyenet.core.platform.file` package and is responsible for managing user authentication within the system. It -implements the `AuthenticationInterface`, providing concrete implementations for user authentication, addition, and logout functionalities. - -### Class Diagram - -```mermaid -classDiagram - class AuthenticationManager { - -HashMap users - +getUser(String accessToken) User - +putUser(String accessToken, User user) User - +logout(String accessToken, User user) void - } - AuthenticationInterface <|.. AuthenticationManager: implements -``` - -### Class Members - -#### Properties - -- `private val users: HashMap`: A private hashmap that stores user objects against their access tokens as keys. - -#### Methods - -##### getUser(accessToken: String?): User? - -- **Description**: Retrieves a `User` object associated with the given access token. -- **Parameters**: - - `accessToken: String?`: The access token used to identify the user. Can be `null`. -- **Returns**: The `User` object if found; otherwise, `null`. -- **Usage Example**: - -```kotlin -val user = authenticationManager.getUser("someAccessToken") -``` - -##### putUser(accessToken: String, user: User): User - -- **Description**: Adds or updates a user in the system with the given access token. -- **Parameters**: - - `accessToken: String`: The access token to associate with the user. - - `user: User`: The user object to add or update. -- **Returns**: The `User` object that was added or updated. -- **Usage Example**: - -```kotlin -val newUser = User("JohnDoe", "john@example.com") -authenticationManager.putUser("newAccessToken", newUser) -``` - -##### logout(accessToken: String, user: User) - -- **Description**: Logs out a user from the system by removing the association between the user and the access token. -- **Parameters**: - - `accessToken: String`: The access token of the user to log out. - - `user: User`: The user object to log out. -- **Preconditions**: The user associated with the given access token must match the user object provided; otherwise, an exception is thrown. -- **Usage Example**: - -```kotlin -authenticationManager.logout("existingAccessToken", existingUser) -``` - -### Sequence Diagram for User Logout - -```mermaid -sequenceDiagram - participant Client - participant AuthenticationManager - Client ->> AuthenticationManager: logout(accessToken, user) - alt valid user and token - AuthenticationManager ->> AuthenticationManager: remove user from users - AuthenticationManager -->> Client: success - else invalid user or token - AuthenticationManager -->> Client: throw exception - end -``` - -This documentation provides an overview of the `AuthenticationManager` class, its properties, methods, and how it interacts with other components for managing user authentication -within the system. - -# kotlin\com\simiacryptus\skyenet\core\platform\ClientManager.kt - -## Developer Documentation for ClientManager - -The `ClientManager` class is a core component designed to manage OpenAI client instances and thread pools for sessions and users within an application. It provides mechanisms to -create and retrieve `OpenAIClient` instances and `ThreadPoolExecutor` instances based on session and user context. This document outlines the structure and functionality of the -`ClientManager` class and its inner classes. - -### Overview - -The `ClientManager` class primarily deals with two types of resources: - -1. **OpenAIClient Instances**: Handles communication with OpenAI's API, customized with user-specific API keys and logging configurations. -2. **ThreadPoolExecutor Instances**: Manages a pool of threads for executing tasks asynchronously, tailored for specific sessions and users. - -### Class Diagram - -```mermaid -classDiagram - class ClientManager { - -clientCache: Map - -poolCache: Map - +getClient(session, user, dataStorage): OpenAIClient - +getPool(session, user, dataStorage): ThreadPoolExecutor - -createClient(session, user, dataStorage): OpenAIClient - -createPool(session, user, dataStorage): ThreadPoolExecutor - } - class SessionKey { - -session: Session - -user: User - } - class RecordingThreadFactory { - -inner: ThreadFactory - -threads: Set - +newThread(r): Thread - } - class MonitoredClient { - -budget: Double - +authorize(request, apiProvider): void - +onUsage(model, tokens): void - } - ClientManager --> SessionKey: uses - ClientManager --> RecordingThreadFactory: uses - ClientManager --> MonitoredClient: creates -``` - -### Key Components - -#### ClientManager - -- **Purpose**: Manages `OpenAIClient` and `ThreadPoolExecutor` instances. -- **Key Methods**: - - `getClient(session, user, dataStorage)`: Retrieves or creates an `OpenAIClient` instance. - - `getPool(session, user, dataStorage)`: Retrieves or creates a `ThreadPoolExecutor` instance. - -#### SessionKey - -- **Purpose**: Acts as a unique identifier for caching clients and pools based on session and user. -- **Fields**: - - `session`: The current session. - - `user`: The current user (optional). - -#### RecordingThreadFactory - -- **Purpose**: Custom `ThreadFactory` that names threads based on session and user, and keeps track of created threads. -- **Key Method**: - - `newThread(r)`: Creates a new thread for the runnable `r`, names it, and adds it to the set of threads. - -#### MonitoredClient - -- **Purpose**: Extends `OpenAIClient` to add usage monitoring and budget enforcement based on session and user. -- **Key Features**: - - Budget tracking and enforcement. - - Custom authorization and usage logging. - -### Usage Flow - -1. **Client Retrieval**: When a client is requested via `getClient`, the `ClientManager` checks the cache. If a client does not exist, it creates one using `createClient`. -2. **Pool Retrieval**: Similarly, when a thread pool is requested via `getPool`, it either retrieves an existing pool from the cache or creates a new one using `createPool`. -3. **Client Creation**: `createClient` may create a `MonitoredClient` instance, which is customized with user-specific API keys and logging configurations. -4. **Pool Creation**: `createPool` initializes a `ThreadPoolExecutor` with a `RecordingThreadFactory` to manage thread naming and tracking. - -### Conclusion - -The `ClientManager` class provides a structured way to manage OpenAI client instances and thread pools within an application, ensuring that resources are efficiently reused and -customized according to session and user contexts. - -# kotlin\com\simiacryptus\skyenet\core\platform\file\AuthorizationManager.kt - -## AuthorizationManager Class Documentation - -The `AuthorizationManager` class is an implementation of the `AuthorizationInterface` designed to manage user authorization for different operations within an application. It -provides a mechanism to check if a user is authorized to perform a specific operation either globally or within the context of a specific application class. - -### Class Overview - -```mermaid -classDiagram - AuthorizationInterface <|-- AuthorizationManager - AuthorizationManager: +isAuthorized(applicationClass, user, operationType) - AuthorizationManager: -isUserAuthorized(permissionPath, user) - AuthorizationManager: +matches(user, line) - AuthorizationManager: -log -``` - -#### Methods - -##### `isAuthorized(applicationClass: Class<*>?, user: User?, operationType: AuthorizationInterface.OperationType): Boolean` - -Checks if a user is authorized to perform a specific operation. It first checks global permissions and then, if necessary, checks permissions specific to the application class. - -- **Parameters:** - - `applicationClass`: The class of the application for which the authorization is being checked. It can be `null` if the check is global. - - `user`: The user for whom the authorization is being checked. - - `operationType`: The type of operation for which authorization is being checked. -- **Returns:** `true` if the user is authorized, otherwise `false`. - -##### `isUserAuthorized(permissionPath: String, user: User?): Boolean` (Private) - -Checks if a user is authorized based on permissions defined in a file located at the specified path. - -- **Parameters:** - - `permissionPath`: The path to the permissions file. - - `user`: The user for whom the authorization is being checked. -- **Returns:** `true` if the user is found in the permissions file, otherwise `false`. - -##### `matches(user: User?, line: String): Boolean` (Open) - -Determines if a line from a permissions file matches the user's email or domain. - -- **Parameters:** - - `user`: The user for whom the match is being checked. - - `line`: A line from the permissions file. -- **Returns:** `true` if the line matches the user's email or domain, otherwise `false`. - -#### Usage Example - -```kotlin -val authorizationManager = AuthorizationManager() -val user = User(email = "example@example.com") -val operationType = AuthorizationInterface.OperationType.READ - -// Check global authorization -val isGloballyAuthorized = authorizationManager.isAuthorized(null, user, operationType) - -// Check authorization for a specific class -val isClassAuthorized = authorizationManager.isAuthorized(MyApplicationClass::class.java, user, operationType) -``` - -#### Error Handling - -The `isAuthorized` method is designed to catch and log any exceptions that occur during the authorization check process. If an exception is caught, the method will return `false`, -indicating that the user is not authorized. - -#### Logging - -The `AuthorizationManager` utilizes SLF4J for logging. It logs debug information about authorization checks and errors. - -### Conclusion - -The `AuthorizationManager` class provides a flexible and extensible way to manage user authorizations. By leveraging permissions files, it allows for granular control over who can -perform specific operations, either globally or within the context of a particular application class. - -# kotlin\com\simiacryptus\skyenet\core\platform\file\DataStorage.kt - -## DataStorage Class Documentation - -The `DataStorage` class is part of the `com.simiacryptus.skyenet.core.platform.file` package and implements the `StorageInterface`. It provides functionalities for managing -session-based data storage, including operations for JSON data manipulation, message handling, and session management within a file system. - -### Overview - -The class is designed to work with a directory structure that segregates data by user and session. It supports operations such as reading and writing JSON data, listing and -managing messages within sessions, and handling session directories and names. - -### Class Diagram - -```mermaid -classDiagram - StorageInterface <|-- DataStorage - StorageInterface: +getJson() - StorageInterface: +getMessages() - StorageInterface: +getSessionDir() - StorageInterface: +getSessionName() - StorageInterface: +getMessageIds() - StorageInterface: +setMessageIds() - StorageInterface: +getSessionTime() - StorageInterface: +listSessions() - StorageInterface: +setJson() - StorageInterface: +updateMessage() - StorageInterface: +deleteSession() - DataStorage: -dataDir File - DataStorage: +getJson() - DataStorage: -getJson() - DataStorage: +getMessages() - DataStorage: +getSessionDir() - DataStorage: +getSessionName() - DataStorage: +getMessageIds() - DataStorage: +setMessageIds() - DataStorage: +getSessionTime() - DataStorage: -messageFiles() - DataStorage: +listSessions() - DataStorage: +setJson() - DataStorage: +updateMessage() - DataStorage: +addMessageID() - DataStorage: +listSessions() - DataStorage: +userRoot() - DataStorage: +deleteSession() -``` - -### Key Methods - -#### JSON Data Management - -- `getJson`: Retrieves a JSON object from a specified file within a session directory and deserializes it into the specified class type. -- `setJson`: Serializes and saves a given object as JSON into a specified file within a session directory. - -#### Message Handling - -- `getMessages`: Retrieves all messages for a given session as a `LinkedHashMap` where keys are message IDs and values are message contents. -- `updateMessage`: Updates or creates a message with a given ID for a specified session. -- `addMessageID`: Adds a new message ID to the list of message IDs for a given session. - -#### Session Management - -- `getSessionDir`: Determines the directory path for a given session based on its ID and potentially the user. -- `getSessionName`: Retrieves or generates a name for a given session. -- `getMessageIds`: Retrieves a list of message IDs for a given session. -- `setMessageIds`: Sets the list of message IDs for a given session. -- `getSessionTime`: Retrieves or calculates the time associated with a given session. -- `listSessions`: Lists all sessions available within a specified directory or globally/user-specific based on the context. - -#### Utility Methods - -- `userRoot`: Determines the root directory for a given user's data. -- `deleteSession`: Deletes all data associated with a given session. - -### Usage Example - -To use the `DataStorage` class, you must first instantiate it with a reference to the root directory where session data will be stored. From there, you can call its methods to -manage session data as needed. - -```kotlin -val dataStorage = DataStorage(File("/path/to/data")) -val user = User("user@example.com") -val session = Session("G-20230101-123456") - -// Retrieve session directory -val sessionDir = dataStorage.getSessionDir(user, session) - -// Update a message -dataStorage.updateMessage(user, session, "messageId", "Hello, World!") - -// Get all messages for a session -val messages = dataStorage.getMessages(user, session) -``` - -### Conclusion - -The `DataStorage` class provides a comprehensive interface for managing session-based data storage in a file system, making it easier to handle user and session-specific data in a -structured and organized manner. - -# kotlin\com\simiacryptus\skyenet\core\platform\file\UserSettingsManager.kt - -## UserSettingsManager Class Documentation - -The `UserSettingsManager` class is a part of the `com.simiacryptus.skyenet.core.platform.file` package and implements the `UserSettingsInterface`. It is designed to manage user -settings, providing functionalities to retrieve and update settings for users. The settings are stored in JSON format in a designated directory on the file system. - -### Class Overview - -```mermaid -classDiagram - class UserSettingsManager { - -HashMap userSettings - -File userConfigDirectory - +getUserSettings(User) UserSettings - +updateUserSettings(User, UserSettings) void - } - UserSettingsManager ..|> UserSettingsInterface: implements -``` - -### Properties - -- `userSettings`: A private HashMap that caches the user settings in memory to avoid frequent file system access. -- `userConfigDirectory`: A private File object representing the directory where user settings files are stored. - -### Methods - -#### getUserSettings - -```kotlin -fun getUserSettings(user: User): UserSettings -``` - -Retrieves the `UserSettings` for a given `User`. If the settings for the user are not found in the cache (`userSettings`), it attempts to load them from a JSON file in the -`userConfigDirectory`. If the file does not exist or an error occurs during loading, it creates and returns new default `UserSettings`. - -##### Parameters - -- `user`: The `User` object for whom the settings are to be retrieved. - -##### Returns - -- `UserSettings`: The user settings for the specified user. - -#### updateUserSettings - -```kotlin -fun updateUserSettings(user: User, settings: UserSettings) -``` - -Updates the settings for a given user both in the cache (`userSettings`) and in the file system by writing the settings to a JSON file in the `userConfigDirectory`. - -##### Parameters - -- `user`: The `User` object for whom the settings are to be updated. -- `settings`: The `UserSettings` object containing the new settings for the user. - -### Usage Example - -```kotlin -val userManager = UserSettingsManager() -val user = User("exampleUser") -val settings = userManager.getUserSettings(user) // Retrieves or creates new settings -settings.someSetting = newValue // Modify settings as needed -userManager.updateUserSettings(user, settings) // Updates the settings in memory and file -``` - -### File Structure - -The user settings are stored in JSON files within the `.skyenet/users` directory. Each user has a separate file named after their username with a `.json` extension, e.g., -`exampleUser.json`. - -### Logging - -The `UserSettingsManager` utilizes SLF4J for logging purposes. It logs information about loading, creating, and updating user settings, as well as warnings in case of errors during -these operations. - ---- - -This documentation provides a comprehensive overview of the `UserSettingsManager` class, its functionalities, and usage within the system for managing user settings. - -# kotlin\com\simiacryptus\skyenet\core\platform\file\UsageManager.kt - -## UsageManager Class Documentation - -The `UsageManager` class is part of the `com.simiacryptus.skyenet.core.platform.file` package and implements the `UsageInterface`. It is designed to manage and track the usage of -various OpenAI models by different users and sessions, logging this information for monitoring and billing purposes. - -### Overview - -The class provides functionality to increment usage counters, retrieve usage summaries for users and sessions, and manage the persistence of this data through logging to files. - -### Class Diagram - -```mermaid -classDiagram - UsageInterface <|-- UsageManager - UsageManager: +File root - UsageManager: -FileWriter txLogFileWriter - UsageManager: -ConcurrentHashMap usagePerSession - UsageManager: -ConcurrentHashMap sessionsByUser - UsageManager: -ConcurrentHashMap usersBySession - UsageManager: +incrementUsage(Session, String, OpenAIModel, ApiModel.Usage) - UsageManager: +getUserUsageSummary(String) Map - UsageManager: +getSessionUsageSummary(Session) Map - UsageManager: +clear() -``` - -### Key Components - -- **root**: The root directory where usage logs and other related files are stored. -- **txLogFileWriter**: A `FileWriter` used to write transaction logs to a CSV file. -- **usagePerSession**: A concurrent hash map tracking usage counters per session. -- **sessionsByUser**: A concurrent hash map associating users with their sessions. -- **usersBySession**: A concurrent hash map associating sessions with their users. - -### Methods - -#### incrementUsage - -Increments the usage counters for a given session, user, and model. This method also logs the transaction to a CSV file. - -**Parameters:** - -- `session`: The session for which usage is being incremented. -- `apiKey`: The API key (or user identifier) associated with the usage. -- `model`: The OpenAI model being used. -- `tokens`: An instance of `com.simiacryptus.jopenai.ApiModel.Usage` detailing the usage metrics. - -#### getUserUsageSummary - -Retrieves a summary of usage for a given user across all sessions. - -**Parameters:** - -- `apiKey`: The API key (or user identifier) for which to retrieve the usage summary. - -**Returns:** A map of `OpenAIModel` to `com.simiacryptus.jopenai.ApiModel.Usage` summarizing the usage. - -#### getSessionUsageSummary - -Retrieves a summary of usage for a given session. - -**Parameters:** - -- `session`: The session for which to retrieve the usage summary. - -**Returns:** A map of `OpenAIModel` to `com.simiacryptus.jopenai.ApiModel.Usage` summarizing the usage. - -#### clear - -Clears all usage counters and associated mappings, and logs the current state before clearing. - -### Persistence Mechanism - -The `UsageManager` class employs a file-based logging mechanism to persist usage data. This includes a CSV file for transaction logs and a JSON file for counter states. The class -also implements a scheduled task to periodically save the counters to disk. - -### Sequence Diagram for incrementUsage - -```mermaid -sequenceDiagram - participant Client - participant UsageManager - participant FileWriter - Client ->> UsageManager: incrementUsage(session, apiKey, model, tokens) - UsageManager ->> FileWriter: Write transaction to log - FileWriter ->> UsageManager: Flush to disk -``` - -### Usage Example - -```kotlin -val rootDir = File("/path/to/usage/logs") -val usageManager = UsageManager(rootDir) -val session = Session("session-id") -val user = "user@example.com" -val model = OpenAIModel.ChatGPT -val usage = ApiModel.Usage(prompt_tokens = 100, completion_tokens = 50) - -usageManager.incrementUsage(session, user, model, usage) -``` - -This documentation provides an overview of the `UsageManager` class, its key components, and how it can be used to track and manage usage of OpenAI models. - -# kotlin\com\simiacryptus\skyenet\core\platform\test\AuthorizationInterfaceTest.kt - -## Developer Documentation: AuthorizationInterfaceTest - -### Overview - -The `AuthorizationInterfaceTest` class is designed to test the authorization functionalities provided by the `AuthorizationInterface`. It aims to ensure that the authorization -checks behave as expected for different users and operations within the system. This documentation provides an overview of the `AuthorizationInterfaceTest` class, its structure, -and how it is used to validate authorization logic. - -### Class Structure - -The `AuthorizationInterfaceTest` class is an open class that requires an instance of `AuthorizationInterface` as a constructor parameter. This design allows for testing different -implementations of the `AuthorizationInterface`. The class contains a predefined `user` object and a test method named `newUser has admin`. - -#### User Object - -The `user` object is an instance of the `User` class with predefined attributes: - -- **Email**: newuser@example.com -- **Name**: Jane Smith -- **ID**: 2 -- **Picture URL**: http://example.com/newpicture.jpg - -This user object represents a new user in the system and is used to test authorization checks. - -#### Test Method: `newUser has admin` - -The test method `newUser has admin` is designed to verify that a new user does not have administrative privileges by default. It uses the `assertFalse` assertion to ensure that the -`isAuthorized` method of the `AuthorizationInterface` returns `false` when checking if the predefined `user` has `Admin` operation type access. - -### Sequence Diagram - -The following sequence diagram illustrates the flow of the `newUser has admin` test method: - -```mermaid -sequenceDiagram - participant Test as AuthorizationInterfaceTest - participant AuthInterface as AuthorizationInterface - Test ->> AuthInterface: isAuthorized(this.javaClass, user, OperationType.Admin) - AuthInterface -->> Test: false - Test ->> Test: assertFalse(result) -``` - -### Usage - -To use the `AuthorizationInterfaceTest` class for testing, follow these steps: - -1. Implement the `AuthorizationInterface` with your custom authorization logic. -2. Instantiate your implementation of `AuthorizationInterface`. -3. Create an instance of `AuthorizationInterfaceTest`, passing your `AuthorizationInterface` implementation to its constructor. -4. Run the `newUser has admin` test method to verify that new users do not have administrative privileges by default. - -### Conclusion - -The `AuthorizationInterfaceTest` class is a crucial component for ensuring that the authorization logic in your application behaves as expected. By testing with predefined user -objects and specific operation types, you can validate the security and correctness of your authorization checks. - -# kotlin\com\simiacryptus\skyenet\core\platform\test\AuthenticationInterfaceTest.kt - -## Authentication Interface Test Documentation - -This documentation provides an overview of the `AuthenticationInterfaceTest` class, which is designed to test the functionality of an `AuthenticationInterface` implementation. The -tests ensure that the interface correctly handles user authentication processes, including adding, retrieving, and removing users based on access tokens. - -### Overview - -The `AuthenticationInterfaceTest` class is an open class that requires an instance of `AuthenticationInterface` to be passed to its constructor. This design allows for testing -different implementations of the `AuthenticationInterface`. The class includes several test methods annotated with `@Test`, each designed to verify a specific aspect of the -authentication process. - -### Test Methods - -The class contains the following test methods: - -- `getUser should return null when no user is associated with access token` -- `putUser should add a new user and return the user` -- `getUser should return User after putUser is called` -- `logout should remove the user associated with the access token` - -#### Diagram: Test Flow - -```mermaid -sequenceDiagram - participant Test as Test Case - participant Auth as AuthenticationInterface - Note over Test, Auth: getUser with no associated user - Test ->> Auth: getUser(validAccessToken) - Auth -->> Test: null - Note over Test, Auth: putUser adds a new user - Test ->> Auth: putUser(validAccessToken, newUser) - Auth -->> Test: newUser - Note over Test, Auth: getUser returns added user - Test ->> Auth: getUser(validAccessToken) - Auth -->> Test: newUser - Note over Test, Auth: logout removes the user - Test ->> Auth: logout(validAccessToken, newUser) - Test ->> Auth: getUser(validAccessToken) - Auth -->> Test: null -``` - -### Test Setup - -Before running the tests, an instance of `AuthenticationInterface` must be provided. This instance is used to perform the actual authentication operations tested by the methods. - -#### Key Variables - -- `validAccessToken`: A randomly generated UUID string used as a mock access token for testing. -- `newUser`: A mock `User` object representing a new user to be added through the authentication interface. - -### Test Methods Explained - -#### `getUser should return null when no user is associated with access token` - -This test verifies that the `getUser` method returns `null` when queried with an access token that has no associated user. - -#### `putUser should add a new user and return the user` - -This test checks if the `putUser` method correctly adds a new user associated with a given access token and returns the added user object. - -#### `getUser should return User after putUser is called` - -This test ensures that after a new user is added with `putUser`, the `getUser` method can retrieve the user object when provided with the same access token. - -#### `logout should remove the user associated with the access token` - -This test confirms that the `logout` method removes the association between the user and the access token, resulting in `getUser` returning `null` for that token. - -### Conclusion - -The `AuthenticationInterfaceTest` class provides a comprehensive suite of tests to ensure the correct behavior of implementations of the `AuthenticationInterface`. By following the -test cases and ensuring all pass, developers can be confident in the reliability and correctness of their authentication processes. - -# kotlin\com\simiacryptus\skyenet\core\platform\test\UsageTest.kt - -## Developer Documentation for `UsageTest` Class - -The `UsageTest` class is an abstract test suite designed to validate the functionality of implementations of the `UsageInterface`. This interface is crucial for tracking and -managing the usage of resources by users, particularly in applications that involve session-based interactions and resource consumption tracking. - -### Overview - -The `UsageTest` class focuses on testing the ability of an implementation to accurately record and report usage statistics for individual sessions and users. It leverages a test -user and predefined usage metrics to ensure that the `incrementUsage` method correctly updates the usage records. - -### Class Diagram - -```mermaid -classDiagram - class UsageTest { - -impl: UsageInterface - -testUser: User - +UsageTest(impl: UsageInterface) - +incrementUsage should increment usage for session(): void - } - class UsageInterface { - +incrementUsage(session: String, user: User, model: ApiModel, usage: ApiModel.Usage): void - +getSessionUsageSummary(session: String): Map~ApiModel, ApiModel.Usage~ - +getUserUsageSummary(user: User): Map~ApiModel, ApiModel.Usage~ - } - class User { - +email: String - +name: String - +id: String - } - class ApiModel { - <> - GPT35Turbo - } - class ApiModel.Usage { -+prompt_tokens: Int -+completion_tokens: Int -+cost: Double -} -UsageTest --> UsageInterface: uses -UsageTest --> User: uses -UsageInterface --> ApiModel: uses -UsageInterface --> ApiModel.Usage: uses -``` - -### Test Method: `incrementUsage should increment usage for session` - -This test method validates the functionality of the `incrementUsage` method within the `UsageInterface`. It follows these steps: - -1. **Setup**: A test user is created with predefined attributes. A session ID is generated using `StorageInterface.newGlobalID()`. A predefined usage object is created to simulate - the consumption of resources. - -2. **Action**: The `incrementUsage` method of the `UsageInterface` implementation is called with the session ID, test user, a model (in this case, `OpenAIModels.GPT35Turbo`), and - the predefined usage object. - -3. **Verification**: - - The method `getSessionUsageSummary` is called with the session ID to retrieve the usage summary for the session. The test verifies that the returned usage summary matches the - predefined usage object. - - The method `getUserUsageSummary` is called with the test user to retrieve the user's usage summary. The test verifies that the returned usage summary for the user matches the - predefined usage object. - -### Sequence Diagram for `incrementUsage should increment usage for session` - -```mermaid -sequenceDiagram - participant Test as UsageTest - participant Interface as UsageInterface - participant Storage as StorageInterface - participant Model as ApiModel - Test ->> Storage: newGlobalID() - Note over Test: Generate session ID - Test ->> Interface: incrementUsage(session, testUser, GPT35Turbo, usage) - Note over Interface: Update usage records - Test ->> Interface: getSessionUsageSummary(session) - Interface ->> Test: Return usage summary for session - Test ->> Interface: getUserUsageSummary(testUser) - Interface ->> Test: Return usage summary for user - Note over Test: Verify usage summaries match predefined usage -``` - -### Conclusion - -The `UsageTest` class provides a structured approach to validate the implementation of the `UsageInterface`. By ensuring that usage statistics are accurately recorded and reported, -developers can maintain the integrity of resource management and usage tracking within their applications. - -# kotlin\com\simiacryptus\skyenet\core\platform\test\UserSettingsTest.kt - -## Developer Documentation: UserSettingsTest - -The `UserSettingsTest` class is designed to validate the functionality of user settings management within a platform. It ensures that custom settings for a user can be updated and -retrieved accurately. This documentation provides an overview of the test cases and their significance in maintaining the integrity of the user settings feature. - -### Overview - -`UserSettingsTest` is an abstract class that requires an implementation of `UserSettingsInterface` to be passed to its constructor. This design allows for testing different -implementations of `UserSettingsInterface` with the same set of tests. - -### Test Cases - -#### 1. `updateUserSettings should store custom settings for user` - -This test verifies that the `updateUserSettings` method correctly stores custom settings for a user. It involves creating a test user, updating their settings, and then retrieving -the settings to ensure they have been updated as expected. - -##### Process Flow - -```mermaid -sequenceDiagram - participant Test - participant UserSettings - Test ->>+ UserSettings: updateUserSettings(testUser, newSettings) - UserSettings -->>- Test: Update Complete - Test ->>+ UserSettings: getUserSettings(testUser) - UserSettings -->>- Test: Return settings - Test ->> Test: Assert settings are updated -``` - -#### 2. `getUserSettings should return updated settings after updateUserSettings is called` - -This test ensures that after updating a user's settings, the `getUserSettings` method returns the updated settings. It checks the initial state of a user's settings, updates the -settings, and then verifies that the new settings are returned upon retrieval. - -##### Process Flow - -```mermaid -sequenceDiagram - participant Test - participant UserSettings - Test ->>+ UserSettings: getUserSettings(testUser) - UserSettings -->>- Test: Return initial settings - Test ->> Test: Assert initial settings - Test ->>+ UserSettings: updateUserSettings(testUser, updatedSettings) - UserSettings -->>- Test: Update Complete - Test ->>+ UserSettings: getUserSettings(testUser) - UserSettings -->>- Test: Return updated settings - Test ->> Test: Assert updated settings -``` - -### Key Components - -- **User**: Represents a user in the system. Each user has an email, name, and ID. -- **UserSettingsInterface**: An interface defining methods for updating and retrieving user settings. - - `updateUserSettings(User, UserSettings)`: Updates the settings for a given user. - - `getUserSettings(User)`: Retrieves the settings for a given user. -- **UserSettingsInterface.UserSettings**: A data class representing user settings. It includes a map of API keys associated with different API providers. - -### Conclusion - -The `UserSettingsTest` class plays a crucial role in ensuring the reliability of the user settings management feature. By abstracting the `UserSettingsInterface`, it allows for -flexible testing across different implementations, ensuring that user settings are correctly updated and retrieved across the platform. - -# kotlin\com\simiacryptus\skyenet\core\platform\test\StorageInterfaceTest.kt - -## StorageInterfaceTest Developer Documentation - -The `StorageInterfaceTest` class is an abstract test suite designed to verify the functionality of implementations of the `StorageInterface`. This documentation provides an -overview of the test methods available in the class and how they interact with the `StorageInterface`. - -### Overview - -The `StorageInterface` is a critical component that abstracts storage operations for sessions, messages, and user-specific data. Implementations of this interface must provide -mechanisms to handle data storage and retrieval in a consistent and reliable manner. - -The `StorageInterfaceTest` class ensures that any implementation of the `StorageInterface` adheres to the expected behavior through a series of unit tests. Each test is designed to -validate specific functionalities such as session management, message handling, and JSON data operations. - -### Test Methods - -Below is a summary of the test methods included in the `StorageInterfaceTest` class: - -- `testGetJson()`: Validates retrieval of JSON data. -- `testGetMessages()`: Checks the retrieval of messages as a `LinkedHashMap`. -- `testGetSessionDir()`: Ensures the session directory is correctly obtained. -- `testGetSessionName()`: Verifies that the session name is correctly retrieved. -- `testGetSessionTime()`: Confirms that the session time is correctly fetched. -- `testListSessions()`: Tests listing of all sessions for a user. -- `testSetJson()`: Validates setting JSON data. -- `testUpdateMessage()`: Checks updating of a message. -- `testListSessionsWithDir()`: Tests listing of sessions from a specific directory. -- `testUserRoot()`: Verifies retrieval of the user's root directory. -- `testDeleteSession()`: Ensures that a session can be deleted without errors. - -### Workflow Diagram - -To better understand the interactions between the tests and the `StorageInterface`, the following Mermaid.js diagram illustrates a simplified workflow for a few selected test -cases: - -```mermaid -sequenceDiagram - participant T as Test - participant S as StorageInterface - participant D as Data Storage - T ->> S: testGetJson() - S ->> D: Fetch JSON - D -->> S: Return JSON - S -->> T: Assert NULL (non-existing) - T ->> S: testSetJson() - S ->> D: Store JSON - D -->> S: Confirm - S -->> T: Assert Equal (settings) - T ->> S: testDeleteSession() - S ->> D: Delete Session Data - D -->> S: Confirm Deletion - S -->> T: Assert No Exception -``` - -### Testing Strategy - -Each test method follows a similar structure: - -1. **Arrange**: Set up any necessary objects and state before the actual test. -2. **Act**: Execute the method under test with the arranged conditions. -3. **Assert**: Verify that the outcome of the action is as expected. - -This structure ensures that each test is isolated, focusing on a single behavior of the `StorageInterface`. - -### Extending Tests - -To extend the `StorageInterfaceTest` with additional tests for new methods in the `StorageInterface`, follow these steps: - -1. **Define the Test Method**: Create a new `@Test` method in the `StorageInterfaceTest` class. -2. **Arrange**: Set up the necessary preconditions for your test. -3. **Act**: Call the method under test. -4. **Assert**: Verify the outcome against the expected result. - -Ensure that each new test method adheres to the testing strategy outlined above to maintain consistency and clarity in the test suite. - -### Conclusion - -The `StorageInterfaceTest` class provides a comprehensive suite of tests to ensure the reliability and correctness of `StorageInterface` implementations. By following the testing -strategy and extending the test suite as needed, developers can ensure that their storage solutions meet the required standards for functionality and robustness. - -# kotlin\com\simiacryptus\skyenet\core\util\ClasspathRelationships.kt - -## ClasspathRelationships Module Documentation - -The `ClasspathRelationships` module is part of the `com.simiacryptus.skyenet.core.util` package. It provides utilities for analyzing and managing relationships between classes -within Java Archive (JAR) files. This documentation outlines the key functionalities provided by this module, including reading classes from JAR files, listing files within a JAR, -and mapping class dependencies. - -### Overview - -The module consists of the `ClasspathRelationships` object, which contains methods for working with JAR files and a data class for representing references or relationships between -classes. - -#### Key Components - -- **Relation**: A sealed class representing a generic relationship between two methods. It has two open properties, `from_method` and `to_method`, which denote the source and - target of the relationship, respectively. -- **Reference**: A data class that encapsulates a relationship between two classes (`from` and `to`) along with the specific `relation` between them. -- **Utility Functions**: Functions for reading classes and files from JAR archives and for mapping class dependencies. - -### Functions - -#### readJarClasses - -```kotlin -fun readJarClasses(jarPath: String): Map -``` - -Reads and returns all classes from a specified JAR file. Each class is represented by its fully qualified name and its bytecode in the form of a `ByteArray`. - -##### Parameters - -- `jarPath`: The file system path to the JAR file. - -##### Returns - -- A `Map` where each key is a fully qualified class name and each value is the class's bytecode as a `ByteArray`. - -#### readJarFiles - -```kotlin -fun readJarFiles(jarPath: String): Array -``` - -Lists all files contained within a specified JAR file. - -##### Parameters - -- `jarPath`: The file system path to the JAR file. - -##### Returns - -- An `Array` of `String` containing the names of all files within the JAR file. - -#### downstreamMap - -```kotlin -fun downstreamMap(dependencies: List): Map> -``` - -Groups a list of `Reference` objects by their source class (`from`). - -##### Parameters - -- `dependencies`: A list of `Reference` objects representing class dependencies. - -##### Returns - -- A `Map` where each key is a class name and each value is a list of `Reference` objects originating from that class. - -### Diagrams - -To better understand the relationships and functionalities provided by this module, the following Mermaid.js diagrams can be used: - -#### Class Relationships - -```mermaid -classDiagram - class Relation { - <> - +String from_method - +String to_method - } - class Reference { - +String from - +String to - +Relation relation - } - Relation <|-- Reference : has -``` - -This diagram illustrates the relationship between the `Relation` and `Reference` classes. `Reference` has a `Relation` indicating the type of relationship between two classes. - -#### Functionality Overview - -```mermaid -graph LR - A[readJarClasses] --> B((JAR File)) - C[readJarFiles] --> B - D[downstreamMap] --> E{References} -``` - -This diagram shows how the main functions interact with JAR files and references between classes. - -### Conclusion - -The `ClasspathRelationships` module provides essential utilities for analyzing class dependencies within JAR files, making it a valuable tool for developers working with Java -applications. By leveraging the functionalities provided, developers can gain insights into the structure and relationships of classes in their applications. - -# kotlin\com\simiacryptus\skyenet\core\util\Ears.kt - -## Ears Class Documentation - -The `Ears` class serves as the auditory interface for the SkyeNet system, facilitating audio input processing, command recognition, and dictation. It leverages the OpenAI API for -audio transcription and command recognition. - -### Overview - -The `Ears` class is designed to capture audio input, transcribe it into text, and recognize specific commands within the transcribed text. It utilizes several components from the -`com.simiacryptus.jopenai` package to achieve this functionality, including audio recording, loudness window buffering, and transcription processing. - -### Class Diagram - -```mermaid -classDiagram - class Ears { - +OpenAIClient api - +Double secondsPerAudioPacket - +CommandRecognizer commandRecognizer - +timeout(ms: Long): () -> Boolean - +listenForCommand(client: OpenAIClient, minCaptureMs: Int, continueFn: () -> Boolean, rawBuffer: Deque~ByteArray~, commandHandler: (command: String) -> Unit) - +startDictationListener(client: OpenAIClient, continueFn: () -> Boolean, rawBuffer: Deque~ByteArray~, textAppend: (String) -> Unit) - +startAudioCapture(continueFn: () -> Boolean): ConcurrentLinkedDeque~ByteArray~ - } - Ears --> "1" CommandRecognizer: Uses -``` - -### Key Components - -#### CommandRecognizer Interface - -Defines the structure for command recognition implementations. It includes a method `listenForCommand` that takes a `DictationBuffer` and returns a `CommandRecognized` object -indicating whether a command was recognized and what the command was. - -#### DictationBuffer Data Class - -Holds the transcribed text buffer for command recognition. - -#### CommandRecognized Data Class - -Represents the outcome of command recognition, indicating whether a command was recognized and the recognized command text. - -### Key Methods - -#### `listenForCommand` - -Initiates listening for commands within the audio input. It captures audio, transcribes it to text, and checks the transcription for commands at intervals specified by -`minCaptureMs`. - -- **Parameters:** - - `client`: The OpenAIClient instance for API interactions. - - `minCaptureMs`: Minimum milliseconds between command checks. - - `continueFn`: A function that returns `true` to continue listening or `false` to stop. - - `rawBuffer`: A deque holding raw audio data. - - `commandHandler`: A callback function that handles recognized commands. - -#### `startDictationListener` - -Starts the dictation listener that transcribes audio input to text. - -- **Parameters:** - - `client`: The OpenAIClient instance for API interactions. - - `continueFn`: A function that returns `true` to continue listening or `false` to stop. - - `rawBuffer`: A deque holding raw audio data. - - `textAppend`: A callback function that appends transcribed text. - -#### `startAudioCapture` - -Begins capturing audio input and stores it in a `ConcurrentLinkedDeque`. - -- **Parameters:** - - `continueFn`: A function that returns `true` to continue capturing or `false` to stop. - -### Usage Example - -```kotlin -val apiClient = OpenAIClient("YourAPIKey") -val ears = Ears(apiClient) -ears.listenForCommand(apiClient) { command -> - println("Command received: $command") -} -``` - -This example initializes the `Ears` class with an `OpenAIClient` instance and starts listening for commands. When a command is recognized, it prints the command to the console. - -### Conclusion - -The `Ears` class provides a comprehensive solution for audio input processing and command recognition in the SkyeNet system. By leveraging the OpenAI API and custom audio -processing components, it enables efficient and effective voice command functionality. - -# kotlin\com\simiacryptus\skyenet\core\util\FunctionWrapper.kt - -## Developer Documentation for Function Interception and Recording - -This documentation provides an overview of the function interception and recording mechanism implemented in the provided Kotlin code. The primary purpose of this mechanism is to -intercept function calls, allowing for operations such as logging, modifying inputs or outputs, and recording function calls for debugging or auditing purposes. - -### Overview - -The codebase introduces several key components to achieve function interception and recording: - -- `FunctionInterceptor`: An interface defining methods for intercepting function calls. -- `FunctionWrapper`: A class that wraps function calls, enabling their interception through the `FunctionInterceptor` interface. -- `NoopFunctionInterceptor`: A no-operation interceptor for cases where interception is not required. -- `JsonFunctionRecorder`: An implementation of `FunctionInterceptor` that records function inputs and outputs as JSON files, useful for debugging and auditing. - -### Components Diagram - -```mermaid -classDiagram - FunctionInterceptor <|-- NoopFunctionInterceptor - FunctionInterceptor <|-- JsonFunctionRecorder - FunctionInterceptor <|-- FunctionWrapper - FunctionInterceptor : +intercept(returnClazz: Class~T~, fn: () -> T) - FunctionInterceptor : +intercept(params: P, returnClazz: Class~T~, fn: (P) -> T) - FunctionInterceptor : +intercept(p1: P1, p2: P2, returnClazz: Class~T~, fn: (P1, P2) -> T) - FunctionInterceptor : +intercept(p1: P1, p2: P2, p3: P3, returnClazz: Class~T~, fn: (P1, P2, P3) -> T) - FunctionInterceptor : +intercept(p1: P1, p2: P2, p3: P3, p4: P4, returnClazz: Class~T~, fn: (P1, P2, P3, P4) -> T) - FunctionWrapper : +wrap(fn: () -> T) - FunctionWrapper : +wrap(p: P, fn: (P) -> T) - FunctionWrapper : +wrap(p1: P1, p2: P2, fn: (P1, P2) -> T) - FunctionWrapper : +wrap(p1: P1, p2: P2, p3: P3, fn: (P1, P2, P3) -> T) - FunctionWrapper : +wrap(p1: P1, p2: P2, p3: P3, p4: P4, fn: (P1, P2, P3, P4) -> T) - JsonFunctionRecorder --|> FunctionInterceptor: implements - JsonFunctionRecorder : +close() - JsonFunctionRecorder : +operationDir() File -``` - -### Usage - -#### Intercepting Functions - -To intercept a function call, wrap the function call using an instance of `FunctionWrapper`, specifying the appropriate `FunctionInterceptor` implementation. For example, to record -function calls as JSON: - -```kotlin -val recorder = JsonFunctionRecorder(File("/path/to/record")) -val wrapper = FunctionWrapper(recorder) - -val result = wrapper.wrap { someFunctionCall() } -``` - -#### Implementing Custom Interceptors - -To implement a custom function interceptor, extend the `FunctionInterceptor` interface and override the `intercept` methods as needed. For example, a custom interceptor could -modify the input parameters before passing them to the original function. - -#### Recording Function Calls - -`JsonFunctionRecorder` is used to record function calls, inputs, and outputs as JSON files. This is particularly useful for debugging and auditing. To use it, instantiate -`JsonFunctionRecorder` with a base directory for storing the records: - -```kotlin -val recorder = JsonFunctionRecorder(File("/path/to/record")) -``` - -Wrap function calls using `FunctionWrapper` as shown in the previous section to record them. - -### Conclusion - -The provided Kotlin code offers a flexible and powerful mechanism for intercepting and recording function calls. By leveraging `FunctionInterceptor` and its implementations, -developers can easily add logging, debugging, and auditing capabilities to their applications. - -# kotlin\com\simiacryptus\skyenet\core\util\LoggingInterceptor.kt - -## LoggingInterceptor Documentation - -The `LoggingInterceptor` class is a utility designed to intercept and capture logging events from specified loggers within an application. This is particularly useful for debugging -or testing purposes, where capturing the output of specific loggers can help diagnose issues or verify that certain conditions are being logged as expected. - -### Overview - -The `LoggingInterceptor` extends `AppenderBase`, allowing it to be attached to SLF4J loggers (via Logback) and capture their logging events. It stores these events -in a `StringBuffer`, which can then be retrieved and inspected. - -### Usage - -#### Basic Usage - -To use the `LoggingInterceptor`, you typically wrap the code block you wish to capture logs from within a call to `LoggingInterceptor.withIntercept(...)`. This method temporarily -attaches the `LoggingInterceptor` to the specified loggers, executes the code block, and then restores the original logger state. - -```kotlin -val capturedLogs = StringBuffer() -LoggingInterceptor.withIntercept(capturedLogs, "com.example.mylogger") { - // Code block where logs from "com.example.mylogger" will be captured -} -// Inspect capturedLogs here -``` - -#### Advanced Usage - -For more control, you can directly instantiate and manage a `LoggingInterceptor` instance, though this requires manually managing the logger state. - -### Methods - -#### `withIntercept` - -```kotlin -fun withIntercept( - stringBuffer: StringBuffer = StringBuffer(), - vararg loggerPrefixes: String, - fn: () -> T, -): T -``` - -Captures logs from loggers with names starting with any of the specified prefixes, executing the provided function block, and then restores the original logger state. - -- **Parameters:** - - `stringBuffer`: The `StringBuffer` to capture logs to. - - `loggerPrefixes`: Vararg parameter specifying the logger name prefixes to intercept. - - `fn`: The function block to execute while capturing logs. - -- **Returns:** The result of the function block `fn`. - -#### `getStringBuffer` - -```kotlin -fun getStringBuffer(): StringBuffer -``` - -Returns the `StringBuffer` containing the captured logs. - -### Internal Mechanics - -#### Appender Attachment - -The `LoggingInterceptor` works by temporarily replacing the appenders of the specified loggers with itself. This allows it to capture all logging events directed to these loggers. - -#### Log Capture - -When a logging event is received, the `LoggingInterceptor` appends the formatted message (and any associated throwable stack trace) to its internal `StringBuffer`. - -#### Restoration - -After the function block has executed, the original logger appenders and levels are restored, ensuring that the logger's original configuration is unaffected by the interception -process. - -### Diagram: Logging Interception Flow - -```mermaid -sequenceDiagram - participant UserCode as User Code - participant LI as LoggingInterceptor - participant Logger as Logger - participant Appender as Original Appender - Note over UserCode, LI: User initiates log capture - UserCode ->> LI: withIntercept(...) - LI ->> Logger: Detach Appender - LI ->> Logger: Attach LoggingInterceptor - Note over UserCode, Logger: User code executes, logs captured - UserCode ->> Logger: Log Event - Logger ->> LI: Forward Log Event - LI ->> UserCode: Capture Log to StringBuffer - Note over UserCode, LI: User code block ends - LI ->> Logger: Detach LoggingInterceptor - LI ->> Logger: Restore Original Appender -``` - -This diagram illustrates the flow of control and data during the interception and capture of log events using the `LoggingInterceptor`. - -# kotlin\com\simiacryptus\skyenet\core\util\RuleTreeBuilder.kt - -## Developer Documentation for RuleTreeBuilder - -The `RuleTreeBuilder` object in the `com.simiacryptus.skyenet.core.util` package is designed to generate Kotlin code expressions for matching and filtering strings based on sets of -inclusion (`toMatch`) and exclusion (`doNotMatch`) criteria. This utility is particularly useful for dynamically generating rules for path matching, filtering collections, or any -scenario where a set of string rules needs to be applied efficiently. - -### Overview - -The core functionality revolves around generating a `when` expression in Kotlin that evaluates to `true` or `false` based on whether a given string (`path`) matches the specified -criteria. The criteria are defined by two sets of strings: `toMatch` (strings that should match) and `doNotMatch` (strings that should not match). The result is a compact, -optimized Kotlin `when` expression that can be used in code to perform the matching. - -### Key Functions - -#### `getRuleExpression` - -- **Description**: Generates the Kotlin `when` expression based on the provided sets of strings to match and not to match. -- **Parameters**: - - `toMatch`: A `Set` containing the strings that should match. - - `doNotMatch`: A `SortedSet` containing the strings that should not match. - - `result`: A `Boolean` indicating the desired result when a match is found. -- **Returns**: A `String` representing the Kotlin `when` expression. - -#### `String Extensions` - -- **`escape`**: Escapes the dollar sign (`$`) in strings, which is necessary to avoid syntax errors in the generated Kotlin code. -- **`safeSubstring`**: Safely extracts a substring, ensuring that the indices are within bounds and returning an empty string if not. - -### Internal Logic - -The `getRuleExpression` function is the entry point for generating the rule expression. It internally calls `getRules` to construct the individual conditions of the `when` -expression. The decision on whether to prioritize matching or not matching strings is made based on the size of the respective sets, aiming to optimize the resulting expression. - -#### Optimization Strategies - -1. **Prefix Matching**: The algorithm attempts to find common prefixes among the strings to match and not to match, optimizing the generated rules by grouping them based on these - prefixes. -2. **Entropy Calculation**: For each potential prefix, an entropy value is calculated to determine the effectiveness of splitting the sets based on that prefix. The prefix with the - best (lowest) entropy is chosen for splitting. - -### Mermaid.js Diagram: Rule Generation Process - -```mermaid -graph TD; - A[Start] --> B{Determine Smaller Set}; - B -->|toMatch Smaller| C[Invert Sets]; - B -->|doNotMatch Smaller| D[Proceed with Original Sets]; - C --> E[Generate Rules]; - D --> E; - E --> F{Any Remaining Items?}; - F -->|Yes| G[Find Best Prefix]; - F -->|No| H[End]; - G -->|Prefix Found| I[Generate Sub-Rule]; - G -->|No Prefix| H; - I --> F; -``` - -### Usage Example - -```kotlin -val toMatch = setOf("path/to/include", "another/path/to/include") -val doNotMatch = sortedSetOf("path/to/exclude", "another/path/to/exclude") -val result = RuleTreeBuilder.getRuleExpression(toMatch, doNotMatch, true) -println(result) -``` - -This will generate a Kotlin `when` expression that can be used to determine if a given path should be included or excluded based on the specified criteria. - -### Conclusion - -The `RuleTreeBuilder` provides a powerful tool for dynamically generating optimized string matching rules in Kotlin. By leveraging prefix analysis and entropy calculations, it -efficiently condenses complex sets of matching criteria into concise `when` expressions, suitable for a wide range of applications where dynamic string filtering is required. - -# kotlin\com\simiacryptus\skyenet\core\util\Selenium.kt - -## Selenium Interface Documentation - -The `Selenium` interface is part of the `com.simiacryptus.skyenet.core.util` package and is designed to provide an abstraction layer for web scraping and automation tasks using -Selenium WebDriver. It extends the `AutoCloseable` interface, ensuring that resources are automatically released when no longer needed. - -### Interface Overview - -```java -package com.simiacryptus.skyenet.core.util; - -import java.net.URL; - -interface Selenium : AutoCloseable { - fun save( - url: URL, - currentFilename: String?, - saveRoot: String - ) -} -``` - -#### Methods - -##### `save` - -The `save` method is responsible for saving the content of a web page to a specified location on the disk. - -- **Parameters:** - - `url`: The URL of the web page to be saved. - - `currentFilename`: An optional filename to use for saving the page. If `null`, a default or generated filename may be used based on the implementation. - - `saveRoot`: The root directory where the web page will be saved. The method may create subdirectories within this root based on the URL or other criteria. - -- **Returns:** This method does not return a value. - -- **Throws:** Implementations may throw exceptions to indicate errors such as invalid URLs, IO errors, or permission issues. - -#### Diagram: Workflow of `save` Method - -To illustrate how the `save` method could be utilized within a system, consider the following sequence diagram created using Mermaid.js syntax: - -```mermaid -sequenceDiagram - participant Client as Client Code - participant Selenium as Selenium Interface - participant FileSystem as File System - - Client->>+Selenium: save(url, currentFilename, saveRoot) - alt if currentFilename is not null - Selenium->>+FileSystem: Create/Overwrite file at saveRoot/currentFilename - else - Selenium->>+FileSystem: Generate filename and create file at saveRoot - end - FileSystem-->>-Selenium: Confirm file saved - Selenium-->>-Client: Return -``` - -This diagram demonstrates the basic flow when a client code calls the `save` method. Depending on whether `currentFilename` is provided, the implementation might either directly -use it to create or overwrite a file at the specified `saveRoot` or generate a new filename based on its logic. - -#### Usage Example - -Below is a hypothetical usage example of the `Selenium` interface: - -```kotlin -class MySeleniumImpl : Selenium { - override fun save(url: URL, currentFilename: String?, saveRoot: String) { - // Implementation details here - } - - override fun close() { - // Cleanup resources here - } -} - -fun main() { - val selenium = MySeleniumImpl() - try { - selenium.save(URL("http://example.com"), "examplePage.html", "/path/to/saveRoot") - } finally { - selenium.close() - } -} -``` - -In this example, `MySeleniumImpl` is a concrete implementation of the `Selenium` interface. The `main` function demonstrates how to use the `save` method to save the content -of "http://example.com" to a specified path, and then it ensures that resources are properly released by calling `close`. - -#### Conclusion - -The `Selenium` interface provides a structured way to interact with web pages for the purpose of saving their content locally. By implementing this interface, developers can create -flexible and reusable web scraping or automation tools that leverage the power of Selenium WebDriver while managing resources efficiently. - -# kotlin\com\simiacryptus\skyenet\core\util\StringSplitter.kt - -## Developer Documentation for `StringSplitter` - -The `StringSplitter` object in the `com.simiacryptus.skyenet.core.util` package provides a utility function for splitting a string based on a set of specified separators and their -associated weights. This document outlines the functionality and usage of the `StringSplitter` object. - -### Overview - -The `StringSplitter` object contains a single public function, `split`, which takes a string and a map of separators with their corresponding weights. It returns a pair of strings, -representing the text split at the optimal point determined by the algorithm. - -#### Functionality - -- **split**: Splits a given string into two parts based on the optimal separator found in the provided map. The optimality is calculated using a weighted scoring system that - considers the position of the separator and its weight. - -#### Usage - -To use the `StringSplitter`, you need to call the `split` function with the text you want to split and a map of separators with their weights. - -```kotlin -val result = StringSplitter.split( - text = "Your text here", - seperators = mapOf( - "." to 2.0, - "," to 1.5, - " " to 1.0 - ) -) -println(result.toList().joinToString("\n")) -``` - -### Algorithm - -The `split` function works as follows: - -1. For each separator in the provided map, it searches the text for occurrences. -2. For each occurrence, it calculates a score based on the position in the text and the weight of the separator. -3. It selects the separator occurrence with the highest score as the split point. -4. The text is split at the selected point, and a pair of strings is returned. - -#### Score Calculation - -The score for each separator occurrence is calculated using the formula: - -``` -score = (b * log(a)) + (a * log(b)) / weight -``` - -where: - -- `a` is the ratio of the separator's position to the text length, -- `b` is `1 - a`, -- `weight` is the weight of the separator from the input map. - -### Mermaid Diagram - -The following Mermaid diagram illustrates the process flow of the `split` function: - -```mermaid -graph TD - A[Start] --> B{For each separator} - B --> C[Find occurrences] - C --> D{For each occurrence} - D --> E[Calculate score] - E --> F{Is it the best score?} - F -->|Yes| G[Update best score] - F -->|No| H[Continue] - G --> H - H --> I{Any more occurrences?} - I -->|Yes| D - I -->|No| J{Any more separators?} - J -->|Yes| B - J -->|No| K[Split at best score] - K --> L[End] -``` - -### Example - -The `main` function in the `StringSplitter` object provides an example of how to use the `split` function: - -```kotlin -@JvmStatic -fun main(args: Array) { - println( - split( - text = "This is a test. This is only a test. If this were a real emergency, you would be instructed to panic.", - seperators = mapOf( - "." to 2.0, - " " to 1.0, - ", " to 2.0, - ) - ).toList().joinToString("\n")) -} -``` - -This example demonstrates splitting a sample text using periods, spaces, and commas as separators with specified weights. - -### Conclusion - -The `StringSplitter` provides a flexible and efficient way to split strings based on weighted separators. Its algorithm ensures that the split point is chosen optimally, -considering both the position of the separators and their assigned importance. - -# kotlin\com\simiacryptus\skyenet\interpreter\Interpreter.kt - -## Skyenet Interpreter Interface Documentation - -The `Interpreter` interface is a core component of the Skyenet project, designed to provide a flexible foundation for implementing various programming language interpreters. This -document outlines the structure, functionality, and usage of the `Interpreter` interface, along with a guide on extending it for custom implementations. - -### Interface Overview - -The `Interpreter` interface defines a set of methods essential for interpreting code written in a specific programming language. Implementations of this interface are expected to -provide the logic for executing code, validating syntax, and managing symbols within the language's scope. - -#### Key Methods - -- `getLanguage()`: Returns the name of the programming language supported by the interpreter. -- `getSymbols()`: Returns a map of symbols (variables, functions, etc.) that are available in the current context of the interpreter. -- `run(code: String)`: Executes the given code string and returns the result. -- `validate(code: String)`: Checks the given code string for syntax errors or other issues. Returns a `Throwable` if any issues are found, or `null` if the code is valid. -- `wrapCode(code: String)`: Provides a default implementation that simply returns the input code. This method can be overridden to modify or preprocess code before execution. -- `wrapExecution(fn: java.util.function.Supplier)`: Executes the supplied function within a context or wrapper defined by the interpreter. This is useful for handling - exceptions, logging, or other cross-cutting concerns. - -#### Companion Object - -The companion object of the `Interpreter` interface contains utility methods and classes for testing implementations of the interface. - -##### `test(factory: java.util.function.Function, Interpreter>)` - -This static method facilitates testing of interpreter implementations. It uses a factory function to create instances of the interpreter with predefined symbols and then runs test -cases to verify the implementation's correctness. - -#### Usage Example - -Implementing a simple interpreter might involve extending the `Interpreter` interface and providing implementations for the abstract methods. Below is a hypothetical example of an -interpreter for a simple scripting language: - -```kotlin -class SimpleScriptInterpreter : Interpreter { - private val symbols = mutableMapOf() - - override fun getLanguage(): String = "SimpleScript" - - override fun getSymbols(): Map = symbols - - override fun run(code: String): Any? { - // Implementation of code execution logic - } - - override fun validate(code: String): Throwable? { - // Implementation of code validation logic - } -} -``` - -### Extending the Interface - -To create a custom interpreter, one must implement all abstract methods of the `Interpreter` interface. Additionally, the `wrapCode` and `wrapExecution` methods can be overridden -to provide custom behavior for code preprocessing and execution wrapping, respectively. - -### Testing Interpreters - -The companion object's `test` method provides a convenient way to test custom interpreter implementations. It automatically sets up test cases based on provided symbols and -expected outcomes, simplifying the process of verifying the correctness of interpreter logic. - -### Diagrams - -To better understand the structure and relationships within the `Interpreter` interface, the following Mermaid diagram illustrates its key components and their interactions: - -```mermaid -classDiagram - class Interpreter { - <> - +getLanguage() String - +getSymbols() Map~String, Any~ - +run(code: String) Any? - +validate(code: String) Throwable? - +wrapCode(code: String) String - +wrapExecution(fn: Supplier~T?~) T? - } - class TestObject { - +square(x: Int) Int - } - class TestInterface { - <> - +square(x: Int) Int - } - Interpreter <|-- TestObject - Interpreter <|-- TestInterface - Interpreter : +test(factory: Function~Map~String, Any~, Interpreter~) -``` - -This diagram shows the `Interpreter` interface, its methods, and how it relates to test utilities provided within the companion object. Implementing classes or objects ( -`TestObject`, `TestInterface`) should provide concrete implementations of the interface's methods to fulfill the contract of an interpreter. - -### Conclusion - -The `Interpreter` interface serves as a foundational component for building interpreters in the Skyenet project. By following the guidelines and utilizing the testing utilities -provided, developers can create robust and flexible interpreters for various programming languages. - -# kotlin\com\simiacryptus\skyenet\interpreter\InterpreterTestBase.kt - -## Interpreter Test Base Documentation - -The `InterpreterTestBase` class serves as an abstract base for testing implementations of an interpreter. It provides a structured way to validate both the execution and validation -capabilities of an interpreter with various inputs, including valid code, invalid code, and code that utilizes variables and tools. - -### Overview - -The `InterpreterTestBase` class includes a series of JUnit tests designed to ensure that an interpreter correctly handles different scenarios. These tests cover: - -- Execution of valid and invalid code -- Validation of code correctness -- Handling of variables and tools within the code - -### Key Methods - -- `newInterpreter(map: Map): Interpreter` - An abstract method that should be implemented to return an instance of the interpreter to be tested, optionally initialized - with a map of variables. - -### Test Cases - -#### Execution Tests - -1. **Valid Code Execution**: Tests if the interpreter can correctly execute a simple arithmetic operation. -2. **Invalid Code Execution**: Ensures the interpreter throws an exception when attempting to execute syntactically incorrect code. -3. **Execution with Variables**: Checks if the interpreter can handle code that uses predefined variables. -4. **Execution with Tool**: Verifies the interpreter's ability to execute code that interacts with an object (referred to as a "tool"). -5. **Invalid Tool Usage**: Tests the interpreter's error handling when code attempts to call a non-existent method on a tool. - -#### Validation Tests - -1. **Valid Code Validation**: Confirms that the interpreter correctly identifies syntactically correct code as valid. -2. **Invalid Code Validation**: Ensures the interpreter identifies syntactically incorrect code as invalid. -3. **Validation with Variables**: Checks if the interpreter correctly validates code that uses predefined variables. -4. **Validation with Tool**: Verifies that the interpreter can validate code that interacts with a tool. -5. **Invalid Tool Usage in Validation**: Tests the interpreter's validation logic when code attempts to call a non-existent method on a tool. -6. **Undefined Variable Validation**: Ensures the interpreter correctly flags code that uses undefined variables as invalid. - -### Diagrams - -#### Test Flow Diagram - -```mermaid -flowchart TD - A[Start Test] --> B{Is Code Valid?} - B -->|Yes| C[Run Code] - B -->|No| D[Validate Code] - C --> E{Does Code Use Variables or Tools?} - D --> E - E -->|No| F[Check Execution/Validation Result] - E -->|Yes| G[Initialize Interpreter with Variables/Tools] - G --> H[Run/Validate Code with Context] - H --> F - F --> I[End Test] -``` - -This diagram illustrates the general flow of tests in the `InterpreterTestBase` class. It highlights the decision points based on code validity and the use of variables or tools, -leading to different paths for execution and validation. - -### Usage - -To use the `InterpreterTestBase` class, follow these steps: - -1. **Extend the Class**: Create a new class that extends `InterpreterTestBase`. -2. **Implement `newInterpreter`**: Provide an implementation for the `newInterpreter` method to return an instance of your interpreter, optionally initialized with variables or - tools. -3. **Run Tests**: Execute the tests to validate your interpreter's functionality. - -### Example - -```kotlin -class MyInterpreterTest : InterpreterTestBase() { - override fun newInterpreter(map: Map): Interpreter { - return MyInterpreter(map) - } -} -``` - -This example demonstrates how to extend the `InterpreterTestBase` class for testing a custom interpreter implementation. - diff --git a/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/AwsPlatform.kt b/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/AwsPlatform.kt index bd778433..0a68078e 100644 --- a/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/AwsPlatform.kt +++ b/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/AwsPlatform.kt @@ -2,6 +2,7 @@ package com.simiacryptus.skyenet.core.platform import com.simiacryptus.skyenet.core.platform.model.CloudPlatformInterface import org.slf4j.LoggerFactory +import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider import software.amazon.awssdk.core.SdkBytes import software.amazon.awssdk.core.sync.RequestBody import software.amazon.awssdk.regions.Region @@ -13,25 +14,29 @@ import software.amazon.awssdk.services.s3.model.PutObjectRequest import java.nio.charset.StandardCharsets import java.util.* + open class AwsPlatform( private val bucket: String = System.getProperty("share_bucket", "share.simiacrypt.us"), - override val shareBase: String = System.getProperty("share_base", "https://share.simiacrypt.us"), - private val region: Region? = Region.US_EAST_1 + override val shareBase: String = System.getProperty("share_base", "https://" + bucket), + private val region: Region? = Region.US_EAST_1, + private val profileName: String = "default", ) : CloudPlatformInterface { + open val credentialsProvider: ProfileCredentialsProvider? = ProfileCredentialsProvider.create(profileName) private val log = LoggerFactory.getLogger(AwsPlatform::class.java) protected open val kmsClient: KmsClient by lazy { log.debug("Initializing KMS client for region: {}", Region.US_EAST_1) - KmsClient.builder().region(Region.US_EAST_1) - //.credentialsProvider(ProfileCredentialsProvider.create("data")) - .build() + var clientBuilder = KmsClient.builder().region(Region.US_EAST_1) + if(null != credentialsProvider) clientBuilder = clientBuilder.credentialsProvider(credentialsProvider) + clientBuilder.build() } protected open val s3Client: S3Client by lazy { log.debug("Initializing S3 client for region: {}", region) - S3Client.builder() - .region(region) - .build() + var clientBuilder = S3Client.builder() + if(null != credentialsProvider) clientBuilder = clientBuilder.credentialsProvider(credentialsProvider) + clientBuilder = clientBuilder.region(region) + clientBuilder.build() } override fun upload( diff --git a/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/ClientManager.kt b/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/ClientManager.kt index 0cf52cc9..4ecafe60 100644 --- a/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/ClientManager.kt +++ b/core/src/main/kotlin/com/simiacryptus/skyenet/core/platform/ClientManager.kt @@ -45,7 +45,7 @@ open class ClientManager { ThreadPoolExecutor( 0, Integer.MAX_VALUE, 500, TimeUnit.MILLISECONDS, - SynchronousQueue(), + ArrayBlockingQueue(1), RecordingThreadFactory(session, user) ) diff --git a/docs/Document_Indexing.md b/docs/Document_Indexing.md deleted file mode 100644 index 8be6e954..00000000 --- a/docs/Document_Indexing.md +++ /dev/null @@ -1,76 +0,0 @@ -# User Guide: Document Data Extraction and Query Index Creation - -This guide covers two main features: Document Data Extraction and Query Index Creation. These tools are designed to help you extract structured data from various document types and -create searchable indexes for efficient querying. - -## 1. Document Data Extraction - -### Overview - -The Document Data Extractor allows you to parse and extract structured information from PDF, TXT, MD, and HTML files. It uses AI to analyze the content and create a hierarchical -JSON representation of the document's structure, entities, and metadata. - -### How to Use - -1. In your IDE, right-click on a supported file (PDF, TXT, MD, or HTML) in the project explorer. -2. Select the "Document Data Extractor" option from the context menu. -3. A configuration dialog will appear with the following options: - -- DPI: Set the resolution for image rendering (for PDFs). -- Max Pages: Limit the number of pages to process. -- Output Format: Choose the format for saved images (PNG, JPEG, GIF, BMP). -- Pages Per Batch: Set how many pages to process in each batch. -- Show Images: Toggle whether to display rendered images in the results. -- Save Image Files: Choose to save rendered images to disk. -- Save Text Files: Choose to save extracted text to disk. -- Save Final JSON: Choose to save the final parsed JSON to disk. - -4. Click "OK" to start the extraction process. -5. A new browser window will open, showing the progress and results of the extraction. - -### Output - -- The extracted data will be displayed in the browser, organized by pages or batches. -- If enabled, image files, text files, and the final JSON will be saved in an "output" directory next to the source file. -- The final JSON file will have a ".parsed.json" extension. - -## 2. Query Index Creation - -### Overview - -The Query Index Creator takes the parsed JSON files from the Document Data Extractor and creates a binary index file that can be efficiently searched using embedding-based -similarity search. - -### How to Use - -1. In your IDE, select one or more ".parsed.json" files in the project explorer. -2. Right-click and choose the "Save As Query Index" option from the context menu. -3. A file chooser dialog will appear. Select the directory where you want to save the index file. -4. Click "OK" to start the conversion process. -5. A progress bar will show the status of the index creation. - -### Output - -- A binary index file named "document.index.data" will be created in the selected output directory. -- This index file can be used for fast similarity searches on the extracted document data. - -## Using the Query Index - -Once you have created the query index, you can use it with the EmbeddingSearchTask to perform similarity searches on your document data. This allows you to quickly find relevant -information across all your indexed documents. - -To use the EmbeddingSearchTask: - -1. Set up your search query and parameters (e.g., distance type, number of results). -2. Point the task to your "document.index.data" file. -3. Run the search to get the most relevant results based on embedding similarity. - -## Tips and Best Practices - -1. For large documents, consider processing them in smaller batches by adjusting the "Max Pages" and "Pages Per Batch" settings. -2. Save the final JSON files when extracting data, as these are required to create the query index. -3. Organize your parsed JSON files in a dedicated folder to make it easier to select them when creating the query index. -4. When creating the query index, choose an output location that is easily accessible for your search tasks. -5. Experiment with different DPI settings for PDFs to balance image quality and processing speed. -6. Use the "Show Images" option during extraction to visually verify the content being processed, especially for PDFs. - diff --git a/docs/actors_intro.md b/docs/actors_intro.md deleted file mode 100644 index 0b033b92..00000000 --- a/docs/actors_intro.md +++ /dev/null @@ -1,101 +0,0 @@ -## Introduction to the LLM Actor Framework - -The LLM (Large Language Model) Actor Framework is a sophisticated and extensible architecture designed to facilitate the -integration and utilization of large language models, such as those provided by OpenAI, in various applications. This -framework abstracts the complexities involved in interacting with these models and provides a structured way to define, -manage, and extend different types of actors that can perform a wide range of tasks, from coding assistance to image -generation and text-to-speech conversion. - -### Core Components - -The framework is built around a few core components, each serving a specific purpose. These components include: - -1. **BaseActor**: The abstract base class that defines the common interface and behavior for all actors. -2. **CodingActor**: An actor specialized in generating and executing code based on natural language instructions. -3. **ImageActor**: An actor designed to transform textual descriptions into images using image generation models. -4. **ParsedActor**: An actor that parses textual input into structured data, such as JSON objects. -5. **SimpleActor**: A straightforward actor that generates text responses based on input questions. -6. **TextToSpeechActor**: An actor that converts text into speech using text-to-speech models. - -### BaseActor - -The `BaseActor` class is the cornerstone of the framework. It defines the essential methods and properties that all -actors must implement or override. These include: - -- **prompt**: A string that defines the initial prompt or instructions for the actor. -- **name**: An optional name for the actor. -- **model**: The language model to be used by the actor. -- **temperature**: A parameter that controls the randomness of the model's output. - -The `BaseActor` class also defines abstract methods such as `respond`, `chatMessages`, and `withModel`, which must be -implemented by subclasses. - -### CodingActor - -The `CodingActor` class extends `BaseActor` and is tailored for coding tasks. It can generate code snippets, execute -them, and handle errors through iterative refinement. Key features include: - -- **interpreterClass**: The class of the interpreter to be used for executing code. -- **symbols**: A map of predefined symbols available in the coding environment. -- **describer**: A type describer that provides descriptions of the available symbols. -- **details**: Additional details or instructions for the actor. -- **fallbackModel**: A fallback language model to be used if the primary model fails. -- **runtimeSymbols**: Symbols available at runtime. - -The `CodingActor` class also defines nested classes and interfaces such as `CodeRequest`, `CodeResult`, -and `ExecutionResult` to encapsulate the request and response structures. - -### ImageActor - -The `ImageActor` class is designed for generating images from textual descriptions. It extends `BaseActor` and adds -properties specific to image generation, such as: - -- **imageModel**: The image generation model to be used. -- **width**: The width of the generated image. -- **height**: The height of the generated image. - -The `ImageActor` class also defines an inner class `ImageResponseImpl` that encapsulates the response, including the -generated image. - -### ParsedActor - -The `ParsedActor` class focuses on parsing textual input into structured data. It extends `BaseActor` and introduces -properties such as: - -- **resultClass**: The class of the result object. -- **exampleInstance**: An example instance of the result class. -- **parsingModel**: The model to be used for parsing. -- **deserializerRetries**: The number of retries for deserialization. -- **describer**: A type describer for the result class. - -The `ParsedActor` class also defines an inner class `ParsedResponseImpl` that encapsulates the parsed response. - -### SimpleActor - -The `SimpleActor` class is a minimalistic actor that generates text responses based on input questions. It -extends `BaseActor` and implements the necessary methods to handle simple question-and-answer interactions. - -### TextToSpeechActor - -The `TextToSpeechActor` class converts text into speech. It extends `BaseActor` and adds properties specific to -text-to-speech conversion, such as: - -- **audioModel**: The audio model to be used. -- **voice**: The voice to be used for speech synthesis. -- **speed**: The speed of the speech. - -The `TextToSpeechActor` class also defines an inner class `SpeechResponseImpl` that encapsulates the speech response, -including the generated audio data. - -### Extensibility and Customization - -The LLM Actor Framework is designed to be highly extensible and customizable. Developers can create new actors by -extending the `BaseActor` class and implementing the required methods. The framework also supports the use of different -language models, interpreters, and type describers, making it adaptable to various use cases and domains. - -### Conclusion - -The LLM Actor Framework provides a robust and flexible foundation for integrating large language models into -applications. By abstracting the complexities of model interaction and providing a structured way to define and manage -actors, the framework empowers developers to harness the full potential of large language models in a wide range of -tasks, from coding assistance to image generation and beyond. \ No newline at end of file diff --git a/docs/apps.md b/docs/apps.md deleted file mode 100644 index a91c0f9e..00000000 --- a/docs/apps.md +++ /dev/null @@ -1,102 +0,0 @@ -## Overview of Base Application Classes - -This document provides an overview of the base application classes used in the provided code snippets. These classes -form the foundation for various application functionalities, including coding assistance, web development, and testing -different types of actors. - -### 1. `CodingAgent` - -The `CodingAgent` class is designed to facilitate coding assistance using an interpreter. It interacts with the OpenAI -API to generate code based on user input and provides mechanisms to display and execute the generated code. The class -extends `ActorSystem` and utilizes a `CodingActor` to handle code generation. - -**Key Features:** - -- **Initialization:** Takes parameters like API, storage, session, user, UI, interpreter class, symbols, temperature, - details, model, and main task. -- **Actor Management:** Manages a map of actors, with `CodingActor` being the primary actor. -- **Code Request Handling:** Generates code requests and handles the response. -- **Code Display and Feedback:** Displays the generated code and provides feedback mechanisms. -- **Execution:** Executes the generated code and handles execution errors. - -### 2. `ShellToolAgent` - -The `ShellToolAgent` class extends `CodingAgent` and adds functionalities specific to shell tool operations. It provides -mechanisms to export code prototypes, generate data schemas, and create servlets. - -**Key Features:** - -- **Tool Button Creation:** Adds a button to create tools from the generated code. -- **Schema Actor:** Generates data schemas from code prototypes. -- **Servlet Actor:** Converts code prototypes into servlets. -- **OpenAPI Integration:** Generates OpenAPI definitions for the servlets. -- **Test Page Generation:** Creates test pages for the generated servlets. - -### 3. `WebDevApp` - -The `WebDevApp` class is designed to assist in web development. It provides functionalities to translate user ideas into -detailed web application architectures and generate the necessary HTML, CSS, JavaScript, and image files. - -**Key Features:** - -- **Architecture Discussion:** Translates user ideas into detailed web application architectures. -- **File Drafting:** Drafts HTML, CSS, JavaScript, and image files based on the architecture. -- **Code Review:** Reviews and refines the generated code. -- **Task Management:** Manages tasks for drafting and reviewing code. - -### 4. `ProcessInterpreter` - -The `ProcessInterpreter` class provides an implementation of the `Interpreter` interface for running shell commands. It -wraps the provided code and executes it using a specified command. - -**Key Features:** - -- **Command Execution:** Executes shell commands based on the provided definitions. -- **Output Handling:** Handles the output and errors from the executed commands. -- **Code Wrapping:** Wraps the provided code before execution. - -### 5. `SimpleActorTestApp` - -The `SimpleActorTestApp` class is designed to test `SimpleActor` instances. It provides a simple interface for users to -interact with the actor and view the responses. - -**Key Features:** - -- **User Message Handling:** Handles user messages and passes them to the actor. -- **Response Display:** Displays the actor's responses in a user-friendly format. -- **Settings Management:** Manages settings for the actor. - -### 6. `ParsedActorTestApp` - -The `ParsedActorTestApp` class is designed to test `ParsedActor` instances. It provides functionalities to handle user -messages, parse the responses, and display them. - -**Key Features:** - -- **User Message Handling:** Handles user messages and passes them to the actor. -- **Response Parsing:** Parses the actor's responses and displays them in a structured format. -- **Settings Management:** Manages settings for the actor. - -### 7. `ImageActorTestApp` - -The `ImageActorTestApp` class is designed to test `ImageActor` instances. It provides functionalities to handle user -messages, generate images, and display them. - -**Key Features:** - -- **User Message Handling:** Handles user messages and passes them to the actor. -- **Image Generation:** Generates images based on the actor's responses. -- **Response Display:** Displays the generated images and the actor's responses. -- **Settings Management:** Manages settings for the actor. - -### 8. `CodingActorTestApp` - -The `CodingActorTestApp` class is designed to test `CodingActor` instances. It provides functionalities to handle user -messages, generate code, and execute it. - -**Key Features:** - -- **User Message Handling:** Handles user messages and passes them to the actor. -- **Code Generation:** Generates code based on the actor's responses. -- **Code Execution:** Executes the generated code and displays the results. -- **Settings Management:** Manages settings for the actor. diff --git a/docs/core_platform.md b/docs/core_platform.md deleted file mode 100644 index 42745bda..00000000 --- a/docs/core_platform.md +++ /dev/null @@ -1,184 +0,0 @@ -## Application Services Platform: Manual and Architecture Review - -### Overview - -The Application Services Platform is designed to manage various services related to user authentication, authorization, -data storage, client management, and usage tracking. The platform is modular and extensible, allowing for easy -integration with various backend services and cloud platforms. - -### Components - -1. **ApplicationServicesConfig** -2. **ApplicationServices** -3. **UserSettingsManager** -4. **ClientManager** -5. **HSQLUsageManager** -6. **AuthorizationManager** -7. **AuthenticationManager** -8. **DataStorage** -9. **AwsPlatform** - -### 1. ApplicationServicesConfig - -This object holds configuration settings for the application services. It includes: - -- `isLocked`: A flag to prevent changes after the configuration is locked. -- `dataStorageRoot`: The root directory for data storage. - -### 2. ApplicationServices - -This object is the central hub for managing various services. It includes: - -- `authorizationManager`: Manages user authorization. -- `userSettingsManager`: Manages user settings. -- `authenticationManager`: Manages user authentication. -- `dataStorageFactory`: Factory for creating data storage instances. -- `clientManager`: Manages API clients. -- `cloud`: Interface for cloud platform services. -- `seleniumFactory`: Factory for creating Selenium instances. -- `usageManager`: Manages usage tracking. - -### 3. UserSettingsManager - -This class manages user settings, including loading and saving settings to disk. It implements `UserSettingsInterface`. - -### 4. ClientManager - -This class manages API clients, including creating and caching clients for sessions and users. It includes: - -- `getClient`: Fetches or creates a client for a session and user. -- `getPool`: Fetches or creates a thread pool for a session and user. -- `getScheduledPool`: Fetches or creates a scheduled thread pool for a session and user. -- `MonitoredClient`: A subclass of `OpenAIClient` that tracks usage and enforces budget limits. - -### 5. HSQLUsageManager - -This class manages usage tracking using an HSQL database. It implements `UsageInterface` and includes methods for -incrementing usage, getting usage summaries, and clearing usage data. - -### 6. AuthorizationManager - -This class manages user authorization, checking if a user is authorized for specific operations. It -implements `AuthorizationInterface`. - -### 7. AuthenticationManager - -This class manages user authentication, including storing and retrieving users based on access tokens. It -implements `AuthenticationInterface`. - -### 8. DataStorage - -This class manages data storage, including storing and retrieving messages, sessions, and other data. It -implements `StorageInterface`. - -### 9. AwsPlatform - -This class provides integration with AWS services, including S3 for file storage and KMS for encryption. It -implements `CloudPlatformInterface`. - -## Manual - -### Setup - -1. **Configuration** - - Set the `dataStorageRoot` in `ApplicationServicesConfig` to the desired root directory for data storage. - - Lock the configuration by setting `isLocked` to `true`. - -2. **Service Initialization** - - Initialize the various services in `ApplicationServices` as needed. For example: - ```kotlin - ApplicationServices.userSettingsManager = UserSettingsManager() - ApplicationServices.authorizationManager = AuthorizationManager() - ApplicationServices.authenticationManager = AuthenticationManager() - ApplicationServices.dataStorageFactory = { DataStorage(it) } - ApplicationServices.clientManager = ClientManager() - ApplicationServices.cloud = AwsPlatform.get() - ApplicationServices.usageManager = HSQLUsageManager(File(dataStorageRoot, "usage")) - ``` - -### Usage - -1. **User Authentication** - - To authenticate a user, use the `AuthenticationManager`: - ```kotlin - val user = User(email = "user@example.com", name = "John Doe") - val accessToken = "someAccessToken" - ApplicationServices.authenticationManager.putUser(accessToken, user) - ``` - -2. **User Authorization** - - To check if a user is authorized for an operation, use the `AuthorizationManager`: - ```kotlin - val isAuthorized = ApplicationServices.authorizationManager.isAuthorized( - applicationClass = SomeClass::class.java, - user = user, - operationType = AuthorizationInterface.OperationType.Read - ) - ``` - -3. **User Settings** - - To get and update user settings, use the `UserSettingsManager`: - ```kotlin - val settings = ApplicationServices.userSettingsManager.getUserSettings(user) - ApplicationServices.userSettingsManager.updateUserSettings(user, settings.copy(apiKeys = newApiKeys)) - ``` - -4. **Data Storage** - - To store and retrieve data, use the `DataStorage`: - ```kotlin - val session = Session("someSessionId") - val messages = ApplicationServices.dataStorageFactory(dataStorageRoot).getMessages(user, session) - ``` - -5. **Client Management** - - To get an API client, use the `ClientManager`: - ```kotlin - val client = ApplicationServices.clientManager.getClient(session, user) - ``` - -6. **Usage Tracking** - - To track usage, use the `UsageManager`: - ```kotlin - ApplicationServices.usageManager.incrementUsage(session, user, model, tokens) - ``` - -### Architecture Review - -#### Strengths - -1. **Modularity**: The platform is highly modular, with clear separation of concerns between different services. -2. **Extensibility**: The use of interfaces and factory methods makes it easy to extend and customize the platform. -3. **Logging**: Extensive logging is used throughout the platform, aiding in debugging and monitoring. -4. **Cloud Integration**: The platform includes integration with AWS services, making it suitable for cloud-based - applications. - -#### Weaknesses - -1. **Complexity**: The platform's modularity and extensibility come at the cost of increased complexity, which may make - it harder to understand and maintain. -2. **Error Handling**: While there is some error handling, it could be improved to provide more robust and user-friendly - error messages. -3. **Thread Safety**: Some classes, such as `DataStorage`, use synchronized blocks, but the overall thread safety of the - platform should be reviewed and tested. - -#### Opportunities - -1. **Documentation**: Improving the documentation, including code comments and usage examples, would make the platform - more accessible to developers. -2. **Testing**: Adding unit tests and integration tests would improve the reliability and maintainability of the - platform. -3. **Performance Optimization**: Profiling and optimizing the performance of the platform, especially in areas like data - storage and client management, could improve its efficiency. - -#### Threats - -1. **Dependency Management**: The platform relies on several external libraries and services, which may introduce - compatibility and security issues. -2. **Security**: Ensuring the security of user data and credentials, especially when integrating with cloud services, is - critical. - -### Conclusion - -The Application Services Platform is a powerful and flexible solution for managing various backend services. By -addressing the identified weaknesses and opportunities, it can be further improved to meet the needs of modern -applications. \ No newline at end of file diff --git a/docs/core_user_documentation.md b/docs/core_user_documentation.md deleted file mode 100644 index 99f59d2a..00000000 --- a/docs/core_user_documentation.md +++ /dev/null @@ -1,1621 +0,0 @@ - - -* [Skyenet Core Actors Package Documentation](#skyenet-core-actors-package-documentation) - * [Package Overview](#package-overview) - * [Core Classes](#core-classes) - * [Supporting Classes and Interfaces](#supporting-classes-and-interfaces) - * [Documentation and Testing Base Classes](#documentation-and-testing-base-classes) -* [BaseActor.kt](#baseactorkt) - * [User Documentation for BaseActor Class](#user-documentation-for-baseactor-class) - * [Overview](#overview) - * [Key Features](#key-features) - * [Constructor Parameters](#constructor-parameters) - * [Methods](#methods) - * [`respond(input: I, api: API, vararg messages: ApiModel.ChatMessage): R`](#respondinput-i-api-api-vararg-messages-apimodelchatmessage-r) - * [ - `response(vararg input: ApiModel.ChatMessage, model: OpenAIModel = this.model, api: API): List`](#responsevararg-input-apimodelchatmessage-model-openaimodel--thismodel-api-api-listapimodelchatmessage) - * [`answer(input: I, api: API): R`](#answerinput-i-api-api-r) - * [`chatMessages(questions: I): Array`](#chatmessagesquestions-i-arrayapimodelchatmessage) - * [`withModel(model: ChatModels): BaseActor`](#withmodelmodel-chatmodels-baseactorir) - * [Usage Example](#usage-example) - * [Conclusion](#conclusion) -* [CodingActor.kt](#codingactorkt) - * [CodingActor Documentation](#codingactor-documentation) - * [Overview](#overview-1) - * [Key Features](#key-features-1) - * [Usage](#usage) - * [Initialization](#initialization) - * [Processing Requests](#processing-requests) - * [Handling Results](#handling-results) - * [Advanced Features](#advanced-features) - * [Conclusion](#conclusion-1) -* [ImageActor.kt](#imageactorkt) - * [ImageActor Documentation](#imageactor-documentation) - * [Features](#features) - * [Usage](#usage-1) - * [Initialization](#initialization-1) - * [Generating Images](#generating-images) - * [Accessing the Generated Image](#accessing-the-generated-image) - * [Customizing the Actor](#customizing-the-actor) - * [Interfaces and Classes](#interfaces-and-classes) - * [Conclusion](#conclusion-2) -* [ActorSystem.kt](#actorsystemkt) - * [ActorSystem Documentation](#actorsystem-documentation) - * [Overview](#overview-2) - * [Key Components](#key-components) - * [Features](#features-1) - * [Actor Retrieval](#actor-retrieval) - * [Interceptors](#interceptors) - * [Function Wrapping](#function-wrapping) - * [Usage](#usage-2) - * [Conclusion](#conclusion-3) -* [opt\Expectation.kt](#optexpectationkt) - * [Skyenet Core Expectation Module Documentation](#skyenet-core-expectation-module-documentation) - * [Overview](#overview-3) - * [VectorMatch Class](#vectormatch-class) - * [Parameters:](#parameters) - * [Methods:](#methods-1) - * [ContainsMatch Class](#containsmatch-class) - * [Parameters:](#parameters-1) - * [Methods:](#methods-2) - * [Usage](#usage-3) - * [Example](#example) -* [ParsedActor.kt](#parsedactorkt) - * [ParsedActor Class Documentation](#parsedactor-class-documentation) - * [Features](#features-2) - * [Constructor Parameters](#constructor-parameters-1) - * [Methods](#methods-3) - * [`chatMessages(questions: List)`](#chatmessagesquestions-liststring) - * [`getParser(api: API)`](#getparserapi-api) - * [`respond(input: List, api: API, vararg messages: ApiModel.ChatMessage)`](#respondinput-liststring-api-api-vararg-messages-apimodelchatmessage) - * [`withModel(model: ChatModels)`](#withmodelmodel-chatmodels) - * [Inner Classes](#inner-classes) - * [`ParsedResponseImpl`](#parsedresponseimpl) - * [Usage Example](#usage-example-1) -* [ParsedResponse.kt](#parsedresponsekt) - * [User Documentation for ParsedResponse Class](#user-documentation-for-parsedresponse-class) - * [Overview](#overview-4) - * [Key Features](#key-features-2) - * [Properties](#properties) - * [Methods](#methods-4) - * [Usage](#usage-4) - * [Example](#example-1) - * [Conclusion](#conclusion-4) -* [record\CodingActorInterceptor.kt](#recordcodingactorinterceptorkt) - * [CodingActorInterceptor Documentation](#codingactorinterceptor-documentation) - * [Features](#features-3) - * [Usage](#usage-5) - * [Initialization](#initialization-2) - * [Overridden Methods](#overridden-methods) - * [Example: Interception Logic](#example-interception-logic) - * [Integration](#integration) - * [Conclusion](#conclusion-5) -* [opt\ActorOptimization.kt](#optactoroptimizationkt) - * [ActorOptimization Class Documentation](#actoroptimization-class-documentation) - * [Features](#features-4) - * [Key Components](#key-components-1) - * [TestCase Class](#testcase-class) - * [GeneticApi Interface](#geneticapi-interface) - * [Main Methods](#main-methods) - * [runGeneticGenerations](#rungeneticgenerations) - * [regenerate](#regenerate) - * [Usage Example](#usage-example-2) - * [Customization](#customization) - * [Conclusion](#conclusion-6) -* [record\ImageActorInterceptor.kt](#recordimageactorinterceptorkt) - * [User Documentation for ImageActorInterceptor](#user-documentation-for-imageactorinterceptor) - * [Overview](#overview-5) - * [Key Features](#key-features-3) - * [Usage](#usage-6) - * [Initialization](#initialization-3) - * [Interception and Custom Processing](#interception-and-custom-processing) - * [Methods](#methods-5) - * [`response`](#response) - * [`render`](#render) - * [Conclusion](#conclusion-7) -* [record\ParsedActorInterceptor.kt](#recordparsedactorinterceptorkt) - * [ParsedActorInterceptor Documentation](#parsedactorinterceptor-documentation) - * [Features](#features-5) - * [Usage](#usage-7) - * [Initialization](#initialization-4) - * [Responding to Inputs](#responding-to-inputs) - * [Custom Response Processing](#custom-response-processing) - * [Key Methods](#key-methods) - * [Example](#example-2) - * [Conclusion](#conclusion-8) -* [record\TextToSpeechActorInterceptor.kt](#recordtexttospeechactorinterceptorkt) - * [TextToSpeechActorInterceptor Documentation](#texttospeechactorinterceptor-documentation) - * [Features](#features-6) - * [Usage](#usage-8) - * [Initialization](#initialization-5) - * [Interception](#interception) - * [Response Interception](#response-interception) - * [Render Interception](#render-interception) - * [Customization](#customization-1) - * [Conclusion](#conclusion-9) -* [record\SimpleActorInterceptor.kt](#recordsimpleactorinterceptorkt) - * [SimpleActorInterceptor Documentation](#simpleactorinterceptor-documentation) - * [Features](#features-7) - * [Usage](#usage-9) - * [Example](#example-3) - * [Parameters](#parameters-2) - * [Methods](#methods-6) - * [Conclusion](#conclusion-10) -* [test\ImageActorTestBase.kt](#testimageactortestbasekt) - * [User Documentation for ImageActorTestBase](#user-documentation-for-imageactortestbase) - * [Overview](#overview-6) - * [Key Features](#key-features-4) - * [Getting Started](#getting-started) - * [Example Usage](#example-usage) - * [Conclusion](#conclusion-11) -* [test\CodingActorTestBase.kt](#testcodingactortestbasekt) - * [CodingActorTestBase Documentation](#codingactortestbase-documentation) - * [Overview](#overview-7) - * [Key Components](#key-components-2) - * [Properties](#properties-1) - * [Methods](#methods-7) - * [Usage](#usage-10) - * [Example](#example-4) - * [Conclusion](#conclusion-12) -* [SimpleActor.kt](#simpleactorkt) - * [SimpleActor Class Documentation](#simpleactor-class-documentation) - * [Constructor Parameters](#constructor-parameters-2) - * [Methods](#methods-8) - * [respond](#respond) - * [chatMessages](#chatmessages) - * [withModel](#withmodel) - * [Usage Example](#usage-example-3) -* [test\ActorTestBase.kt](#testactortestbasekt) - * [ActorTestBase Documentation](#actortestbase-documentation) - * [Overview](#overview-8) - * [Key Components](#key-components-3) - * [Properties](#properties-2) - * [Methods](#methods-9) - * [`opt()`](#opt) - * [`testOptimize()`](#testoptimize) - * [`testRun()`](#testrun) - * [`answer()`](#answer) - * [Companion Object](#companion-object) - * [Usage](#usage-11) -* [test\ParsedActorTestBase.kt](#testparsedactortestbasekt) - * [User Documentation for ParsedActorTestBase](#user-documentation-for-parsedactortestbase) - * [Overview](#overview-9) - * [Key Components](#key-components-4) - * [Constructor Parameters](#constructor-parameters-3) - * [Methods](#methods-10) - * [actorFactory](#actorfactory) - * [getPrompt](#getprompt) - * [resultMapper](#resultmapper) - * [Usage](#usage-12) - * [Example](#example-5) - * [Conclusion](#conclusion-13) -* [TextToSpeechActor.kt](#texttospeechactorkt) - * [TextToSpeechActor Documentation](#texttospeechactor-documentation) - * [Features](#features-8) - * [Constructor Parameters](#constructor-parameters-4) - * [Methods](#methods-11) - * [`chatMessages(questions: List)`](#chatmessagesquestions-liststring-1) - * [`render(text: String, api: API)`](#rendertext-string-api-api) - * [`respond(input: List, api: API, vararg messages: ChatMessage)`](#respondinput-liststring-api-api-vararg-messages-chatmessage) - * [`withModel(model: AudioModels)`](#withmodelmodel-audiomodels) - * [Interfaces](#interfaces) - * [`SpeechResponse`](#speechresponse) - * [Usage Example](#usage-example-4) - - - -# Skyenet Core Actors Package Documentation - -The Skyenet Core Actors package is a comprehensive framework designed to facilitate interaction with OpenAI's GPT models and other functionalities such as text-to-speech and image -generation. This package is structured to provide a robust foundation for building applications that leverage OpenAI's capabilities, including conversational agents, code -generators, and visual content creators. - -## Package Overview - -The package is organized into several key components, each serving a specific purpose within the framework. Below is an overview of the primary classes and their relationships. - -### Core Classes - -- **`BaseActor`**: Serves as the abstract base class for all actors. It defines the basic structure and functionalities common to all types of actors. -- **`SimpleActor`**: Extends `BaseActor` to provide simple interaction with OpenAI's GPT models, focusing on sending prompts and receiving text responses. -- **`ParsedActor`**: A specialized actor that parses responses into specific data types, extending the `BaseActor` functionalities. -- **`CodingActor`**: Designed for code generation and execution, this actor extends `BaseActor` and integrates with code interpreters. -- **`ImageActor`**: Focuses on generating images from textual prompts, extending the capabilities of `BaseActor` to support visual content creation. -- **`TextToSpeechActor`**: Converts text into speech, extending `BaseActor` to utilize OpenAI's text-to-speech models. - -### Supporting Classes and Interfaces - -- **`ActorSystem`**: Manages and interacts with various types of actors within a session, applying specific interceptors based on the actor type. -- **`ParsedResponse`**: Represents responses parsed from various sources in a structured manner. -- **Interceptors (`*Interceptor`)**: Specialized classes designed to intercept and potentially modify the behavior of actor instances (e.g., `CodingActorInterceptor`, - `ImageActorInterceptor`). - -### Documentation and Testing Base Classes - -- **`ActorTestBase`**: An abstract class designed to facilitate the testing of actors within the system. -- **`ParsedActorTestBase`, `CodingActorTestBase`, `ImageActorTestBase`**: Extend `ActorTestBase` to provide testing functionalities tailored to specific actor types. - -# BaseActor.kt - -## User Documentation for BaseActor Class - -The `BaseActor` class serves as an abstract foundation for creating actors that interact with OpenAI's GPT models via the `com.simiacryptus.jopenai` API. This class is designed to -facilitate the development of applications that require conversational capabilities or complex decision-making processes based on the input provided to the model. - -### Overview - -The `BaseActor` class abstracts the process of sending prompts to an OpenAI model and receiving responses. It is designed to be extended by more specific actor classes that define -the logic for generating prompts based on input and processing the model's responses. - -### Key Features - -- **Customizable Prompts and Models:** Allows specifying the prompt and the model (e.g., GPT-3.5 Turbo) to use for generating responses. -- **Temperature Control:** Enables setting the temperature parameter to control the randomness of the model's responses. -- **Convenient Response Handling:** Simplifies the process of sending requests to the OpenAI API and handling responses. - -### Constructor Parameters - -- `prompt`: The initial prompt or question to be sent to the model. -- `name`: An optional name for the actor. Useful for identification purposes in more complex scenarios. -- `model`: The OpenAI model to be used. Defaults to `OpenAIModels.GPT35Turbo`. -- `temperature`: Controls the randomness of the model's responses. Lower values make responses more deterministic. - -### Methods - -#### `respond(input: I, api: API, vararg messages: ApiModel.ChatMessage): R` - -An abstract method that must be implemented by subclasses. It defines how the actor should respond to a given input, using the specified API and any additional chat messages. - -#### `response(vararg input: ApiModel.ChatMessage, model: OpenAIModel = this.model, api: API): List` - -Sends a chat request to the OpenAI API using the provided messages and returns the response. This method can be overridden if necessary. - -#### `answer(input: I, api: API): R` - -Generates a response based on the input by internally calling `respond`. It automatically handles the conversion of input to chat messages. - -#### `chatMessages(questions: I): Array` - -An abstract method that subclasses must implement. It should convert the input into an array of `ApiModel.ChatMessage` instances, which are then used to generate the model's -response. - -#### `withModel(model: ChatModels): BaseActor` - -An abstract method that should be implemented to return a new instance of the actor with the specified model. - -### Usage Example - -To use the `BaseActor` class, you must extend it and implement the abstract methods. Here's a simplified example: - -```kotlin -class SimpleActor(prompt: String) : BaseActor(prompt) { - override fun respond(input: String, api: API, vararg messages: ApiModel.ChatMessage): String { - val response = response(*messages, api = api) - return response.joinToString(separator = "\n") { it.text } - } - - override fun chatMessages(questions: String): Array { - return arrayOf(ApiModel.ChatMessage(questions)) - } - - override fun withModel(model: ChatModels): BaseActor { - return SimpleActor(prompt).apply { this.model = model } - } -} -``` - -This example demonstrates a basic actor that sends a single message to the model and returns the concatenated text of the responses. - -### Conclusion - -The `BaseActor` class provides a flexible and powerful foundation for building applications that interact with OpenAI's models. By extending this class and implementing its -abstract methods, developers can create customized actors tailored to their specific needs. - -# CodingActor.kt - -## CodingActor Documentation - -The `CodingActor` class is a versatile component designed to facilitate the interaction between natural language instructions and code execution within a specified programming -language context. It leverages the capabilities of OpenAI's GPT models to translate instructions into executable code, evaluate the code, and provide feedback or corrections as -necessary. This document provides an overview of the `CodingActor` class, its functionalities, and how to utilize it effectively. - -### Overview - -`CodingActor` extends the `BaseActor` class and specializes in processing coding requests (`CodeRequest`) and generating code results (`CodeResult`). It integrates with an -interpreter to execute the generated code and handles the interaction with OpenAI's API for code generation and correction. - -#### Key Features - -- **Code Generation**: Translates natural language instructions into executable code. -- **Code Execution**: Executes the generated code within the context of a specified interpreter. -- **Error Handling and Correction**: Attempts to correct errors in the generated code through iterative feedback loops with the OpenAI API. -- **Customizable Code Formatting**: Supports customizable code formatting instructions. -- **Extensible**: Allows for the specification of additional symbols and runtime context to be used during code generation and execution. - -### Usage - -#### Initialization - -To create an instance of `CodingActor`, you need to provide the following parameters: - -- `interpreterClass`: The class of the interpreter to be used for code execution. -- `symbols`: A map of predefined symbols (variables, functions, etc.) available during code generation. -- `describer`: An instance of `TypeDescriber` used to describe the types of the provided symbols. -- `name`: An optional name for the actor. -- `details`: Optional additional details to be included in the prompt sent to the OpenAI API. -- `model`: The OpenAI model to be used for code generation (default is `OpenAIModels.GPT35Turbo`). -- `fallbackModel`: A fallback OpenAI model to be used in case of failure with the primary model (default is `OpenAIModels.GPT4o`). -- `temperature`: The temperature parameter for the OpenAI API requests (default is `0.1`). -- `runtimeSymbols`: Additional symbols to be added at runtime. - -Example: - -```kotlin -val codingActor = CodingActor( - interpreterClass = MyCustomInterpreter::class, - symbols = mapOf("exampleSymbol" to Any()), - name = "MyCodingActor", - details = "This is a detailed description of what the actor does.", - model = OpenAIModels.GPT35Turbo -) -``` - -#### Processing Requests - -To process a coding request, create an instance of `CodeRequest` with the desired parameters, such as the list of messages (instructions and responses), code prefix, and whether to -auto-evaluate the code. Then, call the `respond` method of your `CodingActor` instance with the request. - -Example: - -```kotlin -val request = CodingActor.CodeRequest( - messages = listOf(Pair("Please write a function to add two numbers.", CodingActor.Role.user)), - autoEvaluate = true -) - -val result = codingActor.respond(request, apiInstance) -``` - -#### Handling Results - -The result of a coding request is an instance of `CodeResult`, which includes the generated code, its execution status, the execution result, and a rendered response. - -Example: - -```kotlin -println("Status: ${result.status}") -println("Generated Code: ${result.code}") -println("Execution Result: ${result.result.resultValue}") -``` - -### Advanced Features - -- **Custom Type Describers**: Implement custom `TypeDescriber` to control how symbols are described in the prompt. -- **Error Correction**: Utilize the error correction mechanism to automatically attempt to fix errors in the generated code. -- **Execution Result**: Access detailed execution results, including output and returned values. - -### Conclusion - -The `CodingActor` class provides a powerful interface for integrating natural language coding instructions with executable code, leveraging the capabilities of OpenAI's models. By -customizing the interpreter, symbols, and other parameters, developers can tailor the coding actor to fit a wide range of coding tasks and workflows. - -# ImageActor.kt - -## ImageActor Documentation - -The `ImageActor` class is a powerful tool designed to transform textual prompts into visually compelling images using OpenAI's image generation models. This class is part of the -`com.simiacryptus.skyenet.core.actors` package and leverages the OpenAI API to fulfill image generation requests based on user input. - -### Features - -- **Customizable Prompts**: Allows setting a default prompt to guide the image generation process. -- **Flexible Image Models**: Supports different OpenAI image models, including DALLยทE 2, for generating images. -- **Adjustable Image Dimensions**: Users can specify the desired width and height for the generated images. -- **Text-to-Image Transformation**: Converts user requests into image generation prompts and subsequently into images. -- **Model Switching**: Provides methods to switch between different chat and image models while retaining other settings. - -### Usage - -#### Initialization - -To create an instance of `ImageActor`, you can use the following constructor: - -```kotlin -val imageActor = ImageActor( - prompt = "Transform the user request into an image generation prompt that the user will like", - name = null, - textModel = OpenAIModels.GPT35Turbo, - imageModel = ImageModels.DallE2, - temperature = 0.3, - width = 1024, - height = 1024 -) -``` - -#### Generating Images - -To generate an image based on a list of textual prompts, use the `respond` method: - -```kotlin -val api: API = // Initialize your OpenAIClient here -val inputPrompts: List = listOf("A futuristic cityscape", "under a purple sky") -val imageResponse: ImageResponse = imageActor.respond(inputPrompts, api) -``` - -The `respond` method processes the input prompts, interacts with the OpenAI API to generate a textual description for the image, and then generates the image based on this -description. - -#### Accessing the Generated Image - -The `ImageResponse` object returned by the `respond` method contains both the textual description used for the image generation and the generated image itself: - -```kotlin -val description: String = imageResponse.text -val image: BufferedImage = imageResponse.image -``` - -#### Customizing the Actor - -You can customize the `ImageActor` by changing its model settings: - -- To change the chat model: - -```kotlin -val newChatModel: ChatModels = OpenAIModels.GPT4 -val updatedActor = imageActor.withModel(newChatModel) -``` - -- To change the image model: - -```kotlin -val newImageModel: ImageModels = ImageModels.Craiyon -val updatedActor = imageActor.withModel(newImageModel) -``` - -### Interfaces and Classes - -- **`ImageActor`**: The main class responsible for transforming text prompts into images. -- **`ImageResponse`**: An interface representing the response from an image generation request, containing both the textual description and the generated image. - -### Conclusion - -The `ImageActor` class offers a convenient and flexible way to integrate OpenAI's image generation capabilities into your applications. By customizing prompts, models, and image -dimensions, you can tailor the image generation process to meet a wide range of creative needs. - -# ActorSystem.kt - -## ActorSystem Documentation - -The `ActorSystem` class is a core component of the framework designed to manage and interact with various types of actors within a session. It provides a structured way to access -and utilize actors, ensuring that the necessary interceptors and wrappers are applied for enhanced functionality and monitoring. This document outlines the key features and usage -of the `ActorSystem` class. - -### Overview - -An `ActorSystem` is initialized with a collection of actors, data storage interface, an optional user, and a session. It facilitates the retrieval and interaction with different -types of actors, applying specific interceptors based on the actor type. This system is designed to work within the context of a session, providing a seamless way to manage actors' -lifecycle and their interactions. - -### Key Components - -- **actors**: A map of actor identifiers to `BaseActor` instances. These are the actors managed by the `ActorSystem`. -- **dataStorage**: An implementation of `StorageInterface` to handle data storage and retrieval. -- **user**: An optional `User` instance representing the user interacting with the actors. -- **session**: A `Session` instance representing the current session. - -### Features - -#### Actor Retrieval - -- **getActor(actor: T)**: Retrieves an instance of `BaseActor` for the specified actor identifier. If the actor has not been initialized yet, it applies the appropriate interceptor - based on the actor's type and caches the result for future retrievals. - -#### Interceptors - -Depending on the actor type, one of the following interceptors is applied to enhance its functionality: - -- `SimpleActorInterceptor` -- `ParsedActorInterceptor` -- `CodingActorInterceptor` -- `ImageActorInterceptor` -- `TextToSpeechActorInterceptor` - -These interceptors wrap the original actor to provide additional features such as logging, monitoring, or custom behavior modifications. - -#### Function Wrapping - -For each actor, a `FunctionWrapper` is associated to enable recording and monitoring of function calls. This is facilitated through a `JsonFunctionRecorder` which stores the -function call data in a structured JSON format for later analysis or debugging. - -### Usage - -To use the `ActorSystem`, you must first initialize it with the required components: - -```kotlin -val actorSystem = ActorSystem( - actors = mapOf(/* Actor identifiers and instances */), - dataStorage = /* An implementation of StorageInterface */, - user = /* An optional User instance */, - session = /* A Session instance */ -) -``` - -Once initialized, you can retrieve and interact with actors through the system: - -```kotlin -val actor = actorSystem.getActor(/* Actor identifier */) -// Use the actor for its intended purpose -``` - -### Conclusion - -The `ActorSystem` class provides a robust and flexible way to manage actors within a session. By abstracting the complexities of actor initialization and management, it allows -developers to focus on the core logic and interactions of their application. - -# opt\Expectation.kt - -## Skyenet Core Expectation Module Documentation - -The Skyenet Core Expectation module is a part of the Skyenet framework designed to facilitate the evaluation of responses from an AI model, specifically using the OpenAI API. This -module provides a flexible way to define expectations for responses and score them based on different criteria. It includes two primary classes: `VectorMatch` and `ContainsMatch`, -each serving a unique purpose in evaluating responses. - -### Overview - -The Expectation module is abstract and contains a companion object for logging purposes. It defines two abstract methods, `matches` and `score`, which must be implemented by -subclasses to determine if a response meets a certain expectation and to assign a numerical score to the response, respectively. - -#### VectorMatch Class - -The `VectorMatch` class is designed to compare the semantic similarity between a given example text and a response text using embeddings. It utilizes the OpenAI API to generate -embeddings for both texts and calculates the distance between them based on a specified metric. - -##### Parameters: - -- `example`: The example text to compare against. -- `metric`: The distance metric to use for comparison. Defaults to `DistanceType.Cosine`. - -##### Methods: - -- `matches(api: OpenAIClient, response: String)`: Always returns `true` as it is not used for matching but for scoring based on distance. -- `score(api: OpenAIClient, response: String)`: Calculates and returns the negative distance between the embeddings of the example text and the response text, effectively scoring - the response based on its semantic similarity to the example. - -#### ContainsMatch Class - -The `ContainsMatch` class checks if a given pattern (regular expression) is contained within the response text. It is useful for scenarios where the presence of specific patterns -or keywords is critical or of interest. - -##### Parameters: - -- `pattern`: The regular expression pattern to search for within the response. -- `critical`: A boolean indicating if the match is critical. If `false`, `matches` always returns `true`. Defaults to `true`. - -##### Methods: - -- `matches(api: OpenAIClient, response: String)`: Returns `true` if the pattern is found in the response and `critical` is `true`, otherwise returns `true` or `false` based on the - `critical` flag. -- `score(api: OpenAIClient, response: String)`: Returns `1.0` if the pattern matches the response, otherwise returns `0.0`. - -### Usage - -To use the Expectation module, instantiate either `VectorMatch` or `ContainsMatch` with the appropriate parameters and call the `matches` and/or `score` methods with an instance of -`OpenAIClient` and the response text to evaluate. - -#### Example - -```kotlin -val apiClient = OpenAIClient(...) -val expectation = VectorMatch("Example text") -val response = "Response text from the AI model" -val score = expectation.score(apiClient, response) -println("Score: $score") -``` - -This module provides a powerful and flexible way to evaluate AI model responses, enabling developers to implement custom logic for scoring and matching responses based on semantic -similarity or the presence of specific patterns. - -# ParsedActor.kt - -## ParsedActor Class Documentation - -The `ParsedActor` class is a specialized actor designed to parse responses from chat models into specific data types. It extends the functionality of a base actor to not only -interact with chat models but also to process the responses into a more structured format. This class is particularly useful when working with chat models to perform specific tasks -that require structured output. - -### Features - -- **Custom Parsing**: Allows for the specification of a custom parser to convert chat model responses into a desired data type. -- **Flexible Chat Model Interaction**: Supports interaction with different chat models, including specifying the model for parsing responses. -- **Error Handling**: Implements retry logic for deserialization to handle potential parsing errors gracefully. -- **Customizable Prompts**: Enables the use of custom prompts for initiating chat model conversations. - -### Constructor Parameters - -- `parserClass`: The class of the parser function used to convert chat model responses into the desired data type. -- `prompt`: The initial prompt to send to the chat model. -- `name`: An optional name for the actor. Defaults to the simple name of the parser class if not provided. -- `model`: The chat model to use for generating responses. Defaults to `OpenAIModels.GPT35Turbo`. -- `temperature`: The temperature setting for the chat model, affecting the randomness of responses. Defaults to `0.3`. -- `parsingModel`: The chat model to use specifically for parsing responses. Defaults to `OpenAIModels.GPT35Turbo`. -- `deserializerRetries`: The number of retries for deserialization in case of parsing errors. Defaults to `2`. - -### Methods - -#### `chatMessages(questions: List)` - -Generates an array of chat messages to be sent to the chat model, including the initial system prompt and user questions. - -#### `getParser(api: API)` - -Creates and returns a parser function based on the specified parser class and chat model settings. - -#### `respond(input: List, api: API, vararg messages: ApiModel.ChatMessage)` - -Processes the given input questions and returns a `ParsedResponse` containing the parsed data type. - -#### `withModel(model: ChatModels)` - -Creates a new instance of `ParsedActor` with the specified chat model, retaining other settings. - -### Inner Classes - -#### `ParsedResponseImpl` - -An implementation of `ParsedResponse` that holds the parsed object and the original text response from the chat model. - -### Usage Example - -```kotlin -// Define a parser function class that converts a String to a custom data type -class MyParser : Function { - override fun apply(t: String): MyDataType { - // Implement parsing logic here - } -} - -// Initialize a ParsedActor with the custom parser and a prompt -val parsedActor = ParsedActor( - parserClass = MyParser::class.java, - prompt = "Please provide information about X." -) - -// Use the actor to send questions and receive parsed responses -val response = parsedActor.respond(listOf("Question about X"), api) -``` - -This documentation provides an overview of the `ParsedActor` class and its capabilities. For more detailed information on specific methods and parameters, refer to the source code -documentation. - -# ParsedResponse.kt - -## User Documentation for ParsedResponse Class - -### Overview - -The `ParsedResponse` class is an abstract class designed to handle and represent responses parsed from various sources in a structured manner. It is part of the -`com.simiacryptus.skyenet.core.actors` package. This class is generic and can be used to represent any type of parsed data by specifying the type parameter `T` during -implementation. The class is designed to encapsulate both the raw text of the response and its parsed object representation. - -### Key Features - -- **Type-Safe Responses**: By specifying the type parameter `T`, users can ensure that the responses handled by instances of `ParsedResponse` are type-safe, making the code more - robust and easier to maintain. -- **Dual Representation**: The class stores both the raw text of the response (`text`) and its object representation (`obj`), allowing users to access the data in the format most - convenient for their needs. -- **Abstract Design**: Being an abstract class, `ParsedResponse` serves as a template for creating more specific response handlers tailored to particular data types or parsing - requirements. - -### Properties - -- `clazz: Class`: The class literal of the type parameter `T`. This property is used to specify the type of the parsed object and is essential for type safety and - reflection-based operations. -- `text: String`: An abstract property that should be overridden to return the raw text of the parsed response. -- `obj: T`: An abstract property that should be overridden to return the object representation of the parsed response. - -### Methods - -- `toString()`: Overrides the `toString` method to return the raw text of the response. This method provides a convenient way to access the raw response as a string. - -### Usage - -To use the `ParsedResponse` class, you need to extend it with a concrete implementation that specifies the type of the parsed object and provides implementations for the abstract -properties `text` and `obj`. - -#### Example - -```kotlin -class JsonResponse(parsedText: String) : ParsedResponse(JSONObject::class.java) { - override val text: String = parsedText - override val obj: JSONObject = JSONObject(parsedText) - - // Additional functionality specific to JSON responses can be added here -} -``` - -In this example, `JsonResponse` is a concrete implementation of `ParsedResponse` for handling JSON responses. It specifies `JSONObject` as the type parameter `T` and provides -implementations for the `text` and `obj` properties. This class can now be used to handle JSON responses in a type-safe and structured manner. - -### Conclusion - -The `ParsedResponse` class provides a flexible and type-safe framework for handling parsed responses. By extending this class, developers can create customized response handlers -that encapsulate both the raw and object representations of responses, tailored to specific data types or parsing requirements. - -# record\CodingActorInterceptor.kt - -## CodingActorInterceptor Documentation - -The `CodingActorInterceptor` class is a specialized implementation of the `CodingActor` interface, designed to intercept and potentially modify the behavior of another`CodingActor` -instance. This class is part of the `com.simiacryptus.skyenet.core.actors.record` package and is intended for use in scenarios where additional processing or logging of the -interactions with a `CodingActor` is required. - -### Features - -- **Function Interception**: Allows for the interception and modification of the responses generated by the wrapped `CodingActor` instance. -- **Compatibility**: Seamlessly integrates with existing `CodingActor` instances without requiring changes to their implementation. -- **Customizable Behavior**: Through the use of a `FunctionWrapper`, users can define custom behavior to be executed before or after the wrapped `CodingActor`'s methods are - invoked. - -### Usage - -#### Initialization - -To use the `CodingActorInterceptor`, you must first instantiate it with a reference to an existing `CodingActor` instance and a `FunctionWrapper` that defines the interception -logic. - -```kotlin -val innerActor: CodingActor = // Initialize your CodingActor -val functionInterceptor: FunctionWrapper = // Define your FunctionWrapper -val interceptor = CodingActorInterceptor(innerActor, functionInterceptor) -``` - -#### Overridden Methods - -The `CodingActorInterceptor` overrides several methods from the `CodingActor` interface, allowing it to intercept calls to these methods. The key methods are: - -- `response(...)`: Intercepts the generation of responses to chat messages. -- `respond(...)`: Intercepts the generation of code responses to code requests. -- `execute(...)`: Intercepts the execution of code. - -#### Example: Interception Logic - -The interception logic is defined within the `FunctionWrapper` provided at initialization. This example demonstrates a simple logging wrapper: - -```kotlin -val functionInterceptor = FunctionWrapper { args, proceed -> - println("Before invocation with args: ${args.joinToString()}") - val result = proceed(*args) - println("After invocation, result: $result") - result -} -``` - -This `FunctionWrapper` logs the arguments before the invocation of the wrapped method and the result after the invocation. - -### Integration - -To integrate the `CodingActorInterceptor` into your system, replace direct references to the original `CodingActor` instance with references to the `CodingActorInterceptor` -instance. This allows all interactions with the `CodingActor` to be intercepted according to the logic defined in your `FunctionWrapper`. - -### Conclusion - -The `CodingActorInterceptor` provides a powerful mechanism for augmenting the behavior of `CodingActor` instances, enabling scenarios such as logging, monitoring, and dynamic -behavior modification. By leveraging the `FunctionWrapper`, developers can easily inject custom logic into the processing pipeline of a `CodingActor`, enhancing its capabilities -and facilitating advanced use cases. - -# opt\ActorOptimization.kt - -## ActorOptimization Class Documentation - -The `ActorOptimization` class is designed to facilitate the optimization of actor responses using genetic algorithms and OpenAI's GPT models. This class allows for the generation, -mutation, and recombination of prompts to produce optimized responses based on defined test cases and expectations. - -### Features - -- **Genetic Algorithm Implementation**: Utilizes genetic algorithms for optimizing prompts to achieve desired responses. -- **Flexible Actor Factory**: Supports any actor implementation that can generate responses based on prompts. -- **Customizable Mutation and Recombination**: Allows for the customization of mutation rates and types, as well as recombination logic. -- **Test Case Evaluation**: Supports the definition of test cases with expectations to guide the optimization process. - -### Key Components - -#### TestCase Class - -Represents a test case with user messages, expectations, and the number of retries. It is used to evaluate the effectiveness of a prompt. - -- `userMessages`: A list of messages simulating a conversation leading up to the response. -- `expectations`: A list of expectations to evaluate the response. -- `retries`: The number of retries allowed for the test case. - -#### GeneticApi Interface - -Defines the operations for mutating and recombining prompts. - -- `mutate`: Mutates a given prompt based on a directive (e.g., Rephrase, Randomize). -- `recombine`: Recombines two prompts to produce a new prompt. - -#### Main Methods - -##### runGeneticGenerations - -Runs multiple generations of genetic optimization to improve prompts based on test cases. - -Parameters: - -- `prompts`: Initial list of prompts to optimize. -- `testCases`: List of `TestCase` instances to evaluate prompts. -- `actorFactory`: Factory function to create actors based on prompts. -- `resultMapper`: Function to map actor responses to strings for evaluation. -- `selectionSize`: Number of top prompts to select for the next generation. -- `populationSize`: Size of the prompt population for each generation. -- `generations`: Number of generations to run. - -Returns: - -- A list of optimized prompts. - -##### regenerate - -Generates a new set of prompts based on progenitors through mutation or recombination. - -Parameters: - -- `progenetors`: List of parent prompts. -- `desiredCount`: Desired number of prompts in the new generation. - -Returns: - -- A list of new prompts. - -### Usage Example - -```kotlin -val apiClient = OpenAIClient("your_api_key") -val actorOptimization = ActorOptimization(api = apiClient) - -val initialPrompts = listOf("How can I help you today?") -val testCases = listOf( - ActorOptimization.TestCase( - userMessages = listOf(ApiModel.ChatMessage(role = ApiModel.Role.user, content = "I need assistance with my order.")), - expectations = listOf(/* Define your expectations here */) - ) -) - -val optimizedPrompts = actorOptimization.runGeneticGenerations( - prompts = initialPrompts, - testCases = testCases, - actorFactory = { prompt -> /* Your actor factory logic here */ }, - resultMapper = { response -> /* Your result mapping logic here */ } -) - -println("Optimized Prompts: $optimizedPrompts") -``` - -### Customization - -You can customize the mutation rate and types by adjusting the `mutationRate` and `mutatonTypes` parameters during the instantiation of the `ActorOptimization` class. - -### Conclusion - -The `ActorOptimization` class provides a powerful tool for optimizing conversational prompts using genetic algorithms and OpenAI's GPT models. By defining test cases and -expectations, users can iteratively improve prompts to achieve desired responses. - -# record\ImageActorInterceptor.kt - -## User Documentation for ImageActorInterceptor - -The `ImageActorInterceptor` class is a specialized component designed to intercept and potentially modify the behavior of an `ImageActor` instance. This class is part of the -`com.simiacryptus.skyenet.core.actors.record` package and is intended for advanced users who need to customize or extend the functionality of image generation or processing within -a system that utilizes the Skyenet framework. - -### Overview - -`ImageActorInterceptor` acts as a wrapper around an existing `ImageActor` instance, allowing users to intercept and modify the inputs and outputs of the `ImageActor`'s methods. -This is particularly useful for debugging, logging, or applying custom transformations to the data processed by the `ImageActor`. - -### Key Features - -- **Function Interception**: Allows interception of the `response` and `render` methods of the `ImageActor`. -- **Custom Processing**: Enables custom processing of inputs and outputs without modifying the original `ImageActor` code. -- **Compatibility**: Seamlessly integrates with existing `ImageActor` instances, preserving their properties and behaviors. - -### Usage - -#### Initialization - -To use the `ImageActorInterceptor`, you must first have an instance of `ImageActor` that you wish to intercept. You then create an instance of `ImageActorInterceptor` by passing -the `ImageActor` instance and a `FunctionWrapper` instance to its constructor. - -```kotlin -val originalImageActor = ImageActor(...) -val functionInterceptor = FunctionWrapper(...) -val imageActorInterceptor = ImageActorInterceptor(originalImageActor, functionInterceptor) -``` - -#### Interception and Custom Processing - -The `FunctionWrapper` provided during initialization is responsible for defining how the interception and potential modification of the `ImageActor`'s behavior occur. You must -implement the `wrap` method of the `FunctionWrapper` to specify the custom processing logic. - -```kotlin -val functionInterceptor = object : FunctionWrapper { - override fun wrap(input: T, function: (T) -> R): R { - // Implement custom processing logic here - return function(input) - } -} -``` - -#### Methods - -##### `response` - -Intercepts the `response` method calls of the `ImageActor`, allowing for custom processing of chat messages and model interactions. - -```kotlin -imageActorInterceptor.response(inputMessages, model, api) -``` - -##### `render` - -Intercepts the `render` method calls of the `ImageActor`, enabling custom processing of text inputs for image rendering. - -```kotlin -val image = imageActorInterceptor.render(text, api) -``` - -### Conclusion - -The `ImageActorInterceptor` provides a powerful mechanism for customizing the behavior of `ImageActor` instances in the Skyenet framework. By leveraging function interception, -developers can implement custom logic for processing inputs and outputs, enhancing the flexibility and capabilities of their image generation and processing systems. - -# record\ParsedActorInterceptor.kt - -## ParsedActorInterceptor Documentation - -The `ParsedActorInterceptor` class is designed to act as a middleware that intercepts and modifies the behavior of a `ParsedActor` in a flexible manner. It allows for the -interception and manipulation of responses generated by the `ParsedActor` through a custom `FunctionWrapper`. This class is part of the`com.simiacryptus.skyenet.core.actors.record` -package. - -### Features - -- **Interception and Modification**: Allows for the interception of the response generation process, enabling the modification or enhancement of responses. -- **Flexible Parsing**: Utilizes a custom parser to process and transform the response text into a desired object format. -- **Lazy Evaluation**: Employs lazy evaluation for response objects to optimize performance and resource utilization. -- **Seamless Integration**: Designed to seamlessly integrate with existing `ParsedActor` instances without requiring significant modifications. - -### Usage - -#### Initialization - -To initialize a `ParsedActorInterceptor`, you need to provide an instance of `ParsedActor` and a `FunctionWrapper`. The `FunctionWrapper` is responsible for defining how the -responses are intercepted and processed. - -```kotlin -val parsedActorInterceptor = ParsedActorInterceptor( - inner = parsedActorInstance, - functionInterceptor = customFunctionWrapper -) -``` - -#### Responding to Inputs - -The `ParsedActorInterceptor` overrides the `respond` method to intercept the response generation process. It utilizes the provided `FunctionWrapper` to modify or enhance the -response based on custom logic. - -```kotlin -val response = parsedActorInterceptor.respond( - input = listOf("Your input here"), - api = apiInstance -) -``` - -#### Custom Response Processing - -The interceptor uses a lazy evaluation strategy for processing response objects. The actual parsing and processing of the response text into the desired object format are deferred -until the `obj` property is accessed. This approach optimizes performance by avoiding unnecessary computations. - -### Key Methods - -- **respond**: Intercepts the response generation process, allowing for custom processing of inputs and modification of the generated response. -- **response**: A method designed to wrap the response generation process, enabling the interception and modification of responses for a given set of input messages and a specified - model. - -### Example - -```kotlin -// Initialize your ParsedActor instance -val parsedActor = ... - -// Define your custom function wrapper for intercepting responses -val functionWrapper = FunctionWrapper { ... } - -// Initialize the ParsedActorInterceptor with the ParsedActor and FunctionWrapper -val interceptor = ParsedActorInterceptor( - inner = parsedActor, - functionInterceptor = functionWrapper -) - -// Use the interceptor to respond to inputs -val customResponse = interceptor.respond( - input = listOf("Hello, world!"), - api = apiInstance -) - -// Access the modified response text and object -println(customResponse.text) -println(customResponse.obj) -``` - -### Conclusion - -The `ParsedActorInterceptor` provides a powerful mechanism for intercepting and modifying the behavior of `ParsedActor` instances. By leveraging custom function wrappers, -developers can implement sophisticated logic to enhance and tailor the responses generated by their actors, enabling more dynamic and context-aware applications. - -# record\TextToSpeechActorInterceptor.kt - -## TextToSpeechActorInterceptor Documentation - -The `TextToSpeechActorInterceptor` class is a part of the `com.simiacryptus.skyenet.core.actors.record` package, designed to intercept and modify the behavior of a -`TextToSpeechActor` instance. This class allows for additional processing or modification of inputs and outputs to the `TextToSpeechActor` through a provided `FunctionWrapper`. It -is particularly useful for debugging, logging, or applying custom transformations to the text-to-speech process. - -### Features - -- **Interception and Modification**: Allows for the interception and potential modification of both the input to and output from the `TextToSpeechActor` methods. -- **Function Wrapping**: Utilizes a `FunctionWrapper` to apply custom logic around the invocation of the `TextToSpeechActor`'s methods. -- **Compatibility**: Seamlessly integrates with existing `TextToSpeechActor` instances without requiring modifications to their implementation. - -### Usage - -#### Initialization - -To use the `TextToSpeechActorInterceptor`, you must first instantiate it with an existing `TextToSpeechActor` and a `FunctionWrapper` that defines the interception logic. - -```kotlin -val originalActor = TextToSpeechActor(name, audioModel, "alloy", 1.0) -val functionInterceptor = FunctionWrapper(/* custom interception logic here */) -val interceptor = TextToSpeechActorInterceptor(originalActor, functionInterceptor) -``` - -#### Interception - -Once the interceptor is initialized, it can be used in place of the original `TextToSpeechActor`. Calls to `response` and `render` will be intercepted according to the logic -defined in the `FunctionWrapper`. - -##### Response Interception - -The `response` method allows for the interception of chat message processing. The custom logic can modify the input messages or the processing behavior before delegating to the -original actor's `response` method. - -```kotlin -interceptor.response(inputMessages, model, api) -``` - -##### Render Interception - -The `render` method enables the interception of text rendering to audio. The interception logic can modify the input text or the rendering behavior before calling the original -actor's `render` method. - -```kotlin -val audioBytes = interceptor.render(text, api) -``` - -### Customization - -The behavior of the `TextToSpeechActorInterceptor` is largely determined by the `FunctionWrapper` provided during initialization. By implementing custom logic within this wrapper, -users can achieve a wide range of effects, from simple logging to complex input/output transformations. - -### Conclusion - -The `TextToSpeechActorInterceptor` provides a powerful and flexible mechanism for augmenting the behavior of `TextToSpeechActor` instances. By leveraging function wrapping, it -offers a seamless way to integrate custom logic into the text-to-speech process, making it an invaluable tool for developers looking to extend or customize the functionality of -their text-to-speech applications. - -# record\SimpleActorInterceptor.kt - -## SimpleActorInterceptor Documentation - -The `SimpleActorInterceptor` class is a part of the `com.simiacryptus.skyenet.core.actors.record` package, designed to act as a wrapper around instances of `SimpleActor`. This -class allows for the interception and potential modification of function calls, specifically the `response` function of the `SimpleActor` class. It is particularly useful for -scenarios where additional processing or logging is required for the inputs and outputs of the `response` function. - -### Features - -- **Function Interception**: Enables the interception of the `response` function calls to `SimpleActor`, allowing for pre-processing or post-processing of inputs and outputs. -- **Custom Processing**: Through the use of a `FunctionWrapper`, custom logic can be applied to the inputs and outputs of the `response` function. -- **Seamless Integration**: Inherits from `SimpleActor`, ensuring compatibility and ease of integration with existing systems that utilize `SimpleActor` instances. - -### Usage - -To use the `SimpleActorInterceptor`, you must first have an instance of `SimpleActor` that you wish to wrap. You will also need to define or have a `FunctionWrapper` that specifies -the custom logic to be applied during the interception. - -#### Example - -Below is a simple example demonstrating how to create and use a `SimpleActorInterceptor`: - -```kotlin -import com.simiacryptus.skyenet.core.actors.SimpleActor -import com.simiacryptus.skyenet.core.actors.record.SimpleActorInterceptor -import com.simiacryptus.skyenet.core.util.FunctionWrapper - -// Create an instance of SimpleActor -val simpleActor = SimpleActor( - prompt = "Your prompt here", - name = "ActorName", - model = yourModelInstance, // Replace with your OpenAIModel instance - temperature = 0.5 -) - -// Define your custom function logic -val functionInterceptor = FunctionWrapper { input, model, function -> - // Custom pre-processing logic here - println("Before calling response: $input") - - // Call the original function - val result = function(input, model) - - // Custom post-processing logic here - println("After calling response") - - // Return the result - result -} - -// Create an instance of SimpleActorInterceptor -val interceptor = SimpleActorInterceptor(simpleActor, functionInterceptor) - -// Now you can use the interceptor as you would use a SimpleActor -// The interceptor will apply your custom logic before and after calling the response function -``` - -### Parameters - -- `inner`: The `SimpleActor` instance that is being wrapped. -- `functionInterceptor`: An instance of `FunctionWrapper` that defines the custom logic to be applied during the interception. - -### Methods - -- `response(vararg input: com.simiacryptus.jopenai.ApiModel.ChatMessage, model: OpenAIModel, api: API)`: Overrides the `response` method of `SimpleActor` to apply the custom logic - defined in `functionInterceptor` before and after the original `response` function is called. - -### Conclusion - -The `SimpleActorInterceptor` class provides a powerful way to add custom processing logic to the inputs and outputs of the `response` function of `SimpleActor` instances. By -leveraging the `FunctionWrapper`, developers can easily implement custom pre-processing and post-processing logic to enhance the functionality of their `SimpleActor` instances. - -# test\ImageActorTestBase.kt - -## User Documentation for ImageActorTestBase - -### Overview - -The `ImageActorTestBase` class is part of the testing framework for the `ImageActor` component within the Skyenet project. This abstract class extends `ActorTestBase`, specifically -tailored for testing actors that process a list of strings as input and produce an `ImageResponse`. It provides a foundational setup for creating and testing instances of -`ImageActor` based on different prompts. - -### Key Features - -- **Abstract Testing Framework**: Designed to facilitate the testing of `ImageActor` instances, ensuring they function as expected when given various prompts. -- **Prompt-Based Actor Factory**: Implements a method to instantiate `ImageActor` objects using specific prompts, allowing for flexible testing scenarios. - -### Getting Started - -To utilize `ImageActorTestBase` in your testing suite, follow these steps: - -1. **Extend `ImageActorTestBase`**: Since `ImageActorTestBase` is an abstract class, you'll need to create a concrete subclass that implements any abstract methods (if any exist - beyond those implemented in `ActorTestBase`). - -2. **Implement Required Methods**: Ensure all abstract methods inherited from `ActorTestBase` are implemented. In the case of `ImageActorTestBase`, the `actorFactory` method is - already implemented, but you may need to provide implementations for other abstract methods depending on your specific testing needs. - -3. **Define Test Cases**: Create test cases within your subclass that utilize the `actorFactory` method to generate `ImageActor` instances with various prompts. These instances can - then be used to verify the actor's response to different inputs. - -4. **Run Tests**: Execute your tests to validate the behavior of `ImageActor` instances. This can help identify any issues or unexpected behavior in the actor's implementation. - -### Example Usage - -Below is a hypothetical example of how one might extend `ImageActorTestBase` to create a specific test suite for `ImageActor`: - -```kotlin -package com.example.tests - -import com.simiacryptus.skyenet.core.actors.test.ImageActorTestBase -import com.simiacryptus.skyenet.core.actors.ImageResponse -import org.junit.Assert -import org.junit.Test - -class MyImageActorTests : ImageActorTestBase() { - - @Test - fun testImageActorWithSamplePrompt() { - val prompt = "A sample prompt" - val imageActor = actorFactory(prompt) - val response = imageActor.process(listOf(prompt)) - - // Perform assertions on the response - Assert.assertNotNull("The response should not be null", response) - Assert.assertTrue("The response should be of type ImageResponse", response is ImageResponse) - // Add more assertions as needed to validate the response - } -} -``` - -In this example, `MyImageActorTests` extends `ImageActorTestBase` and defines a test case that checks whether `ImageActor` produces a non-null response of the correct type when -given a sample prompt. - -### Conclusion - -`ImageActorTestBase` provides a structured approach to testing `ImageActor` instances within the Skyenet project. By extending this class and implementing specific test cases, -developers can ensure their image processing actors behave as expected across a variety of scenarios. - -# test\CodingActorTestBase.kt - -## CodingActorTestBase Documentation - -The `CodingActorTestBase` class is an abstract base class designed for testing coding actors in the Skyenet framework. It extends the functionality of `ActorTestBase` by -specifically catering to actors that generate or interpret code, providing a streamlined way to create, test, and evaluate coding actors. - -### Overview - -Coding actors are specialized actors within the Skyenet framework that deal with generating or interpreting code based on given prompts. The `CodingActorTestBase` class facilitates -the testing of these actors by setting up a common testing infrastructure that includes creating actors, sending prompts, and evaluating the generated code. - -### Key Components - -#### Properties - -- `abstract val interpreterClass: KClass`: This abstract property must be implemented by subclasses to specify the class of the interpreter that the coding actor - should use. The interpreter class must extend the `Interpreter` interface. - -#### Methods - -- `actorFactory(prompt: String): CodingActor`: This method overrides the `actorFactory` method from `ActorTestBase`. It is responsible for creating an instance of `CodingActor`with - the specified interpreter class and details (prompt). This allows for the dynamic creation of coding actors based on the test requirements. - -- `getPrompt(actor: BaseActor): String`: This method overrides the `getPrompt` method from `ActorTestBase`. It retrieves the prompt (details) - from the given coding actor. This prompt is what the actor uses to generate or interpret code. - -- `resultMapper(result: CodeResult): String`: This method overrides the `resultMapper` method from `ActorTestBase`. It maps the `CodeResult` (the output of the coding actor) to a - string representation of the code. This is useful for evaluating the actor's output or for further processing. - -### Usage - -To use the `CodingActorTestBase`, you need to create a subclass that implements the `interpreterClass` property. This subclass will specify the interpreter that the coding actors -should use during testing. Once the subclass is defined, you can create instances of your coding actors by providing prompts, and then use the provided methods to test and evaluate -their code generation capabilities. - -#### Example - -```kotlin -class MyCodingActorTest : CodingActorTestBase() { - override val interpreterClass = MyInterpreter::class // Specify your interpreter class here - - fun testMyCodingActor() { - val prompt = "Generate a greeting message" - val actor = actorFactory(prompt) - val result = actor.generateCode(CodeRequest(prompt)) - println(resultMapper(result)) - } -} -``` - -In this example, `MyCodingActorTest` extends `CodingActorTestBase` and specifies `MyInterpreter` as the interpreter class. It includes a test method that creates a coding actor -with a given prompt, generates code based on that prompt, and then prints the generated code. - -### Conclusion - -The `CodingActorTestBase` class provides a foundational framework for testing coding actors within the Skyenet framework. By abstracting common tasks such as actor creation, prompt -handling, and result mapping, it simplifies the process of developing and testing coding actors. - -# SimpleActor.kt - -## SimpleActor Class Documentation - -The `SimpleActor` class is part of the `com.simiacryptus.skyenet.core.actors` package and extends the functionality of the `BaseActor` class. It is designed to facilitate easy -interaction with OpenAI's GPT models through the `com.simiacryptus.jopenai` API. This class simplifies the process of sending prompts to the model and receiving responses. - -### Constructor Parameters - -- `prompt`: A `String` representing the initial prompt or context to be sent to the model. -- `name`: An optional `String` parameter that specifies the name of the actor. It defaults to `null` if not provided. -- `model`: Specifies the model to be used for generating responses. It defaults to `OpenAIModels.GPT35Turbo`. -- `temperature`: A `Double` value that controls the randomness of the model's responses. Lower values make the model more deterministic. It defaults to `0.3`. - -### Methods - -#### respond - -```kotlin -override fun respond(input: List, api: API, vararg messages: ApiModel.ChatMessage): String -``` - -Generates a response based on the provided input and messages. - -- `input`: A list of `String` representing the user's input or questions. -- `api`: An instance of `API` used to communicate with the OpenAI API. -- `messages`: Vararg parameter of `ApiModel.ChatMessage` representing additional context or messages to be considered by the model. - -**Returns**: A `String` representing the model's response. - -**Throws**: `RuntimeException` if no response is received from the model. - -#### chatMessages - -```kotlin -override fun chatMessages(questions: List): Array -``` - -Converts a list of questions into an array of `ApiModel.ChatMessage`, including the initial prompt as part of the system's role. - -- `questions`: A list of `String` representing the questions or inputs from the user. - -**Returns**: An array of `ApiModel.ChatMessage` ready to be sent to the model. - -#### withModel - -```kotlin -override fun withModel(model: ChatModels): SimpleActor -``` - -Creates a new instance of `SimpleActor` with the specified model while retaining the other properties. - -- `model`: The `ChatModels` instance specifying the new model to be used. - -**Returns**: A new instance of `SimpleActor` configured with the specified model. - -### Usage Example - -```kotlin -val simpleActor = SimpleActor( - prompt = "Hello, how can I assist you today?", - name = "Assistant", - model = OpenAIModels.GPT35Turbo, - temperature = 0.3 -) - -val api = API("your_api_key") // Initialize your API instance with your API key -val questions = listOf("What is the weather like today?", "Can you tell me a joke?") -val response = simpleActor.respond(questions, api) -println(response) -``` - -This example demonstrates how to create an instance of `SimpleActor`, initialize it with a prompt, and use it to generate responses to a list of questions. - -# test\ActorTestBase.kt - -## ActorTestBase Documentation - -The `ActorTestBase` class is an abstract class designed to facilitate the testing of actors within a system. It provides a structured way to optimize and test actors through -predefined test cases and optimization strategies. This class is part of the `com.simiacryptus.skyenet.core.actors.test` package. - -### Overview - -The `ActorTestBase` class serves as a foundation for testing different types of actors. An actor, in this context, refers to a component that takes input, performs some processing, -and produces output. The class is designed to be extended by concrete test classes that specify the behavior of the actors being tested. - -### Key Components - -#### Properties - -- `api`: An instance of `OpenAIClient` used for making API calls. It is initialized with a debug log level. -- `testCases`: An abstract property that should be overridden to provide a list of test cases for optimization. -- `actor`: An abstract property representing the actor being tested. -- `actorFactory`: An abstract function that takes a prompt string and returns a new instance of the actor. - -#### Methods - -##### `opt()` - -This method runs the optimization process for the actor. It uses genetic algorithms to evolve the actor based on the provided test cases. The method parameters allow for -customization of the optimization process, including specifying a different actor or test cases. - -Parameters: - -- `actor`: The actor to optimize. Defaults to the actor defined in the class. -- `testCases`: The test cases to use for optimization. Defaults to the test cases defined in the class. -- `actorFactory`: A factory function to create new actor instances. Defaults to the `actorFactory` method defined in the class. -- `resultMapper`: A function to map the actor's output to a string. Defaults to the `resultMapper` method defined in the class. - -##### `testOptimize()` - -A convenience method that calls the `opt()` method with default parameters. It starts the optimization process for the actor using the predefined test cases and actor factory. - -##### `testRun()` - -This method iterates over the test cases and tests the actor's response to each case. It constructs the input messages for the actor, calls the `answer()` method to get the actor's -response, and logs the result. - -##### `answer()` - -Takes an array of `ApiModel.ChatMessage` objects as input and returns the actor's response. This method is responsible for converting the chat messages into the appropriate input -format for the actor, calling the actor's `respond()` method, and returning the result. - -#### Companion Object - -Contains a logger instance for logging information related to the test execution. - -### Usage - -To use the `ActorTestBase` class, you need to extend it with a concrete class that specifies the actor and test cases. Here's a simplified example: - -```kotlin -class MyActorTest : ActorTestBase() { - override val testCases = listOf(/* Define your test cases here */) - override val actor = MyActor() - - override fun actorFactory(prompt: String): BaseActor { - // Implement the logic to create a new instance of your actor - } - - override fun getPrompt(actor: BaseActor): String { - // Implement the logic to generate a prompt for your actor - } - - override fun resultMapper(result: MyResultType): String { - // Implement the logic to convert your actor's result to a string - } -} -``` - -In this example, you would replace `MyInputType` and `MyResultType` with the actual types used by your actor. You would also implement the abstract methods to provide the necessary -functionality for testing your actor. - -# test\ParsedActorTestBase.kt - -## User Documentation for ParsedActorTestBase - -The `ParsedActorTestBase` class is an abstract class designed to facilitate the testing of actors that parse responses using a specified parser. This class is part of the -`com.simiacryptus.skyenet.core.actors.test` package and extends the functionality provided by `ActorTestBase`. It is tailored for use with actors that take a list of strings as -input and produce a `ParsedResponse` object containing a generic type `T`. - -### Overview - -The `ParsedActorTestBase` class is designed to streamline the process of testing parsed actors by providing a structured framework. It abstracts away common testing -functionalities, allowing developers to focus on the specifics of their parsed actor implementations. The class requires specifying the type of parser to be used for parsing -responses through the `parserClass` parameter. - -### Key Components - -#### Constructor Parameters - -- `parserClass`: This parameter expects a class that implements the `Function` interface. It defines the parser that will be used to parse the responses from the actor. - The parser class should take a string as input and return an instance of type `T`. - -#### Methods - -##### actorFactory - -```kotlin -override fun actorFactory(prompt: String): ParsedActor -``` - -- **Description**: Creates an instance of `ParsedActor` with the specified prompt and parser class. The parsing model is set to `OpenAIModels.GPT35Turbo` by default. -- **Parameters**: - - `prompt`: A string representing the prompt to be used by the actor. -- **Returns**: An instance of `ParsedActor` configured with the provided prompt and parser. - -##### getPrompt - -```kotlin -override fun getPrompt(actor: BaseActor, ParsedResponse>): String -``` - -- **Description**: Retrieves the prompt associated with the given actor. -- **Parameters**: - - `actor`: The actor whose prompt is to be retrieved. -- **Returns**: The prompt string used by the specified actor. - -##### resultMapper - -```kotlin -override fun resultMapper(result: ParsedResponse): String -``` - -- **Description**: Maps the `ParsedResponse` object to a string representation. This method is used to extract the text content from the `ParsedResponse` object. -- **Parameters**: - - `result`: The `ParsedResponse` object containing the parsed result. -- **Returns**: A string representation of the parsed result. - -### Usage - -To use the `ParsedActorTestBase` class, you need to extend it in your test class and specify the type of the parser class as well as the generic type `T` that your parser returns. -You will also need to implement any abstract methods if required. - -#### Example - -```kotlin -class MyActorTest : ParsedActorTestBase(MyResponseParser::class.java) { - // Implement additional test methods or override existing ones if necessary -} -``` - -In this example, `MyResponseType` is the type returned by the parser, and `MyResponseParser` is the class that implements the `Function` interface to parse -the actor's responses. - -### Conclusion - -The `ParsedActorTestBase` class provides a structured and efficient way to test actors that require response parsing. By abstracting common functionalities and providing a clear -framework, it simplifies the testing process and allows developers to focus on the specifics of their actor implementations. - -# TextToSpeechActor.kt - -## TextToSpeechActor Documentation - -The `TextToSpeechActor` class is a part of the `com.simiacryptus.skyenet.core.actors` package, designed to convert text into speech using OpenAI's API. This class extends the -`BaseActor` class, allowing it to process lists of strings and generate speech responses. - -### Features - -- **Customizable Voice and Speed**: Users can specify the voice and speed of the speech. -- **Support for Different Audio Models**: The class supports various audio models provided by OpenAI. -- **Lazy Loading of Speech Data**: The speech data is loaded lazily, meaning it's only processed when needed, optimizing resource usage. - -### Constructor Parameters - -- `name`: Optional. The name of the actor. -- `audioModel`: The audio model to use for text-to-speech conversion. Defaults to `AudioModels.TTS_HD`. -- `voice`: The voice to be used. Defaults to `"alloy"`. -- `speed`: The speed of the speech. Defaults to `1.0`. - -### Methods - -#### `chatMessages(questions: List)` - -Converts a list of questions into an array of `ChatMessage` objects, which are then used by the OpenAI API for processing. - -- **Parameters**: `questions` - A list of strings representing the questions or text to be converted into speech. -- **Returns**: An array of `ChatMessage` objects. - -#### `render(text: String, api: API)` - -Converts the given text into speech data using the specified OpenAI API client. - -- **Parameters**: - - `text`: The text to be converted into speech. - - `api`: The OpenAI API client to use for the conversion. -- **Returns**: A byte array containing the speech data. - -#### `respond(input: List, api: API, vararg messages: ChatMessage)` - -Processes the input text and generates a `SpeechResponse` containing the speech data. - -- **Parameters**: - - `input`: A list of strings to be processed. - - `api`: The OpenAI API client to use for processing. - - `messages`: Additional chat messages to consider in the processing. -- **Returns**: A `SpeechResponseImpl` object containing the speech data. - -#### `withModel(model: AudioModels)` - -Creates a new instance of `TextToSpeechActor` with the specified audio model. - -- **Parameters**: `model` - The audio model to use. -- **Returns**: A new instance of `TextToSpeechActor`. - -### Interfaces - -#### `SpeechResponse` - -An interface representing the response from the text-to-speech conversion. - -- **Properties**: - - `mp3data`: A byte array containing the MP3 data of the converted speech. - -### Usage Example - -```kotlin -val ttsActor = TextToSpeechActor() -val apiClient = OpenAIClient("your_api_key_here") -val questions = listOf("How is the weather today?", "What is the time?") -val speechResponse = ttsActor.respond(questions, apiClient) - -// Access the MP3 data -val mp3Data = speechResponse.mp3data -``` - -This example demonstrates how to create an instance of `TextToSpeechActor`, process a list of questions, and access the generated speech data. Remember to replace -`"your_api_key_here"` with your actual OpenAI API key. - diff --git a/docs/core_utilities.md b/docs/core_utilities.md deleted file mode 100644 index a93ceaf6..00000000 --- a/docs/core_utilities.md +++ /dev/null @@ -1,163 +0,0 @@ -Sure! Here is a detailed description of each utility provided in the code snippets: - -### 1. **StringSplitter.kt** - -The `StringSplitter` object provides a utility to split a string into two parts based on specified separators and their -associated weights. The main functionality is encapsulated in the `split` function. - -- **split(text: String, separators: Map): Pair** - - **Parameters:** - - `text`: The input string to be split. - - `separators`: A map where keys are separator strings and values are their associated weights. - - **Returns:** - - A pair of strings, representing the two parts of the split text. - - **Functionality:** - - The function iterates over the provided separators and calculates potential split points based on the - positions of these separators in the text. - - For each separator, it computes a score using a logarithmic formula that takes into account the position of - the separator and its weight. - - The split point with the highest score is chosen, and the text is split at this point. - - If no suitable split point is found, the text is split in the middle by default. - -- **main(args: Array)** - - A sample main function to demonstrate the usage of the `split` function. - - It prints the result of splitting a sample text using specified separators and their weights. - -### 2. **Selenium.kt** - -The `Selenium` interface defines a contract for saving web content and potentially setting cookies (commented out). - -- **save(url: URL, currentFilename: String?, saveRoot: String)** - - **Parameters:** - - `url`: The URL of the web content to be saved. - - `currentFilename`: The current filename for saving the content. - - `saveRoot`: The root directory where the content should be saved. - - **Functionality:** - - This function is intended to save the web content from the specified URL to a file in the given directory. - -- **AutoCloseable** - - The interface extends `AutoCloseable`, indicating that any implementing class should provide a mechanism to - release resources when no longer needed. - -### 3. **OutputInterceptor.java** - -The `OutputInterceptor` class provides a utility to intercept and capture standard output (`System.out`) and standard -error (`System.err`) streams. - -- **setupInterceptor()** - - Sets up the interceptor by redirecting `System.out` and `System.err` to custom `PrintStream` instances that route - output to both the original streams and internal buffers. - -- **getThreadOutputStream()** - - Retrieves a thread-local output stream for capturing output specific to the current thread. - -- **getThreadOutput()** - - Returns the captured output for the current thread as a string. - -- **clearThreadOutput()** - - Clears the captured output for the current thread. - -- **getGlobalOutput()** - - Returns the globally captured output as a string. - -- **clearGlobalOutput()** - - Clears the globally captured output. - -- **OutputStreamRouter** - - A custom `ByteArrayOutputStream` that routes written data to both the original stream and internal buffers (global - and thread-local). - - Ensures that buffers do not exceed specified maximum sizes, resetting them if necessary. - -### 4. **LoggingInterceptor.kt** - -The `LoggingInterceptor` class provides a utility to intercept and capture logging output. - -- **withIntercept(stringBuffer: StringBuffer, vararg loggerPrefixes: String, fn: () -> T): T** - - A companion object function that temporarily replaces the appenders of specified loggers with a custom appender - that writes to a `StringBuffer`. - - Executes the provided function `fn` and restores the original loggers' state afterward. - -- **append(event: ILoggingEvent)** - - Appends the formatted log message and any associated throwable information to the `StringBuffer`. - -- **getStringBuffer()** - - Returns the `StringBuffer` containing the captured log messages. - -### 5. **FunctionWrapper.kt** - -The `FunctionWrapper` class provides a utility to wrap function calls and intercept their execution. - -- **wrap(fn: () -> T)** - - Wraps a no-argument function and intercepts its execution using the provided `FunctionInterceptor`. - -- **wrap(p: P, fn: (P) -> T)** - - Wraps a single-argument function and intercepts its execution. - -- **wrap(p1: P1, p2: P2, fn: (P1, P2) -> T)** - - Wraps a two-argument function and intercepts its execution. - -- **wrap(p1: P1, p2: P2, p3: P3, fn: (P1, P2, P3) -> T)** - - Wraps a three-argument function and intercepts its execution. - -- **wrap(p1: P1, p2: P2, p3: P3, p4: P4, fn: (P1, P2, P3, P4) -> T)** - - Wraps a four-argument function and intercepts its execution. - -### 6. **FunctionInterceptor Interface** - -Defines methods for intercepting function calls with varying numbers of parameters. - -- **intercept(returnClazz: Class, fn: () -> T)** - - Intercepts a no-argument function. - -- **intercept(params: P, returnClazz: Class, fn: (P) -> T)** - - Intercepts a single-argument function. - -- **intercept(p1: P1, p2: P2, returnClazz: Class, fn: (P1, P2) -> T)** - - Intercepts a two-argument function. - -- **intercept(p1: P1, p2: P2, p3: P3, returnClazz: Class, fn: (P1, P2, P3) -> T)** - - Intercepts a three-argument function. - -- **intercept(p1: P1, p2: P2, p3: P3, p4: P4, returnClazz: Class, fn: (P1, P2, P3, P4) -> T)** - - Intercepts a four-argument function. - -### 7. **NoopFunctionInterceptor** - -A no-operation implementation of `FunctionInterceptor` that simply executes the provided function without any -interception logic. - -### 8. **JsonFunctionRecorder** - -A class that intercepts function calls and records their inputs and outputs in JSON format. - -- **JsonFunctionRecorder(baseDir: File)** - - Initializes the recorder with a base directory for storing JSON files. - -- **intercept(returnClazz: Class, fn: () -> T)** - - Intercepts a no-argument function, records its output (or error) in JSON format, and returns the result. - -- **intercept(params: P, returnClazz: Class, fn: (P) -> T)** - - Intercepts a single-argument function, records its input and output (or error) in JSON format, and returns the - result. - -- **operationDir()** - - Creates a directory for storing JSON files for the current operation, based on a sequence ID and timestamp. - -- **close()** - - Closes the recorder (no resources to close in this implementation). - -### 9. **getModel(modelName: String?): OpenAIModel?** - -A utility function that retrieves an `OpenAIModel` instance based on the provided model name. - -- **Parameters:** - - `modelName`: The name of the model to retrieve. -- **Returns:** - - An `OpenAIModel` instance if a matching model is found, or `null` otherwise. -- **Functionality:** - - Searches through available `ChatModels`, `EmbeddingModels`, and `ImageModels` to find a model with the specified - name. - -These utilities provide a range of functionalities, including string splitting, web content saving, output interception, -logging interception, function call interception, and model retrieval. They are designed to be flexible and reusable -across different parts of an application. \ No newline at end of file diff --git a/docs/interpreter.md b/docs/interpreter.md deleted file mode 100644 index 1d0b5d1c..00000000 --- a/docs/interpreter.md +++ /dev/null @@ -1,216 +0,0 @@ -# Interpreter Subsystem Manual - -## Overview - -The interpreter subsystem provides a framework for executing and validating code snippets in various scripting -languages. It supports Scala, Kotlin, and Groovy, allowing dynamic execution of code with predefined variables and -functions. - -## Components - -### 1. Core Interfaces and Utilities - -#### `Interpreter.kt` - -This is the core interface that defines the contract for any interpreter implementation. It includes methods for running -and validating code, as well as utility methods for wrapping code and execution. - -- **Methods:** - - `getLanguage(): String`: Returns the language of the interpreter. - - `getSymbols(): Map`: Returns the predefined symbols (variables and functions) available in the - interpreter. - - `run(code: String): Any?`: Executes the given code and returns the result. - - `validate(code: String): Throwable?`: Validates the given code and returns any validation errors. - - `wrapCode(code: String): String`: Wraps the code (default implementation returns the code as-is). - - ` wrapExecution(fn: java.util.function.Supplier): T?`: Wraps the execution of a function (default - implementation simply executes the function). - -- **Companion Object:** - - Contains utility methods for testing interpreter implementations. - -#### `InterpreterTestBase.kt` - -This is an abstract base class for unit testing interpreter implementations. It provides a set of standardized tests to -ensure that interpreters conform to expected behavior. - -- **Methods:** - - `test run with valid code` - - `test run with invalid code` - - `test validate with valid code` - - `test validate with invalid code` - - `test run with variables` - - `test validate with variables` - - `test run with tool Any` - - `test validate with tool Any` - - `test run with tool Any and invalid code` - - `test validate with tool Any and invalid code` - - `test validate with undefined variable` - -- **Abstract Method:** - - `newInterpreter(map: Map): Interpreter`: Must be implemented by subclasses to provide an instance of - the interpreter being tested. - -### 2. Scala Interpreter - -#### `ScalaLocalInterpreter.scala` - -This class provides an implementation of the `Interpreter` interface for the Scala language. It uses the Scala REPL ( -Read-Eval-Print Loop) to execute and validate code. - -- **Components:** - - **Object `ScalaLocalInterpreter`:** - - `getTypeTag(value: Any): Type`: Utility method to get the type tag of a value. - - **Class `ScalaLocalInterpreter`:** - - Constructor: Accepts a map of predefined variables and functions. - - `CustomReplReporter`: Custom reporter to handle errors and warnings during code execution. - - `getClasspathFromManifest(jarPath: String): String`: Utility method to get classpath from a JAR manifest. - - `run(code: String): Any`: Executes the given code and returns the result. - - `validate(code: String): Exception`: Validates the given code and returns any validation errors. - - `wrapCode(code: String): String`: Wraps the code (default implementation returns the code as-is). - - `wrapExecution[T](fn: Supplier[T]): T`: Wraps the execution of a function (default implementation simply - executes the function). - - `getSymbols(): util.Map[String, AnyRef]`: Returns the predefined symbols (variables and functions) available - in the interpreter. - -### 3. Kotlin Interpreter - -#### `KotlinInterpreter.kt` - -This class provides an implementation of the `Interpreter` interface for the Kotlin language. It uses the Kotlin JSR223 -scripting engine to execute and validate code. - -- **Components:** - - **Class `KotlinInterpreter`:** - - Constructor: Accepts a map of predefined variables and functions. - - `scriptEngine`: Provides the Kotlin scripting engine. - - `validate(code: String): Throwable?`: Validates the given code and returns any validation errors. - - `run(code: String): Any?`: Executes the given code and returns the result. - - `wrapException(cause: ScriptException, wrappedCode: String, code: String): CodingActor.FailedToImplementException`: - Wraps a script exception with additional information. - - `wrapCode(code: String): String`: Wraps the code (default implementation returns the code as-is). - - **Companion Object:** - - `errorMessage(code: String, line: Int, column: Int, message: String)`: Utility method to format error - messages. - - `classLoader`: Class loader for the Kotlin interpreter. - -### 4. Groovy Interpreter - -#### `GroovyInterpreter.kt` - -This class provides an implementation of the `Interpreter` interface for the Groovy language. It uses the GroovyShell to -execute and validate code. - -- **Components:** - - **Class `GroovyInterpreter`:** - - Constructor: Accepts a map of predefined variables and functions. - - `shell`: Provides the Groovy shell for executing code. - - `run(code: String): Any?`: Executes the given code and returns the result. - - `validate(code: String): Exception?`: Validates the given code and returns any validation errors. - - `wrapCode(code: String): String`: Wraps the code (default implementation returns the code as-is). - -## Usage - -### Creating an Interpreter - -To create an interpreter, you need to instantiate the appropriate class and provide a map of predefined variables and -functions. - -#### Scala Interpreter - -```scala -val scalaInterpreter = new ScalaLocalInterpreter(java.util.Map.of("x", 2.asInstanceOf[AnyRef], "y", 3.asInstanceOf[AnyRef])) -``` - -#### Kotlin Interpreter - -```kotlin -val kotlinInterpreter = KotlinInterpreter(mapOf("x" to 2, "y" to 3)) -``` - -#### Groovy Interpreter - -```kotlin -val groovyInterpreter = GroovyInterpreter(java.util.Map.of("x", 2 as Any, "y", 3 as Any)) -``` - -### Running Code - -To run code using an interpreter, use the `run` method. - -```scala -val result = scalaInterpreter.run("x * y") -println(result) // Output: 6 -``` - -```kotlin -val result = kotlinInterpreter.run("x * y") -println(result) // Output: 6 -``` - -```kotlin -val result = groovyInterpreter.run("x * y") -println(result) // Output: 6 -``` - -### Validating Code - -To validate code using an interpreter, use the `validate` method. - -```scala -val error = scalaInterpreter.validate("x * y") -if (error != null) { - println("Validation failed: " + error.getMessage) -} else { - println("Validation succeeded") -} -``` - -```kotlin -val error = kotlinInterpreter.validate("x * y") -if (error != null) { - println("Validation failed: " + error.message) -} else { - println("Validation succeeded") -} -``` - -```kotlin -val error = groovyInterpreter.validate("x * y") -if (error != null) { - println("Validation failed: " + error.message) -} else { - println("Validation succeeded") -} -``` - -## Testing - -To test an interpreter implementation, extend the `InterpreterTestBase` class and implement the `newInterpreter` method. - -```kotlin -class ScalaInterpreterTest : InterpreterTestBase() { - override fun newInterpreter(map: Map): Interpreter { - return ScalaLocalInterpreter(map) - } -} - -class KotlinInterpreterTest : InterpreterTestBase() { - override fun newInterpreter(map: Map): Interpreter { - return KotlinInterpreter(map) - } -} - -class GroovyInterpreterTest : InterpreterTestBase() { - override fun newInterpreter(map: Map): Interpreter { - return GroovyInterpreter(map) - } -} -``` - -Run the tests to ensure your interpreter implementation conforms to the expected behavior. - -## Conclusion - -The interpreter subsystem provides a flexible and extensible framework for executing and validating code snippets in -various scripting languages. By implementing the `Interpreter` interface, you can create custom interpreters for -additional languages and integrate them seamlessly into the existing framework. \ No newline at end of file diff --git a/docs/patch.md b/docs/patch.md deleted file mode 100644 index 98b5bb61..00000000 --- a/docs/patch.md +++ /dev/null @@ -1,314 +0,0 @@ -Certainly! Here's a detailed documentation of the diff/patch classes, including a comparison of strategies and a -technical summary of each class. - ---- - -## Overview - -This project provides several utilities for computing and applying differences (diffs) and patches between text files. -The main classes involved are: - -1. **DiffMatchPatch**: Implements Google's diff-match-patch algorithm. -2. **IterativePatchUtil**: Applies patches iteratively using line-by-line matching. -3. **ApxPatchUtil**: Applies patches using an approximate matching strategy. -4. **DiffUtil**: Generates and formats diffs between two lists of strings. -5. **PatchUtil**: Alias for `IterativePatchUtil`. - ---- - -## DiffMatchPatch - -### Description - -The `DiffMatchPatch` class provides methods to compute the differences between two texts, create patches from these -differences, and apply patches to texts. It is based on Google's diff-match-patch algorithm. - -### Key Features - -- **Diff Computation**: Computes differences between two texts. -- **Patch Creation**: Creates patches from the computed differences. -- **Patch Application**: Applies patches to texts, allowing for errors. -- **Efficiency and Semantic Cleanup**: Cleans up diffs to reduce the number of edits and eliminate trivial equalities. - -### Technical Summary - -- **diff_main**: Main method to compute differences between two texts. -- **diff_compute**: Core method that handles the actual diff computation. -- **diff_lineMode**: Performs a quick line-level diff for faster results. -- **diff_bisect**: Uses the Myers' O(ND) algorithm to find the 'middle snake' and split the problem. -- **diff_halfMatch**: Checks if the texts share a substring that is at least half the length of the longer text. -- **diff_cleanupSemantic**: Reduces the number of edits by eliminating semantically trivial equalities. -- **diff_cleanupEfficiency**: Reduces the number of edits by eliminating operationally trivial equalities. -- **patch_make**: Creates patches from the differences. -- **patch_apply**: Applies patches to a given text. -- **patch_addContext**: Increases the context of a patch until it is unique. -- **patch_splitMax**: Splits patches that are longer than the maximum limit of the match algorithm. - -### Example Usage - -```kotlin -val dmp = DiffMatchPatch() -val diffs = dmp.diff_main("Hello World", "Goodbye World") -val patches = dmp.patch_make(diffs) -val result = dmp.patch_apply(patches, "Hello World") -``` - ---- - -## IterativePatchUtil - -### Description - -The `IterativePatchUtil` class applies patches to text by iteratively matching lines between the source and patch. It -uses a line-by-line matching strategy. - -### Key Features - -- **Line-by-Line Matching**: Matches lines between the source and patch iteratively. -- **Unique Line Linking**: Links unique lines that match exactly. -- **Adjacent Line Linking**: Links lines that are adjacent to already linked lines. -- **Proximity-Based Matching**: Uses Levenshtein distance and proximity to establish links. - -### Technical Summary - -- **patch**: Main method to apply a patch to the source text. -- **generatePatchedTextUsingLinks**: Generates the final patched text using established links. -- **linkUniqueMatchingLines**: Links lines between the source and patch that are unique and match exactly. -- **linkAdjacentMatchingLines**: Links lines that are adjacent to already linked lines. -- **linkByLevenshteinDistance**: Establishes links based on Levenshtein distance and proximity to established links. -- **calculateProximityDistance**: Calculates the proximity distance between a source line and a patch line. -- **parseLines**: Parses the given text into a list of line records. -- **parsePatchLines**: Parses the patch text into a list of line records, identifying the type of each line (ADD, - DELETE, CONTEXT). - -### Example Usage - -```kotlin -val source = "Hello World" -val patch = """ - @@ -1,1 +1,1 @@ - -Hello - +Goodbye -""".trimIndent() -val result = IterativePatchUtil.patch(source, patch) -``` - ---- - -## ApxPatchUtil - -### Description - -The `ApxPatchUtil` class applies patches using an approximate matching strategy. It uses Levenshtein distance to match -lines approximately. - -### Key Features - -- **Approximate Matching**: Matches lines approximately using Levenshtein distance. -- **Context Line Handling**: Handles context lines and advances the source cursor accordingly. -- **Deletion Handling**: Skips lines marked for deletion in the patch. - -### Technical Summary - -- **patch**: Main method to apply a patch to the source text. -- **onDelete**: Handles deletion lines in the patch. -- **onContextLine**: Handles context lines in the patch. -- **lookAheadFor**: Looks ahead for a matching line in the source text. -- **lineMatches**: Checks if two lines match approximately using Levenshtein distance. - -### Example Usage - -```kotlin -val source = "Hello World" -val patch = """ - @@ -1,1 +1,1 @@ - -Hello - +Goodbye -""".trimIndent() -val result = ApxPatchUtil.patch(source, patch) -``` - ---- - -## DiffUtil - -### Description - -The `DiffUtil` class generates and formats diffs between two lists of strings. It categorizes each line as added, -deleted, or unchanged. - -### Key Features - -- **Diff Generation**: Generates a list of diffs representing the differences between two lists of strings. -- **Diff Formatting**: Formats the list of diffs into a human-readable string representation. - -### Technical Summary - -- **generateDiff**: Generates a list of diffs representing the differences between two lists of strings. -- **formatDiff**: Formats the list of diffs into a human-readable string representation. - -### Example Usage - -```kotlin -val original = listOf("Hello World") -val modified = listOf("Goodbye World") -val diffs = DiffUtil.generateDiff(original, modified) -val formattedDiff = DiffUtil.formatDiff(diffs) -``` - ---- - -## PatchUtil - -### Description - -`PatchUtil` is an alias for `IterativePatchUtil`. It provides the same functionality as `IterativePatchUtil`. - -### Example Usage - -```kotlin -val source = "Hello World" -val patch = """ - @@ -1,1 +1,1 @@ - -Hello - +Goodbye -""".trimIndent() -val result = PatchUtil.patch(source, patch) -``` - ---- - -## Comparison of Strategies - -### DiffMatchPatch - -- **Algorithm**: Google's diff-match-patch algorithm. -- **Strengths**: Highly optimized, supports semantic cleanup, and efficient diff computation. -- **Use Case**: Suitable for applications requiring precise and efficient diff computation and patch application. - -### IterativePatchUtil - -- **Algorithm**: Iterative line-by-line matching. -- **Strengths**: Simple and effective for line-based patches, handles unique and adjacent line matching. -- **Use Case**: Suitable for applications requiring straightforward line-based patch application. - -### ApxPatchUtil - -- **Algorithm**: Approximate matching using Levenshtein distance. -- **Strengths**: Handles approximate matches, useful for patches with minor differences. -- **Use Case**: Suitable for applications requiring approximate matching and handling of minor differences. - -Certainly! Below is a detailed documentation of the UI decorators found in the provided Kotlin code, including an -overview of the patch verification process. - ---- - -## UI Decorators Overview - -### 1. `addApplyDiffLinks` - -This function adds "Apply Diff" links to code blocks in a markdown response. It processes the response to find diff -blocks and appends links that allow users to apply the diffs to the provided code. - -#### Parameters: - -- `code: () -> String`: A lambda function that returns the current code as a string. -- `response: String`: The markdown response containing diff blocks. -- `handle: (String) -> Unit`: A function to handle the new code after applying the diff. -- `task: SessionTask`: The current session task. -- `ui: ApplicationInterface`: The UI interface for creating tasks and rendering markdown. - -#### Process: - -1. **Regex Matching**: Uses a regex pattern to find all diff blocks in the response. -2. **Task Creation**: For each diff block, creates a new task to handle the application of the diff. -3. **Apply Diff**: Adds a link to apply the diff. If the diff is applied successfully, updates the link text to "Diff - Applied". -4. **Reverse Diff**: If the reverse diff is different, adds an additional link to apply the diff in reverse order. -5. **Verification**: Verifies the applied diff by generating and formatting the diff between the original and patched - code. - -#### Patch Verification: - -- **Forward Verification**: Applies the diff and generates a diff between the original and patched code. -- **Reverse Verification**: Applies the diff in reverse order and generates a diff between the original and - reverse-patched code. -- **Display**: Displays the original diff, forward verification diff, and reverse verification diff in tabs. - -### 2. `addApplyFileDiffLinks` - -This function adds "Apply Diff" and "Save File" links to file diffs and code blocks in a markdown response. It processes -the response to find headers, diff blocks, and code blocks, and appends links that allow users to apply the diffs or -save the code to files. - -#### Parameters: - -- `root: Path`: The root path for resolving file paths. -- `code: () -> Map`: A lambda function that returns the current code as a map of file paths to code - strings. -- `response: String`: The markdown response containing headers, diff blocks, and code blocks. -- `handle: (Map) -> Unit`: A function to handle the new code after applying the diff or saving the file. -- `ui: ApplicationInterface`: The UI interface for creating tasks and rendering markdown. - -#### Process: - -1. **Regex Matching**: Uses regex patterns to find headers, diff blocks, and code blocks in the response. -2. **Task Creation**: For each diff block, creates a new task to handle the application of the diff. -3. **Apply Diff**: Adds a link to apply the diff. If the diff is applied successfully, updates the link text to "Diff - Applied". -4. **Save File**: Adds a link to save the code block to a file. If the file is saved successfully, updates the link text - to "Saved ". -5. **Verification**: Verifies the applied diff by generating and formatting the diff between the original and patched - code. - -#### Patch Verification: - -- **Forward Verification**: Applies the diff and generates a diff between the original and patched code. -- **Reverse Verification**: Applies the diff in reverse order and generates a diff between the original and - reverse-patched code. -- **Display**: Displays the original diff, forward verification diff, and reverse verification diff in tabs. - -### 3. `addSaveLinks` - -This function adds "Save File" links to code blocks in a markdown response. It processes the response to find code -blocks and appends links that allow users to save the code to files. - -#### Parameters: - -- `response: String`: The markdown response containing code blocks. -- `task: SessionTask`: The current session task. -- `ui: ApplicationInterface`: The UI interface for creating tasks and rendering markdown. -- `handle: (Path, String) -> Unit`: A function to handle the new code after saving the file. - -#### Process: - -1. **Regex Matching**: Uses regex patterns to find code blocks in the response. -2. **Task Creation**: For each code block, creates a new task to handle the saving of the file. -3. **Save File**: Adds a link to save the code block to a file. If the file is saved successfully, updates the link text - to "Saved ". - -## Patch Verification Process - -The patch verification process ensures that the applied diff is correct and consistent. It involves the following steps: - -1. **Forward Verification**: - -- **Apply Diff**: The diff is applied to the original code. -- **Generate Diff**: A diff is generated between the original code and the patched code. -- **Format Diff**: The generated diff is formatted for display. - -2. **Reverse Verification**: - -- **Apply Reverse Diff**: The diff is applied in reverse order to the reversed original code. -- **Generate Diff**: A diff is generated between the original code and the reverse-patched code. -- **Format Diff**: The generated diff is formatted for display. - -3. **Display**: - -- **Tabs**: The original diff, forward verification diff, and reverse verification diff are displayed in tabs for easy - comparison. - -4. **Update Verification**: - -- **Scheduled Task**: A scheduled task periodically checks if the file content has changed and updates the verification - diffs accordingly. diff --git a/docs/path_structure.md b/docs/path_structure.md deleted file mode 100644 index ae768b9c..00000000 --- a/docs/path_structure.md +++ /dev/null @@ -1,57 +0,0 @@ -Based on the provided code, here's a document detailing the servlet path layout for the application: - -# Servlet Path Layout - -The application uses a hierarchical structure for its servlet paths, with different functionalities mapped to specific URLs. Below is a breakdown of the main servlet paths and -their purposes: - -## Root Level Paths - -1. `/` - Welcome page (WelcomeServlet) -2. `/api` - API endpoint (WelcomeServlet) -3. `/logout` - Logout functionality (LogoutServlet) -4. `/proxy` - Proxy HTTP requests (ProxyHttpServlet) -5. `/userInfo` - User information (UserInfoServlet) -6. `/userSettings` - User settings (UserSettingsServlet) -7. `/usage` - Usage information (UsageServlet) -8. `/apiKeys` - API key management (ApiKeyServlet) -9. `/stats` - Server statistics (StatisticsServlet) - -## Authentication - -* `/oauth2callback` - OAuth2 callback URL for Google authentication - -## Application-Specific Paths - -For each child web application defined in `childWebApps`, a new path is created. These paths are dynamically generated based on the `ApplicationServer` instances. - -## Common Paths for Each Application - -Within each application-specific path, the following common servlets are typically available: - -1. `/` - Default servlet -2. `/ws` - WebSocket endpoint -3. `/newSession` - Create a new session -4. `/appInfo` - Application information -5. `/userInfo` - User information (may overlap with root level) -6. `/usage` - Usage information (may overlap with root level) -7. `/fileIndex/*` - File indexing -8. `/fileZip` - File compression -9. `/sessions` - Session listing -10. `/settings` - Session settings -11. `/threads` - Session threads -12. `/share` - Session sharing -13. `/delete` - Delete session -14. `/cancel` - Cancel threads - -## Notes - -* The exact paths may vary depending on the specific implementation of each `ApplicationServer` instance. -* Some paths (like `/userInfo` and `/usage`) appear both at the root level and within application-specific contexts. The application-specific versions may provide more tailored - information. -* The application uses filters for Cross-Origin Resource Sharing (CORS) and authentication/authorization checks. -* WebSocket support is configured for real-time communication. -* The server is set up to handle both HTTP and HTTPS connections, with the latter being used when running in server mode. - -This layout provides a comprehensive structure for handling various aspects of the application, from user management and authentication to application-specific functionalities and -session handling. diff --git a/docs/planning_agent.md b/docs/planning_agent.md deleted file mode 100644 index f04c5f95..00000000 --- a/docs/planning_agent.md +++ /dev/null @@ -1,763 +0,0 @@ -# **Plan Feature User and Developer Guide** - -Welcome to the comprehensive guide for the **Plan** feature, designed to assist both end-users and developers in -effectively utilizing and extending the functionality provided by the `com.simiacryptus.skyenet.apps.plan` package. This -guide covers everything from high-level overviews to detailed implementation insights. - - - -* [**Plan Feature User and Developer Guide**](#plan-feature-user-and-developer-guide) - * [**Introduction**](#introduction) - * [**Features Overview**](#features-overview) - * [**User Guide**](#user-guide) - * [**Accessing the Plan Feature**](#accessing-the-plan-feature) - * [**Creating and Managing Plans**](#creating-and-managing-plans) - * [**Understanding Task Dependencies**](#understanding-task-dependencies) - * [**Viewing Execution Logs**](#viewing-execution-logs) - * [**Developer Guide**](#developer-guide) - * [**Architecture Overview**](#architecture-overview) - * [**Task Types**](#task-types) - * [**1. CommandAutoFixTask**](#1-commandautofixtask) - * [**2. InquiryTask**](#2-inquirytask) - * [**3. FileModificationTask**](#3-filemodificationtask) - * [**4. RunShellCommandTask**](#4-runshellcommandtask) - * [**5. RefactorTask**](#5-refactortask) - * [**6. SecurityAuditTask**](#6-securityaudittask) - * [**7. CodeOptimizationTask**](#7-codeoptimizationtask) - * [**8. CodeReviewTask**](#8-codereviewtask) - * [**9. DocumentationTask**](#9-documentationtask) - * [**10. TestGenerationTask**](#10-testgenerationtask) - * [**Adding New Task Types**](#adding-new-task-types) - * [**Configuring Plan Settings**](#configuring-plan-settings) - * [**Extending PlanCoordinator**](#extending-plancoordinator) - * [**Utilizing PlanUtil**](#utilizing-planutil) - * [**Handling Asynchronous Task Execution**](#handling-asynchronous-task-execution) - * [**Customizing Task Execution**](#customizing-task-execution) - * [**Best Practices**](#best-practices) - * [**Troubleshooting**](#troubleshooting) - * [**Frequently Asked Questions (FAQs)**](#frequently-asked-questions-faqs) - - - -## **Introduction** - -The **Plan** feature is a sophisticated system designed to break down high-level user requests into manageable, -actionable tasks. It leverages OpenAI's capabilities to automate task planning, execution, and management, ensuring -efficient workflow and seamless integration with existing systems. - ---- - -## **Features Overview** - -* **Dynamic Task Breakdown:** Automatically decompose user requests into smaller tasks with defined dependencies. -* **Task Execution:** Execute tasks such as code modifications, running shell commands, conducting code reviews, and - more. -* **Dependency Management:** Visualize and manage task dependencies using Mermaid diagrams. -* **Asynchronous Processing:** Execute tasks concurrently where possible, respecting dependencies to optimize - performance. -* **Customizable Settings:** Tailor the planning and execution process through configurable settings. -* **Extensible Architecture:** Easily add new task types and extend existing functionalities to fit unique requirements. - ---- - -## **User Guide** - -### **Accessing the Plan Feature** - -To utilize the Plan feature, navigate to the application interface where task planning and management functionalities -are exposed. This could be within a web UI or another integrated platform provided by your organization. - -### **Creating and Managing Plans** - -1. **Initiate a New Plan:** - -* Enter your high-level request or goal into the provided input field. -* For example: "Set up a new React project with TypeScript and implement authentication." - -2. **Task Breakdown:** - -* The system will analyze your request and automatically generate a list of actionable tasks. -* Each task will include details such as task type, description, dependencies, input files, and output files. - -3. **Review Generated Tasks:** - -* Examine the generated tasks to ensure they align with your expectations. -* Modify or adjust tasks if necessary through the interface (if supported). - -4. **Execute the Plan:** - -* Start the execution process, which will carry out each task based on the defined dependencies. -* Monitor the progress through real-time updates and logs. - -### **Understanding Task Dependencies** - -* **Visual Representation:** - * The Plan feature utilizes Mermaid to render task dependency graphs. - * Access the "Task Dependency Graph" section to visualize how tasks interrelate. - -* **Interpreting the Graph:** - * Nodes represent individual tasks. - * Arrows indicate dependencies; a task will only execute after its dependent tasks are completed. - * Styles (e.g., color-coding) indicate the state of each task (Pending, In Progress, Completed). - -### **Viewing Execution Logs** - -* **Accessing Logs:** - * Execution logs are automatically generated and can be accessed through the UI. - * Logs provide detailed information about each task's execution, including API interactions and any errors - encountered. - -* **Log Files:** - * Logs are stored in the `.logs` directory within your working directory. - * Each task execution generates a unique log file for traceability. - ---- - -## **Developer Guide** - -This section provides in-depth information for developers looking to understand, maintain, or extend the Plan feature. - -### **Architecture Overview** - -The Plan feature is built around several core components: - -1. **TaskType:** Defines various types of tasks that can be executed. -2. **PlanSettings:** Configures settings related to task execution, such as models, command environments, and - task-specific configurations. -3. **PlanCoordinator:** Manages the execution of tasks, handling dependencies, threading, and interaction with external - APIs. -4. **PlanUtil:** Contains utility functions for rendering, filtering, and managing task plans. -5. **PlanningTask:** Represents individual planning tasks and manages their execution logic. -6. **AbstractTask:** A base class for all tasks, providing common functionality and state management. - -### **Task Types** - -`TaskType.kt` defines various tasks like `FileModification`, `RunShellCommand`, `CodeReview`, etc. Each task type is -associated with a specific constructor that defines how the task is instantiated and executed. - -##### **1. CommandAutoFixTask** - -* **Description:** - - The `CommandAutoFixTask` is designed to execute specified shell commands and automatically address any issues that - arise during their execution. This task type streamlines the process of running commands by integrating automated - troubleshooting and fixes based on predefined configurations. - -* **Functionality:** - - * **Command Execution:** Runs a user-specified command with provided arguments. - * **Automatic Fixing:** Analyzes the command's output and applies fixes if issues are detected. - * **Configurability:** Allows customization of available commands and auto-fix behavior through `PlanSettings`. - * **Logging and Notifications:** Logs the execution results and notifies users of successes or failures. - -* **Key Components:** - - * **`planSettings.commandAutoFixCommands`:** A list of available commands that can be auto-fixed. - * **`CmdPatchApp`:** Utilized to execute the command and handle patching based on the output. - * **Semaphore Mechanism:** Ensures that the task waits for the command execution and auto-fixing process to complete - before proceeding. - -* **Configuration:** - - * **Available Commands:** Defined in `PlanSettings.kt` under `commandAutoFixCommands`. Developers can add or remove - commands as needed. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic fixing. - -**Implementation Details:** - -The `run` method orchestrates the execution by: - -1. **Identifying the Command:** Matches the alias provided in the task with available commands. -2. **Setting the Working Directory:** Resolves the specified working directory or defaults to the root. -3. **Executing the Command:** Utilizes `CmdPatchApp` to run the command with the specified arguments. -4. **Handling Results:** Based on the exit code, it either applies auto-fixes or notifies the user of failures. - -* **Extending the Task:** - - * **Adding New Commands:** Update `commandAutoFixCommands` in `PlanSettings.kt` with the new command paths. - * **Custom Fix Logic:** Modify the behavior within the `run` method or extend `CmdPatchApp` for specialized fixing - mechanisms. - ---- - -##### **2. InquiryTask** - -* **Description:** - - The `InquiryTask` facilitates answering user-defined questions by reading relevant files and providing comprehensive - summaries. This task type is essential for gathering information, generating insights, and ensuring that responses are - well-informed and contextually accurate. - -* **Functionality:** - - * **Question Handling:** Processes user questions and objectives to generate meaningful answers. - * **Contextual Analysis:** Reads and analyzes input files to provide informed responses. - * **Markdown Rendering:** Formats the output in markdown for easy readability and discussion. - * **Blocking and Asynchronous Execution:** Supports both blocking and non-blocking modes based on configuration. - -* **Key Components:** - - * **`SimpleActor`:** Utilizes a prompt tailored to generate accurate and relevant information based on user inquiries. - * **`Discussable`:** Handles interactive discussions with the user, allowing for response revisions and approvals. - * **Semaphore Mechanism:** Ensures proper synchronization during the inquiry process. - -* **Configuration:** - - * **Enabled Task Types:** Filters which task types are supported during the inquiry to align responses with system - capabilities. - * **Model Selection:** Determines which OpenAI model to use for generating responses, configurable via `PlanSettings`. - -* **Implementation Details:** - - The `run` method executes by: - - 1. **Preparing Input:** Combines user messages, plan details, prior code, and input files into a structured input. - 2. **Generating Response:** Leveraging `inquiryActor` to produce a detailed and formatted answer. - 3. **Handling Blocking Mode:** If enabled, facilitates an interactive discussion allowing user revisions; otherwise, - provides a direct response. - 4. **Updating Task State:** Stores the inquiry result in the processing state for record-keeping and further actions. - -* **Extending the Task:** - - * **Custom Prompts:** Modify the prompt within `SimpleActor` to tailor the inquiry process. - * **Response Handling:** Adjust how responses are processed and displayed by altering the `outputFn` or integrating - additional formatting. - ---- - -##### **3. FileModificationTask** - -* **Description:** - - The `FileModificationTask` automates the process of modifying existing files or creating new ones based on specified - requirements and contexts. It ensures that code changes are efficient, maintainable, and adhere to best practices by - generating precise patches or new file structures. - -* **Functionality:** - - * **Patch Generation:** Creates patches for existing files, highlighting additions and deletions. - * **New File Creation:** Generates new files with appropriate structuring and syntax highlighting. - * **Summary Reporting:** Provides a concise summary of all modifications made. - * **Auto-Application of Changes:** Depending on configuration, can automatically apply the generated patches or prompt - for user approval. - -* **Key Components:** - - * **`SimpleActor`:** Uses a detailed prompt to generate accurate file modifications or new file contents. - * **`CmdPatchApp`:** Manages the application of patches to existing files. - * **`addApplyFileDiffLinks`:** Integrates with the UI to allow users to apply or review changes interactively. - -* **Configuration:** - - * **Input and Output Files:** Defines which files are to be examined or created. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to determine whether changes should be applied - automatically. - * **Language Specifications:** For new files, specifies the programming language to ensure correct syntax - highlighting. - -* **Implementation Details:** - - The `run` method operates by: - - 1. **Validating Input Files:** Ensures that there are files specified for modification or creation. - 2. **Generating Code Changes:** Uses `fileModificationActor` to produce the necessary patches or new file contents. - 3. **Applying Changes:** Depending on `autoFix`, either applies changes automatically or provides links for user - approval. - 4. **Logging and Tracking:** Updates the task state with the results of the modifications for future reference. - -* **Extending the Task:** - - * **Custom Modification Logic:** Enhance the prompt within `SimpleActor` to define specific modification behaviors. - * **Integration with Additional Services:** Extend the task to interact with other systems or services as needed. - * **Advanced Patch Handling:** Modify how patches are applied or reviewed by customizing `addApplyFileDiffLinks` and - related methods. - ---- - -##### **4. RunShellCommandTask** - -* **Description:** - The `RunShellCommandTask` is designed to execute specified shell commands within a designated working directory. - It facilitates automation of system commands, ensuring outputs are captured and handled appropriately. -* **Functionality:** - * **Command Execution:** Executes user-specified shell commands with provided arguments in the configured working directory. - * **Output Handling:** Captures and processes the standard output and error streams from command execution. - * **Error Management:** Handles any errors or exceptions that occur during command execution gracefully. - * **Configurability:** Allows customization of environment variables and command behavior through `PlanSettings`. - * **Logging and Notifications:** Logs command outputs and notifies users of command execution statuses. -* **Key Components:** - * **`planSettings.command`:** Defines the shell commands that can be executed. Developers can add custom commands as needed. - * **`ProcessInterpreter`:** Utilized to execute shell commands and manage their execution lifecycle. - * **Semaphore Mechanism:** Ensures synchronization during the execution of shell commands and processing of their outputs. -* **Configuration:** - * **Command Definitions:** Defined in `PlanSettings.kt` under `command`. Developers can specify which commands are available. - * **Working Directory:** Configured via `planTask.execution_task?.workingDir` or defaults to the root directory. - * **Environment Variables:** Set through `planSettings.env` to customize the execution environment. -* **Implementation Details:** - The `run` method orchestrates the execution by: - 1. **Preparing the Environment:** Sets up the working directory and environment variables as specified. - 2. **Executing the Command:** Uses `ProcessInterpreter` to run the specified shell command with provided arguments. - 3. **Handling Outputs:** Captures the output and error streams, processing them for logging and user notifications. - 4. **Managing Execution State:** Updates the task state based on the success or failure of the command execution. -* **Extending the Task:** - * **Adding New Commands:** Update the `command` list in `PlanSettings.kt` with new shell commands. - * **Custom Output Processing:** Modify the `displayFeedback` method to handle outputs differently or integrate with other systems. - * **Advanced Error Handling:** Enhance the `run` method to manage complex error scenarios or integrate with monitoring tools. - -##### **5. RefactorTask** - -* **Description:** - - The `RefactorTask` is designed to analyze existing code and suggest refactoring improvements to enhance code structure, readability, and maintainability. - -* **Functionality:** - - * **Code Analysis:** Examines the provided code files to identify areas for improvement. - * **Refactoring Suggestions:** Provides detailed recommendations for refactoring, including code reorganization, reducing duplication, and applying design patterns where - appropriate. - * **Change Application:** Utilizes diff patches to implement the suggested refactoring changes. - * **Logging and Reporting:** Logs the refactoring process and results, providing traceability and auditability. - -* **Key Components:** - - * **`actorName` and `actorPrompt`:** Define the behavior and instructions for the refactoring analysis. - * **`AbstractAnalysisTask`:** Provides the foundational structure for the task execution logic. - * **`CommandPatchApp`:** Handles the application of generated refactoring patches. - * **Semaphore Mechanism:** Ensures synchronization and proper task execution flow. - -* **Configuration:** - - * **Refactoring Focus Areas:** Specify particular aspects to focus on during refactoring, such as modularity or naming conventions. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic application of refactoring suggestions. - -* **Implementation Details:** - - The `run` method orchestrates the refactoring process by: - - 1. **Analyzing Code:** Uses `analysisActor` to generate refactoring suggestions based on the current codebase. - 2. **Generating Patches:** Creates diff patches representing the suggested changes. - 3. **Applying Changes:** Applies patches automatically if `autoFix` is enabled or prompts the user for approval. - 4. **Logging Results:** Updates logs and task states based on the outcome of the refactoring process. - -* **Extending the Task:** - - * **Custom Refactoring Logic:** Modify the prompt to target specific refactoring goals or integrate additional analysis tools. - * **Integration with Development Tools:** Enhance `CommandPatchApp` to interact with other systems or IDEs. - -##### **6. SecurityAuditTask** - -* **Description:** - - The `SecurityAuditTask` performs comprehensive security audits on provided code files, identifying potential vulnerabilities and ensuring compliance with security best practices. - -* **Functionality:** - - * **Vulnerability Detection:** Scans code for potential security issues such as injection attacks, insecure data handling, and authentication flaws. - * **Compliance Checking:** Ensures code adheres to established security standards and guidelines. - * **Recommendations:** Provides specific recommendations to address identified security vulnerabilities. - * **Reporting:** Generates detailed audit reports with actionable insights. - -* **Key Components:** - - * **`actorName` and `actorPrompt`:** Tailor the analysis behavior for security auditing. - * **`AbstractAnalysisTask`:** Provides the foundational structure for the task execution logic. - * **`CommandPatchApp`:** Facilitates the application of security fixes. - * **Semaphore Mechanism:** Ensures synchronization during the audit process. - -* **Configuration:** - - * **Audit Focus Areas:** Specify particular security aspects to prioritize during audits. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic application of security fixes. - -* **Implementation Details:** - - The `run` method manages the security auditing by: - - 1. **Analyzing Code:** Uses `analysisActor` to identify security vulnerabilities in the codebase. - 2. **Generating Fixes:** Creates diff patches to address the identified security issues. - 3. **Applying Changes:** Applies patches automatically if `autoFix` is enabled or prompts the user for approval otherwise. - 4. **Logging Results:** Updates logs and task states based on the outcome of the audit. - -* **Extending the Task:** - - * **Custom Security Rules:** Modify or extend the audit criteria to include additional security checks. - * **Integration with Security Tools:** Enhance `CommandPatchApp` to work with external security scanning or fixing tools. - -##### **7. CodeOptimizationTask** - -* **Description:** - - The `CodeOptimizationTask` analyzes existing code to suggest optimizations that enhance performance, reduce resource consumption, and improve overall code quality. - -* **Functionality:** - - * **Performance Analysis:** Identifies performance bottlenecks and inefficient code patterns. - * **Optimization Suggestions:** Recommends specific code changes to improve efficiency and performance. - * **Code Refactoring:** Applies optimized changes using diff patches. - * **Impact Assessment:** Evaluates the potential benefits and trade-offs of suggested optimizations. - -* **Key Components:** - - * **`actorName` and `actorPrompt`:** Define the behavior and instructions for code optimization analysis. - * **`AbstractAnalysisTask`:** Provides the foundational structure for the task execution logic. - * **`CommandPatchApp`:** Facilitates the application of optimization changes. - * **Semaphore Mechanism:** Manages synchronization during the optimization process. - -* **Configuration:** - - * **Optimization Targets:** Define areas of focus, such as memory usage, execution speed, or algorithm efficiency. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic application of optimizations. - -* **Implementation Details:** - - The `run` method orchestrates the optimization process by: - - 1. **Analyzing Code:** Utilizes `analysisActor` to identify optimization opportunities within the codebase. - 2. **Generating Patches:** Creates diff patches representing the suggested optimizations. - 3. **Applying Changes:** Applies patches automatically if `autoFix` is enabled or prompts the user for approval. - 4. **Logging Results:** Updates logs and task states based on the outcome of the optimization process. - -* **Extending the Task:** - - * **Custom Optimization Criteria:** Modify the prompt to target specific optimization goals or integrate performance profiling tools. - * **Advanced Patch Handling:** Enhance how patches are applied to support more complex optimization scenarios. - -##### **8. CodeReviewTask** - -* **Description:** - - The `CodeReviewTask` conducts thorough code reviews on provided code files, assessing code quality, adherence to standards, and identifying potential issues. - -* **Functionality:** - - * **Quality Assessment:** Evaluates code for readability, maintainability, and adherence to coding standards. - * **Issue Identification:** Detects potential bugs, logical errors, and performance issues within the codebase. - * **Best Practices Enforcement:** Ensures that code follows industry best practices and organizational guidelines. - * **Recommendations:** Provides actionable suggestions for improving code quality and resolving identified issues. - -* **Key Components:** - - * **`actorName` and `actorPrompt`:** Tailor the analysis behavior for code reviews. - * **`AbstractAnalysisTask`:** Provides the foundational structure for the task execution logic. - * **`CommandPatchApp`:** Facilitates the application of code review suggestions. - * **Semaphore Mechanism:** Ensures synchronization during the review process. - -* **Configuration:** - - * **Review Scope:** Define which aspects of the code (e.g., security, performance) to prioritize during the review. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic application of review suggestions. - -* **Implementation Details:** - - The `run` method manages the code review by: - - 1. **Analyzing Code:** Uses `analysisActor` to assess code quality and identify issues. - 2. **Generating Suggestions:** Creates diff patches with recommended code changes to improve quality. - 3. **Applying Changes:** Applies patches automatically if `autoFix` is enabled or prompts the user for approval. - 4. **Logging Results:** Updates logs and task states based on the outcome of the code review. - -* **Extending the Task:** - - * **Custom Review Criteria:** Modify the prompt to focus on specific review aspects or integrate additional analysis tools. - * **Integration with CI/CD Pipelines:** Enhance `CommandPatchApp` to work seamlessly with continuous integration systems. - -##### **9. DocumentationTask** - -* **Description:** - - The `DocumentationTask` generates comprehensive documentation for provided code files, ensuring clarity, completeness, and ease of understanding for future developers. - -* **Functionality:** - - * **Automatic Documentation Generation:** Creates detailed documentation covering code purpose, functionality, inputs, outputs, and usage examples. - * **Structured Formatting:** Ensures that documentation is organized in a consistent and readable format. - * **Code Examples:** Includes code snippets to illustrate usage and functionality. - * **Update Tracking:** Monitors changes in code to automatically update relevant documentation sections. - -* **Key Components:** - - * **`documentationGeneratorActor`:** Generates the documentation content based on provided code files. - * **`MarkdownUtil`:** Formats the generated documentation into markdown for consistency and readability. - * **Semaphore Mechanism:** Manages synchronization during the documentation generation process. - -* **Configuration:** - - * **Documentation Scope:** Define which aspects of the code to document, such as functions, classes, and modules. - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic acceptance of generated documentation. - -* **Implementation Details:** - - The `run` method orchestrates the documentation generation by: - - 1. **Generating Documentation:** Utilizes `documentationGeneratorActor` to create detailed documentation based on the codebase. - 2. **Formatting Output:** Formats the documentation using `MarkdownUtil` for integration into the UI. - 3. **Applying or Approving Documentation:** Automatically accepts the documentation if `autoFix` is enabled, or provides an interface for user approval. - 4. **Logging Results:** Updates task states and logs based on the outcome of the documentation process. - -* **Extending the Task:** - - * **Custom Documentation Templates:** Modify the prompt to generate documentation in specific formats or styles. - * **Integration with Documentation Systems:** Enhance the task to integrate with existing documentation platforms or tools. - -##### **10. TestGenerationTask** - -* **Description:** - - The `TestGenerationTask` generates comprehensive unit tests for provided code files, ensuring functionality correctness and high code coverage. - -* **Functionality:** - - * **Unit Test Creation:** Produces unit tests covering all public methods and functions, including positive, negative, and edge cases. - * **Framework Integration:** Utilizes appropriate testing frameworks and assertion libraries based on the target language. - * **Setup and Teardown:** Includes necessary setup and teardown methods to prepare the testing environment. - * **Commenting:** Adds comments explaining the purpose and functionality of each test case. - -* **Key Components:** - - * **`actorName` and `actorPrompt`:** Define the behavior and instructions for test generation analysis. - * **`AbstractAnalysisTask`:** Provides the foundational structure for the task execution logic. - * **`CommandPatchApp`:** Facilitates the creation and application of test files. - * **Semaphore Mechanism:** Manages synchronization during the test generation process. - -* **Configuration:** - - * **Test Coverage Targets:** Define the desired code coverage levels and test case types (e.g., boundary cases). - * **Auto-Fix Toggle:** Controlled via `planSettings.autoFix` to enable or disable automatic creation of test files. - -* **Implementation Details:** - - The `run` method manages the test generation process by: - - 1. **Generating Tests:** Utilizes `analysisActor` to generate unit tests based on the provided code files. - 2. **Creating Test Files:** Formats and structures generated tests into runnable code files using appropriate testing frameworks. - 3. **Applying Tests:** Automatically creates test files if `autoFix` is enabled or prompts the user for approval. - 4. **Logging Results:** Updates logs and task states based on the outcome of the test generation process. - -* **Extending the Task:** - - * **Custom Test Case Logic:** Modify the prompt to generate specific types of tests or integrate with additional testing tools. - * **Integration with CI/CD Pipelines:** Enhance `CommandPatchApp` to work with continuous integration systems and testing frameworks. - -#### **Adding New Task Types** - -To introduce a new task type: - -1. **Define the Task Type:** - ```kotlin - val NewTaskType = TaskType("NewTaskType") - ``` - -2. **Create the Task Class:** - -* Extend `AbstractTask` and implement necessary methods. - - ```kotlin - class NewTaskTypeTask(settings: PlanSettings, task: PlanningTask.PlanTask) : AbstractTask(settings, task) { - override fun promptSegment(): String { - return "Description for NewTaskType" - } - - override fun run( - agent: PlanCoordinator, - taskId: String, - userMessage: String, - plan: TaskBreakdownInterface, - planProcessingState: PlanProcessingState, - task: SessionTask, - api: API - ) { - // Implementation for task execution - } - } - ``` - -3. **Register the Task Type:** - -* In `TaskType.kt`, within the `init` block, register the constructor. - - ```kotlin - registerConstructor(NewTaskType) { settings, task -> NewTaskTypeTask(settings, task) } - ``` - -4. **Update PlanSettings:** - -* Ensure that the new task type is configurable within `PlanSettings` if necessary. - -### **Configuring Plan Settings** - -`PlanSettings.kt` allows for extensive configuration of how tasks are planned and executed. - -* **Default and Parsing Models:** - * Define which OpenAI models to use for default operations and parsing tasks. - -* **Command Environment:** - * Configure the command-line environment, such as using `bash` or `powershell` based on the operating system. - -* **Temperature and Budget:** - * Adjust the creativity (`temperature`) and resource allocation (`budget`) for task planning. - -* **Task-Specific Settings:** - * Enable or disable specific task types and assign models to them. - * Example: - ```kotlin - val taskSettings: MutableMap = TaskType.values().associateWith { taskType -> - TaskSettings( - when (taskType) { - TaskType.FileModification, TaskType.Inquiry -> true - else -> false - } - ) - }.toMutableMap() - ``` - -* **Auto-Fix and Blocking Behavior:** - * Configure whether the system should attempt to auto-fix issues and whether to allow blocking operations. - -### **Extending PlanCoordinator** - -`PlanCoordinator.kt` is pivotal in managing the lifecycle of plans and their tasks. - -* **Initialization:** - * Handles the initial breakdown of tasks based on user input. - * Utilizes `PlanUtil` for filtering and organizing tasks. - -* **Task Execution:** - * Manages threading and asynchronous task execution using `ThreadPoolExecutor`. - * Handles dependencies to ensure tasks execute in the correct order. - -* **Logging and Error Handling:** - * Integrates logging for monitoring task executions. - * Captures and logs errors to assist in debugging. - -**Extending Functionality:** - -* **Custom Execution Strategies:** - * Modify how tasks are queued and executed by overriding methods in `PlanCoordinator`. - -* **Enhancing Logging:** - * Implement additional logging mechanisms or integrate with external monitoring systems. - -### **Utilizing PlanUtil** - -`PlanUtil.kt` provides utility functions essential for task management and visualization. - -* **Diagram Generation:** - * Uses Mermaid to create visual representations of task dependencies. - -* **Rendering Plans:** - * Provides functions to render plans in various formats (Text, JSON, Diagram). - -* **Plan Filtering:** - * Filters out invalid or circular dependencies in task plans. - -**Custom Utilities:** - -* **Extend `PlanUtil` with Additional Helpers:** - * Implement new utility functions as needed to support extended features. - -### **Handling Asynchronous Task Execution** - -The system leverages a `ThreadPoolExecutor` to manage task execution asynchronously. - -* **Concurrency Management:** - * Configure the thread pool size and behavior based on system capabilities and workload. - -* **Task Dependencies:** - * Ensure tasks wait for their dependencies to complete before execution. - -* **Error Propagation:** - * Implement mechanisms to handle exceptions in asynchronous tasks gracefully. - -### **Customizing Task Execution** - -Developers can customize how individual tasks execute by: - -1. **Overriding the `run` Method:** - -* Implement specific logic for new or existing tasks by overriding the `run` method in task classes. - -2. **Integrating with External Systems:** - -* Extend task execution to interact with databases, APIs, or other services as required. - -3. **Enhancing the Execution Flow:** - -* Modify the execution flow in `PlanCoordinator` to support complex scenarios or additional dependencies. - ---- - -## **Best Practices** - -* **Modular Design:** - * Keep task implementations modular to facilitate easy maintenance and extension. - -* **Robust Error Handling:** - * Implement comprehensive error handling within tasks to ensure the system remains stable. - -* **Efficient Dependency Management:** - * Clearly define task dependencies to avoid circular dependencies and ensure smooth execution. - -* **Logging and Monitoring:** - * Maintain detailed logs for all task executions to aid in monitoring and troubleshooting. - -* **Security Considerations:** - * Ensure that executing shell commands or modifying files does not introduce security vulnerabilities. - ---- - -## **Troubleshooting** - -1. **Circular Dependency Detected:** - -* **Issue:** The system throws a "Circular dependency detected in task breakdown" error. -* **Solution:** Review the task definitions to ensure that dependencies are acyclic. Modify task dependencies to - eliminate cycles. - -2. **Unknown Task Type Error:** - -* **Issue:** An error stating "Unknown task type" is encountered. -* **Solution:** Ensure that all custom task types are properly defined and registered in `TaskType.kt`. - -3. **API Errors:** - -* **Issue:** Failures in API interactions during task execution. -* **Solution:** Check API credentials, network connectivity, and API service status. Review logs for detailed error - messages. - -4. **File Reading Errors:** - -* **Issue:** Errors while reading input or output files. -* **Solution:** Verify file paths, permissions, and the existence of the specified files. - -5. **Task Execution Failures:** - -* **Issue:** Individual tasks fail during execution. -* **Solution:** Examine the logs associated with the failed task for error details. Ensure that task configurations are - correct and dependencies are met. - ---- - -## **Frequently Asked Questions (FAQs)** - -**Q1: How do I enable or disable specific task types?** - -* **A:** Modify the `taskSettings` in `PlanSettings.kt` by setting the `enabled` property for each `TaskType`. - -**Q2: Can I add custom commands for shell tasks?** - -* **A:** Yes. When defining a new task type or configuring existing ones, specify the custom commands in the - `ExecutionTask` data class. - -**Q3: How are task dependencies managed?** - -* **A:** Dependencies are defined in the `task_dependencies` field of each `PlanTask`. The system ensures that dependent - tasks execute only after their dependencies have been completed. - -**Q4: Is it possible to log API interactions for auditing purposes?** - -* **A:** Yes. The `PlanCoordinator` creates log files for each API interaction, stored in the `.logs` directory. - -**Q5: How can I visualize the task execution flow?** - -* **A:** The Plan feature generates Mermaid diagrams that visually represent task dependencies and execution flow, - accessible within the application interface. \ No newline at end of file diff --git a/docs/planning_agent_bespoke.html b/docs/planning_agent_bespoke.html deleted file mode 100644 index 7c42c4cd..00000000 --- a/docs/planning_agent_bespoke.html +++ /dev/null @@ -1,245 +0,0 @@ - - - - - - Plan Feature User and Developer Guide - - - - - -
-
-

Plan Feature User and Developer Guide

-
- -
-

Introduction

-

The Plan feature is a sophisticated system designed to break down high-level user requests into manageable, actionable tasks. It leverages OpenAI's - capabilities to automate task planning, execution, and management, ensuring efficient workflow and seamless integration with existing systems.

-
- -
-

Features Overview

-
    -
  • Dynamic Task Breakdown
    • -
  • Task Execution
    • -
  • Dependency Management
    • -
  • Asynchronous Processing
    • -
  • Customizable Settings
    • -
  • Extensible Architecture
    • -
-
- -
-

User Guide

-

Accessing the Plan Feature

-

Navigate to the application interface where task planning and management functionalities are exposed.

-
- -
-

Creating and Managing Plans

-
    -
  1. Initiate a New Plan
    2. -
  2. Task Breakdown
    3. -
  3. Review Generated Tasks
    4. -
  4. Execute the Plan
    5. -
-
- -
-

Understanding Task Dependencies

-
    -
  • Visual Representation using Mermaid
    • -
  • Interpreting the Graph
    • -
-
- -
-

Viewing Execution Logs

-
    -
  • Accessing Logs through UI
    • -
  • Log Files stored in .logs directory
    • -
-
- -
-

Developer Guide

-

Architecture Overview

-
    -
  • TaskType
    • -
  • PlanSettings
    • -
  • PlanCoordinator
    • -
  • PlanUtil
    • -
  • PlanningTask
    • -
  • AbstractTask
    • -
-
- -
-

Task Types

-
    -
  1. CommandAutoFixTask
    2. -
  2. InquiryTask
    3. -
  3. FileModificationTask
    4. -
  4. RunShellCommandTask
    5. -
  5. RefactorTask
    6. -
  6. SecurityAuditTask
    7. -
  7. CodeOptimizationTask
    8. -
  8. CodeReviewTask
    9. -
  9. DocumentationTask
    10. -
  10. TestGenerationTask
    11. -
-
- -
-

Adding New Task Types

- 

-val NewTaskType = TaskType("NewTaskType")
-
-class NewTaskTypeTask(settings: PlanSettings, task: PlanningTask.PlanTask) : AbstractTask(settings, task) {
-    override fun promptSegment(): String {
-        return "Description for NewTaskType"
-    }
-
-    override fun run(
-        agent: PlanCoordinator,
-        taskId: String,
-        userMessage: String,
-        plan: TaskBreakdownInterface,
-        planProcessingState: PlanProcessingState,
-        task: SessionTask,
-        api: API
-    ) {
-        // Implementation for task execution
-    }
-}
-
-
- -
-

Configuring Plan Settings

-
    -
  • Default and Parsing Models
    • -
  • Command Environment
    • -
  • Temperature and Budget
    • -
  • Task-Specific Settings
    • -
  • Auto-Fix and Blocking Behavior
    • -
-
- -
-

Extending PlanCoordinator

-
    -
  • Initialization
    • -
  • Task Execution
    • -
  • Logging and Error Handling
    • -
-
- -
-

Utilizing PlanUtil

-
    -
  • Diagram Generation
    • -
  • Rendering Plans
    • -
  • Plan Filtering
    • -
-
- -
-

Handling Asynchronous Task Execution

-
    -
  • Concurrency Management
    • -
  • Task Dependencies
    • -
  • Error Propagation
    • -
-
- -
-

Customizing Task Execution

-
    -
  1. Overriding the `run` Method
    2. -
  2. Integrating with External Systems
    3. -
  3. Enhancing the Execution Flow
    4. -
-
- -
-

Best Practices

-
    -
  • Modular Design
    • -
  • Robust Error Handling
    • -
  • Efficient Dependency Management
    • -
  • Logging and Monitoring
    • -
  • Security Considerations
    • -
-
- -
-

Troubleshooting

-
    -
  1. Circular Dependency Detected
    2. -
  2. Unknown Task Type Error
    3. -
  3. API Errors
    4. -
  4. File Reading Errors
    5. -
  5. Task Execution Failures
    6. -
-
- -
-

Frequently Asked Questions (FAQs)

-

Q1: How do I enable or disable specific task types?

-

Q2: Can I add custom commands for shell tasks?

-

Q3: How are task dependencies managed?

-

Q4: Is it possible to log API interactions for auditing purposes?

-

Q5: How can I visualize the task execution flow?

-
-
- - - - - - - - - - - diff --git a/docs/planning_agent_deck.html b/docs/planning_agent_deck.html deleted file mode 100644 index 044a1329..00000000 --- a/docs/planning_agent_deck.html +++ /dev/null @@ -1,225 +0,0 @@ -+ - - - - - Plan Feature User and Developer Guide - - - - - - - - - -
-

Plan Feature User and Developer Guide

-
- -
-

Introduction

-

The Plan feature is a sophisticated system designed to break down high-level user requests into manageable, actionable tasks. It leverages OpenAI's - capabilities to automate task planning, execution, and management, ensuring efficient workflow and seamless integration with existing systems.

-
- -
-

Features Overview

-
    -
  • Dynamic Task Breakdown
    • -
  • Task Execution
    • -
  • Dependency Management
    • -
  • Asynchronous Processing
    • -
  • Customizable Settings
    • -
  • Extensible Architecture
    • -
-
- -
-

User Guide

-

Accessing the Plan Feature

-

Navigate to the application interface where task planning and management functionalities are exposed.

-
- -
-

Creating and Managing Plans

-
    -
  1. Initiate a New Plan
    2. -
  2. Task Breakdown
    3. -
  3. Review Generated Tasks
    4. -
  4. Execute the Plan
    5. -
-
- -
-

Understanding Task Dependencies

-
    -
  • Visual Representation using Mermaid
    • -
  • Interpreting the Graph
    • -
-
- -
-

Viewing Execution Logs

-
    -
  • Accessing Logs through UI
    • -
  • Log Files stored in .logs directory
    • -
-
- -
-

Developer Guide

-

Architecture Overview

-
    -
  • TaskType
    • -
  • PlanSettings
    • -
  • PlanCoordinator
    • -
  • PlanUtil
    • -
  • PlanningTask
    • -
  • AbstractTask
    • -
-
- -
-

Task Types

-
    -
  1. CommandAutoFixTask
    2. -
  2. InquiryTask
    3. -
  3. FileModificationTask
    4. -
  4. RunShellCommandTask
    5. -
  5. RefactorTask
    6. -
-
- -
-

Task Types (continued)

-
    -
  1. SecurityAuditTask
    2. -
  2. CodeOptimizationTask
    3. -
  3. CodeReviewTask
    4. -
  4. DocumentationTask
    5. -
  5. TestGenerationTask
    6. -
-
- -
-

Adding New Task Types

-
    -
  1. Define the Task Type
    2. -
  2. Create the Task Class
    3. -
  3. Register the Task Type
    4. -
  4. Update PlanSettings
    5. -
-
- -
-

Configuring Plan Settings

-
    -
  • Default and Parsing Models
    • -
  • Command Environment
    • -
  • Temperature and Budget
    • -
  • Task-Specific Settings
    • -
  • Auto-Fix and Blocking Behavior
    • -
-
- -
-

Extending PlanCoordinator

-
    -
  • Initialization
    • -
  • Task Execution
    • -
  • Logging and Error Handling
    • -
-
- -
-

Utilizing PlanUtil

-
    -
  • Diagram Generation
    • -
  • Rendering Plans
    • -
  • Plan Filtering
    • -
-
- -
-

Handling Asynchronous Task Execution

-
    -
  • Concurrency Management
    • -
  • Task Dependencies
    • -
  • Error Propagation
    • -
-
- -
-

Customizing Task Execution

-
    -
  1. Overriding the run Method
    2. -
  2. Integrating with External Systems
    3. -
  3. Enhancing the Execution Flow
    4. -
-
- -
-

Best Practices

-
    -
  • Modular Design
    • -
  • Robust Error Handling
    • -
  • Efficient Dependency Management
    • -
  • Logging and Monitoring
    • -
  • Security Considerations
    • -
-
- -
-

Troubleshooting

-
    -
  1. Circular Dependency Detected
    2. -
  2. Unknown Task Type Error
    3. -
  3. API Errors
    4. -
  4. File Reading Errors
    5. -
  5. Task Execution Failures
    6. -
-
- -
-

Frequently Asked Questions (FAQs)

-
    -
  • How do I enable or disable specific task types?
    • -
  • Can I add custom commands for shell tasks?
    • -
  • How are task dependencies managed?
    • -
  • Is it possible to log API interactions for auditing purposes?
    • -
  • How can I visualize the task execution flow?
    • -
-
- - - - - - - - - - diff --git a/docs/planning_agent_impress.html b/docs/planning_agent_impress.html deleted file mode 100644 index a0d8d3fe..00000000 --- a/docs/planning_agent_impress.html +++ /dev/null @@ -1,158 +0,0 @@ - - - - - - Plan Feature User and Developer Guide - - - - -
-
-

Plan Feature User and Developer Guide

-

Welcome to the comprehensive guide for the Plan feature, designed to assist both end-users and developers.

-
- -
-

Introduction

-

The Plan feature is a sophisticated system designed to break down high-level user requests into manageable, actionable tasks.

-
- -
-

Features Overview

-
    -
  • Dynamic Task Breakdown
    • -
  • Task Execution
    • -
  • Dependency Management
    • -
  • Asynchronous Processing
    • -
  • Customizable Settings
    • -
  • Extensible Architecture
    • -
-
- -
-

User Guide

-

Accessing the Plan Feature

-

Navigate to the application interface where task planning and management functionalities are exposed.

-
- -
-

Creating and Managing Plans

-
    -
  1. Initiate a New Plan
    2. -
  2. Task Breakdown
    3. -
  3. Review Generated Tasks
    4. -
  4. Execute the Plan
    5. -
-
- -
-

Understanding Task Dependencies

-

The Plan feature utilizes Mermaid to render task dependency graphs.

-
- -
-

Viewing Execution Logs

-

Execution logs are automatically generated and can be accessed through the UI.

-
- -
-

Developer Guide

-

Architecture Overview

-

The Plan feature is built around several core components: TaskType, PlanSettings, PlanCoordinator, PlanUtil, PlanningTask, and AbstractTask.

-
- -
-

Task Types

-

Various task types are defined, including CommandAutoFixTask, InquiryTask, FileModificationTask, and more.

-
- -
-

Adding New Task Types

-

To introduce a new task type:

-
    -
  1. Define the Task Type
    2. -
  2. Create the Task Class
    3. -
  3. Register the Task Type
    4. -
  4. Update PlanSettings
    5. -
-
- -
-

Configuring Plan Settings

-

PlanSettings.kt allows for extensive configuration of how tasks are planned and executed.

-
- -
-

Extending PlanCoordinator

-

PlanCoordinator.kt is pivotal in managing the lifecycle of plans and their tasks.

-
- -
-

Utilizing PlanUtil

-

PlanUtil.kt provides utility functions essential for task management and visualization.

-
- -
-

Handling Asynchronous Task Execution

-

The system leverages a ThreadPoolExecutor to manage task execution asynchronously.

-
- -
-

Customizing Task Execution

-

Developers can customize how individual tasks execute by overriding methods and integrating with external systems.

-
- -
-

Best Practices

-
    -
  • Modular Design
    • -
  • Robust Error Handling
    • -
  • Efficient Dependency Management
    • -
  • Logging and Monitoring
    • -
  • Security Considerations
    • -
-
- -
-

Troubleshooting

-

Common issues and their solutions are provided, including circular dependencies, unknown task types, API errors, file reading errors, and task execution failures.

-
- -
-

Frequently Asked Questions (FAQs)

-

Answers to common questions about enabling/disabling task types, adding custom commands, managing dependencies, logging API interactions, and visualizing task execution - flow.

-
-
- - - - - diff --git a/docs/planning_agent_reveal.html b/docs/planning_agent_reveal.html deleted file mode 100644 index 2fd66669..00000000 --- a/docs/planning_agent_reveal.html +++ /dev/null @@ -1,154 +0,0 @@ -+ - - - - - - Plan Feature User and Developer Guide - - - - - - - - - -
-
-
-

Plan Feature User and Developer Guide

-
- -
-

Introduction

-

The Plan feature is a sophisticated system designed to break down high-level user requests into manageable, actionable tasks. It leverages OpenAI's - capabilities to automate task planning, execution, and management, ensuring efficient workflow and seamless integration with existing systems.

-
- -
-

Features Overview

-
    -
  • Dynamic Task Breakdown
    • -
  • Task Execution
    • -
  • Dependency Management
    • -
  • Asynchronous Processing
    • -
  • Customizable Settings
    • -
  • Extensible Architecture
    • -
-
- -
-

User Guide

-
-

Accessing the Plan Feature

-

Navigate to the application interface where task planning and management functionalities are exposed.

-
-
-

Creating and Managing Plans

-
    -
  1. Initiate a New Plan
    2. -
  2. Task Breakdown
    3. -
  3. Review Generated Tasks
    4. -
  4. Execute the Plan
    5. -
-
-
-

Understanding Task Dependencies

-

The Plan feature utilizes Mermaid to render task dependency graphs.

-
-
-

Viewing Execution Logs

-

Execution logs are automatically generated and can be accessed through the UI.

-
-
- -
-

Developer Guide

-
-

Architecture Overview

-
    -
  • TaskType
    • -
  • PlanSettings
    • -
  • PlanCoordinator
    • -
  • PlanUtil
    • -
  • PlanningTask
    • -
  • AbstractTask
    • -
-
-
-

Task Types

-
    -
  1. CommandAutoFixTask
    2. -
  2. InquiryTask
    3. -
  3. FileModificationTask
    4. -
  4. RunShellCommandTask
    5. -
  5. RefactorTask
    6. -
  6. SecurityAuditTask
    7. -
  7. CodeOptimizationTask
    8. -
  8. CodeReviewTask
    9. -
  9. DocumentationTask
    10. -
  10. TestGenerationTask
    11. -
-
-
-

Adding New Task Types

-
    -
  1. Define the Task Type
    2. -
  2. Create the Task Class
    3. -
  3. Register the Task Type
    4. -
  4. Update PlanSettings
    5. -
-
-
- -
-

Best Practices

-
    -
  • Modular Design
    • -
  • Robust Error Handling
    • -
  • Efficient Dependency Management
    • -
  • Logging and Monitoring
    • -
  • Security Considerations
    • -
-
- -
-

Troubleshooting

-
    -
  1. Circular Dependency Detected
    2. -
  2. Unknown Task Type Error
    3. -
  3. API Errors
    4. -
  4. File Reading Errors
    5. -
  5. Task Execution Failures
    6. -
-
- -
-

Frequently Asked Questions (FAQs)

-

Q1: How do I enable or disable specific task types?

-

Q2: Can I add custom commands for shell tasks?

-

Q3: How are task dependencies managed?

-

Q4: Is it possible to log API interactions for auditing purposes?

-

Q5: How can I visualize the task execution flow?

-
-
-
- - - - - - - - diff --git a/docs/prompts/project_info.md b/docs/prompts/project_info.md deleted file mode 100644 index 45bbcdfe..00000000 --- a/docs/prompts/project_info.md +++ /dev/null @@ -1,216 +0,0 @@ -# Software Project Formalization and Documentation Questionnaire - -## General Information - -1. **Project Name:** - Skyenet LLM Agent Platform - -2. **Project Description:** - The Skyenet LLM Agent Platform is designed to facilitate the integration and utilization of large language models (LLMs) in various applications. It provides a framework for - managing LLM interactions, handling user inputs, and displaying dynamic content. - -3. **Project Objectives:** - * To create a robust and flexible platform that supports real-time LLM interactions. - * To provide a seamless user experience with dynamic content updates. - * To integrate various backend processes and user interactions using LLMs. - -4. **Target Audience:** - Developers building applications that leverage large language models for tasks such as coding assistance, image generation, text-to-speech conversion, and more. - -## Scope and Deliverables - -1. **Assumptions and Context:** - * What assumptions are being made about the project? - * The system will be used in applications that require real-time LLM interactions. - * Users will have access to modern LLM models and necessary computational resources. - * What is the context in which this project is being developed? - * The project is being developed to enhance user interaction in applications by providing a dynamic and responsive LLM-based platform. - * What problem is this project aiming to solve? - * The project aims to solve the problem of integrating and utilizing large language models in applications, by providing a flexible and robust LLM agent platform. - * The project aims to solve the problem of static web pages that require full page reloads for updates, by - providing a real-time, interactive UI framework. - -2. **Scope:** - The project includes the development of an LLM agent platform that supports real-time interactions using large language models, dynamic content updates, and integration with - various backend processes. - -3. **Deliverables:** - * An LLM-based agent platform. - * Documentation for developers. - * Example implementations and usage guides. - * Integration with backend processes. - -4. **Out of Scope:** - * Development of LLM models. - * Support for outdated computational resources that cannot handle modern LLMs. - -5. **Use Case Details:** - * **Coding assistance**: Generate and execute code snippets based on natural language instructions. - * **Image generation**: Transform textual descriptions into images using image generation models. - * **Text-to-speech conversion**: Convert text into speech using text-to-speech models. - * **Interactive discussions**: Facilitate user interactions and collect feedback to refine responses. - -## Developer Documentation - -1. **Code Repository:** - The code repository will be hosted on GitHub at https://github.com/your-repo/skyenet-llm-agent-platform - - * Repository URL: - * Branching strategy: - * Contribution guidelines: - -2. **Development Setup:** - * Clone the repository from GitHub. - * Install the required dependencies using the provided setup script. - * Follow the development environment setup guide in the repository's README file. - -3. **Contribution Guidelines:** - * Follow the coding standards and best practices outlined in the CONTRIBUTING.md file. - * Submit pull requests for code reviews before merging changes. - * Write unit tests for new features and bug fixes. - -4. **API Documentation:** - API documentation can be found in the docs/api directory of the repository. - -5. **Coding Standards:** - What coding standards or best practices should be followed? - * Follow the Kotlin coding conventions. - * Use meaningful variable and function names. - * Write clear and concise comments. - Are there any specific linting or formatting tools to be used? - * Use ktlint for linting and code formatting. - Are there any version control guidelines? - * Use Git for version control. - Are there any specific branch naming conventions or commit message guidelines? - * Use feature/branch-name for feature branches. - * Use fix/branch-name for bug fix branches. - * Follow the commit message guidelines outlined in the CONTRIBUTING.md file. - -## Architecture and Design - -1. **System Architecture:** - * Overview of the system architecture: - The Skyenet LLM Agent Platform is designed with a modular architecture to support real-time, interactive applications leveraging large language models. The core components - include the LLM manager, session management, and dynamic content rendering. The system leverages LLMs for real-time interaction between the client and server, ensuring low - latency and high - responsiveness. The architecture is divided into several layers: the presentation layer (UI components), the - application layer (session and task management), and the data layer (storage and retrieval of user data and - settings). - - * Diagrams (if any): - ![System Architecture Diagram](path/to/diagram.png) // Add a path to the actual diagram if available - -2. **Design Patterns:** - * What design patterns are used? - The Skyenet LLM Agent Platform employs several design patterns to ensure a robust and maintainable codebase: - -**Observer Pattern**: Used in the LLM manager to handle real-time updates and notifications. -**Factory Pattern**: Utilized for creating instances of various components like session tasks and UI elements. -**Singleton Pattern**: Ensures a single instance of core managers like `ApplicationServices` -- **Observer Pattern**: Used in the WebSocket manager to handle real-time updates and notifications. -- **Factory Pattern**: Utilized for creating instances of various components like session tasks and UI elements. -- **Singleton Pattern**: Ensures a single instance of core managers like `ApplicationServices` -and `SocketManagerBase`. -**Strategy Pattern**: Allows for flexible implementation of different retry mechanisms in the `Retryable` class. -**Decorator Pattern**: Used to extend the functionality of UI components dynamically. -- **Strategy Pattern**: Allows for flexible implementation of different retry mechanisms in the `Retryable` class. -- **Decorator Pattern**: Used to extend the functionality of UI components dynamically. - - * Why were they chosen? - These design patterns were chosen to promote code reusability, flexibility, and separation of concerns. The - Observer Pattern is ideal for real-time LLM interaction scenarios, while the Factory Pattern simplifies the creation - of complex objects. The Singleton Pattern ensures efficient resource management, and the Strategy Pattern allows - for easy customization of retry logic. The Decorator Pattern enables dynamic enhancement of UI components without - modifying their core functionality. - -3. **Technology Stack:** - * List of technologies and frameworks used: - - - **Frontend**: HTML, CSS, JavaScript, and Kotlin for the UI components and LLM interactions. - - **Backend**: Kotlin for server-side logic, leveraging Ktor for building asynchronous servers and managing LLM interactions. - - **LLMs**: Used for real-time interaction between the client and server. - - **Database**: HSQLDB for usage tracking, data storage, and managing LLM interactions. - **Cloud Services**: AWS S3 for file storage, AWS KMS for encryption. - **CI/CD**: GitHub Actions for continuous integration and deployment, Docker for containerization. - - **Cloud Services**: AWS S3 for file storage, AWS KMS for encryption. - - **CI/CD**: GitHub Actions for continuous integration and deployment, Docker for containerization. - - * Reasons for choosing these technologies: - The chosen technologies and frameworks provide a balance of performance, scalability, and developer productivity. Kotlin is used for both frontend and backend development to - ensure consistency and leverage its modern language features. Ktor is selected for its lightweight and asynchronous capabilities, making it suitable for real-time LLM - interactions. LLMs are essential for maintaining low-latency communication. HSQLDB is chosen for its - simplicity and ease of integration. AWS services are utilized for their reliability, security, and scalability. - GitHub Actions and Docker streamline the CI/CD process, ensuring efficient and reliable deployments. - -## Components and Modules - -1. **Component Overview:** - * **LLMManagerBase**: Manages LLM interactions, message queuing, and broadcasting. It handles the core LLM functionality, ensuring reliable communication between the client and - server. - * **ApplicationInterface**: Provides methods to create interactive HTML elements and manage tasks. It acts as a - bridge between the task management logic and the LLM communication layer. - * **SessionTask**: Represents a task that can display progress and messages. It allows for dynamic updates to the UI - as tasks progress. - * **TabbedDisplay**: Manages a tabbed interface for displaying content. It supports adding, updating, and rendering - tabs dynamically. - * **Retryable**: Extends `TabbedDisplay` to add a retry mechanism for tasks. It allows for re-running a block of - code and capturing the new output in a new tab. - * **AgentPatterns**: Provides utility functions for displaying content in tabs. It helps in organizing and rendering - content efficiently. - * **Discussable**: Facilitates interactive discussions with users, allowing for feedback and revisions. It manages - user interactions and refines responses based on feedback. - * **UserSettingsManager**: Manages user settings, including loading and saving settings to disk. It ensures that - user preferences are maintained across sessions. - * **ClientManager**: Manages API clients, including creating and caching clients for sessions and users. It handles - the lifecycle of API clients and ensures efficient resource usage. - * **HSQLUsageManager**: Manages usage tracking using an HSQL database. It tracks resource usage and provides - summaries for monitoring and billing purposes. - * **AuthorizationManager**: Manages user authorization, checking if a user is authorized for specific operations. It - enforces access control policies. - * **AuthenticationManager**: Manages user authentication, including storing and retrieving users based on access - tokens. It ensures secure access to the system. - * **DataStorage**: Manages data storage, including storing and retrieving messages, sessions, and other data. It - provides a persistent storage layer for the application. - * **AwsPlatform**: Provides integration with AWS services, including S3 for file storage and KMS for encryption. It - leverages cloud services for scalability and security. - -2. **Inter-component Communication:** - * **LLM Communication**: `LLMManagerBase` handles the LLM interactions and message broadcasting. It - interacts with `ApplicationInterface` to send and receive messages. - * **Task Management**: `ApplicationInterface` creates and manages `SessionTask` instances. These tasks - use `TabbedDisplay` and `Retryable` to display content and handle retries. - * **User Interaction**: `Discussable` manages user interactions and collects feedback. It works with `SessionTask` - to update the UI based on user input. - * **Data Storage and Retrieval**: `DataStorage` handles the storage and retrieval of data. It interacts - with `UserSettingsManager`, `ClientManager`, and `HSQLUsageManager` to manage user settings, API clients, and - usage tracking. - * **Authorization and Authentication**: `AuthorizationManager` and `AuthenticationManager` enforce security - policies. They interact with other components to ensure secure access and operations. - * **Cloud Integration**: `AwsPlatform` provides cloud services for storage and encryption. It integrates - with `DataStorage` and other components to leverage AWS services. - -## Testing and Quality Assurance - -1. **Testing Strategy:** - What is the overall testing strategy? - * The testing strategy includes unit testing, integration testing, system testing, and acceptance testing. - What types of testing will be performed (unit, integration, system, acceptance)? - * **Unit Testing**: Testing individual components and functions. - * **Integration Testing**: Testing interactions between components. - * **System Testing**: Testing the entire system as a whole. - * **Acceptance Testing**: Testing the system against user requirements. - -2. **Test Cases:** - Test cases will be documented in the docs/test-cases directory of the repository. - -3. **CI/CD Pipeline:** - What is the CI/CD pipeline setup? - * The CI/CD pipeline includes automated testing, code quality checks, and deployment processes. - What tools and services are used for CI/CD? - * The pipeline uses GitHub Actions for automation, Docker for containerization, and AWS for deployment. - -4. **Code Reviews:** - What is the process for code reviews? - * Code reviews are conducted through pull requests on GitHub. - Who is responsible for code reviews? - * The development team is responsible for reviewing each other's code. \ No newline at end of file diff --git a/docs/prompts/project_info_template.md b/docs/prompts/project_info_template.md deleted file mode 100644 index 7c666486..00000000 --- a/docs/prompts/project_info_template.md +++ /dev/null @@ -1,138 +0,0 @@ -# Software Project Formalization and Documentation Questionnaire - -## General Information - -1. **Project Name:** - What is the name of the project? - -2. **Project Description:** - Provide a brief description of the project. - -3. **Project Objectives:** - What are the main objectives of the project? - -4. **Target Audience:** - Who are the intended users of the project? - -5. **Stakeholders:** - Who are the key stakeholders involved in the project? - -6. **Assumptions and Context:** - - What assumptions are being made about the project? - - What is the context in which this project is being developed? - - What problem is this project aiming to solve? - -## Scope and Deliverables - -1. **Scope:** - What is the scope of the project? - -2. **Deliverables:** - What are the key deliverables of the project? - -3. **Out of Scope:** - What is explicitly out of scope for the project? - -4. **Use Case Details:** - - What are the primary use cases for the project? - - Provide detailed descriptions of each use case. - -5. **Comparative Product Analysis:** - - Are there existing products that solve a similar problem? - - How does this project compare to those products? - -## Timeline - -1. **Start Date:** - When is the project starting? - -2. **End Date:** - When is the project expected to be completed? - -3. **Key Dates:** - Are there any key dates or deadlines that need to be met? - -4. **Milestones:** - List the major milestones and their expected completion dates. - -## Risks and Assumptions - -1. **Risks:** - Identify potential risks and their mitigation strategies. - -2. **Assumptions:** - - List any assumptions made during the planning of the project. - -## Project Closure - -1. **Closure Criteria:** - - What are the criteria for project closure? - - How will success be measured? - -2. **Post-Implementation Review:** - - Will there be a post-implementation review? - - What will be the focus of this review? - -3. **Lessons Learned:** - How will lessons learned be documented and shared? - -## Developer Documentation - -1. **Code Repository:** - Where will the code repository be hosted? - -2. **Development Setup:** - What are the steps to set up the development environment? - -3. **Contribution Guidelines:** - What are the guidelines for contributing to the project? - -4. **API Documentation:** - Where can developers find the API documentation? - -5. **Coding Standards:** - What coding standards or best practices should be followed? - Are there any specific linting or formatting tools to be used? - Are there any version control guidelines? - Are there any specific branch naming conventions or commit message guidelines? - -6. **Architectural Perspectives:** - - What are the different architectural perspectives of the project? - - Provide diagrams and descriptions for each perspective (e.g., logical, physical, deployment, etc.). - - What coding standards or best practices should be followed? - - Are there any specific linting or formatting tools to be used? - - Are there any version control guidelines? - - Are there any specific branch naming conventions or commit message guidelines? - -## Testing and Quality Assurance - -1. **Testing Strategy:** - - What is the overall testing strategy? - - What types of testing will be performed (unit, integration, system, acceptance)? - -2. **Test Cases:** - Where are the test cases documented? - -3. **CI/CD Pipeline:** - - What is the CI/CD pipeline setup? - - What tools and services are used for CI/CD? - -4. **Code Reviews:** - - What is the process for code reviews? - - Who is responsible for code reviews? - -## Maintenance and Support - -1. **Maintenance Plan:** - - What is the plan for ongoing maintenance? - - How will updates and patches be handled? - -2. **Support Plan:** - - What is the support plan post-deployment? - - Who will provide support and how can they be contacted? - -3. **Documentation:** - - What user documentation will be provided? - - Where can users find help and support documentation? - - What coding standards or best practices should be followed? - - List any assumptions made during the planning of the project. \ No newline at end of file diff --git a/docs/prompts/project_info_template2.md b/docs/prompts/project_info_template2.md deleted file mode 100644 index cca8486d..00000000 --- a/docs/prompts/project_info_template2.md +++ /dev/null @@ -1,67 +0,0 @@ -# Software Project Formalization and Documentation Questionnaire - -## General Information - -1. **Project Name:** -2. **Project Description:** - - What problem does this project solve? - - What are the key features and functionalities? -3. **Project Objectives:** - - What are the short-term and long-term goals? - - How will success be measured? -4. **Target Audience:** - - Who are the primary users? - - What are their needs and pain points? -5. **Scope:** - - What are the boundaries of the project? - - What will not be included in this project? - -## Developer Documentation - -1. **Code Repository:** - - Repository URL: - - Branching strategy: - - Contribution guidelines: -2. **Development Setup:** - - Prerequisites: - - Installation steps: - - Configuration details: - - How to run the project locally: - - How to run tests: - -## Architecture and Design - -1. **System Architecture:** - - Overview of the system architecture: - - Diagrams (if any): -2. **Design Patterns:** - - What design patterns are used? - - Why were they chosen? -3. **Technology Stack:** - - List of technologies and frameworks used: - - Reasons for choosing these technologies: - -## Components and Modules - -1. **Component Overview:** - - List of main components/modules: - - Brief description of each component/module: -2. **Inter-component Communication:** - - How do components/modules interact with each other? - - Data flow between components/modules: - -## Testing and Quality Assurance - -1. **Testing Strategy:** - - Types of tests (unit, integration, end-to-end): - - Tools and frameworks used for testing: - - Test coverage goals: -2. **CI/CD Pipeline:** - - Overview of the CI/CD process: - - Tools and services used: - - Steps in the pipeline: -3. **Release Strategy:** - - Versioning scheme: - - Release frequency: - - Deployment process: - - Rollback plan: \ No newline at end of file diff --git a/docs/ui/discussable.md b/docs/ui/discussable.md deleted file mode 100644 index ef77633e..00000000 --- a/docs/ui/discussable.md +++ /dev/null @@ -1,70 +0,0 @@ -## Discussable Class Documentation - -### Overview - -The `Discussable` class in Kotlin is designed to facilitate user interactions within a session-based application, -particularly for tasks that require user confirmation or iterative refinement based on user feedback. It is highly -useful in scenarios where an initial response needs to be generated based on user input, and then potentially revised -multiple times based on further user interactions. - -### Key Features - -- **Generic Type Support**: The class is generic, allowing it to work with any type of response object. -- **User Interaction**: Manages user interactions and collects feedback to refine the response. -- **Concurrency Handling**: Utilizes semaphores to manage concurrent access to the response object, ensuring thread - safety. -- **Session Management**: Integrates with session tasks to maintain state across multiple interactions within a user - session. - -### Constructor Parameters - -- `task: SessionTask`: The session task associated with the current user interaction. -- `userMessage: String`: The initial message from the user that triggers the response generation. -- `initialResponse: (String) -> T`: A function that generates the initial response based on the user message. -- `outputFn: (T) -> String`: A function that converts the response object into a string for display. -- `ui: ApplicationInterface`: The interface for interacting with the application's user interface. -- `reviseResponse: (List>) -> T`: A function to revise the response based on a list of user messages - and their roles. -- `atomicRef: AtomicReference = AtomicReference()`: An atomic reference to hold the response object securely for - concurrent access. -- `semaphore: Semaphore = Semaphore(0)`: A semaphore to control the release of the response once it is discussed. -- `heading: String`: A heading for the interaction, typically used for display purposes. - -### Methods - -- `call()`: This is the main method that executes the interaction logic. It orchestrates the display of the initial - response, handles user feedback, and waits for the user to discuss a final response. -- `main(tabIndex: Int, task: SessionTask)`: Handles the main interaction logic for a specific tab in the user interface. -- `feedbackForm(...)`: Generates a form for user feedback based on the current state of the response. -- `discuss(...)`: Handles the response, releasing the semaphore and setting the response in the atomic - reference. - -### Usage Example - -```kotlin -val discussable = com.simiacryptus.skyenet.Discussable( - task = sessionTask, - userMessage = "What is the weather like today?", - initialResponse = { message -> weatherService.getWeather(message) }, - outputFn = { response -> "The weather is ${response.description} with a temperature of ${response.temperature}" }, - ui = applicationInterface, - reviseResponse = { interactions -> - weatherService.reviseWeather(interactions.last().first) - }, - heading = "Weather Inquiry" -) - -val finalWeather = discussable.call() -println("Final weather response: $finalWeather") -``` - -### Notes - -- The `Discussable` class is designed to be flexible and reusable for different types of interactions where user - confirmation is required. -- It uses advanced Kotlin features like generics, lambdas, and concurrency utilities to provide a robust solution for - interactive applications. -- Proper error handling and user interface management are crucial for implementing a smooth user experience. - -This documentation provides a comprehensive overview of the `Discussable` class, detailing its construction, methods, and -typical usage within an application that requires interactive user sessions. \ No newline at end of file diff --git a/docs/ui/task_ui.md b/docs/ui/task_ui.md deleted file mode 100644 index 906e5e8b..00000000 --- a/docs/ui/task_ui.md +++ /dev/null @@ -1,214 +0,0 @@ -# Task UI - -The `SessionTask` class provides a way to display long-running tasks with progress updates in the UI. It allows you to -append messages, headers, errors, and images to the task output. - -## Methods - -### add(message: String, showSpinner: Boolean = true, tag: String = "div", className: String = "response-message") - -Adds a message to the task output. The message will be wrapped in the specified HTML tag with the given CSS class. -If `showSpinner` is true, a loading spinner will be displayed after the message to indicate ongoing processing. -The `tag` parameter allows customization of the HTML tag used, and `className` specifies the CSS class for styling. - -### hideable(ui: ApplicationInterface?, message: String, showSpinner: Boolean = true, tag: String = "div", className: String = "response-message"): StringBuilder? - -Adds a hideable message to the task output. The message will include a close button, allowing the user to manually hide -the message. This method returns a `StringBuilder` instance containing the message, which can be manipulated further if -needed. The `ui` parameter is used to handle UI interactions for the close button. - -### echo(message: String, showSpinner: Boolean = true, tag: String = "div") - -Echos a user message to the task output. -This method is typically used for echoing user inputs or commands back to the UI for confirmation or logging purposes. - -### header(message: String, showSpinner: Boolean = true, tag: String = "div", classname: String = "response-header") - -Adds a header to the task output with the specified CSS class. -Headers are useful for separating sections of output or introducing new stages of task progress. - -### verbose(message: String, showSpinner: Boolean = true, tag: String = "pre") - -Adds a verbose message to the task output. Verbose messages are hidden by default and wrapped in a `
` tag.
-This method is ideal for displaying detailed diagnostic or debug information that can be expanded or collapsed by the
-user.
-
-### error(ui: ApplicationInterface?, e: Throwable, showSpinner: Boolean = false, tag: String = "div")
-
-Displays an error in the task output. This method is specialized to handle different types of exceptions:
-
-- `ValidationError`: Displays a message indicating a validation error along with a detailed stack trace.
-- `FailedToImplementException`: Shows a message indicating a failure in implementation, including relevant code snippets
-  and language details.
-- Other exceptions: Displays the exception name and a complete stack trace to aid in debugging. The `showSpinner`
-  parameter can be set to `false` to not show a spinner, as errors typically denote the end of processing.
-
-### complete(message: String = "", tag: String = "div", className: String = "response-message")
-
-Displays a final message in the task output and hides the spinner, indicating that the task has been completed. If no
-message is provided, only the spinner will be hidden without any additional text.
-
-### Placeholder Mechanism
-
-Each `SessionTask` instance generates a unique placeholder in the UI, represented by an HTML `div` element with an `id`
-attribute set to the task's `operationID`. This placeholder serves as a dynamic container where task-related updates and
-outputs are rendered in real-time. This mechanism is crucial for maintaining a responsive and interactive user interface
-during the execution of tasks.
-
-### Non-Root Tasks
-
-When creating a new task with the `newTask(root: Boolean)` method, setting `root` to `false` allows the creation of a
-subordinate task. Non-root tasks are typically used for operations that are part of a larger task or workflow. They
-inherit the context and permissions of the parent task, enabling structured and hierarchical task management within the
-application.
-
-### image(image: BufferedImage)
-
-Displays an image in the task output. The image is saved as a PNG file, and the URL of the saved image is embedded
-within an `` tag to be displayed in the UI.
-
-## Saving Files
-
-The `saveFile(relativePath: String, data: ByteArray): String` method allows saving file data and returns the URL of the
-saved file. This is useful for displaying images or providing file downloads in the task output.
-This method is crucial for managing file outputs in tasks that involve file generation or manipulation, ensuring that
-users can access or download the generated files directly from the task UI.
-
-## Overview of the Task UI API
-
-The Task UI API in the provided Kotlin codebase is designed to facilitate the creation and management of user interface
-tasks within a web application. This API is part of a larger system that likely involves real-time interaction with
-users through a web interface. The main components involved in the Task UI API
-include `SessionTask`, `ApplicationInterface`, and utility functions and classes that support task management and
-display.
-The API's design focuses on providing a seamless and dynamic user experience, where tasks can be monitored and
-controlled interactively, enhancing the overall user engagement and efficiency of the web application.
-
-### Key Components
-
-#### 1. `SessionTask`
-
-`SessionTask` is an abstract class that represents a task session in the UI. It is designed to handle the dynamic output
-of content to the user interface during the execution of a task. Key functionalities include:
-
-- **Progress Tracking**: Allows real-time tracking of task progress through various states, providing immediate feedback
-  to the user.
-- **Interactive Elements**: Supports adding interactive elements like buttons and links within the task output, enabling
-  user actions directly from the task interface.
-
-- **Dynamic Content Management**: It manages a buffer that aggregates output content which can be dynamically updated
-  and displayed in the UI.
-- **Abstract Methods**:
-    - `send(html: String)`: Sends the compiled HTML content to the UI.
-    - `saveFile(relativePath: String, data: ByteArray)`: Saves a file and returns a URL to access it, used for handling
-      file outputs like images.
-- **Utility Methods**:
-    - `add(message: String, showSpinner: Boolean, tag: String, className: String)`: Adds a message to the UI with
-      configurable HTML wrapping and CSS styling.
-    - `hideable(...)`, `echo(...)`, `header(...)`, `verbose(...)`, `error(...)`, `complete(...)`, `image(...)`: These
-      methods provide specialized ways to add different types of content to the UI, such as hideable messages, errors,
-      headers, and images.
-
-#### 2. `ApplicationInterface`
-
-`ApplicationInterface` serves as a bridge between the task management logic and the socket communication
-layer (`SocketManagerBase`). It provides methods to interact with the user through hyperlinks and form inputs, and to
-manage tasks:
-
-- **Dynamic Task Creation**: Dynamically creates tasks based on user interactions or automated triggers, ensuring that
-  each task is tailored to the specific needs of the operation.
-- **Enhanced User Interaction**: Facilitates richer user interaction models by providing utility methods for creating
-  interactive UI components like hyperlinks and text inputs.
-
-- **Task Creation**: `newTask(root: Boolean)`: Creates a new `SessionTask` instance.
-- **UI Element Creation**:
-    - `hrefLink(...)`: Generates HTML for a clickable link that triggers a specified handler.
-    - `textInput(...)`: Generates HTML for a text input form that triggers a specified handler upon submission.
-
-#### 3. Utility Functions and Classes
-
-- **`AgentPatterns`**: Contains utility functions like `displayMapInTabs(...)`, which helps in displaying data in a
-  tabbed interface.
-- **Error Handling**: Includes mechanisms to gracefully handle and display errors within the UI, ensuring that users are
-  well-informed about any issues during task execution.
-- **Session Management**: Provides robust session management capabilities to maintain the state and continuity of user
-  tasks.
-- **Image Handling**: The `toPng()` extension function for `BufferedImage` converts an image to PNG format, useful in
-  tasks that involve image processing.
-
-### Usage Example
-
-To use the Task UI API, a developer would typically instantiate a `SessionTask` through `ApplicationInterface` and use
-the task's methods to dynamically add content to the UI based on the application's logic and user interactions. For
-example:
-
-```kotlin
-val appInterface = ApplicationInterface(socketManager)
-val task = appInterface.newTask()
-task.header("Processing Data")
-// Perform some data processing
-task.add("Data processed successfully", showSpinner = false)
-task.complete("Task completed.")
-```
-
-### Conclusion
-
-The Task UI API provides a robust set of tools for managing interactive tasks in a web application. By abstracting the
-complexities of real-time UI updates and task management, it allows developers to focus on the core logic of their
-applications while providing a responsive and interactive user experience.
-
-The `SessionTask` class in the Kotlin codebase provides a mechanism to manage and display linked tasks in a web UI,
-particularly through the use of placeholders. This functionality is crucial for tasks that are interdependent or need to
-be executed in a sequence, allowing the UI to dynamically update as tasks progress or complete.
-
-## Placeholders
-
-The placeholder is a unique identifier used to represent a task in the UI dynamically. It allows the system to update
-the task's output in real-time without reloading the entire page. Here's how it is implemented and used:
-
-#### Placeholder Generation
-
-Each `SessionTask` instance has an `operationID`, which is a unique identifier for that task. The `placeholder` is an
-HTML `div` element with its `id` attribute set to the `operationID`. This `div` acts as a placeholder in the HTML
-document where the task's output will be dynamically inserted or updated.
-
-```kt
-val placeholder: String get() = "
"
-```
-
-#### Using the Placeholder
-
-When a new task is created, its placeholder is initially empty. As the task progresses, messages, errors, images, or
-other outputs are dynamically inserted into this placeholder using JavaScript and WebSocket communication. This approach
-allows the UI to remain responsive and update in real-time as the task outputs change.
-
-### Example of Placeholder Usage
-
-Hereโ€™s a simplified example to illustrate how placeholders are typically used in the system:
-
-1. **Task Creation**: A new `SessionTask` is instantiated, and its placeholder is added to the web page.
-   ```kt
-   val task = appInterface.newTask()
-   val placeholderHtml = task.placeholder
-   ```
-
-2. **Task Execution**: The task performs its operations, during which it may use methods like `add`, `error`,
-   or `complete` to update its output.
-   ```kt
-   task.add("Processing data...")
-   // Some processing happens here
-   task.complete("Processing complete.")
-   ```
-
-3. **Dynamic UI Update**: As the task updates its output, these changes are sent to the client's browser using WebSocket
-   messages. The JavaScript on the client side listens for these messages and updates the inner HTML of the `div` with
-   the corresponding `operationID`.
-
-4. **Final Output**: Once the task completes, the final message is displayed in the placeholder, and no further updates
-   occur unless a new task is linked or started.
-
-### Conclusion
-
-The placeholder mechanism in `SessionTask` is a powerful feature that supports dynamic and real-time updates to the web
-UI without requiring page refreshes. It is especially useful in applications that involve complex or long-running tasks,
-providing users with immediate feedback and enhancing the interactivity of the application.
\ No newline at end of file
diff --git a/docs/ui/ui_protocol.md b/docs/ui/ui_protocol.md
deleted file mode 100644
index 4bb1bb4a..00000000
--- a/docs/ui/ui_protocol.md
+++ /dev/null
@@ -1,170 +0,0 @@
-# Developer Guide for the Skyenet UI System
-
-## Overview
-
-The Skyenet UI system is designed to facilitate real-time, interactive web applications using WebSockets. It provides a
-framework for managing WebSocket connections, handling user interactions, and displaying dynamic content. The core
-components of this system
-include `SocketManagerBase`, `ApplicationInterface`, `SessionTask`, `TabbedDisplay`, `Retryable`, `AgentPatterns`,
-and `Discussable`.
-
-## Components
-
-### 1. SocketManagerBase
-
-`SocketManagerBase` is an abstract class that manages WebSocket connections, message queuing, and message broadcasting.
-
-#### Key Methods:
-
-- **addSocket(socket: ChatSocket, session: org.eclipse.jetty.websocket.api.Session)**: Adds a new WebSocket connection.
-- **removeSocket(socket: ChatSocket)**: Removes an existing WebSocket connection.
-- **send(out: String)**: Queues a message for sending.
-- **getReplay(): List**: Retrieves a list of messages for replay.
-- **onWebSocketText(socket: ChatSocket, message: String)**: Handles incoming WebSocket messages.
-- **hrefLink(linkText: String, classname: String = "href-link", id: String? = null, handler: Consumer)**: Creates
-  an HTML link that triggers a handler when clicked.
-- **textInput(handler: Consumer)**: Creates an HTML text input form that triggers a handler when submitted.
-
-#### Example Usage:
-
-```kotlin
-class MySocketManager(
-    session: Session,
-    dataStorage: StorageInterface?,
-    owner: User?,
-    applicationClass: Class<*>
-) : SocketManagerBase(session, dataStorage, owner, applicationClass) {
-
-    override fun onRun(userMessage: String, socket: ChatSocket) {
-        // Handle user message
-    }
-}
-```
-
-### 2. ApplicationInterface
-
-`ApplicationInterface` provides methods to create interactive HTML elements and manage tasks.
-
-#### Key Methods:
-
-- **hrefLink(linkText: String, classname: String = "href-link", id: String? = null, handler: Consumer)**: Creates
-  an HTML link.
-- **textInput(handler: Consumer)**: Creates an HTML text input form.
-- **newTask(root: Boolean = true): SessionTask**: Creates a new task for displaying progress.
-
-#### Example Usage:
-
-```kotlin
-val appInterface = ApplicationInterface(socketManager)
-val linkHtml = appInterface.hrefLink("Click me", handler = Consumer { println("Link clicked") })
-val inputHtml = appInterface.textInput(handler = Consumer { input -> println("Input received: $input") })
-```
-
-### 3. SessionTask
-
-`SessionTask` represents a task that can display progress and messages.
-
-#### Key Methods:
-
-- **add(message: String, showSpinner: Boolean = true, tag: String = "div", className: String = "response-message")**:
-  Adds a message to the task output.
-- **hideable(ui: ApplicationInterface?, message: String, showSpinner: Boolean = true, tag: String = "div", className:
-  String = "response-message")**: Adds a hideable message.
-- **echo(message: String, showSpinner: Boolean = true, tag: String = "div")**: Echos a user message.
-- **header(message: String, showSpinner: Boolean = true, tag: String = "div", classname: String = "response-header")**:
-  Adds a header message.
-- **verbose(message: String, showSpinner: Boolean = true, tag: String = "pre")**: Adds a verbose message.
-- **error(ui: ApplicationInterface?, e: Throwable, showSpinner: Boolean = false, tag: String = "div")**: Displays an
-  error message.
-- **complete(message: String = "", tag: String = "div", className: String = "response-message")**: Completes the task
-  and hides the spinner.
-- **image(image: BufferedImage)**: Displays an image.
-
-#### Example Usage:
-
-```kotlin
-val task = appInterface.newTask()
-task.add("Processing started")
-task.complete("Processing complete")
-```
-
-### 4. TabbedDisplay
-
-`TabbedDisplay` manages a tabbed interface for displaying content.
-
-#### Key Methods:
-
-- **render()**: Renders the tabbed interface as HTML.
-- **clear()**: Clears all tabs.
-- **update()**: Updates the content of the tabbed interface.
-- **operator fun set(name: String, content: String)**: Sets the content of a tab by name.
-
-#### Example Usage:
-
-```kotlin
-val tabbedDisplay = TabbedDisplay(task)
-tabbedDisplay["Tab 1"] = "Content for Tab 1"
-tabbedDisplay.update()
-```
-
-### 5. Retryable
-
-`Retryable` extends `TabbedDisplay` to add a retry mechanism for tasks.
-
-#### Key Methods:
-
-- **renderTabButtons()**: Renders the tab buttons with a retry link.
-
-#### Example Usage:
-
-```kotlin
-val retryable = Retryable(appInterface, task) { content ->
-    // Process content
-    "Processed content"
-}
-```
-
-### 6. AgentPatterns
-
-`AgentPatterns` provides utility functions for displaying content in tabs.
-
-#### Key Methods:
-
-- **displayMapInTabs(map: Map, ui: ApplicationInterface? = null, split: Boolean = map.entries.map {
-  it.value.length + it.key.length }.sum() > 10000)**: Displays a map of content in tabs.
-
-#### Example Usage:
-
-```kotlin
-val contentMap = mapOf("Tab 1" to "Content 1", "Tab 2" to "Content 2")
-val tabsHtml = AgentPatterns.displayMapInTabs(contentMap, appInterface)
-```
-
-### 7. Discussable
-
-`Discussable` facilitates interactive discussions with users, allowing for feedback and revisions.
-
-#### Key Methods:
-
-- **call(): T**: Starts the discussion and returns the final result.
-
-#### Example Usage:
-
-```kotlin
-val discussable = Discussable(
-    task = task,
-    userMessage = { "User message" },
-    initialResponse = { message -> "Initial response" },
-    outputFn = { response -> "Output: $response" },
-    ui = appInterface,
-    reviseResponse = { history -> "Revised response" },
-    heading = "Discussion"
-)
-val result = discussable.call()
-```
-
-## Conclusion
-
-The Skyenet UI system provides a robust framework for building interactive web applications with WebSocket support. By
-leveraging the provided components, developers can easily manage WebSocket connections, handle user interactions, and
-display dynamic content in a structured and efficient manner.
\ No newline at end of file
diff --git a/docs/ui_code_guide.md b/docs/ui_code_guide.md
deleted file mode 100644
index ec107c78..00000000
--- a/docs/ui_code_guide.md
+++ /dev/null
@@ -1,241 +0,0 @@
-1. Server Architecture Overview
-
-The architecture is based on an ApplicationServer class, which serves as the foundation for various specialized applications. Key components include:
-
-* ApplicationServer: The base class for all applications
-* Session: Represents a user session
-* User: Represents a user of the application
-* ApplicationInterface: Provides methods for interacting with the UI
-* SessionTask: Represents a task within a session
-* SocketManager: Manages WebSocket connections for real-time communication
-* StorageInterface: Handles data persistence
-
-2. Creating a New Application
-
-To create a new application:
-
-a) Extend the ApplicationServer class:
-
-```kotlin
-class MyApp(
-    applicationName: String = "My Application",
-    path: String = "/myapp"
-) : ApplicationServer(
-    applicationName = applicationName,
-    path = path,
-    showMenubar = true
-)
-```
-
-b1) Override the userMessage method to handle user input:
-
-```kotlin
-override fun userMessage(
-    session: Session,
-    user: User?,
-    userMessage: String,
-    ui: ApplicationInterface,
-    api: API
-) {
-    // Handle user input and generate response
-}
-```
-
-b2) Somtimes, when a user prompt isn't needed, we can implement the newSession method directly:
-
-```kotlin
-override fun newSession(user: User?, session: Session): SocketManager {
-   val newSession = super.newSession(user, session)
-   // Perform additional logic, e.g. start an asynchronous task
-   return newSession
-}
-```
-
-3. UI Components and Interactions
-
-a) Creating Tasks:
-Use ui.newTask() to create new tasks for organizing content:
-
-```kotlin
-val task = ui.newTask()
-```
-
-b) Adding Content:
-Add content to tasks using various methods:
-
-```kotlin
-task.add(MarkdownUtil.renderMarkdown("# My Header", ui = ui))
-task.echo("Simple text output")
-task.complete("Task completed message")
-```
-
-c) Creating Tabs:
-Use TabbedDisplay for creating tabbed interfaces:
-
-```kotlin
-val tabDisplay = object : TabbedDisplay(task) {
-    override fun renderTabButtons(): String {
-        // Define tab buttons
-    }
-}
-
-// Add content to tabs
-val tabTask = ui.newTask(false).apply { tabDisplay["Tab 1"] = it.placeholder }
-```
-
-d) Handling Errors:
-Use task.error() to display error messages:
-
-```kotlin
-task.error(ui, Exception("An error occurred"))
-```
-
-e) Creating Links:
-Generate clickable links:
-
-```kotlin
-ui.hrefLink("Click me", "href-link custom-class") {
-    // Action when clicked
-}
-```
-
-4. Asynchronous Operations
-
-Use threads or the application's thread pool for long-running operations:
-
-```kotlin
-ui.socketManager?.pool!!.submit {
-    // Perform long-running task
-}
-```
-
-5. File Handling
-
-a) Reading Files:
-
-```kotlin
-val fileContent = File(path).readText(Charsets.UTF_8)
-```
-
-b) Saving Files:
-
-```kotlin
-task.saveFile(filename, content.toByteArray(Charsets.UTF_8))
-```
-
-c) Accessing Application Data:
-
-```kotlin
-val dataStorage: StorageInterface = dataStorageFactory(dataStorageRoot)
-dataStorage.setJson(
-    user, session, "info.json", mapOf(
-        "session" to session.toString(),
-        "application" to applicationName,
-        "path" to path,
-        "startTime" to System.currentTimeMillis(),
-    )
-)
-```
-
-6. API Integration
-
-Most applications use an API client for external communications:
-
-```kotlin
-val api: API = // ... initialize API client
-    (api as ChatClient).budget = settings.budget ?: 2.00
-```
-
-7. Settings Management
-
-Create a Settings data class and use it to manage application settings:
-
-```kotlin
-data class Settings(
-    val budget: Double? = 2.00,
-    // Other settings...
-)
-
-override val settingsClass: Class<*> get() = Settings::class.java
-
-@Suppress("UNCHECKED_CAST")
-override fun  initSettings(session: Session): T? = Settings() as T
-```
-
-To retrieve settings:
-
-```kotlin
-fun  getSettings(
-    session: Session,
-    userId: User?,
-    clazz: Class = settingsClass as Class
-): T? {
-    val settingsFile = getSettingsFile(session, userId)
-    // ... (implementation details)
-}
-```
-
-8. Advanced Features
-
-a) Discussable Pattern:
-Use the Discussable class for creating interactive, revisable content:
-
-```kotlin
-Discussable(
-    task = task,
-    userMessage = { userMessage },
-    initialResponse = { /* Generate initial response */ },
-    outputFn = { /* Render output */ },
-    ui = ui,
-    reviseResponse = { /* Handle revisions */ }
-).call()
-```
-
-b) Actor System:
-Implement an ActorSystem for complex, multi-step processes:
-
-```kotlin
-class MyAgent(/* parameters */) : ActorSystem(/* parameters */) {
-    enum class ActorTypes {
-        // Define actor types
-    }
-
-    // Implement actor logic
-}
-```
-
-9. Security and Authorization
-   Implement authorization checks using filters:
-
-```kotlin
-webAppContext.addFilter(
-    FilterHolder { request, response, chain ->
-        val user = authenticationManager.getUser((request as HttpServletRequest).getCookie())
-        val canRead = authorizationManager.isAuthorized(
-            applicationClass = this@ApplicationServer.javaClass,
-            user = user,
-            operationType = OperationType.Read
-        )
-        if (canRead) {
-            chain?.doFilter(request, response)
-        } else {
-            response?.writer?.write("Access Denied")
-            (response as HttpServletResponse?)?.status = HttpServletResponse.SC_FORBIDDEN
-        }
-    }, "/*", null
-)
-```
-
-10. Best Practices
-
-* Use meaningful names for classes, methods, and variables
-* Implement error handling and logging
-* Break down complex functionality into smaller, manageable functions
-* Use Kotlin's null safety features to prevent null pointer exceptions
-* Leverage Kotlin's coroutines for managing asynchronous operations when appropriate
-* Implement proper security measures, including authentication and authorization
-* Use the StorageInterface for persistent data storage
-* Leverage the SocketManager for real-time communication with clients
-
-By following this guide and studying the provided examples, you can create robust and interactive UI applications using this server architecture. Remember to adapt the concepts to
-your specific use case and requirements.
\ No newline at end of file
diff --git a/docs/webui_documentation.md b/docs/webui_documentation.md
deleted file mode 100644
index 424166e8..00000000
--- a/docs/webui_documentation.md
+++ /dev/null
@@ -1,6282 +0,0 @@
-# Skyenet Project Documentation
-
-## Overview
-
-Skyenet is a comprehensive framework designed to facilitate the development, integration, and deployment of AI-driven
-applications. It leverages OpenAI's API to provide a suite of tools and services that enable developers to create
-sophisticated AI models, interpreters for various programming languages, and utilities for web scraping, audio
-processing, and more. The project aims to streamline the process of building intelligent applications by offering a
-modular, extensible architecture.
-
-## Key Components
-
-### Core Actors
-
-Skyenet's architecture is built around the concept of "actors" - modular components that interact with OpenAI's API to
-perform specific tasks. These actors can generate responses based on input, execute code, process images, and handle
-natural language understanding tasks.
-
-- **BaseActor**: Serves as the abstract base for all actors, defining common properties and methods.
-- **CodingActor**: Specializes in translating natural language instructions into executable code.
-- **ImageActor**: Focuses on generating images based on textual prompts.
-- **TextToSpeechActor**: Converts text to speech, facilitating auditory interfaces.
-
-### Actor System
-
-The `ActorSystem` class manages the lifecycle and interactions of various actors within the system. It supports
-session-based context, user-specific data storage, and dynamic behavior modification through interceptors.
-
-### Platform Integration
-
-Skyenet provides integration with cloud platforms and utilities for enhanced functionality:
-
-- **AWS Platform**: Facilitates interactions with AWS services, including S3 for storage and KMS for encryption.
-- **OutputInterceptor**: Captures and redirects standard output and error streams for logging or testing purposes.
-- **Selenium**: Offers an abstraction for web scraping and automation tasks using Selenium WebDriver.
-
-### Utility Classes
-
-A collection of utility classes and interfaces support various functionalities across the project:
-
-- **ClasspathRelationships**: Analyzes and manages relationships between classes within JAR files.
-- **FunctionWrapper**: Allows for interception and recording of function calls.
-- **LoggingInterceptor**: Captures logging events from specified loggers.
-- **RuleTreeBuilder**: Generates Kotlin code expressions for matching and filtering strings.
-
-## Development Guide
-
-### Setting Up the Development Environment
-
-1. **Clone the Repository**: Start by cloning the Skyenet project repository to your local machine.
-2. **Install Dependencies**: Ensure you have Kotlin and Java SDKs installed. Additionally, install any required
-   dependencies, such as the OpenAI SDK and Selenium WebDriver.
-3. **Configure API Keys**: For components that interact with external services (e.g., OpenAI, AWS), configure the
-   necessary API keys and access credentials.
-
-### Extending Skyenet
-
-Skyenet's modular architecture makes it easy to extend and customize:
-
-- **Adding New Actors**: Implement the `BaseActor` interface to create new actors for specific tasks.
-- **Custom Interpreters**: Extend the `Interpreter` interface to support additional programming languages or execution
-  environments.
-- **Utility Enhancements**: Contribute new utility classes or enhance existing ones to support broader functionalities.
-
-### Testing
-
-Skyenet includes a comprehensive suite of tests to ensure the reliability and correctness of its components:
-
-- **Unit Tests**: Validate the functionality of individual classes and methods.
-- **Integration Tests**: Test the interactions between different components and external services.
-- **Performance Tests**: Assess the efficiency and scalability of the framework under various conditions.
-
-## Deployment
-
-Skyenet applications can be deployed on various platforms, including cloud services and on-premise servers. The project
-documentation provides guidelines for deployment, including containerization with Docker and orchestration with
-Kubernetes for scalable, distributed applications.
-
-## Contributing
-
-Contributions to Skyenet are welcome! Whether it's adding new features, fixing bugs, or improving documentation, your
-contributions help make Skyenet better for everyone. Please refer to the project's contribution guidelines for more
-information on how to contribute effectively.
-
-## Conclusion
-
-Skyenet aims to be a versatile and powerful framework for building AI-driven applications. By providing a rich set of
-tools and services, it enables developers to harness the power of AI more efficiently and effectively. Whether you're
-building web applications, processing multimedia, or developing custom AI models, Skyenet offers the building blocks you
-need to bring your projects to life.
-
-
-
-* [Skyenet Project Documentation](#skyenet-project-documentation)
-    * [Overview](#overview)
-    * [Key Components](#key-components)
-        * [Core Actors](#core-actors)
-        * [Actor System](#actor-system)
-        * [Platform Integration](#platform-integration)
-        * [Utility Classes](#utility-classes)
-    * [Development Guide](#development-guide)
-        * [Setting Up the Development Environment](#setting-up-the-development-environment)
-        * [Extending Skyenet](#extending-skyenet)
-        * [Testing](#testing)
-    * [Deployment](#deployment)
-    * [Contributing](#contributing)
-    * [Conclusion](#conclusion)
-* [kotlin\com\simiacryptus\skyenet\apps\coding\ShellToolAgent.kt](#kotlincomsimiacryptusskyenetappscodingshelltoolagentkt)
-  * [ShellToolAgent Class Documentation](#shelltoolagent-class-documentation)
-  * [Key Features:](#key-features)
-  * [Core Components:](#core-components)
-  * [Usage:](#usage)
-  * [Key Methods:](#key-methods)
-  * [Extensibility:](#extensibility)
-  * [Error Handling:](#error-handling)
-  * [Conclusion](#conclusion-1)
-* [kotlin\com\simiacryptus\skyenet\apps\coding\CodingAgent.kt](#kotlincomsimiacryptusskyenetappscodingcodingagentkt)
-    * [CodingAgent Class Documentation](#codingagent-class-documentation)
-        * [Overview](#overview-1)
-        * [Key Components](#key-components-1)
-        * [Constructor Parameters](#constructor-parameters)
-        * [Methods](#methods)
-            * [start(userMessage: String)](#startusermessage-string)
-            * [displayCode(task: SessionTask, codeRequest: CodingActor.CodeRequest)](#displaycodetask-sessiontask-coderequest-codingactorcoderequest)
-            * [displayFeedback(task: SessionTask, request: CodingActor.CodeRequest, response: CodeResult)](#displayfeedbacktask-sessiontask-request-codingactorcoderequest-response-coderesult)
-            * [execute(task: SessionTask, response: CodeResult, request: CodingActor.CodeRequest)](#executetask-sessiontask-response-coderesult-request-codingactorcoderequest)
-        * [Usage Example](#usage-example)
-        * [Conclusion](#conclusion-2)
-* [kotlin\com\github\simiacryptus\aicoder\util\SimpleDiffUtil.kt](#kotlincomgithubsimiacryptusaicoderutilsimplediffutilkt)
-    * [SimpleDiffUtil and SocketManagerBase Extensions Documentation](#simplediffutil-and-socketmanagerbase-extensions-documentation)
-        * [Overview](#overview-2)
-        * [SimpleDiffUtil](#simplediffutil)
-            * [Functionality](#functionality)
-            * [Methods](#methods-1)
-            * [Internal Mechanics](#internal-mechanics)
-        * [SocketManagerBase Extension Functions](#socketmanagerbase-extension-functions)
-            * [Functionality](#functionality-1)
-            * [Methods](#methods-2)
-            * [Usage Scenarios](#usage-scenarios)
-        * [Conclusion](#conclusion-3)
-* [kotlin\com\simiacryptus\skyenet\apps\general\WebDevApp.kt](#kotlincomsimiacryptusskyenetappsgeneralwebdevappkt)
-    * [Web Development Assistant Application Documentation](#web-development-assistant-application-documentation)
-        * [Overview](#overview-3)
-        * [Key Components](#key-components-2)
-            * [WebDevApp Class](#webdevapp-class)
-                * [Key Methods](#key-methods-1)
-            * [WebDevAgent Class](#webdevagent-class)
-                * [Key Methods](#key-methods-2)
-            * [Actors](#actors)
-        * [Usage](#usage-1)
-        * [Example](#example)
-        * [Conclusion](#conclusion-4)
-* [kotlin\com\simiacryptus\skyenet\apps\coding\ToolAgent.kt](#kotlincomsimiacryptusskyenetappscodingtoolagentkt)
-    * [ToolAgent Class Documentation](#toolagent-class-documentation)
-        * [Overview](#overview-4)
-        * [Key Components](#key-components-3)
-            * [Constructor Parameters](#constructor-parameters-1)
-            * [Methods](#methods-3)
-                * [`displayFeedback`](#displayfeedback)
-                * [`createToolButton`](#createtoolbutton)
-                * [`openAPIParsedActor`](#openapiparsedactor)
-                * [`servletActor`](#servletactor)
-                * [`schemaActor`](#schemaactor)
-                * [`displayCodeFeedback`](#displaycodefeedback)
-                * [`buildTestPage`](#buildtestpage)
-                * [`getInterpreterString`](#getinterpreterstring)
-            * [Utility Methods](#utility-methods)
-        * [Usage](#usage-2)
-        * [Example](#example-1)
-* [kotlin\com\simiacryptus\skyenet\AgentPatterns.kt](#kotlincomsimiacryptusskyenetagentpatternskt)
-    * [AgentPatterns Module Documentation](#agentpatterns-module-documentation)
-        * [Overview](#overview-5)
-            * [1. retryable Function](#1-retryable-function)
-                * [Parameters:](#parameters)
-                * [Returns:](#returns)
-                * [Usage Example:](#usage-example-1)
-            * [2. iterate Function](#2-iterate-function)
-                * [Parameters:](#parameters-1)
-                * [Returns:](#returns-1)
-                * [Usage Example:](#usage-example-2)
-            * [3. iterate Function (Overloaded Version)](#3-iterate-function-overloaded-version)
-                * [Parameters:](#parameters-2)
-                * [Returns:](#returns-2)
-                * [Usage Example:](#usage-example-3)
-        * [Conclusion](#conclusion-5)
-* [kotlin\com\simiacryptus\skyenet\interpreter\ProcessInterpreter.kt](#kotlincomsimiacryptusskyenetinterpreterprocessinterpreterkt)
-    * [ProcessInterpreter Class Documentation](#processinterpreter-class-documentation)
-        * [Constructor](#constructor)
-            * [ProcessInterpreter](#processinterpreter)
-        * [Properties](#properties)
-            * [command](#command)
-        * [Methods](#methods-4)
-            * [getLanguage](#getlanguage)
-            * [getSymbols](#getsymbols)
-            * [validate](#validate)
-            * [run](#run)
-        * [Usage Example](#usage-example-4)
-        * [Notes](#notes)
-* [kotlin\com\simiacryptus\skyenet\webui\application\ApplicationInterface.kt](#kotlincomsimiacryptusskyenetwebuiapplicationapplicationinterfacekt)
-    * [ApplicationInterface Documentation](#applicationinterface-documentation)
-        * [Constructor](#constructor-1)
-            * [ApplicationInterface(SocketManagerBase socketManager)](#applicationinterfacesocketmanagerbase-socketmanager)
-        * [Methods](#methods-5)
-            * [hrefLink](#hreflink)
-            * [textInput](#textinput)
-            * [newTask](#newtask)
-        * [Companion Object Methods](#companion-object-methods)
-            * [oneAtATime](#oneatatime)
-        * [Usage Example](#usage-example-5)
-* [kotlin\com\simiacryptus\skyenet\webui\application\ApplicationDirectory.kt](#kotlincomsimiacryptusskyenetwebuiapplicationapplicationdirectorykt)
-    * [Developer Documentation for `ApplicationDirectory` Class](#developer-documentation-for-applicationdirectory-class)
-        * [Overview](#overview-6)
-        * [Key Components](#key-components-4)
-            * [Properties](#properties-1)
-            * [Inner Classes](#inner-classes)
-            * [Methods](#methods-6)
-                * [Abstract and Open Methods](#abstract-and-open-methods)
-                * [Protected Methods](#protected-methods)
-            * [Companion Object](#companion-object)
-        * [Usage](#usage-3)
-        * [Conclusion](#conclusion-6)
-* [kotlin\com\simiacryptus\skyenet\webui\application\ApplicationServer.kt](#kotlincomsimiacryptusskyenetwebuiapplicationapplicationserverkt)
-    * [Developer Documentation for ApplicationServer](#developer-documentation-for-applicationserver)
-        * [Overview](#overview-7)
-        * [Key Components](#key-components-5)
-            * [Fields and Properties](#fields-and-properties)
-            * [Servlets and Filters](#servlets-and-filters)
-            * [Session Management](#session-management)
-            * [Settings Management](#settings-management)
-            * [Utility Methods](#utility-methods-1)
-        * [Usage](#usage-4)
-            * [Example](#example-2)
-        * [Conclusion](#conclusion-7)
-* [kotlin\com\simiacryptus\skyenet\webui\application\ApplicationSocketManager.kt](#kotlincomsimiacryptusskyenetwebuiapplicationapplicationsocketmanagerkt)
-    * [ApplicationSocketManager Class Documentation](#applicationsocketmanager-class-documentation)
-        * [Overview](#overview-8)
-        * [Constructor](#constructor-2)
-        * [Key Methods and Properties](#key-methods-and-properties)
-            * [onRun](#onrun)
-            * [userMessage](#usermessage)
-            * [applicationInterface](#applicationinterface)
-        * [Companion Object](#companion-object-1)
-            * [spinner](#spinner)
-        * [Usage](#usage-5)
-        * [Conclusion](#conclusion-8)
-* [kotlin\com\simiacryptus\skyenet\webui\chat\ChatServer.kt](#kotlincomsimiacryptusskyenetwebuichatchatserverkt)
-    * [ChatServer Class Documentation](#chatserver-class-documentation)
-        * [Overview](#overview-9)
-        * [Key Components](#key-components-6)
-            * [Properties](#properties-2)
-            * [Inner Classes](#inner-classes-1)
-                * [WebSocketHandler](#websockethandler)
-                    * [Key Methods](#key-methods-3)
-            * [Abstract Methods](#abstract-methods)
-            * [Open Properties](#open-properties)
-            * [Methods](#methods-7)
-        * [Usage](#usage-6)
-        * [Companion Object](#companion-object-2)
-            * [Properties](#properties-3)
-            * [Extension Functions](#extension-functions)
-        * [Conclusion](#conclusion-9)
-* [kotlin\com\simiacryptus\skyenet\webui\chat\ChatSocket.kt](#kotlincomsimiacryptusskyenetwebuichatchatsocketkt)
-    * [ChatSocket Class Documentation](#chatsocket-class-documentation)
-        * [Dependencies](#dependencies)
-        * [Class Overview](#class-overview)
-        * [Usage](#usage-7)
-            * [Example](#example-3)
-        * [Conclusion](#conclusion-10)
-* [kotlin\com\simiacryptus\skyenet\webui\chat\ChatSocketManager.kt](#kotlincomsimiacryptusskyenetwebuichatchatsocketmanagerkt)
-    * [ChatSocketManager Class Documentation](#chatsocketmanager-class-documentation)
-        * [Constructor](#constructor-3)
-            * [Parameters](#parameters-3)
-        * [Methods](#methods-8)
-            * [onRun](#onrun-1)
-                * [Parameters](#parameters-4)
-            * [renderResponse](#renderresponse)
-                * [Parameters](#parameters-5)
-                * [Returns](#returns-3)
-            * [onResponse](#onresponse)
-                * [Parameters](#parameters-6)
-        * [Properties](#properties-4)
-        * [Companion Object](#companion-object-3)
-        * [Usage](#usage-8)
-        * [Error Handling](#error-handling-1)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\ApiKeyServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletapikeyservletkt)
-    * [API Key Servlet Documentation](#api-key-servlet-documentation)
-        * [Overview](#overview-10)
-        * [Data Model](#data-model)
-            * [ApiKeyRecord](#apikeyrecord)
-        * [Supported Operations](#supported-operations)
-            * [Handling GET Requests](#handling-get-requests)
-            * [Handling POST Requests](#handling-post-requests)
-        * [Interacting with the Servlet](#interacting-with-the-servlet)
-            * [Editing an API Key Record](#editing-an-api-key-record)
-            * [Deleting an API Key Record](#deleting-an-api-key-record)
-            * [Creating a New API Key Record](#creating-a-new-api-key-record)
-            * [Inviting a User to Use an API Key](#inviting-a-user-to-use-an-api-key)
-        * [Utility Methods](#utility-methods-2)
-        * [Storage](#storage)
-        * [Conclusion](#conclusion-11)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\AppInfoServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletappinfoservletkt)
-    * [AppInfoServlet Class Documentation](#appinfoservlet-class-documentation)
-        * [Generics](#generics)
-        * [Constructor](#constructor-4)
-        * [Methods](#methods-9)
-            * [`doGet(HttpServletRequest req, HttpServletResponse resp)`](#dogethttpservletrequest-req-httpservletresponse-resp)
-        * [Usage](#usage-9)
-        * [Dependencies](#dependencies-1)
-        * [Conclusion](#conclusion-12)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\CancelThreadsServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletcancelthreadsservletkt)
-    * [CancelThreadsServlet Documentation](#cancelthreadsservlet-documentation)
-        * [Class Overview](#class-overview-1)
-        * [Constructor](#constructor-5)
-            * [`CancelThreadsServlet(ApplicationServer server)`](#cancelthreadsservletapplicationserver-server)
-        * [Methods](#methods-10)
-            * [`doGet(HttpServletRequest req, HttpServletResponse resp)`](#dogethttpservletrequest-req-httpservletresponse-resp-1)
-            * [`doPost(HttpServletRequest req, HttpServletResponse resp)`](#doposthttpservletrequest-req-httpservletresponse-resp)
-        * [Usage](#usage-10)
-        * [Security Considerations](#security-considerations)
-        * [Error Handling](#error-handling-2)
-        * [Conclusion](#conclusion-13)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\CorsFilter.kt](#kotlincomsimiacryptusskyenetwebuiservletcorsfilterkt)
-    * [CorsFilter Class Documentation](#corsfilter-class-documentation)
-        * [Features](#features)
-        * [Usage](#usage-11)
-            * [Configuration](#configuration)
-            * [Methods](#methods-11)
-            * [CORS Headers Added](#cors-headers-added)
-        * [Example](#example-4)
-        * [Logging](#logging)
-        * [Conclusion](#conclusion-14)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\FileServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletfileservletkt)
-    * [FileServlet Class Documentation](#fileservlet-class-documentation)
-        * [Constructor](#constructor-6)
-        * [Methods](#methods-12)
-            * [Public Methods](#public-methods)
-            * [Private Methods](#private-methods)
-        * [Utility Methods](#utility-methods-3)
-        * [Companion Object](#companion-object-4)
-        * [Cache Configuration](#cache-configuration)
-        * [Exception Handling](#exception-handling)
-        * [Usage](#usage-12)
-        * [Example](#example-5)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\DeleteSessionServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletdeletesessionservletkt)
-    * [DeleteSessionServlet Documentation](#deletesessionservlet-documentation)
-        * [Overview](#overview-11)
-        * [Usage](#usage-13)
-            * [Initialization](#initialization)
-            * [Handling GET Requests](#handling-get-requests-1)
-            * [Handling POST Requests](#handling-post-requests-1)
-        * [Security Considerations](#security-considerations-1)
-        * [Dependencies](#dependencies-2)
-        * [Conclusion](#conclusion-15)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\OAuthBase.kt](#kotlincomsimiacryptusskyenetwebuiservletoauthbasekt)
-    * [OAuthBase Class Documentation](#oauthbase-class-documentation)
-        * [Overview](#overview-12)
-        * [Usage](#usage-14)
-            * [Parameters](#parameters-7)
-            * [Methods](#methods-13)
-                * [`configure`](#configure)
-                    * [Parameters:](#parameters-8)
-                    * [Returns:](#returns-4)
-        * [Example](#example-6)
-        * [Conclusion](#conclusion-16)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\LogoutServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletlogoutservletkt)
-    * [LogoutServlet Documentation](#logoutservlet-documentation)
-        * [Overview](#overview-13)
-        * [Usage](#usage-15)
-            * [Key Methods](#key-methods-4)
-            * [Logout Process](#logout-process)
-            * [Error Handling](#error-handling-3)
-        * [Example Deployment Descriptor Configuration](#example-deployment-descriptor-configuration)
-        * [Conclusion](#conclusion-17)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\OAuthGoogle.kt](#kotlincomsimiacryptusskyenetwebuiservletoauthgooglekt)
-    * [OAuthGoogle Class Documentation](#oauthgoogle-class-documentation)
-        * [Overview](#overview-14)
-        * [Key Components](#key-components-7)
-        * [Usage](#usage-16)
-        * [Configuration Parameters](#configuration-parameters)
-        * [Example](#example-7)
-        * [Dependencies](#dependencies-3)
-        * [Important Notes](#important-notes)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\NewSessionServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletnewsessionservletkt)
-    * [NewSessionServlet Documentation](#newsessionservlet-documentation)
-        * [Usage](#usage-17)
-        * [Implementation Details](#implementation-details)
-            * [Import Statements](#import-statements)
-            * [Class Definition](#class-definition)
-            * [doGet Method](#doget-method)
-                * [Parameters](#parameters-9)
-                * [Implementation](#implementation)
-        * [Example Usage](#example-usage)
-        * [Conclusion](#conclusion-18)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\SessionIdFilter.kt](#kotlincomsimiacryptusskyenetwebuiservletsessionidfilterkt)
-    * [SessionIdFilter Class Documentation](#sessionidfilter-class-documentation)
-        * [Package](#package)
-        * [Imports](#imports)
-        * [Constructor](#constructor-7)
-        * [Methods](#methods-14)
-            * [init](#init)
-            * [doFilter](#dofilter)
-            * [destroy](#destroy)
-        * [Usage Example](#usage-example-6)
-        * [Conclusion](#conclusion-19)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\ProxyHttpServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletproxyhttpservletkt)
-    * [ProxyHttpServlet Developer Documentation](#proxyhttpservlet-developer-documentation)
-        * [Overview](#overview-15)
-        * [Setup](#setup)
-        * [Key Components](#key-components-8)
-            * [Fields](#fields)
-            * [Methods](#methods-15)
-                * [`service(HttpServletRequest req, HttpServletResponse resp)`](#servicehttpservletrequest-req-httpservletresponse-resp)
-                * [`getProxyRequest(HttpServletRequest req)`](#getproxyrequesthttpservletrequest-req)
-                * [`onResponse(...)`](#onresponse-1)
-                * [`onRequest(HttpServletRequest req, ByteArray bytes)`](#onrequesthttpservletrequest-req-bytearray-bytes)
-            * [Usage Example](#usage-example-7)
-        * [Extending `ProxyHttpServlet`](#extending-proxyhttpservlet)
-        * [Conclusion](#conclusion-20)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\SessionSettingsServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletsessionsettingsservletkt)
-    * [SessionSettingsServlet Documentation](#sessionsettingsservlet-documentation)
-        * [Overview](#overview-16)
-        * [Dependencies](#dependencies-4)
-        * [Constructor](#constructor-8)
-        * [Fields](#fields-1)
-        * [HTTP Methods](#http-methods)
-            * [doGet(HttpServletRequest req, HttpServletResponse resp)](#dogethttpservletrequest-req-httpservletresponse-resp-2)
-                * [Process Flow](#process-flow)
-            * [doPost(HttpServletRequest req, HttpServletResponse resp)](#doposthttpservletrequest-req-httpservletresponse-resp-1)
-                * [Process Flow](#process-flow-1)
-        * [Usage Example](#usage-example-8)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\SessionListServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletsessionlistservletkt)
-    * [SessionListServlet Class Documentation](#sessionlistservlet-class-documentation)
-        * [Dependencies](#dependencies-5)
-        * [Constructor Parameters](#constructor-parameters-2)
-        * [Key Methods](#key-methods-5)
-            * [doGet(HttpServletRequest req, HttpServletResponse resp)](#dogethttpservletrequest-req-httpservletresponse-resp-3)
-                * [Parameters](#parameters-10)
-                * [Functionality](#functionality-2)
-        * [Usage Example](#usage-example-9)
-        * [Notes](#notes-1)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\SessionShareServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletsessionshareservletkt)
-    * [SessionShareServlet Documentation](#sessionshareservlet-documentation)
-        * [Overview](#overview-17)
-        * [Key Components](#key-components-9)
-            * [Dependencies](#dependencies-6)
-            * [Main Methods](#main-methods)
-                * [doGet(HttpServletRequest req, HttpServletResponse resp)](#dogethttpservletrequest-req-httpservletresponse-resp-4)
-            * [Helper Methods](#helper-methods)
-            * [Usage](#usage-18)
-            * [Error Handling](#error-handling-4)
-            * [Security Considerations](#security-considerations-2)
-        * [Conclusion](#conclusion-21)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\SessionThreadsServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletsessionthreadsservletkt)
-    * [SessionThreadsServlet Documentation](#sessionthreadsservlet-documentation)
-        * [Overview](#overview-18)
-        * [Usage](#usage-19)
-            * [Request Parameters](#request-parameters)
-            * [Response](#response)
-            * [Example Request](#example-request)
-        * [Implementation Details](#implementation-details-1)
-            * [Key Components](#key-components-10)
-            * [HTML Response Generation](#html-response-generation)
-            * [Error Handling](#error-handling-5)
-        * [Security Considerations](#security-considerations-3)
-        * [Conclusion](#conclusion-22)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\ToolServlet.kt](#kotlincomsimiacryptusskyenetwebuiservlettoolservletkt)
-    * [ToolServlet Developer Documentation](#toolservlet-developer-documentation)
-        * [Overview](#overview-19)
-        * [Key Components](#key-components-11)
-            * [Tool Data Class](#tool-data-class)
-            * [Main Methods](#main-methods-1)
-                * [doGet(HttpServletRequest?, HttpServletResponse?)](#dogethttpservletrequest-httpservletresponse)
-                * [doPost(HttpServletRequest?, HttpServletResponse?)](#doposthttpservletrequest-httpservletresponse)
-                * [service(HttpServletRequest?, HttpServletResponse?)](#servicehttpservletrequest-httpservletresponse)
-            * [Utility Methods](#utility-methods-4)
-                * [indexPage(): String](#indexpage-string)
-                * [toolDetailsPage(tool: Tool): String](#tooldetailspagetool-tool-string)
-                * [serveEditPage(HttpServletRequest, HttpServletResponse, Tool)](#serveeditpagehttpservletrequest-httpservletresponse-tool)
-        * [Security](#security)
-        * [Extensibility](#extensibility-1)
-        * [Example Usage](#example-usage-1)
-        * [Conclusion](#conclusion-23)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\UserSettingsServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletusersettingsservletkt)
-    * [UserSettingsServlet Documentation](#usersettingsservlet-documentation)
-        * [Overview](#overview-20)
-        * [Functionality](#functionality-3)
-            * [doGet(HttpServletRequest req, HttpServletResponse resp)](#dogethttpservletrequest-req-httpservletresponse-resp-5)
-            * [doPost(HttpServletRequest req, HttpServletResponse resp)](#doposthttpservletrequest-req-httpservletresponse-resp-2)
-        * [Security Considerations](#security-considerations-4)
-        * [Usage](#usage-20)
-        * [Conclusion](#conclusion-24)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\WelcomeServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletwelcomeservletkt)
-    * [WelcomeServlet Class Documentation](#welcomeservlet-class-documentation)
-        * [Constructor](#constructor-9)
-        * [Methods](#methods-16)
-            * [doGet(HttpServletRequest req, HttpServletResponse resp)](#dogethttpservletrequest-req-httpservletresponse-resp-6)
-            * [doPost(HttpServletRequest req, HttpServletResponse resp)](#doposthttpservletrequest-req-httpservletresponse-resp-3)
-            * [homepage(User user)](#homepageuser-user)
-            * [appRow(ApplicationDirectory.ChildWebApp app, User user)](#approwapplicationdirectorychildwebapp-app-user-user)
-        * [Fields](#fields-2)
-        * [Usage](#usage-21)
-        * [Security](#security-1)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\UserInfoServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletuserinfoservletkt)
-    * [UserInfoServlet Documentation](#userinfoservlet-documentation)
-        * [Package](#package-1)
-        * [Dependencies](#dependencies-7)
-        * [Class Overview](#class-overview-2)
-            * [UserInfoServlet](#userinfoservlet)
-                * [Methods](#methods-17)
-        * [Usage](#usage-22)
-        * [Example Response](#example-response)
-        * [Security Considerations](#security-considerations-5)
-        * [Conclusion](#conclusion-25)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\UsageServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletusageservletkt)
-    * [UsageServlet Documentation](#usageservlet-documentation)
-        * [Overview](#overview-21)
-        * [Key Methods](#key-methods-6)
-            * [doGet(HttpServletRequest req, HttpServletResponse resp)](#dogethttpservletrequest-req-httpservletresponse-resp-7)
-            * [serve(HttpServletResponse resp, Map usage)](#servehttpservletresponse-resp-mapopenaimodel-apimodelusage-usage)
-        * [Usage Example](#usage-example-10)
-        * [Security Considerations](#security-considerations-6)
-        * [Dependencies](#dependencies-8)
-* [kotlin\com\simiacryptus\skyenet\webui\servlet\ZipServlet.kt](#kotlincomsimiacryptusskyenetwebuiservletzipservletkt)
-    * [ZipServlet Class Documentation](#zipservlet-class-documentation)
-        * [Dependencies](#dependencies-9)
-        * [Constructor](#constructor-10)
-        * [Public Methods](#public-methods-1)
-            * [`doGet(HttpServletRequest req, HttpServletResponse resp)`](#dogethttpservletrequest-req-httpservletresponse-resp-8)
-        * [Private Methods](#private-methods-1)
-            * [`write(basePath: File, file: File, zip: ZipOutputStream)`](#writebasepath-file-file-file-zip-zipoutputstream)
-        * [Usage Example](#usage-example-11)
-        * [Security Considerations](#security-considerations-7)
-* [kotlin\com\simiacryptus\skyenet\webui\session\SocketManager.kt](#kotlincomsimiacryptusskyenetwebuisessionsocketmanagerkt)
-    * [SocketManager Interface Documentation](#socketmanager-interface-documentation)
-        * [Interface Overview](#interface-overview)
-        * [Methods](#methods-18)
-            * [`removeSocket(socket: ChatSocket)`](#removesocketsocket-chatsocket)
-            * [`addSocket(socket: ChatSocket, session: Session)`](#addsocketsocket-chatsocket-session-session)
-            * [`getReplay(): List`](#getreplay-liststring)
-            * [`onWebSocketText(socket: ChatSocket, message: String)`](#onwebsockettextsocket-chatsocket-message-string)
-        * [Usage Example](#usage-example-12)
-        * [Conclusion](#conclusion-26)
-* [kotlin\com\simiacryptus\skyenet\webui\session\SocketManagerBase.kt](#kotlincomsimiacryptusskyenetwebuisessionsocketmanagerbasekt)
-    * [SocketManagerBase Class Documentation](#socketmanagerbase-class-documentation)
-        * [Overview](#overview-22)
-        * [Key Components](#key-components-12)
-            * [Fields](#fields-3)
-            * [Constructor](#constructor-11)
-            * [Methods](#methods-19)
-                * [Public](#public)
-                * [Protected](#protected)
-                * [Private](#private)
-            * [Inner Classes](#inner-classes-2)
-        * [Utility Functions](#utility-functions)
-        * [Usage](#usage-23)
-        * [Example](#example-8)
-* [kotlin\com\simiacryptus\skyenet\webui\test\ImageActorTestApp.kt](#kotlincomsimiacryptusskyenetwebuitestimageactortestappkt)
-    * [ImageActorTestApp Developer Documentation](#imageactortestapp-developer-documentation)
-        * [Overview](#overview-23)
-        * [Key Components](#key-components-13)
-            * [Constructor](#constructor-12)
-            * [Settings Data Class](#settings-data-class)
-            * [Overridden Methods](#overridden-methods)
-                * [initSettings](#initsettings)
-                * [userMessage](#usermessage-1)
-            * [Companion Object](#companion-object-5)
-        * [Usage](#usage-24)
-        * [Error Handling](#error-handling-6)
-        * [Conclusion](#conclusion-27)
-* [kotlin\com\simiacryptus\skyenet\webui\test\CodingActorTestApp.kt](#kotlincomsimiacryptusskyenetwebuitestcodingactortestappkt)
-    * [CodingActorTestApp Class Documentation](#codingactortestapp-class-documentation)
-        * [Constructor](#constructor-13)
-            * [Parameters:](#parameters-11)
-        * [Methods](#methods-20)
-            * [userMessage](#usermessage-2)
-                * [Parameters:](#parameters-12)
-        * [Usage](#usage-25)
-        * [Companion Object](#companion-object-6)
-        * [Example](#example-9)
-        * [Note](#note)
-* [kotlin\com\simiacryptus\skyenet\webui\test\ParsedActorTestApp.kt](#kotlincomsimiacryptusskyenetwebuitestparsedactortestappkt)
-    * [ParsedActorTestApp Class Documentation](#parsedactortestapp-class-documentation)
-        * [Overview](#overview-24)
-        * [Constructor](#constructor-14)
-            * [Parameters](#parameters-13)
-        * [Methods](#methods-21)
-            * [userMessage](#usermessage-3)
-                * [Parameters](#parameters-14)
-        * [Companion Object](#companion-object-7)
-            * [Properties](#properties-5)
-        * [Usage Example](#usage-example-13)
-        * [Conclusion](#conclusion-28)
-* [kotlin\com\simiacryptus\skyenet\webui\test\SimpleActorTestApp.kt](#kotlincomsimiacryptusskyenetwebuitestsimpleactortestappkt)
-    * [SimpleActorTestApp Documentation](#simpleactortestapp-documentation)
-        * [Overview](#overview-25)
-        * [Key Components](#key-components-14)
-            * [Constructor](#constructor-15)
-            * [Settings Data Class](#settings-data-class-1)
-            * [User Message Handling](#user-message-handling)
-            * [Logging](#logging-1)
-        * [Usage](#usage-26)
-        * [Conclusion](#conclusion-29)
-* [kotlin\com\simiacryptus\skyenet\webui\util\MarkdownUtil.kt](#kotlincomsimiacryptusskyenetwebuiutilmarkdownutilkt)
-    * [MarkdownUtil Documentation](#markdownutil-documentation)
-        * [Functions](#functions)
-            * [renderMarkdown](#rendermarkdown)
-                * [Parameters:](#parameters-15)
-                * [Returns:](#returns-5)
-                * [Usage Example:](#usage-example-14)
-            * [defaultOptions](#defaultoptions)
-                * [Returns:](#returns-6)
-        * [Implementation Details](#implementation-details-2)
-        * [Notes](#notes-2)
-* [kotlin\com\simiacryptus\skyenet\webui\session\SessionTask.kt](#kotlincomsimiacryptusskyenetwebuisessionsessiontaskkt)
-    * [SessionTask Class Documentation](#sessiontask-class-documentation)
-        * [Overview](#overview-26)
-        * [Properties](#properties-6)
-        * [Methods](#methods-22)
-            * [Abstract Methods](#abstract-methods-1)
-            * [Public Methods](#public-methods-2)
-            * [Companion Object](#companion-object-8)
-        * [Usage](#usage-27)
-            * [Example](#example-10)
-* [kotlin\com\simiacryptus\skyenet\webui\util\EncryptFiles.kt](#kotlincomsimiacryptusskyenetwebuiutilencryptfileskt)
-    * [EncryptFiles Utility Documentation](#encryptfiles-utility-documentation)
-        * [Overview](#overview-27)
-        * [Usage](#usage-28)
-            * [Prerequisites](#prerequisites)
-            * [Encrypting and Writing Data](#encrypting-and-writing-data)
-            * [Extension Functions](#extension-functions-1)
-        * [Example](#example-11)
-        * [Conclusion](#conclusion-30)
-* [kotlin\com\simiacryptus\skyenet\webui\util\OpenAPI.kt](#kotlincomsimiacryptusskyenetwebuiutilopenapikt)
-    * [Skyenet WebUI Util - OpenAPI Data Classes Documentation](#skyenet-webui-util---openapi-data-classes-documentation)
-        * [Overview](#overview-28)
-            * [OpenAPI](#openapi)
-            * [Info](#info)
-            * [Contact](#contact)
-            * [License](#license)
-            * [PathItem](#pathitem)
-            * [Operation](#operation)
-            * [Response](#response-1)
-            * [Components](#components)
-            * [Schema](#schema)
-            * [Parameter](#parameter)
-            * [Example, RequestBody, Header, SecurityScheme, Link, Callback, MediaType](#example-requestbody-header-securityscheme-link-callback-mediatype)
-* [kotlin\com\simiacryptus\skyenet\webui\util\TensorflowProjector.kt](#kotlincomsimiacryptusskyenetwebuiutiltensorflowprojectorkt)
-    * [TensorflowProjector Class Documentation](#tensorflowprojector-class-documentation)
-        * [Constructor](#constructor-16)
-        * [Methods](#methods-23)
-            * [`toVectorMap(vararg words: String): Map`](#tovectormapvararg-words-string-mapstring-doublearray)
-            * [`writeTensorflowEmbeddingProjectorHtml(vararg words: String): String`](#writetensorflowembeddingprojectorhtmlvararg-words-string-string)
-        * [Usage Example](#usage-example-15)
-        * [Notes](#notes-3)
-* [kotlin\com\simiacryptus\skyenet\webui\util\Selenium2S3.kt](#kotlincomsimiacryptusskyenetwebuiutilselenium2s3kt)
-    * [Developer Documentation for Selenium2S3 Class](#developer-documentation-for-selenium2s3-class)
-        * [Overview](#overview-29)
-        * [Key Features](#key-features-1)
-        * [Initialization](#initialization-1)
-        * [Core Methods](#core-methods)
-            * [save](#save)
-            * [process](#process)
-            * [getHtml, getJson, getMedia](#gethtml-getjson-getmedia)
-            * [saveJS, saveHTML](#savejs-savehtml)
-        * [Utility Methods](#utility-methods-5)
-        * [Closing Resources](#closing-resources)
-        * [Companion Object](#companion-object-9)
-            * [chromeDriver](#chromedriver)
-            * [setCookies](#setcookies)
-        * [Usage Example](#usage-example-16)
-* [resources\application\chat.js](#resourcesapplicationchatjs)
-    * [WebSocket Communication Module](#websocket-communication-module)
-        * [Functions](#functions-1)
-            * [getSessionId()](#getsessionid)
-            * [send(message)](#sendmessage)
-            * [connect(sessionId, customReceiveFunction)](#connectsessionid-customreceivefunction)
-            * [showDisconnectedOverlay(show)](#showdisconnectedoverlayshow)
-        * [Usage Example](#usage-example-17)
-* [resources\application\index.html](#resourcesapplicationindexhtml)
-    * [WebSocket Client Web Application Documentation](#websocket-client-web-application-documentation)
-        * [Overview](#overview-30)
-        * [Dependencies](#dependencies-10)
-            * [External Libraries and Stylesheets](#external-libraries-and-stylesheets)
-        * [HTML Structure](#html-structure)
-            * [Head Section](#head-section)
-            * [Body Section](#body-section)
-                * [Toolbar](#toolbar)
-                * [Namebar](#namebar)
-                * [Session](#session)
-                * [Modal](#modal)
-                * [Footer](#footer)
-        * [Functionality](#functionality-4)
-        * [Scripts](#scripts)
-            * [Prism.js](#prismjs)
-            * [Mermaid.js](#mermaidjs)
-            * [Application Scripts](#application-scripts)
-        * [Conclusion](#conclusion-31)
-* [resources\application\main.js](#resourcesapplicationmainjs)
-    * [Developer Documentation](#developer-documentation)
-        * [Overview](#overview-31)
-        * [Functions](#functions-2)
-            * [showModal(endpoint, useSession = true)](#showmodalendpoint-usesession--true)
-            * [closeModal()](#closemodal)
-            * [async fetchData(endpoint, useSession = true)](#async-fetchdataendpoint-usesession--true)
-            * [onWebSocketText(event)](#onwebsockettextevent)
-            * [updateTabs()](#updatetabs)
-            * [toggleVerbose()](#toggleverbose)
-            * [refreshReplyForms()](#refreshreplyforms)
-            * [refreshVerbose()](#refreshverbose)
-        * [Event Listeners](#event-listeners)
-            * [Theme Change](#theme-change)
-            * [Modal Triggers](#modal-triggers)
-            * [Form Submission](#form-submission)
-            * [Input Field Auto-Resize](#input-field-auto-resize)
-            * [Fetch User Information](#fetch-user-information)
-            * [Privacy and Terms Links](#privacy-and-terms-links)
-        * [Conclusion](#conclusion-32)
-* [resources\shared\schemes\_alien_spaceship.scss](#resourcessharedschemes_alien_spaceshipscss)
-    * [Alien Spaceship Theme - Developer Documentation](#alien-spaceship-theme---developer-documentation)
-        * [Base Colors](#base-colors)
-        * [Derived Colors](#derived-colors)
-        * [Usage](#usage-29)
-            * [Buttons](#buttons)
-            * [Messages](#messages)
-            * [Modals](#modals)
-        * [Conclusion](#conclusion-33)
-* [resources\shared\schemes\_forest.scss](#resourcessharedschemes_forestscss)
-    * [Forest Canopy Theme - Developer Documentation](#forest-canopy-theme---developer-documentation)
-        * [Importing the Theme](#importing-the-theme)
-        * [Typography Variables](#typography-variables)
-        * [Base Colors](#base-colors-1)
-        * [Derived Colors](#derived-colors-1)
-        * [Buttons](#buttons-1)
-        * [Forms](#forms)
-        * [Messages and Modals](#messages-and-modals)
-        * [Customization](#customization)
-        * [Conclusion](#conclusion-34)
-* [resources\shared\schemes\_night.scss](#resourcessharedschemes_nightscss)
-    * [Nighttime Theme Color Scheme Documentation](#nighttime-theme-color-scheme-documentation)
-        * [Base Colors](#base-colors-2)
-        * [Derived Colors](#derived-colors-2)
-        * [Component Styling](#component-styling)
-            * [Buttons](#buttons-2)
-            * [Forms](#forms-1)
-            * [Messages and Modals](#messages-and-modals-1)
-        * [Utility Variables](#utility-variables)
-        * [Usage](#usage-30)
-* [resources\shared\schemes\_normal.scss](#resourcessharedschemes_normalscss)
-    * [Developer Documentation: UI Theme Variables](#developer-documentation-ui-theme-variables)
-        * [Typography Variables](#typography-variables-1)
-        * [Base Colors](#base-colors-3)
-        * [Derived Colors](#derived-colors-3)
-        * [Buttons](#buttons-3)
-        * [Forms](#forms-2)
-        * [Messages and Modals](#messages-and-modals-2)
-* [resources\shared\schemes\_pony.scss](#resourcessharedschemes_ponyscss)
-    * [Developer Documentation: Theme Customization](#developer-documentation-theme-customization)
-        * [Typography Variables](#typography-variables-2)
-        * [Base Colors](#base-colors-4)
-        * [Derived Colors](#derived-colors-4)
-        * [Buttons](#buttons-4)
-        * [Forms](#forms-3)
-        * [Messages and Modals](#messages-and-modals-3)
-        * [Usage](#usage-31)
-* [resources\application\favicon.svg](#resourcesapplicationfaviconsvg)
-  * [Developer Documentation: SVG Graphic Implementation](#developer-documentation-svg-graphic-implementation)
-  * [Overview](#overview-32)
-  * [SVG Graphic Description](#svg-graphic-description)
-  * [File Information](#file-information)
-  * [Implementation Guide](#implementation-guide)
-  * [Conclusion](#conclusion-35)
-* [resources\shared\_main.scss](#resourcesshared_mainscss)
-    * [Developer Documentation: SCSS Mixins and Styles](#developer-documentation-scss-mixins-and-styles)
-        * [Mixins](#mixins)
-            * [1. `typography`](#1-typography)
-            * [2. `flex-container`](#2-flex-container)
-            * [3. `fixed-full`](#3-fixed-full)
-            * [4. `link-hover-transition`](#4-link-hover-transition)
-            * [5. `message-style`](#5-message-style)
-        * [Key Styling Sections](#key-styling-sections)
-            * [Body](#body)
-            * [Messages Container](#messages-container)
-            * [Input Fields](#input-fields)
-            * [Disconnected Overlay](#disconnected-overlay)
-            * [Buttons](#buttons-5)
-        * [Keyframes](#keyframes)
-* [resources\welcome\index.html](#resourceswelcomeindexhtml)
-    * [Developer Documentation: HTML Page Auto-Redirect](#developer-documentation-html-page-auto-redirect)
-        * [Overview](#overview-33)
-        * [Implementation](#implementation-1)
-            * [HTML Structure](#html-structure-1)
-                * [Meta Refresh Tag](#meta-refresh-tag)
-            * [Fallback Link](#fallback-link)
-        * [Usage](#usage-32)
-        * [Considerations](#considerations)
-        * [Conclusion](#conclusion-36)
-* [resources\welcome\main.js](#resourceswelcomemainjs)
-    * [Developer Documentation](#developer-documentation-1)
-        * [Overview](#overview-34)
-        * [Functions](#functions-3)
-            * [showModal(endpoint)](#showmodalendpoint)
-            * [closeModal()](#closemodal-1)
-            * [async fetchData(endpoint)](#async-fetchdataendpoint)
-            * [updateTabs()](#updatetabs-1)
-            * [Event Listeners Setup](#event-listeners-setup)
-        * [Usage](#usage-33)
-        * [Conclusion](#conclusion-37)
-* [resources\welcome\main.scss](#resourceswelcomemainscss)
-    * [Developer Documentation: App Type Styles](#developer-documentation-app-type-styles)
-        * [Import Statements](#import-statements-1)
-        * [`.app-type` Class Definition](#app-type-class-definition)
-        * [Usage](#usage-34)
-        * [Example](#example-12)
-* [resources\welcome\favicon.svg](#resourceswelcomefaviconsvg)
-    * [Developer Documentation for SVG Illustration](#developer-documentation-for-svg-illustration)
-        * [Overview](#overview-35)
-        * [Styles](#styles)
-        * [Paths](#paths)
-        * [Understanding Path Coordinates](#understanding-path-coordinates)
-        * [Modifying the SVG](#modifying-the-svg)
-        * [Reusability](#reusability)
-        * [Conclusion](#conclusion-38)
-* [resources\welcome\favicon.png](#resourceswelcomefaviconpng)
-    * [Developer Documentation](#developer-documentation-2)
-        * [Overview](#overview-36)
-        * [System Requirements](#system-requirements)
-        * [Setup Instructions](#setup-instructions)
-        * [Key Functionalities](#key-functionalities)
-        * [Contributing](#contributing-1)
-        * [Support](#support)
-
-
-
-# kotlin\com\simiacryptus\skyenet\apps\coding\ShellToolAgent.kt
-
-#### ShellToolAgent Class Documentation
-
-The `ShellToolAgent` class is an abstract class designed to facilitate the creation and execution of shell tools within
-a web application. It extends the `CodingAgent` class, integrating with an interpreter to execute code, manage session
-tasks, and interact with the user interface. This class is part of a larger framework aimed at enabling dynamic code
-execution and tool generation within a web environment.
-
-##### Key Features:
-
-- **Dynamic Code Execution**: Executes shell commands and scripts, allowing for real-time interaction and feedback.
-- **Tool Generation**: Dynamically generates tools based on code execution results, including OpenAPI documentation and
-  test pages.
-- **User Interaction**: Provides a web-based interface for users to input commands, view execution results, and interact
-  with generated tools.
-
-##### Core Components:
-
-- **Interpreter Integration**: Utilizes a Kotlin interpreter to execute shell commands and scripts.
-- **Session Management**: Manages user sessions, storing execution history and generated tools.
-- **UI Interaction**: Generates HTML forms and buttons for user interaction, including execution feedback and tool
-  interaction.
-
-##### Usage:
-
-The `ShellToolAgent` class is abstract and is intended to be extended by specific implementations that define the
-interpreter and tool generation logic. Here's a simplified usage example:
-
-```kotlin
-class MyShellToolAgent : ShellToolAgent(/* constructor parameters */) {
-  override fun getInterpreterString(): String {
-    // Return a string representation of the interpreter used
-  }
-
-  // Implement other abstract methods and any additional functionality
-}
-```
-
-##### Key Methods:
-
-- **displayFeedback**: Generates and displays feedback for code execution, including interactive buttons for further
-  actions.
-- **execute**: Executes the provided code within the context of the current session and user.
-- **createToolButton**: Generates a button for exporting execution results as a tool, including OpenAPI documentation
-  and test pages.
-
-##### Extensibility:
-
-The class is designed to be extensible, allowing developers to customize the execution environment, tool generation
-logic, and user interface components. By overriding methods and providing custom implementations, developers can tailor
-the functionality to meet specific requirements.
-
-##### Error Handling:
-
-The class includes error handling mechanisms to manage exceptions during code execution and tool generation. It provides
-feedback to the user and logs errors for debugging purposes.
-
-#### Conclusion
-
-The `ShellToolAgent` class is a powerful component for creating dynamic web-based tools that execute shell commands and
-scripts. Its integration with an interpreter, session management capabilities, and interactive user interface make it a
-versatile foundation for developing custom tooling solutions within web applications.
-
-# kotlin\com\simiacryptus\skyenet\apps\coding\CodingAgent.kt
-
-## CodingAgent Class Documentation
-
-The `CodingAgent` class is a part of the `com.simiacryptus.skyenet.apps.code` package, designed to facilitate code
-generation and execution within a specific application context. It leverages the OpenAI API for generating code snippets
-based on user input and provides mechanisms for executing and providing feedback on the generated code. This class is
-built to be integrated within a larger system that supports user interactions, code storage, and execution environments.
-
-### Overview
-
-`CodingAgent` extends the `ActorSystem` class, specializing in handling coding-related tasks. It interacts with users
-through a defined `ApplicationInterface`, receives code generation requests, processes them using OpenAI's GPT models,
-and manages the execution of generated code. Additionally, it provides functionalities for user feedback and code
-regeneration.
-
-### Key Components
-
-- **API Integration**: Utilizes the OpenAI API for generating code snippets.
-- **User Interaction**: Interfaces with users through tasks and messages, allowing for code requests and feedback.
-- **Code Execution**: Supports executing generated code and handling execution results or errors.
-- **Feedback Mechanism**: Allows users to provide feedback on generated code, facilitating iterative improvement.
-
-### Constructor Parameters
-
-- `api`: Instance of `API` for interacting with OpenAI services.
-- `dataStorage`: Storage interface for persisting data.
-- `session`: Current user session information.
-- `user`: The user object, may be null.
-- `ui`: Interface for application-specific UI interactions.
-- `interpreter`: The class of the interpreter to be used for code execution.
-- `symbols`: A map of predefined symbols available for code generation.
-- `temperature`: Controls the randomness of the code generation. Lower values make the output more deterministic.
-- `details`: Optional details for further customizing the code generation process.
-- `model`: Specifies the OpenAI model to be used for code generation.
-- `actorMap`: A map of actor types to their corresponding `CodingActor` instances.
-
-### Methods
-
-#### start(userMessage: String)
-
-Initiates the code generation process based on the user's message. It creates a new task, processes the user's request,
-and displays the generated code or any encountered errors.
-
-#### displayCode(task: SessionTask, codeRequest: CodingActor.CodeRequest)
-
-Displays the generated code to the user. If the user's message indicates that code is provided (enclosed in triple
-backticks), it directly processes this code; otherwise, it requests code generation.
-
-#### displayFeedback(task: SessionTask, request: CodingActor.CodeRequest, response: CodeResult)
-
-Displays options for the user to interact with the generated code, including buttons for executing, regenerating code,
-or providing feedback.
-
-#### execute(task: SessionTask, response: CodeResult, request: CodingActor.CodeRequest)
-
-Executes the generated code and displays the execution result or any errors.
-
-### Usage Example
-
-```kotlin
-val codingAgent = CodingAgent(
-  api = openAIClient,
-  dataStorage = storageInterface,
-  session = currentSession,
-  user = currentUser,
-  ui = applicationInterface,
-  interpreter = MyInterpreter::class,
-  symbols = mapOf("exampleSymbol" to Any()),
-  temperature = 0.1,
-  details = "Example details",
-  model = ChatModels.davinci
-)
-
-codingAgent.start("Generate a hello world program in Python")
-```
-
-This example creates an instance of `CodingAgent` configured with necessary dependencies and initiates a code generation
-request with a simple user message.
-
-### Conclusion
-
-The `CodingAgent` class provides a comprehensive solution for integrating AI-powered code generation and execution into
-applications. It abstracts the complexities of interacting with the OpenAI API, managing user interactions, and
-executing code, making it easier to build intelligent coding assistants and automation tools.
-
-# kotlin\com\github\simiacryptus\aicoder\util\SimpleDiffUtil.kt
-
-## SimpleDiffUtil and SocketManagerBase Extensions Documentation
-
-### Overview
-
-The provided code consists of two main components: `SimpleDiffUtil` and extension functions for `SocketManagerBase` to
-add functionality for applying diffs to code. `SimpleDiffUtil` is a utility object that provides functionality to apply
-patch strings to source code strings, simulating a simple version of a diff patch application. The `SocketManagerBase`
-extension functions enhance a web UI session by allowing diffs to be applied directly from the session interface.
-
-### SimpleDiffUtil
-
-#### Functionality
-
-- **Patch Application**: Applies a patch string to a source string, returning the modified source string.
-
-#### Methods
-
-- `fun patch(source: String, patch: String): String`
-    - Applies a patch to the given source string.
-    - **Parameters**:
-        - `source`: The original source code as a string.
-        - `patch`: The patch string, following a simplified diff format.
-    - **Returns**: The modified source code as a string after applying the patch.
-
-#### Internal Mechanics
-
-- **Deletion Handling**: Identifies lines to be deleted from the source based on the patch and removes them.
-- **Addition Handling**: Identifies lines to be added to the source based on the patch and inserts them.
-- **Context Line Handling**: Ensures that non-modified lines (context lines) are correctly maintained in the source.
-- **Line Matching**: Uses a Levenshtein Distance algorithm to match lines with a certain tolerance, allowing for minor
-  discrepancies.
-
-### SocketManagerBase Extension Functions
-
-#### Functionality
-
-- **Add Apply Diff Links**: Enhances a web UI session by adding links to apply diffs to code directly from the session
-  interface.
-
-#### Methods
-
-1. `fun SocketManagerBase.addApplyDiffLinks(code: String, response: String, fullPatch: MutableList = mutableListOf(), handle: (String) -> Unit): String`
-    - Adds links to apply diffs to a single piece of code.
-    - **Parameters**:
-        - `code`: The original code as a string.
-        - `response`: The session response content, where diff blocks are identified and links are added.
-        - `fullPatch`: A mutable list of strings to keep track of applied patches.
-        - `handle`: A callback function that handles the updated code.
-    - **Returns**: The modified session response content with diff application links.
-
-2. `fun SocketManagerBase.addApplyDiffLinks(code: Map, response: String, handle: (Map) -> Unit): String`
-    - Adds links to apply diffs to multiple pieces of code, each identified by a filename.
-    - **Parameters**:
-        - `code`: A map of filenames to their corresponding code as strings.
-        - `response`: The session response content, where diff blocks are identified and links are added.
-        - `handle`: A callback function that handles the updated code map.
-    - **Returns**: The modified session response content with diff application links.
-
-#### Usage Scenarios
-
-- **Code Review Sessions**: Allows reviewers to suggest changes directly in the session interface, which can then be
-  applied to the code.
-- **Collaborative Editing**: Enables real-time collaborative editing and versioning of code within a web UI session.
-
-### Conclusion
-
-The `SimpleDiffUtil` and the `SocketManagerBase` extension functions provide a foundational framework for applying diffs
-to code within a web UI session. This setup facilitates easier code reviews and collaborative editing by allowing direct
-application of changes from the session interface.
-
-# kotlin\com\simiacryptus\skyenet\apps\general\WebDevApp.kt
-
-## Web Development Assistant Application Documentation
-
-### Overview
-
-The Web Development Assistant Application is designed to facilitate the development of web applications by automating
-the generation of code and architecture planning through interaction with AI models. It leverages the OpenAI API to
-generate HTML, CSS, and JavaScript code based on user inputs, and provides tools for code review and architecture
-discussion.
-
-### Key Components
-
-#### WebDevApp Class
-
-The `WebDevApp` class extends `ApplicationServer` and serves as the main entry point for the application. It initializes
-the application with a name and a set of symbols, and defines the behavior for handling user messages.
-
-##### Key Methods
-
-- `userMessage(session: Session, user: User?, userMessage: String, ui: ApplicationInterface, api: API)`: Processes
-  messages from users and initiates the generation of web development artifacts based on the user's input.
-
-- `initSettings(session: Session)`: Initializes default settings for a session.
-
-#### WebDevAgent Class
-
-The `WebDevAgent` class extends `ActorSystem` and is responsible for managing the interaction with different actors to
-generate and review code. It defines various actor types for handling specific tasks such as HTML coding, JavaScript
-coding, CSS coding, architecture discussion, and code review.
-
-##### Key Methods
-
-- `start(userMessage: String)`: Initiates the process of generating web development artifacts based on the user's
-  message. It orchestrates the interaction with different actors to produce the desired output.
-
-- `draftResourceCode(...)`: Generates code for a specific resource (HTML, CSS, JavaScript) based on the user's input and
-  the architecture plan.
-
-#### Actors
-
-Actors are specialized components that handle specific tasks in the code generation and review process.
-The `WebDevAgent` class defines several actors for different purposes:
-
-- `HtmlCodingActor`
-- `JavascriptCodingActor`
-- `CssCodingActor`
-- `ArchitectureDiscussionActor`
-- `CodeReviewer`
-
-Each actor type is associated with a `SimpleActor` or `ParsedActor` instance, which defines the prompt and model to be
-used for generating or reviewing code.
-
-### Usage
-
-To use the Web Development Assistant Application, instantiate the `WebDevApp` class and configure it with the necessary
-settings. Then, interact with the application through user messages, which will be processed by the `WebDevApp`
-and `WebDevAgent` classes to generate and review web development artifacts.
-
-### Example
-
-```kotlin
-val webDevApp = WebDevApp(
-  applicationName = "My Web Dev Assistant",
-  symbols = mapOf("symbol1" to "value1", "symbol2" to "value2"),
-  temperature = 0.1
-)
-
-webDevApp.start()
-```
-
-This example creates an instance of the `WebDevApp` with a custom name and symbols, and starts the application. Users
-can then interact with the application to generate and review web development artifacts.
-
-### Conclusion
-
-The Web Development Assistant Application provides a powerful tool for automating the generation and review of web
-development code. By leveraging AI models, it simplifies the process of web application development and helps developers
-to quickly prototype and refine their applications.
-
-# kotlin\com\simiacryptus\skyenet\apps\coding\ToolAgent.kt
-
-## ToolAgent Class Documentation
-
-The `ToolAgent` class is an abstract class designed to facilitate the creation and manipulation of tools within a web
-application. It extends the `CodingAgent` class, incorporating additional functionalities specific to handling code
-generation, servlet creation, and OpenAPI documentation generation based on user interactions within a session.
-
-### Overview
-
-`ToolAgent` is tailored for applications that require dynamic code generation and evaluation, particularly in the
-context of web development and API documentation. It leverages the capabilities of the `CodingActor` class to process
-and generate code snippets, servlet implementations, and OpenAPI specifications.
-
-### Key Components
-
-#### Constructor Parameters
-
-- `api`: An instance of `API` used for interacting with OpenAI services.
-- `dataStorage`: Interface for data storage operations.
-- `session`: Represents the current user session.
-- `user`: Optional user information.
-- `ui`: Interface for application UI interactions.
-- `interpreter`: The Kotlin class of the interpreter to be used.
-- `symbols`: A map of predefined symbols for code generation.
-- `temperature`: Controls the randomness of the code generation.
-- `details`: Optional details for further customization.
-- `model`: Specifies the OpenAI model to be used for code generation.
-- `actorMap`: A map of `ActorTypes` to `CodingActor` instances for different code generation tasks.
-
-#### Methods
-
-##### `displayFeedback`
-
-Displays feedback to the user based on the code generation task's result. It includes play, regenerate, and tool
-creation buttons for user interaction.
-
-##### `createToolButton`
-
-Generates a button in the UI that, when clicked, initiates the process of exporting the generated code as a servlet and
-creating OpenAPI documentation.
-
-##### `openAPIParsedActor`
-
-Returns a `ParsedActor` instance configured for parsing OpenAPI specifications.
-
-##### `servletActor`
-
-Returns a `CodingActor` instance configured for generating servlet implementations.
-
-##### `schemaActor`
-
-Returns a `CodingActor` instance configured for generating data schema definitions.
-
-##### `displayCodeFeedback`
-
-Displays the generated code and provides options for acceptance, regeneration, or revision based on user feedback.
-
-##### `buildTestPage`
-
-Generates a test page for the created servlet based on the OpenAPI specification.
-
-##### `getInterpreterString`
-
-An abstract method that subclasses must implement to return the interpreter string representation.
-
-#### Utility Methods
-
-- `execWrap`: A companion object method that ensures the execution context is correctly set up for code generation and
-  evaluation tasks.
-
-### Usage
-
-While `ToolAgent` is abstract and cannot be instantiated directly, it serves as a base for creating specific tool agents
-within an application. Subclasses must implement the `getInterpreterString` method and can override other methods as
-needed to customize behavior.
-
-Subclasses of `ToolAgent` are responsible for handling specific tool-related tasks, such as generating code snippets,
-creating servlets, and generating OpenAPI documentation based on user interactions. They utilize the
-provided `CodingActor` instances for code generation and leverage the application UI for displaying feedback and options
-to the user.
-
-### Example
-
-To create a specific tool agent, extend the `ToolAgent` class and implement the required methods. For instance:
-
-```kotlin
-class MyToolAgent(
-  api: API,
-  dataStorage: StorageInterface,
-  session: Session,
-  user: User?,
-  ui: ApplicationInterface,
-  interpreter: KClass,
-  symbols: Map,
-  temperature: Double = 0.1,
-  details: String? = null,
-  model: ChatModels
-) : ToolAgent(api, dataStorage, session, user, ui, interpreter, symbols, temperature, details, model) {
-  override fun getInterpreterString(): String {
-    return "MyInterpreter"
-  }
-}
-```
-
-This example demonstrates how to create a specific tool agent by extending `ToolAgent` and providing the necessary
-constructor parameters and method implementations.
-
-# kotlin\com\simiacryptus\skyenet\AgentPatterns.kt
-
-## AgentPatterns Module Documentation
-
-The `AgentPatterns` module in the `com.simiacryptus.skyenet` package provides utility functions designed to facilitate
-interactive and iterative processes within a web UI context. These functions are particularly useful for applications
-that require user feedback loops, such as chatbots or interactive design tools. This documentation outlines the key
-functionalities provided by the module and how to utilize them in your projects.
-
-### Overview
-
-The module contains three primary functions:
-
-1. `retryable`: Allows the execution of a process with the option to retry, updating the UI with each attempt.
-2. `iterate`: Facilitates an iterative feedback loop between the user and the application, allowing for the refinement
-   of responses based on user input.
-3. `iterate` (overloaded version): An extension of the `iterate` function, tailored for use with `BaseActor` instances,
-   enabling more complex interactions such as those involving AI models.
-
-#### 1. retryable Function
-
-The `retryable` function is designed to execute a given process and provide the user with the option to retry the
-operation. It is useful in scenarios where an operation may not succeed on the first attempt or where the user may wish
-to try different inputs.
-
-##### Parameters:
-
-- `ui`: An instance of `ApplicationInterface`, which provides methods for interacting with the web UI.
-- `task`: An optional `SessionTask` instance representing the current task. If not provided, a new task is created.
-- `process`: A lambda function representing the process to be executed and retried as needed. It should return
-  a `String` that will be displayed in the UI.
-
-##### Returns:
-
-- A `String` representing the final result of the process after any retries.
-
-##### Usage Example:
-
-```kotlin
-val result = AgentPatterns.retryable(ui) {
-  // Your process logic here, returning a String to display in the UI
-}
-```
-
-#### 2. iterate Function
-
-The `iterate` function enables an iterative feedback loop, allowing the application to refine its responses based on
-user input. It is particularly useful for applications that require user feedback to improve the accuracy or relevance
-of the response.
-
-##### Parameters:
-
-- `ui`: An instance of `ApplicationInterface`.
-- `userMessage`: The initial user message or query.
-- `heading`: An optional heading for the feedback loop, displayed in the UI. Defaults to the rendered markdown
-  of `userMessage`.
-- `initialResponse`: A lambda function that takes the user message and returns an initial response of type `T`.
-- `reviseResponse`: A lambda function that takes the user message, the current response, and the user's feedback,
-  returning a revised response of type `T`.
-- `outputFn`: An optional lambda function that takes a `SessionTask` and the response of type `T`, and outputs it to the
-  UI. By default, it renders the response as markdown.
-
-##### Returns:
-
-- The final response of type `T` after the iterative feedback loop.
-
-##### Usage Example:
-
-```kotlin
-val finalResponse = AgentPatterns.iterate(
-  ui = ui,
-  userMessage = "Initial query",
-  initialResponse = { query -> "Initial response" },
-  reviseResponse = { query, currentResponse, userFeedback -> "Revised response based on $userFeedback" }
-)
-```
-
-#### 3. iterate Function (Overloaded Version)
-
-This version of the `iterate` function is specifically designed for use with `BaseActor` instances, facilitating more
-complex interactions such as those involving AI models.
-
-##### Parameters:
-
-- Inherits all parameters from the base `iterate` function.
-- `actor`: An instance of `BaseActor`, where `I` is the input type and `T` is the response type.
-- `toInput`: A lambda function that converts a `String` (typically the user message) into the input type `I` expected by
-  the `actor`.
-- `api`: An instance of `API`, used by the `actor` to interact with external services or models.
-
-##### Returns:
-
-- The final response of type `T` after the iterative feedback loop.
-
-##### Usage Example:
-
-```kotlin
-val finalResponse = AgentPatterns.iterate(
-  input = "Initial query",
-  actor = myActorInstance,
-  toInput = { query -> /* Convert query to actor's input type */ },
-  api = myApiInstance,
-  ui = ui
-)
-```
-
-### Conclusion
-
-The `AgentPatterns` module provides powerful utilities for creating interactive and iterative user experiences in web
-applications. By leveraging these functions, developers can implement complex feedback loops, retry mechanisms, and
-AI-driven interactions with ease.
-
-# kotlin\com\simiacryptus\skyenet\interpreter\ProcessInterpreter.kt
-
-## ProcessInterpreter Class Documentation
-
-The `ProcessInterpreter` class is part of the `com.simiacryptus.skyenet.interpreter` package and extends the
-functionality of the `Interpreter` interface. This class is designed to execute shell commands or scripts in various
-programming languages by wrapping the code and running it in a separate process. It provides a flexible way to execute
-code dynamically, supporting different languages and custom execution environments.
-
-### Constructor
-
-#### ProcessInterpreter
-
-```kotlin
-ProcessInterpreter(defs: Map< String, Any > = mapOf ())
-```
-
-Initializes a new instance of the `ProcessInterpreter` class.
-
-- **Parameters:**
-    - `defs`: A map containing configuration options for the interpreter, such as the command to run, the working
-      directory, environment variables, and the language. Defaults to an empty map.
-
-### Properties
-
-#### command
-
-```kotlin
-val command: List
-```
-
-A read-only property that returns the command to be executed as a list of strings. The command is determined based on
-the `defs` map passed to the constructor. It supports both a single string (which will be split into parts) and a list
-of strings. If no command is specified, it defaults to `bash`.
-
-### Methods
-
-#### getLanguage
-
-```kotlin
- override fun getLanguage(): String
-```
-
-Returns the programming language of the code to be interpreted. The language is specified in the `defs` map. If not
-specified, it defaults to `bash`.
-
-#### getSymbols
-
-```kotlin
-override fun getSymbols(): Map
-```
-
-Returns the symbols (configuration options) provided to the interpreter. This includes all the definitions passed
-through the `defs` map.
-
-#### validate
-
-```kotlin
-override fun validate(code: String): Throwable?
-```
-
-Validates the given code. This implementation always considers the code valid and returns `null`.
-
-- **Parameters:**
-    - `code`: The code to validate.
-
-- **Returns:** Always `null`, indicating the code is valid.
-
-#### run
-
-```kotlin
-override fun run(code: String): Any?
-```
-
-Executes the given code by wrapping it and running it in a separate process. It supports custom working directories,
-environment variables, and execution timeouts.
-
-- **Parameters:**
-    - `code`: The code to execute.
-
-- **Returns:** The output of the executed code. In case of an error, it returns a formatted string containing both the
-  error and the output. If the process times out, it throws a `RuntimeException`.
-
-### Usage Example
-
-```kotlin
-val interpreter = ProcessInterpreter(
-  mapOf(
-    "command" to "python",
-    "workingDir" to "/path/to/working/dir",
-    "env" to mapOf("KEY" to "value"),
-    "language" to "python"
-  )
-)
-
-val code = """
-print("Hello, World!")
-""".trimIndent()
-
-try {
-  val result = interpreter.run(code)
-  println(result)
-} catch (e: Exception) {
-  e.printStackTrace()
-}
-```
-
-This example creates an instance of `ProcessInterpreter` configured to run Python code, sets a working directory,
-environment variables, and specifies the language. It then executes a simple Python script and prints the result.
-
-### Notes
-
-- The `run` method writes the code to the process's output stream and reads the result from the input stream. It handles
-  process timeouts and captures both standard output and error output.
-- The class supports customization through the `defs` map, allowing for flexible execution environments and commands.
-
-# kotlin\com\simiacryptus\skyenet\webui\application\ApplicationInterface.kt
-
-## ApplicationInterface Documentation
-
-The `ApplicationInterface` class serves as a bridge between the web UI components and the underlying socket management
-and session handling logic. It provides methods to create interactive web elements such as links and text input forms,
-and to manage tasks representing long-running operations.
-
-### Constructor
-
-#### ApplicationInterface(SocketManagerBase socketManager)
-
-Initializes a new instance of the `ApplicationInterface` class.
-
-- **Parameters:**
-    - `socketManager`: An instance of `SocketManagerBase` that handles socket connections and interactions.
-
-### Methods
-
-#### hrefLink
-
-```kotlin
-open fun hrefLink(
-  linkText: String,
-  classname: String = "href-link",
-  handler: Consumer
-): String
-```
-
-Generates HTML for a hyperlink that triggers a specified handler when clicked.
-
-- **Parameters:**
-    - `linkText`: The text to display for the link.
-    - `classname`: The CSS class to apply to the link. Defaults to `"href-link"`.
-    - `handler`: A `Consumer` that is triggered when the link is clicked.
-- **Returns:** A `String` containing the HTML for the hyperlink.
-
-#### textInput
-
-```kotlin
-open fun textInput(
-  handler: Consumer
-): String
-```
-
-Generates HTML for a text input form that triggers a specified handler when the form is submitted.
-
-- **Parameters:**
-    - `handler`: A `Consumer` that is triggered with the text input value when the form is submitted.
-- **Returns:** A `String` containing the HTML for the text input form.
-
-#### newTask
-
-```kotlin
-open fun newTask(): SessionTask
-```
-
-Creates a new `SessionTask` instance that can be used to display the progress of a long-running operation. Currently,
-the method does not support cancelable tasks and defaults to non-cancelable tasks.
-
-- **Returns:** A `SessionTask` instance representing the new task.
-
-### Companion Object Methods
-
-#### oneAtATime
-
-```kotlin
-fun  oneAtATime(handler: Consumer): Consumer
-```
-
-Wraps a given handler to ensure that it is executed one at a time. If the handler is already in execution, subsequent
-calls will be ignored until the current execution completes.
-
-- **Parameters:**
-    - `handler`: The `Consumer` to be wrapped.
-- **Returns:** A `Consumer` that ensures the handler is executed one at a time.
-
-### Usage Example
-
-```kotlin
-val socketManager = MySocketManager()
-val appInterface = ApplicationInterface(socketManager)
-
-val linkHtml = appInterface.hrefLink("Click me", "custom-class") {
-  println("Link clicked")
-}
-
-val textInputHtml = appInterface.textInput { inputText ->
-  println("Form submitted with text: $inputText")
-}
-
-val task = appInterface.newTask()
-// Use the task for long-running operations
-```
-
-This documentation provides an overview of the `ApplicationInterface` class, its constructor, methods, and a usage
-example to help developers understand how to use it in their projects.
-
-# kotlin\com\simiacryptus\skyenet\webui\application\ApplicationDirectory.kt
-
-## Developer Documentation for `ApplicationDirectory` Class
-
-The `ApplicationDirectory` class serves as a foundational component for creating web applications with embedded Jetty
-servers, integrating various services including chat servers, OAuth authentication, and dynamic web content serving.
-This class is designed to be extended and customized for specific web application needs.
-
-### Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.application`
-- **Imports**: Various, including Jetty server components, servlets, and utility classes for handling OAuth and API
-  keys.
-- **Abstract Class**: Yes
-
-### Key Components
-
-#### Properties
-
-- `localName`: The hostname used when the server is not in "server mode" (default: `"localhost"`).
-- `publicName`: The hostname used in "server mode" (default: `"localhost"`).
-- `port`: The port on which the server will listen (default: `8081`).
-- `domainName`: The resolved domain name, set during initialization.
-- `childWebApps`: A list of `ChildWebApp` instances representing child web applications to be hosted.
-
-#### Inner Classes
-
-- `ChildWebApp`: A data class representing a child web application, including its path and an instance of `ChatServer`.
-
-#### Methods
-
-##### Abstract and Open Methods
-
-- `authenticatedWebsite()`: Returns an instance of `OAuthBase` for handling OAuth authentication. Can be overridden.
-- `setupPlatform()`: Configures platform-specific services, such as Selenium. Can be overridden.
-- `init(isServer: Boolean)`: Initializes the application, setting up the domain name and interceptors. Can be
-  overridden.
-- `start(port: Int, vararg webAppContexts: WebAppContext)`: Starts the Jetty server with the specified port and web
-  application contexts. Can be overridden.
-- `httpConnectionFactory()`: Creates and returns an `HttpConnectionFactory` with custom configuration. Can be
-  overridden.
-- `newWebAppContext(path: String, server: ChatServer)`: Creates a `WebAppContext` for a given path and `ChatServer`. Can
-  be overridden.
-- `newWebAppContext(path: String, baseResource: Resource, resourceBase: String, indexServlet: Servlet?)`: Creates
-  a `WebAppContext` with a specified base resource and optional index servlet. Can be overridden.
-- `newWebAppContext(path: String, servlet: Servlet)`: Creates a `WebAppContext` for a given path and servlet. Can be
-  overridden.
-
-##### Protected Methods
-
-- `_main(args: Array)`: The main method to start the application. It sets up the platform, initializes the
-  application, configures the server, and starts it.
-
-#### Companion Object
-
-- Contains a logger instance and a utility method `allResources(resourceName: String)` for loading resources.
-
-### Usage
-
-To use the `ApplicationDirectory` class, you should extend it in your application and implement the abstract properties
-and methods as needed. Here's a simplified example:
-
-```kotlin
-class MyApplicationDirectory : ApplicationDirectory() {
-  override val childWebApps = listOf(
-    ChildWebApp("/chat", ChatServer())
-  )
-
-  override fun setupPlatform() {
-    // Custom platform setup
-  }
-
-  override fun authenticatedWebsite(): OAuthBase? {
-    // Return an OAuthBase instance for authentication
-  }
-
-  // Implement other abstract/open methods as needed
-}
-```
-
-After extending `ApplicationDirectory`, you can instantiate your subclass and call its `_main` method with the necessary
-arguments to start the application.
-
-### Conclusion
-
-The `ApplicationDirectory` class provides a robust framework for building web applications with embedded Jetty servers,
-offering out-of-the-box support for chat functionalities, OAuth authentication, and more. By extending and customizing
-this class, developers can rapidly develop and deploy web applications tailored to their specific requirements.
-
-# kotlin\com\simiacryptus\skyenet\webui\application\ApplicationServer.kt
-
-## Developer Documentation for ApplicationServer
-
-The `ApplicationServer` class is an abstract base class designed to facilitate the creation and management of web
-applications within the Skyenet framework. It extends the `ChatServer` class, inheriting its chat functionalities, and
-provides a structured way to set up web applications with authentication, authorization, session management, and static
-file serving.
-
-### Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.application`
-- **Dependencies**: Requires various Skyenet core platform services and utilities, as well as external libraries such as
-  Jetty for web serving and SLF4J for logging.
-
-### Key Components
-
-#### Fields and Properties
-
-- `applicationName`: The name of the application.
-- `path`: The base URL path for the application.
-- `resourceBase`: The base directory for serving static resources.
-- `root`: The root directory for application data storage.
-- `description`: A brief description of the application.
-- `singleInput`, `stickyInput`: Flags to control input behavior.
-- `appInfo`: A lazy-initialized map containing application information.
-- `dataStorage`: A storage interface instance for data persistence.
-
-#### Servlets and Filters
-
-The class lazily initializes several `ServletHolder` instances for handling different aspects of the application:
-
-- `appInfoServlet`: Serves application information.
-- `userInfo`: Manages user information.
-- `usageServlet`: Tracks application usage.
-- `fileZip`, `fileIndex`: Serve and manage files.
-- `sessionSettingsServlet`, `sessionShareServlet`, `sessionThreadsServlet`, `deleteSessionServlet`, `cancelSessionServlet`:
-  Manage sessions and their settings.
-
-A filter is added to the web application context to enforce read access control based on user authentication and
-authorization.
-
-#### Session Management
-
-- `newSession(user: User?, session: Session)`: Creates a new `SocketManager` instance for managing WebSocket connections
-  for a session.
-- `userMessage(...)`: An abstract method intended to be overridden to handle user messages.
-
-#### Settings Management
-
-- `settingsClass`: Specifies the class type for application settings.
-- `initSettings(session: Session)`: Initializes settings for a session.
-- `getSettings(...)`: Retrieves or initializes settings for a session.
-
-#### Utility Methods
-
-- `sessionsServlet(path: String)`: Creates a servlet for listing sessions.
-- `configure(webAppContext: WebAppContext)`: Configures the web application context, adding filters and servlets.
-- `getMimeType(filename: String)`: Utility method to determine the MIME type based on file extension.
-- `HttpServletRequest.getCookie(name: String)`: Extension function to retrieve a cookie value by name.
-
-### Usage
-
-To create a web application using the `ApplicationServer` class, you must extend it and implement the abstract methods,
-such as `userMessage(...)`. You may also override open properties and methods to customize the application's behavior.
-
-#### Example
-
-```kotlin
-class MyApplicationServer : ApplicationServer("MyApp", "/myapp") {
-  override fun userMessage(session: Session, user: User?, userMessage: String, ui: ApplicationInterface, api: API) {
-    // Handle user messages here
-  }
-}
-```
-
-In this example, `MyApplicationServer` extends `ApplicationServer`, providing an implementation for handling user
-messages. Additional customization and functionality can be added by overriding other open methods and properties.
-
-### Conclusion
-
-The `ApplicationServer` class provides a robust foundation for building web applications within the Skyenet framework,
-offering out-of-the-box support for essential features like authentication, authorization, session management, and file
-serving. By extending this class and implementing the required abstract methods, developers can efficiently create
-feature-rich web applications.
-
-# kotlin\com\simiacryptus\skyenet\webui\application\ApplicationSocketManager.kt
-
-## ApplicationSocketManager Class Documentation
-
-The `ApplicationSocketManager` class is an abstract class designed to manage WebSocket connections for a specific
-application within a web-based platform. It extends the `SocketManagerBase` class, inheriting its basic WebSocket
-management functionalities and adding application-specific behaviors.
-
-### Overview
-
-This class is part of the `com.simiacryptus.skyenet.webui.application` package and integrates various components of the
-platform, including user sessions, data storage, and external API interactions. It is designed to handle messages
-received from users through WebSocket connections and to facilitate interaction with the application's backend services.
-
-### Constructor
-
-The constructor of the `ApplicationSocketManager` class requires the following parameters:
-
-- `session: Session`: The current user session.
-- `owner: User?`: The owner of the session, which can be null.
-- `dataStorage: StorageInterface?`: The interface for data storage operations, which can be null.
-- `applicationClass: Class<*>`: The class of the application for which this socket manager is being instantiated.
-
-### Key Methods and Properties
-
-#### onRun
-
-```kotlin
-override fun onRun(userMessage: String, socket: ChatSocket)
-```
-
-This method is called when a message is received from a user through the WebSocket. It processes the user message and
-interacts with the application's backend services as necessary.
-
-- `userMessage: String`: The message received from the user.
-- `socket: ChatSocket`: The WebSocket connection through which the message was received.
-
-#### userMessage
-
-```kotlin
-abstract fun userMessage(
-  session: Session,
-  user: User?,
-  userMessage: String,
-  socketManager: ApplicationSocketManager,
-  api: API
-)
-```
-
-An abstract method that must be implemented by subclasses to define how user messages are handled within the application
-context.
-
-- `session: Session`: The current user session.
-- `user: User?`: The user who sent the message, which can be null.
-- `userMessage: String`: The message received from the user.
-- `socketManager: ApplicationSocketManager`: The instance of the socket manager handling the message.
-- `api: API`: The API client for interacting with backend services.
-
-#### applicationInterface
-
-```kotlin
-open val applicationInterface by lazy { ApplicationInterface(this) }
-```
-
-A lazily initialized property that provides an interface to the application-specific functionalities. It is intended to
-be used for interactions between the WebSocket management and the application's backend services.
-
-### Companion Object
-
-The companion object of the `ApplicationSocketManager` class contains static properties and methods that can be accessed
-without an instance of the class.
-
-#### spinner
-
-```kotlin
-val spinner: String
-```
-
-A static property that provides HTML content for a loading spinner, which can be used in the user interface to indicate
-ongoing operations.
-
-### Usage
-
-To use the `ApplicationSocketManager` class, one must extend it and implement the abstract `userMessage` method to
-define the application-specific behavior for handling user messages. The subclass can then be instantiated with the
-necessary parameters (session, owner, dataStorage, and applicationClass) and used to manage WebSocket connections for
-the application.
-
-### Conclusion
-
-The `ApplicationSocketManager` class provides a structured way to manage WebSocket connections and facilitate
-communication between the user interface and the application's backend services. By extending this class and
-implementing its abstract methods, developers can integrate real-time features into their web applications efficiently.
-
-# kotlin\com\simiacryptus\skyenet\webui\chat\ChatServer.kt
-
-## ChatServer Class Documentation
-
-The `ChatServer` class is an abstract class designed to facilitate the creation of a chat server using Jetty web
-sockets. It provides a structured way to handle web socket connections, manage sessions, and integrate with a storage
-interface for data persistence.
-
-### Overview
-
-The `ChatServer` class encapsulates the core functionalities required to set up a chat server, including session
-management, web socket configuration, and servlet setup. It leverages Jetty's web socket and servlet capabilities to
-create a robust chat server framework.
-
-### Key Components
-
-#### Properties
-
-- `applicationName`: An abstract property that should be overridden to specify the name of the application.
-- `dataStorage`: An open property that can be overridden to provide an implementation of `StorageInterface` for data
-  storage purposes. It is nullable and defaults to `null`.
-- `sessions`: A mutable map that tracks active sessions and their corresponding `SocketManager` instances.
-
-#### Inner Classes
-
-##### WebSocketHandler
-
-An inner class that extends `JettyWebSocketServlet`. It is responsible for configuring the web socket factory and
-creating web socket instances for incoming connections.
-
-###### Key Methods
-
-- `configure(factory: JettyWebSocketServletFactory)`: Configures the web socket factory with specific settings such as
-  timeouts, buffer sizes, and message sizes. It also sets up the web socket creator to handle incoming connections.
-
-#### Abstract Methods
-
-- `newSession(user: User?, session: Session): SocketManager`: An abstract method that must be implemented to create a
-  new `SocketManager` instance for a given session. It allows for custom session initialization logic.
-
-#### Open Properties
-
-- `baseResource`: An open property that provides access to the base resource for the web server. It can be overridden to
-  customize the resource base.
-
-#### Methods
-
-- `configure(webAppContext: WebAppContext)`: Configures the `WebAppContext` with servlets for handling default requests,
-  web socket connections, and session creation.
-
-### Usage
-
-To use the `ChatServer` class, one must extend it and provide implementations for the abstract properties and methods.
-Here is a simplified example:
-
-```kotlin
-class MyChatServer(resourceBase: String) : ChatServer(resourceBase) {
-  override val applicationName = "MyChatApp"
-
-  override fun newSession(user: User?, session: Session): SocketManager {
-    // Implement session initialization logic here
-    return MySocketManager(session)
-  }
-}
-```
-
-Once extended, the server can be configured and started using Jetty's standard server setup procedures.
-
-### Companion Object
-
-#### Properties
-
-- `log`: A logger instance for logging purposes.
-
-#### Extension Functions
-
-- `JettyServerUpgradeRequest.getCookie(name: String)`: An extension function for `JettyServerUpgradeRequest` to simplify
-  cookie retrieval.
-
-### Conclusion
-
-The `ChatServer` class provides a foundational framework for building chat servers with Jetty. By extending this class
-and implementing the required abstract methods, developers can create customized chat server applications tailored to
-their specific needs.
-
-# kotlin\com\simiacryptus\skyenet\webui\chat\ChatSocket.kt
-
-## ChatSocket Class Documentation
-
-The `ChatSocket` class is part of the `com.simiacryptus.skyenet.webui.chat` package and is designed to handle WebSocket
-connections for a chat application. It extends the `WebSocketAdapter` class provided by the Eclipse Jetty WebSocket API,
-enabling it to manage WebSocket events such as connection, message reception, and disconnection.
-
-### Dependencies
-
-- `SocketManager`: A custom class that manages WebSocket sessions and their interactions.
-- `SocketManagerBase`: A base class for `SocketManager` that provides utility methods, such as retrieving a user from a
-  session.
-- `Session`: Part of the Eclipse Jetty WebSocket API, representing a WebSocket session.
-- `WebSocketAdapter`: A convenience class from the Eclipse Jetty WebSocket API that can be extended to create WebSocket
-  endpoints.
-
-### Class Overview
-
-- **Constructor**: The class constructor takes a `SocketManager` instance, which is used to manage the WebSocket
-  sessions.
-
-- **Properties**:
-    - `user`: A read-only property that retrieves the user associated with the current WebSocket session.
-
-- **Methods**:
-    - `onWebSocketConnect(session: Session)`: Called when a WebSocket connection is established. It registers the new
-      connection with the `SocketManager` and sends any messages that need to be replayed to the newly connected client.
-    - `onWebSocketText(message: String)`: Invoked when a text message is received from the client. It forwards the
-      message to the `SocketManager` for further processing.
-    - `onWebSocketClose(statusCode: Int, reason: String?)`: Triggered when the WebSocket connection is closed. It
-      removes the connection from the `SocketManager`.
-
-### Usage
-
-The `ChatSocket` class is intended to be used as part of a server-side application that utilizes WebSockets for
-real-time communication, specifically in a chat application context. It requires an instance of `SocketManager` to
-handle session management and message broadcasting.
-
-#### Example
-
-To use `ChatSocket` in a server application, you would typically set it up as an endpoint that clients can connect to.
-Here's a simplified example of how it might be integrated into a server setup:
-
-```kotlin
-import org.eclipse.jetty.server.Server
-import org.eclipse.jetty.websocket.server.WebSocketHandler
-import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory
-
-fun main() {
-  val server = Server(8080)
-  val socketManager = SocketManager()
-
-  val wsHandler = object : WebSocketHandler() {
-    override fun configure(factory: WebSocketServletFactory) {
-      factory.register(ChatSocket::class.java) { ChatSocket(socketManager) }
-    }
-  }
-
-  server.handler = wsHandler
-  server.start()
-  server.join()
-}
-```
-
-This example sets up a basic Jetty server that listens on port 8080. It configures a WebSocket handler to create
-instances of `ChatSocket`, passing in a shared `SocketManager` instance to manage the chat sessions.
-
-### Conclusion
-
-The `ChatSocket` class facilitates the creation of WebSocket endpoints for real-time chat applications, handling
-connection, message reception, and disconnection events. It leverages the `SocketManager` class to manage chat sessions
-and broadcast messages to connected clients.
-
-# kotlin\com\simiacryptus\skyenet\webui\chat\ChatSocketManager.kt
-
-## ChatSocketManager Class Documentation
-
-The `ChatSocketManager` class is part of the `com.simiacryptus.skyenet.webui.chat` package and is responsible for
-managing chat interactions within a web UI, leveraging the OpenAI API for generating chat responses. This class
-extends `SocketManagerBase`, integrating chat functionality with a session-based web application.
-
-### Constructor
-
-```kotlin
-ChatSocketManager(
-  session: Session,
-  model: ChatModels,
-  userInterfacePrompt: String,
-  initialAssistantPrompt: String = "",
-systemPrompt: String,
-api: OpenAIClient,
-temperature: Double = 0.3,
-applicationClass: Class,
-storage: StorageInterface?
-)
-```
-
-#### Parameters
-
-- `session`: The current user session.
-- `model`: The OpenAI model to be used for generating chat responses.
-- `userInterfacePrompt`: A prompt displayed to the user at the start of the chat session.
-- `initialAssistantPrompt`: An optional initial message from the assistant.
-- `systemPrompt`: A system-level prompt used to initialize the chat context.
-- `api`: The OpenAI client used for making API requests.
-- `temperature`: Controls randomness in the response generation. Lower values make responses more deterministic.
-- `applicationClass`: The application server class that this socket manager is associated with.
-- `storage`: An optional storage interface for persisting chat data.
-
-### Methods
-
-#### onRun
-
-```kotlin
-override fun onRun(userMessage: String, socket: ChatSocket)
-```
-
-Handles the reception of a user message, generates a response using the OpenAI API, and sends the response back to the
-user.
-
-##### Parameters
-
-- `userMessage`: The message received from the user.
-- `socket`: The chat socket through which the message was received.
-
-#### renderResponse
-
-```kotlin
-open fun renderResponse(response: String): String
-```
-
-Formats the response from the OpenAI API for display in the web UI.
-
-##### Parameters
-
-- `response`: The raw response string from the OpenAI API.
-
-##### Returns
-
-- A `String` containing the formatted response.
-
-#### onResponse
-
-```kotlin
-open fun onResponse(response: String, responseContents: String)
-```
-
-A hook method that can be overridden to perform additional actions after a response has been generated and sent.
-
-##### Parameters
-
-- `response`: The response generated by the OpenAI API.
-- `responseContents`: The formatted response contents that were sent to the user.
-
-### Properties
-
-- `messages`: A lazy-initialized list of chat messages, including system and assistant prompts, used to maintain the
-  chat context.
-
-### Companion Object
-
-Contains a logger for logging information and errors.
-
-### Usage
-
-To use `ChatSocketManager`, instantiate it with the required parameters, including the session, OpenAI model, and
-prompts. The class handles the chat interaction flow, including sending initial prompts, receiving user messages,
-generating responses using the OpenAI API, and sending those responses back to the user.
-
-This class is designed to be extended, allowing developers to customize the response rendering and handling through
-overriding the `renderResponse` and `onResponse` methods.
-
-### Error Handling
-
-Errors during chat, such as issues with the OpenAI API requests, are logged using the class's logger. Override
-the `onResponse` method to implement custom error handling or additional logging as needed.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\ApiKeyServlet.kt
-
-## API Key Servlet Documentation
-
-The `ApiKeyServlet` class is a part of the web application that manages API key records. It allows users to create,
-edit, delete, and invite others to use API keys. This documentation provides an overview of its functionality, including
-the data model, supported operations, and how to interact with the servlet.
-
-### Overview
-
-The servlet extends `HttpServlet` and overrides the `doGet` and `doPost` methods to handle HTTP GET and POST requests,
-respectively. It interacts with the application's authentication, user settings, and usage management services to
-perform its operations.
-
-### Data Model
-
-#### ApiKeyRecord
-
-The `ApiKeyRecord` data class represents an API key record with the following properties:
-
-- `owner`: The email of the user who owns the API key.
-- `apiKey`: The API key string.
-- `mappedKey`: A mapped or alias key for the original API key.
-- `budget`: A budget limit for the API key usage.
-- `comment`: A description or comment about the API key.
-- `welcomeMessage`: A welcome message for users invited to use the API key.
-
-### Supported Operations
-
-The servlet supports the following operations based on the `action` parameter in the request:
-
-- **Edit**: Allows users to edit an existing API key record.
-- **Delete**: Enables users to delete an API key record.
-- **Create**: Facilitates the creation of a new API key record.
-- **Invite**: Provides a mechanism to invite other users to use an API key.
-
-#### Handling GET Requests
-
-The `doGet` method handles HTTP GET requests and supports the following actions: `edit`, `delete`, `create`,
-and `invite`. It uses the `action` and `apiKey` parameters from the request to determine the operation to perform.
-
-#### Handling POST Requests
-
-The `doPost` method handles HTTP POST requests for accepting invitations and updating API key records. It supports
-the `acceptInvite` action and updates to existing records.
-
-### Interacting with the Servlet
-
-#### Editing an API Key Record
-
-To edit an API key record, send a GET request with the `action` parameter set to `edit` and the `apiKey` parameter set
-to the API key you want to edit.
-
-#### Deleting an API Key Record
-
-To delete an API key record, send a GET request with the `action` parameter set to `delete` and the `apiKey` parameter
-set to the API key you want to delete.
-
-#### Creating a New API Key Record
-
-To create a new API key record, send a GET request with the `action` parameter set to `create`.
-
-#### Inviting a User to Use an API Key
-
-To invite a user to use an API key, send a GET request with the `action` parameter set to `invite` and the `apiKey`
-parameter set to the API key for which you want to send an invitation.
-
-### Utility Methods
-
-The servlet includes private utility methods for generating HTML
-pages (`indexPage`, `serveInviteConfirmationPage`, `serveEditPage`) and for saving API key records to a JSON
-file (`saveRecords`).
-
-### Storage
-
-API key records are stored in a JSON file located in the `.skyenet/apiKeys` directory under the application's data
-storage root. The `apiKeyRecords` companion object property lazily loads the records from this file when accessed.
-
-### Conclusion
-
-The `ApiKeyServlet` class provides a comprehensive solution for managing API key records within the web application. By
-supporting operations such as creation, editing, deletion, and invitation, it enables users to effectively manage their
-API keys and control access to their services.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\AppInfoServlet.kt
-
-## AppInfoServlet Class Documentation
-
-The `AppInfoServlet` class is a custom servlet that extends `HttpServlet` and is designed to serve JSON data about an
-application's specific information. This class is part of the `com.simiacryptus.skyenet.webui.servlet` package and
-utilizes the `JsonUtil` class for JSON serialization.
-
-### Generics
-
-- `T`: The type parameter `T` represents the type of information the servlet will provide. This allows for flexibility
-  in the type of data the servlet can handle and return as JSON.
-
-### Constructor
-
-- `AppInfoServlet(T info)`: Constructs an instance of `AppInfoServlet` with the specified information to be returned by
-  the servlet.
-
-  | Parameter | Type | Description |
-      |-----------|------|-------------|
-  | info      | `T`  | The information of type `T` that this servlet will return when handling GET requests. |
-
-### Methods
-
-#### `doGet(HttpServletRequest req, HttpServletResponse resp)`
-
-Overrides the `doGet` method from `HttpServlet` to handle HTTP GET requests.
-
-- **Parameters:**
-    - `req`: `HttpServletRequest` - The request sent by the client to the server.
-    - `resp`: `HttpServletResponse` - The response that the servlet sends back to the client.
-
-- **Functionality:** When a GET request is received, this method sets the response content type to `text/json`, sets the
-  HTTP status code to `200 OK`, and writes the JSON representation of the `info` object to the response. The JSON
-  serialization is performed using the `JsonUtil.objectMapper()` utility.
-
-- **Usage Example:** This method is automatically called by the servlet container when a GET request is made to the
-  servlet's URL. It is not meant to be called directly.
-
-### Usage
-
-To use the `AppInfoServlet`, you need to instantiate it with the specific type of information you want to serve as JSON.
-For example, if you have a class `AppInfo` that contains application metadata, you can create an instance
-of `AppInfoServlet` like so:
-
-```java
-AppInfo appInfo = new AppInfo("MyApp", "1.0");
-AppInfoServlet servlet = new AppInfoServlet<>(appInfo);
-```
-
-After instantiation, the servlet needs to be registered with a servlet container or web server to handle requests to a
-specific path.
-
-### Dependencies
-
-- `com.simiacryptus.jopenai.util.JsonUtil`: Used for converting the `info` object into its JSON representation.
-- `jakarta.servlet.http.HttpServlet`: The base class for HTTP servlets.
-- `jakarta.servlet.http.HttpServletRequest`: Represents the request sent by the client to the server.
-- `jakarta.servlet.http.HttpServletResponse`: Represents the response that the servlet sends back to the client.
-
-### Conclusion
-
-The `AppInfoServlet` class provides a simple and flexible way to serve application-specific information as JSON over
-HTTP. By leveraging generic types, it can be used to serve various kinds of information, making it a versatile component
-in web applications.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\CancelThreadsServlet.kt
-
-## CancelThreadsServlet Documentation
-
-The `CancelThreadsServlet` class is part of the web UI module for managing session-based operations within an
-application server environment. It extends `HttpServlet` and provides mechanisms to cancel running threads associated
-with a specific session through a web interface. This servlet is designed to handle both GET and POST requests, allowing
-users to initiate the cancellation process and confirm cancellation, respectively.
-
-### Class Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.servlet`
-- **Dependencies**:
-    - `com.simiacryptus.jopenai.util.JsonUtil`
-    - `com.simiacryptus.skyenet.core.platform.*`
-    - `jakarta.servlet.http.*`
-    - Other internal components related to application server and session management.
-
-### Constructor
-
-#### `CancelThreadsServlet(ApplicationServer server)`
-
-Initializes a new instance of the `CancelThreadsServlet` with a reference to the `ApplicationServer`.
-
-- **Parameters**:
-    - `server`: An instance of `ApplicationServer` to which this servlet belongs.
-
-### Methods
-
-#### `doGet(HttpServletRequest req, HttpServletResponse resp)`
-
-Handles the GET request by displaying an HTML form where the user can confirm the cancellation of a session.
-
-- **Parameters**:
-    - `req`: The `HttpServletRequest` object that contains the request the client made to the servlet.
-    - `resp`: The `HttpServletResponse` object that contains the response the servlet returns to the client.
-
-#### `doPost(HttpServletRequest req, HttpServletResponse resp)`
-
-Handles the POST request by processing the cancellation confirmation. It validates the session ID and confirmation
-input, checks user authorization, and then proceeds to shut down the thread pool associated with the session.
-
-- **Parameters**:
-    - `req`: The `HttpServletRequest` object that contains the request the client made to the servlet.
-    - `resp`: The `HttpServletResponse` object that contains the response the servlet returns to the client.
-
-### Usage
-
-1. **Initiating Cancellation**: A user initiates the cancellation process by navigating to the servlet's URL with
-   a `sessionId` parameter in the query string. The servlet responds with an HTML form asking the user to confirm the
-   cancellation by typing 'confirm'.
-
-2. **Confirming Cancellation**: The user submits the confirmation form. The servlet validates the input and checks if
-   the user is authorized to cancel the session. If authorized, the servlet cancels the session's threads and redirects
-   the user to the home page.
-
-### Security Considerations
-
-- **Authorization**: The servlet checks if the user is authorized to perform cancellation operations. It ensures that
-  only users with the appropriate permissions can cancel sessions, including global sessions.
-- **Input Validation**: The servlet validates the `sessionId` and confirmation input to prevent unauthorized or
-  accidental operations.
-
-### Error Handling
-
-- The servlet responds with `HttpServletResponse.SC_BAD_REQUEST` if the required parameters are missing or invalid.
-- Throws an exception if the user is not authorized to perform the operation, ensuring that unauthorized attempts are
-  logged and halted.
-
-### Conclusion
-
-The `CancelThreadsServlet` provides a web interface for managing the cancellation of sessions within an application
-server environment. It ensures that only authorized users can perform cancellations and offers a user-friendly way to
-confirm such operations, enhancing the application's manageability and security.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\CorsFilter.kt
-
-## CorsFilter Class Documentation
-
-The `CorsFilter` class is a servlet filter designed to handle Cross-Origin Resource Sharing (CORS) for web applications.
-It is part of the `com.simiacryptus.skyenet.webui.servlet` package. This filter allows or restricts web applications
-running on one domain to request resources from another domain. By default, web browsers restrict cross-origin HTTP
-requests initiated from scripts for security reasons. The `CorsFilter` class provides a way to relax this security
-measure for specific domains or paths.
-
-### Features
-
-- **Cross-Origin Request Handling**: Automatically adds CORS headers to HTTP responses for requests not ending
-  with `/ws`.
-- **Flexible URL Pattern Matching**: Applies to all paths (`/*`) by default, but can be easily adjusted by changing
-  the `urlPatterns` parameter in the `@WebFilter` annotation.
-- **Asynchronous Support**: Supports asynchronous request processing with `asyncSupported = true`.
-
-### Usage
-
-To use the `CorsFilter`, ensure it is included in your web application's deployment descriptor or annotated
-with `@WebFilter` as shown in the code. No additional configuration is required for basic usage. The filter applies to
-all incoming requests but excludes paths ending with `/ws`.
-
-#### Configuration
-
-- **URL Patterns**: The filter is configured to intercept all paths (`/*`). This can be modified by changing
-  the `urlPatterns` attribute in the `@WebFilter` annotation.
-- **Asynchronous Processing**: Enabled by default (`asyncSupported = true`). If your application does not use
-  asynchronous processing, this can be set to `false`.
-
-#### Methods
-
-- `init(filterConfig: FilterConfig?)`: Initializes the filter. No custom initialization is performed in the current
-  implementation.
-- `doFilter(request: ServletRequest?, response: ServletResponse, chain: FilterChain)`: Processes incoming requests and
-  adds CORS headers to the response if the request URI does not end with `/ws`.
-- `destroy()`: Cleans up any resources used by the filter. No custom cleanup is performed in the current implementation.
-
-#### CORS Headers Added
-
-- `Access-Control-Allow-Origin: *`: Allows all domains to request resources.
-- `Access-Control-Allow-Methods: POST, GET, OPTIONS, DELETE, PUT`: Specifies the allowed methods for cross-origin
-  requests.
-- `Access-Control-Max-Age: 3600`: Indicates how long the results of a preflight request can be cached.
-- `Access-Control-Allow-Headers: Content-Type, x-requested-with, authorization`: Specifies the headers allowed in the
-  actual request.
-
-### Example
-
-No additional steps are required to integrate the `CorsFilter` into your application beyond including it in your
-project. Ensure your web application is configured to load servlet filters.
-
-### Logging
-
-The `CorsFilter` utilizes SLF4J for logging. Any exceptions encountered during the filtering process are logged as
-warnings, including the stack trace for debugging purposes.
-
-### Conclusion
-
-The `CorsFilter` class is a straightforward solution for enabling CORS in web applications. By adding this filter,
-developers can easily configure which resources are accessible from different origins, enhancing the interoperability
-and security of web applications.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\FileServlet.kt
-
-## FileServlet Class Documentation
-
-The `FileServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends `HttpServlet` to
-provide a web interface for accessing and managing files within a session-based directory structure. It interacts with
-a `StorageInterface` to resolve file paths and supports both small and large file transfers efficiently.
-
-### Constructor
-
-- `FileServlet(StorageInterface dataStorage)`: Initializes a new instance of the `FileServlet` class using the
-  provided `dataStorage` to manage file paths and access.
-
-### Methods
-
-#### Public Methods
-
-- `void doGet(HttpServletRequest req, HttpServletResponse resp)`: Handles the GET request by serving files or
-  directories based on the request path. It supports file downloading, directory listing, and redirection for directory
-  paths without a trailing slash.
-
-#### Private Methods
-
-- `void writeSmall(FileChannel channel, HttpServletResponse resp, File file, HttpServletRequest req)`: Serves small
-  files (less than 1MB) by reading the file content into memory and writing it to the response output stream.
-- `void writeLarge(FileChannel channel, HttpServletResponse resp, File file, HttpServletRequest req)`: Serves large
-  files using a `MappedByteBuffer` for efficient memory-mapped file I/O.
-- `String directoryHTML(HttpServletRequest req, Session session, String filePath, String folders, String files)`:
-  Generates an HTML page for directory listing, including links to contained files and subdirectories.
-
-### Utility Methods
-
-- `static List parsePath(String path)`: Parses the request path into segments, performing validation to prevent
-  directory traversal attacks and invalid characters.
-- `static String directoryHTML(...)`: Generates the HTML content for directory listings.
-
-### Companion Object
-
-The companion object contains shared resources and utility functions:
-
-- `Logger log`: SLF4J Logger instance for logging.
-- `Cache channelCache`: A Guava cache for managing open `FileChannel` instances, with automatic
-  closing and removal of stale entries.
-
-### Cache Configuration
-
-The `channelCache` is configured with a maximum size of 100 entries and an expiration policy of 10 seconds after the
-last access. It uses a `RemovalListener` to log and close `FileChannel` instances when they are evicted from the cache.
-
-### Exception Handling
-
-The class includes checks and exception handling to ensure safe path resolution, preventing directory traversal attacks
-and handling invalid path characters. It also includes error handling for I/O operations, logging errors encountered
-during file reading or writing.
-
-### Usage
-
-To use the `FileServlet`, it must be registered with a servlet container (e.g., Tomcat, Jetty) and mapped to a URL
-pattern. The servlet requires a `StorageInterface` implementation to manage file storage and session-based directory
-resolution.
-
-### Example
-
-```java
-StorageInterface storage = new MyStorageImplementation();
-FileServlet servlet = new FileServlet(storage);
-// Servlet registration and URL mapping code here
-```
-
-This class is designed to be flexible and efficient, supporting both small and large file transfers while providing a
-user-friendly directory browsing experience.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\DeleteSessionServlet.kt
-
-## DeleteSessionServlet Documentation
-
-`DeleteSessionServlet` is a servlet class designed to handle the deletion of user sessions within a web application. It
-extends `HttpServlet` and overrides the `doGet` and `doPost` methods to provide functionality for confirming and
-processing the deletion of a session, respectively.
-
-### Overview
-
-The servlet is part of a larger web application framework and interacts with various components such
-as `ApplicationServer`, `ApplicationServices`, and `AuthorizationInterface` to perform its duties. It is designed to
-ensure that only authorized users can delete sessions, and it provides both a confirmation mechanism and a direct
-deletion process.
-
-### Usage
-
-#### Initialization
-
-The servlet is initialized with an instance of `ApplicationServer`, which is passed to the constructor:
-
-```java
-public DeleteSessionServlet(ApplicationServer server) {
-    this.server = server;
-}
-```
-
-This `server` instance is used to interact with the application's data storage and authentication systems.
-
-#### Handling GET Requests
-
-The `doGet` method is responsible for presenting the user with a confirmation form when they attempt to delete a
-session. It checks if the request contains a `sessionId` parameter and, if so, displays an HTML form asking the user to
-confirm the deletion by typing 'confirm'.
-
-```java
-
-@Override
-public void doGet(HttpServletRequest req, HttpServletResponse resp) {
-    // Implementation details...
-}
-```
-
-If the `sessionId` parameter is missing, the method responds with a `400 Bad Request` status and a message indicating
-that a session ID is required.
-
-#### Handling POST Requests
-
-The `doPost` method processes the form submission, verifying that the user has typed 'confirm' and that a session ID is
-provided. It also checks if the user is authorized to delete the session, with additional checks for global sessions.
-
-```java
-
-@Override
-public void doPost(HttpServletRequest req, HttpServletResponse resp) {
-    // Implementation details...
-}
-```
-
-If the checks pass, the session is deleted from the application's data storage, and the user is redirected to the
-application's home page.
-
-### Security Considerations
-
-- **Authorization Checks**: The servlet performs thorough authorization checks to ensure that only users with the
-  appropriate permissions can delete sessions. This includes a general authorization check and a specific check for the
-  deletion of global sessions.
-- **Confirmation Requirement**: By requiring users to type 'confirm' before a session can be deleted, the servlet adds
-  an extra layer of protection against accidental or malicious deletions.
-
-### Dependencies
-
-- `ApplicationServer`: Used to access application-specific services and data storage.
-- `ApplicationServices`: Provides access to the application's authentication and authorization systems.
-- `AuthorizationInterface`: Defines the operations and permissions used in authorization checks.
-- `Session`: Represents a user session within the application.
-
-### Conclusion
-
-`DeleteSessionServlet` is a critical component of the web application, providing secure and user-friendly functionality
-for deleting sessions. Its integration with the application's authentication and authorization systems ensures that
-session deletions are handled safely and responsibly.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\OAuthBase.kt
-
-## OAuthBase Class Documentation
-
-The `OAuthBase` class serves as an abstract base for implementing OAuth authentication within a web application using
-the Jetty server. This class is part of the `com.simiacryptus.skyenet.webui.servlet` package.
-
-### Overview
-
-OAuth is an open standard for access delegation, commonly used as a way for Internet users to grant websites or
-applications access to their information on other websites but without giving them the passwords. This class provides a
-structured way to integrate OAuth authentication into your web applications by defining a common interface and
-functionality for OAuth operations.
-
-### Usage
-
-To use the `OAuthBase` class, you need to extend it in your own class and implement the abstract method `configure`.
-This method is designed to configure the Jetty `WebAppContext` for OAuth authentication, including setting up necessary
-filters if required.
-
-#### Parameters
-
-- `redirectUri`: A `String` specifying the URI to redirect to after authentication. This is a constructor parameter for
-  the `OAuthBase` class.
-
-#### Methods
-
-##### `configure`
-
-```kotlin
-abstract fun configure(context: WebAppContext, addFilter: Boolean = true): WebAppContext
-```
-
-This abstract method must be implemented by subclasses. It is intended to configure the provided `WebAppContext` for
-OAuth authentication.
-
-###### Parameters:
-
-- `context`: The `WebAppContext` to be configured. This is the context of your web application where you want to
-  integrate OAuth authentication.
-- `addFilter`: A `Boolean` indicating whether to add the OAuth authentication filter to the `WebAppContext`. The default
-  value is `true`.
-
-###### Returns:
-
-- `WebAppContext`: The configured `WebAppContext` instance.
-
-### Example
-
-Below is an example of how to extend the `OAuthBase` class and implement the `configure` method:
-
-```kotlin
-package com.example.auth
-
-import com.simiacryptus.skyenet.webui.servlet.OAuthBase
-import org.eclipse.jetty.webapp.WebAppContext
-
-class MyOAuthImpl(redirectUri: String) : OAuthBase(redirectUri) {
-  override fun configure(context: WebAppContext, addFilter: Boolean): WebAppContext {
-    // Implement OAuth configuration and filter setup here
-    if (addFilter) {
-      // Add OAuth filter to the context
-    }
-    return context
-  }
-}
-```
-
-In this example, `MyOAuthImpl` extends `OAuthBase` and provides an implementation for the `configure` method. This
-method should include the logic to configure the `WebAppContext` for OAuth, including setting up any necessary filters
-based on the `addFilter` parameter.
-
-### Conclusion
-
-The `OAuthBase` class provides a foundational structure for integrating OAuth authentication into web applications
-running on the Jetty server. By extending this class and implementing the `configure` method, developers can customize
-the OAuth authentication process to meet their application's requirements.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\LogoutServlet.kt
-
-## LogoutServlet Documentation
-
-The `LogoutServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends the `HttpServlet`
-class provided by the Jakarta Servlet API. This servlet is designed to handle logout requests within a web application,
-ensuring that users can securely sign out of their sessions.
-
-### Overview
-
-When a GET request is made to the servlet's mapped URL, the servlet processes the request by attempting to log out the
-current user based on the session or authentication cookie provided in the request. If the logout process is successful,
-the user is redirected to the application's home page. If the logout process fails (e.g., due to an invalid or missing
-cookie), the servlet responds with a `400 Bad Request` status code.
-
-### Usage
-
-To use the `LogoutServlet`, it must be properly mapped in the web application's deployment descriptor (`web.xml`) or
-through annotations, specifying the URL pattern it should respond to. Once mapped, it will automatically handle GET
-requests to that URL for logging out users.
-
-#### Key Methods
-
-- `doGet(HttpServletRequest req, HttpServletResponse resp)`: This method is overridden from the `HttpServlet` class and
-  is called by the server (via the service method) to allow the servlet to handle a GET request. It performs the logout
-  logic and response handling.
-
-#### Logout Process
-
-1. **Cookie Retrieval**: The servlet first attempts to retrieve the authentication cookie from the incoming request
-   using the `getCookie()` method from `ApplicationServer`.
-2. **User Identification**: Using the retrieved cookie, it then attempts to identify the user by
-   calling `getUser(cookie)` on the `authenticationManager` from `ApplicationServices`.
-3. **Logout and Redirection**:
-    - If no user is associated with the provided cookie (i.e., `user` is `null`), the servlet sets the response status
-      to `400 Bad Request`.
-    - If a user is successfully identified, the servlet proceeds to log out the user by calling `logout(cookie, user)`on
-      the `authenticationManager` and then redirects the user to the home page (`"/"`).
-
-#### Error Handling
-
-- If the logout process fails due to issues such as an invalid or missing cookie, the servlet responds with
-  a `400 Bad Request` status, indicating that the request cannot be processed due to client error.
-
-### Example Deployment Descriptor Configuration
-
-```xml
-
-
-    logoutServlet
-    com.simiacryptus.skyenet.webui.servlet.LogoutServlet
-
-
-logoutServlet
-/logout
-
-```
-
-### Conclusion
-
-The `LogoutServlet` provides a straightforward way to handle user logout requests in a web application, ensuring that
-users can securely end their sessions. By leveraging the `ApplicationServices` for user authentication management, it
-integrates seamlessly with the application's overall security framework.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\OAuthGoogle.kt
-
-## OAuthGoogle Class Documentation
-
-The `OAuthGoogle` class provides a comprehensive solution for integrating Google OAuth2 authentication into Java-based
-web applications. It extends the functionality of the `OAuthBase` class and is designed to facilitate user
-authentication and authorization through Google's OAuth2 services.
-
-### Overview
-
-The class is structured to handle the OAuth2 flow, including the login redirection to Google's authorization service,
-callback handling after successful authentication, and user session management. It leverages the Google Client Library
-for Java to simplify the integration process.
-
-### Key Components
-
-- **LoginServlet**: An inner class that handles the redirection of users to Google's OAuth2 authorization page.
-- **CallbackServlet**: An inner class responsible for handling the callback from Google after the user has authorized
-  the application. It processes the authorization code, exchanges it for an access token, retrieves user information,
-  and manages user sessions.
-- **configure**: A method to configure servlets and optional filters in the web application context for handling login
-  and callback URLs.
-
-### Usage
-
-1. **Initialization**: Create an instance of the `OAuthGoogle` class by providing the redirect URI, application name,
-   and a function to retrieve the Google client secrets input stream.
-
-2. **Configuration**: Invoke the `configure` method on the `OAuthGoogle` instance within your web application's
-   initialization code to set up the necessary servlets and filters.
-
-3. **Authentication Flow**: Direct users to the `/login` or `/googleLogin` endpoint to initiate the OAuth2 flow. After
-   successful authentication and authorization by Google, users will be redirected to the specified redirect URI with an
-   authorization code, which is then processed by the `CallbackServlet`.
-
-4. **Session Management**: Upon successful authentication, a session ID is generated, stored, and sent to the client as
-   a secure HTTP cookie. This session ID can be used for managing user sessions within your application.
-
-### Configuration Parameters
-
-- **redirectUri**: The URI to which Google will redirect users after they have authorized your application. This URI
-  must be registered in the Google Cloud Console.
-- **applicationName**: The name of your application. This should match the name registered in the Google Cloud Console.
-- **key**: A function that returns an `InputStream` of the Google client secrets JSON file.
-
-### Example
-
-```java
-WebAppContext context = new WebAppContext();
-OAuthGoogle googleAuth = new OAuthGoogle(
-        "http://yourdomain.com/oauth2callback",
-        "Your Application Name",
-        () -> getClass().getResourceAsStream("/path/to/client_secrets.json")
-);
-googleAuth.
-
-configure(context, true);
-```
-
-### Dependencies
-
-- Google Client Library for Java
-- Jetty Server (for servlet and filter handling)
-- SLF4J (for logging)
-
-### Important Notes
-
-- Ensure that the Google client secrets JSON file is securely stored and accessible by your application.
-- The redirect URI provided during the `OAuthGoogle` class initialization must exactly match one of the URIs registered
-  in the Google Cloud Console for your application.
-- The session management implementation in the `CallbackServlet` is a basic example. Depending on your application's
-  requirements, you may need to implement more sophisticated session handling mechanisms.
-
-This documentation provides a high-level overview of the `OAuthGoogle` class and its usage. For more detailed
-information, refer to the source code and the Google OAuth2 documentation.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\NewSessionServlet.kt
-
-## NewSessionServlet Documentation
-
-The `NewSessionServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends
-the `HttpServlet` class provided by the Jakarta Servlet API. This servlet is designed to handle HTTP GET requests by
-generating a new global session ID, setting the response content type to plain text, and returning the newly generated
-session ID to the client.
-
-### Usage
-
-This servlet is intended to be used in web applications that require session management. When a client sends a GET
-request to the endpoint where this servlet is mapped, it will receive a unique session ID in plain text format. This
-session ID can then be used for tracking the session or for other purposes as required by the application.
-
-### Implementation Details
-
-#### Import Statements
-
-The servlet imports the following classes:
-
-- `com.simiacryptus.skyenet.core.platform.StorageInterface`: Used to generate a new global session ID.
-- `jakarta.servlet.http.HttpServlet`: The base class for HTTP servlets.
-- `jakarta.servlet.http.HttpServletRequest`: Represents the client's request.
-- `jakarta.servlet.http.HttpServletResponse`: Represents the response that the servlet sends back to the client.
-
-#### Class Definition
-
-`NewSessionServlet` extends `HttpServlet` and overrides the `doGet` method to handle GET requests.
-
-#### doGet Method
-
-The `doGet` method has the following signature:
-
-```kotlin
-override fun doGet(req: HttpServletRequest, resp: HttpServletResponse)
-```
-
-##### Parameters
-
-- `req: HttpServletRequest`: The request sent by the client. This parameter is not used in the current implementation
-  but is available for future enhancements.
-- `resp: HttpServletResponse`: The response that the servlet will send back to the client.
-
-##### Implementation
-
-1. **Generate Session ID**: A new global session ID is generated by calling `StorageInterface.newGlobalID()`.
-2. **Set Content Type**: The content type of the response is set to "text/plain" using `resp.contentType`.
-3. **Set Status Code**: The HTTP status code of the response is set to `HttpServletResponse.SC_OK` (200) to indicate a
-   successful request.
-4. **Return Session ID**: The generated session ID is converted to a string and written to the response
-   using `resp.writer.write(sessionId.toString())`.
-
-### Example Usage
-
-This servlet does not require any specific configuration for its basic functionality. Once deployed and mapped to a URL
-pattern in a web application, it can be accessed by sending a GET request to the corresponding URL. The response will
-contain the generated session ID as plain text.
-
-For example, if the servlet is mapped to `/newSession`, a GET request to `http://yourdomain.com/newSession` will return
-a new session ID.
-
-### Conclusion
-
-The `NewSessionServlet` provides a simple and efficient way to generate and return new session IDs for web applications.
-Its implementation can be extended or modified to suit specific requirements, such as adding authentication or logging
-for session creation.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\SessionIdFilter.kt
-
-## SessionIdFilter Class Documentation
-
-The `SessionIdFilter` class is a custom implementation of the `Filter` interface used in Java Servlet applications. It
-is designed to intercept HTTP requests and perform session validation, particularly focusing on secure pages that
-require user authentication. If a user is not authenticated, the filter redirects them to a login page.
-
-### Package
-
-```java
-package com.simiacryptus.skyenet.webui.servlet;
-```
-
-### Imports
-
-The class uses several imports from the `jakarta.servlet` package, the `java.net` package for URL encoding, and specific
-project imports for application services and utilities.
-
-### Constructor
-
-```java
-SessionIdFilter((HttpServletRequest) ->
-        Boolean isSecure, String
-loginRedirect)
-```
-
-- `isSecure`: A lambda function that takes an `HttpServletRequest` object as input and returns a `Boolean` indicating
-  whether the request targets a secure page that requires authentication.
-- `loginRedirect`: A `String` specifying the URL to redirect unauthenticated users to, typically a login page.
-
-### Methods
-
-#### init
-
-```java
-
-@Override
-void init(FilterConfig filterConfig)
-```
-
-- Initializes the filter. This implementation does not perform any initialization logic.
-
-#### doFilter
-
-```java
-
-@Override
-void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
-```
-
-- Core method of the filter. It intercepts HTTP requests and checks if the request is for a secure page and whether the
-  user is authenticated.
-- If the request is for a secure page (`isSecure` returns `true`) and the user is not authenticated (no valid session ID
-  cookie or no user associated with the session ID), the method redirects the user to the login page specified in
-  the `loginRedirect` parameter, appending the original requested URL as a query parameter for potential redirection
-  after successful authentication.
-- If the user is authenticated or the page is not secure, the filter chain proceeds normally
-  with `chain.doFilter(request, response)`.
-
-#### destroy
-
-```java
-
-@Override
-void destroy()
-```
-
-- Cleans up any resources used by the filter. This implementation does not perform any cleanup logic.
-
-### Usage Example
-
-To use `SessionIdFilter` in a web application, you need to configure it in the `web.xml` file or programmatically in the
-application's initialization code. You must provide the `isSecure` lambda function and the `loginRedirect` URL according
-to your application's requirements.
-
-```java
-FilterRegistration.Dynamic sessionFilter = servletContext.addFilter("SessionIdFilter", new SessionIdFilter(
-        request -> request.getRequestURI().startsWith("/secure"),
-        "/login.jsp"
-));
-sessionFilter.
-
-addMappingForUrlPatterns(EnumSet.of(DispatcherType.REQUEST), true,"/*");
-```
-
-In this example, all requests starting with `/secure` are considered to require authentication, and unauthenticated
-users are redirected to `/login.jsp`.
-
-### Conclusion
-
-The `SessionIdFilter` class provides a flexible way to enforce authentication on secure pages within a Java Servlet
-application. By redirecting unauthenticated users to a login page and allowing for custom logic to determine which pages
-are secure, it helps maintain application security and user session management.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\ProxyHttpServlet.kt
-
-## ProxyHttpServlet Developer Documentation
-
-The `ProxyHttpServlet` class is designed to act as a reverse proxy, specifically tailored to support interactions with
-the OpenAI API. It extends `HttpServlet` and provides a mechanism to forward requests from clients to the OpenAI API,
-potentially modifying requests and responses along the way. This documentation outlines the key functionalities, setup,
-and usage of the `ProxyHttpServlet`.
-
-### Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.servlet`
-- **Dependencies**: Jakarta Servlet API, Apache HttpComponents, Jetty Server, and others for handling HTTP requests and
-  JSON manipulation.
-- **Key Features**:
-    - Asynchronous HTTP client for forwarding requests.
-    - Request and response modification capabilities.
-    - API key management and budget checks.
-    - Detailed logging of proxied requests and responses.
-
-### Setup
-
-To use `ProxyHttpServlet`, ensure you have the necessary dependencies in your project's build configuration. The servlet
-is designed to be deployed on a servlet container or application server that supports the Jakarta Servlet API, such as
-Jetty or Tomcat.
-
-### Key Components
-
-#### Fields
-
-- `targetUrl`: The base URL of the OpenAI API. Defaults to `"https://api.openai.com/v1/"`.
-- `asyncClient`: An instance of `CloseableHttpAsyncClient` used for executing asynchronous HTTP requests to the target
-  URL.
-
-#### Methods
-
-##### `service(HttpServletRequest req, HttpServletResponse resp)`
-
-Overrides the `service` method from `HttpServlet` to handle incoming requests. It performs several key operations:
-
-- Starts an asynchronous context for the request.
-- Validates the API key and checks the budget.
-- Forwards the request to the OpenAI API using the asynchronous HTTP client.
-- Modifies the response based on the application's logic before sending it back to the client.
-
-##### `getProxyRequest(HttpServletRequest req)`
-
-Constructs a `SimpleHttpRequest` to be sent to the OpenAI API. It filters and copies headers from the original request
-and sets the request body.
-
-##### `onResponse(...)`
-
-A hook for modifying the response before it is sent back to the client. By default, it logs the request and response but
-can be overridden for custom behavior.
-
-##### `onRequest(HttpServletRequest req, ByteArray bytes)`
-
-A hook for modifying the request before it is forwarded to the OpenAI API. By default, it returns the request body bytes
-unchanged but can be overridden for custom behavior.
-
-#### Usage Example
-
-The `main` method and the `test` function within the `companion object` provide a basic example of setting up a Jetty
-server with the `ProxyHttpServlet` and a simple test servlet. This setup is intended for demonstration purposes and
-should be adapted for production environments.
-
-### Extending `ProxyHttpServlet`
-
-To customize the behavior of the proxy, you can extend `ProxyHttpServlet` and override the `onRequest` and `onResponse`
-methods. This allows you to modify requests and responses according to your application's specific needs.
-
-### Conclusion
-
-The `ProxyHttpServlet` offers a flexible and powerful way to interact with the OpenAI API, providing mechanisms for
-request and response manipulation, API key management, and detailed logging. By extending and customizing the servlet,
-developers can integrate sophisticated proxy logic into their applications.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\SessionSettingsServlet.kt
-
-## SessionSettingsServlet Documentation
-
-The `SessionSettingsServlet` class is part of the web UI module for managing session-specific settings within an
-application. It extends `HttpServlet` and provides mechanisms to both display and update settings associated with a user
-session.
-
-### Overview
-
-This servlet is designed to interact with an `ApplicationServer` instance to retrieve and store settings. It supports
-two HTTP methods:
-
-- `GET`: To fetch and display the current settings for a given session.
-- `POST`: To update the settings for a given session.
-
-### Dependencies
-
-- `com.simiacryptus.jopenai.util.JsonUtil`: For JSON serialization and deserialization.
-- `com.simiacryptus.skyenet.core.platform`: For session management and authentication.
-- `com.simiacryptus.skyenet.webui.application.ApplicationServer`: To interact with application-level services and data
-  storage.
-- `jakarta.servlet.http.HttpServlet`: Base class for handling HTTP requests.
-
-### Constructor
-
-```java
-public SessionSettingsServlet(ApplicationServer server)
-```
-
-- `server`: An instance of `ApplicationServer` to interact with application services.
-
-### Fields
-
-- `settingsClass`: The class type for the settings. Currently set to `Map.class.java`, but it's designed to be easily
-  changed to match the actual settings class used by the server.
-
-### HTTP Methods
-
-#### doGet(HttpServletRequest req, HttpServletResponse resp)
-
-Handles the HTTP GET request to fetch and display the current settings.
-
-- `req`: The `HttpServletRequest` object that contains the request the client made to the servlet.
-- `resp`: The `HttpServletResponse` object that contains the response the servlet returns to the client.
-
-##### Process Flow
-
-1. Sets the response content type to "text/html".
-2. Checks if the request contains a "sessionId" parameter.
-    - If not, responds with a 400 Bad Request status and a message indicating that a session ID is required.
-    - If a session ID is provided:
-        1. Retrieves the session and user information.
-        2. Fetches the settings associated with the session and user.
-        3. Serializes the settings to JSON and embeds them in an HTML form for display and editing.
-
-#### doPost(HttpServletRequest req, HttpServletResponse resp)
-
-Handles the HTTP POST request to update the settings for a given session.
-
-- `req`: The `HttpServletRequest` object that contains the request the client made to the servlet.
-- `resp`: The `HttpServletResponse` object that contains the response the servlet sends to the client.
-
-##### Process Flow
-
-1. Sets the response content type to "text/html".
-2. Checks if the request contains a "sessionId" parameter.
-    - If not, responds with a 400 Bad Request status and a message indicating that a session ID is required.
-    - If a session ID is provided:
-        1. Retrieves the session information.
-        2. Deserializes the updated settings from the request.
-        3. Stores the updated settings in the data storage associated with the session and user.
-        4. Redirects the user to a confirmation or summary page.
-
-### Usage Example
-
-This servlet is typically mapped to a URL pattern within a web application's deployment descriptor (web.xml) or through
-annotations, allowing it to handle requests to view or update session settings.
-
-```xml
-
-
-    sessionSettingsServlet
-    com.simiacryptus.skyenet.webui.servlet.SessionSettingsServlet
-
-
-sessionSettingsServlet
-/settings
-
-```
-
-This setup enables users to navigate to `/settings` to view or modify their session settings, provided they include a
-valid session ID in their request.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\SessionListServlet.kt
-
-## SessionListServlet Class Documentation
-
-The `SessionListServlet` class is a part of the web UI module designed to handle HTTP GET requests to list user sessions
-in a web application. It extends `HttpServlet` from the Jakarta Servlet API, enabling it to respond to HTTP requests.
-This class is specifically tailored for applications that require session management and user authentication.
-
-### Dependencies
-
-- Jakarta Servlet API
-- Application-specific classes such as `StorageInterface`, `ApplicationServer`, and `authenticationManager`
-- Java Standard Library classes like `SimpleDateFormat`
-
-### Constructor Parameters
-
-- `dataStorage`: An instance of `StorageInterface` used to interact with the application's data storage system.
-- `prefix`: A `String` value representing the URL prefix for session links.
-- `applicationServer`: An instance of `ApplicationServer` used to access application-specific settings and descriptions.
-
-### Key Methods
-
-#### doGet(HttpServletRequest req, HttpServletResponse resp)
-
-This method overrides `doGet` from `HttpServlet` to handle GET requests. It generates an HTML page listing all sessions
-associated with the authenticated user.
-
-##### Parameters
-
-- `req`: The `HttpServletRequest` object that contains the request the client has made to the servlet.
-- `resp`: The `HttpServletResponse` object that contains the response the servlet sends to the client.
-
-##### Functionality
-
-1. **Content Type and Status**: Sets the response content type to "text/html" and the HTTP status code to `SC_OK` (200).
-
-2. **User Authentication**: Retrieves the current user based on a cookie from the request, using
-   the `authenticationManager`.
-
-3. **Session Retrieval**: Fetches a list of sessions associated with the authenticated user from `dataStorage`.
-
-4. **HTML Generation**: Dynamically generates an HTML page that lists the user's sessions, including session names and
-   creation times. The page includes a link for each session that, when clicked, redirects the user to a detailed view
-   of that session.
-
-5. **Styling**: The HTML includes basic CSS for styling the table and its contents.
-
-### Usage Example
-
-To use `SessionListServlet`, an instance of it must be created and registered with a servlet container. This is
-typically done within the application's server setup code. Here's a simplified example:
-
-```java
-import com.simiacryptus.skyenet.core.platform.StorageInterface;
-import com.simiacryptus.skyenet.webui.application.ApplicationServer;
-
-// Assuming storage and appServer are already initialized
-StorageInterface storage = ...;
-ApplicationServer appServer = ...;
-String prefix = "/sessions";
-
-SessionListServlet sessionListServlet = new SessionListServlet(storage, prefix, appServer);
-
-// Register the servlet with a servlet container (e.g., Jetty, Tomcat)
-// This step is specific to the servlet container being used
-```
-
-### Notes
-
-- The `SessionListServlet` class is designed to be flexible and can be adapted to different storage backends and
-  application servers by implementing the `StorageInterface` and `ApplicationServer` interfaces accordingly.
-- The class assumes that user authentication and session management logic are handled elsewhere in the application,
-  specifically by the `authenticationManager` and `dataStorage` components.
-- The generated HTML is intended for simple administrative or debugging purposes and may need to be customized or
-  extended for production use.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\SessionShareServlet.kt
-
-## SessionShareServlet Documentation
-
-The `SessionShareServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and
-extends `jakarta.servlet.http.HttpServlet`. It is designed to facilitate the sharing of user sessions within the Skyenet
-web application framework. This servlet handles HTTP GET requests to generate and share session URLs, allowing users to
-share their application state with others.
-
-### Overview
-
-When a GET request is received, the servlet performs several key operations:
-
-1. **Authentication and Authorization**: It authenticates the user making the request and checks if they are authorized
-   to share sessions.
-2. **Session Validation**: Validates the session URL provided in the request parameters.
-3. **Session Sharing**: If the session can be shared, it either retrieves an existing share ID or generates a new one,
-   then saves the session state to a specified location accessible via the generated/shared URL.
-
-### Key Components
-
-#### Dependencies
-
-- **ApplicationServices**: Provides access to various application-level services like authentication, authorization, and
-  cloud storage interfaces.
-- **StorageInterface**: Used for session ID parsing and JSON data storage/retrieval.
-- **Selenium2S3**: A specialized component for saving web session states, potentially including image loading based on
-  request parameters.
-
-#### Main Methods
-
-##### doGet(HttpServletRequest req, HttpServletResponse resp)
-
-Handles the HTTP GET request. It performs the following operations:
-
-1. **User Authentication**: Retrieves the user based on the session cookie.
-2. **URL and Host Validation**: Checks if the request contains a valid URL and if the host is accepted for sharing.
-3. **Session Sharing Process**:
-    - Checks for an existing share ID and reuses it if valid.
-    - If no valid share ID exists and the user is authorized, generates a new share ID and saves the session state.
-
-Parameters:
-
-- `HttpServletRequest req`: The HTTP request object.
-- `HttpServletResponse resp`: The HTTP response object.
-
-#### Helper Methods
-
-- **url(String appName, String shareId)**: Constructs the share URL based on the application name and share ID.
-- **acceptHost(User user, String host)**: Validates the host against allowed values or checks if the user has admin
-  authorization.
-- **validateUrl(String previousShare)**: Checks if a previously shared URL is still processing or validates its HTTP
-  status.
-
-#### Usage
-
-To use this servlet within your web application, ensure it is properly mapped in your web application's deployment
-descriptor (`web.xml`) or through annotations. When making a GET request to the servlet's mapped URL, include the
-session URL as a parameter. The servlet will respond with either the sharing URL or an error message, depending on the
-request's validity and the user's authorization.
-
-#### Error Handling
-
-The servlet responds with appropriate HTTP status codes and messages in case of errors, such as:
-
-- **400 Bad Request**: If the URL parameter is missing or invalid.
-- **403 Forbidden**: If the user is not authorized to share sessions.
-
-#### Security Considerations
-
-Ensure that user authentication and authorization checks are robust to prevent unauthorized access and sharing of
-sensitive session data.
-
-### Conclusion
-
-The `SessionShareServlet` provides a crucial functionality for session sharing within the Skyenet web application
-framework, leveraging the platform's authentication, authorization, and storage services. Proper implementation and
-usage of this servlet enhance collaboration and usability by allowing users to share their application states securely.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\SessionThreadsServlet.kt
-
-## SessionThreadsServlet Documentation
-
-The `SessionThreadsServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends
-the `HttpServlet` class from the Jakarta Servlet API. This servlet is designed to provide a web interface for monitoring
-thread pool statistics and stack traces for a specific session within the Skyenet web application. It is particularly
-useful for debugging and performance monitoring.
-
-### Overview
-
-When a GET request is made to this servlet with a `sessionId` parameter, it responds with an HTML page displaying
-detailed information about the thread pool associated with the given session. This includes general pool statistics such
-as the number of active threads, pool size, queue size, completed task count, and total task count. Additionally, it
-provides a list of all active threads in the pool, including their names and current stack traces.
-
-### Usage
-
-To use this servlet, it must be deployed as part of a Skyenet web application within a servlet container (e.g., Tomcat,
-Jetty). The servlet is mapped to a specific URL pattern within the web application's deployment descriptor (`web.xml`)or
-through annotations.
-
-#### Request Parameters
-
-- **sessionId**: Required. The unique identifier of the session for which thread pool information is requested.
-
-#### Response
-
-- The response is an HTML document containing two main sections:
-    - **Pool Stats**: Displays various statistics about the thread pool associated with the provided session ID.
-    - **Thread Stacks**: Lists all active threads in the pool, including their names and stack traces.
-
-#### Example Request
-
-```
-GET /path/to/servlet?sessionId=12345
-```
-
-This request would return an HTML page with thread pool statistics and stack traces for the session with ID `12345`.
-
-### Implementation Details
-
-#### Key Components
-
-- **ApplicationServer**: The `server` instance variable of type `ApplicationServer` is used to access application-wide
-  services and configurations.
-- **Session**: Represents the session for which information is being requested. It is instantiated using the `sessionId`
-  parameter from the request.
-- **AuthenticationManager**: Used to authenticate the user making the request based on cookies.
-- **ClientManager**: Responsible for managing thread pools associated with sessions. It is used to retrieve the thread
-  pool for the specified session and user.
-
-#### HTML Response Generation
-
-The servlet generates the response HTML dynamically within the `doGet` method. It uses Kotlin's multi-line string
-feature to embed HTML code directly within the Kotlin source code. CSS styles are included within the `` section
-of the HTML to style the output.
-
-#### Error Handling
-
-If the `sessionId` parameter is not provided in the request, the servlet responds with HTTP status code 400 (Bad
-Request) and a simple error message indicating that the session ID is required.
-
-### Security Considerations
-
-- **Authentication**: The servlet uses the application's authentication manager to verify the identity of the user
-  making the request. Ensure that appropriate authentication checks are in place to prevent unauthorized access to
-  sensitive information.
-- **Input Validation**: The servlet should validate the `sessionId` parameter to prevent injection attacks or attempts
-  to access unauthorized information.
-
-### Conclusion
-
-The `SessionThreadsServlet` provides a valuable tool for monitoring and debugging thread pools associated with specific
-sessions in the Skyenet web application. By offering insights into thread activity and pool statistics, it aids
-developers and administrators in ensuring the application's performance and stability.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\ToolServlet.kt
-
-## ToolServlet Developer Documentation
-
-### Overview
-
-`ToolServlet` is an abstract class extending `HttpServlet` designed to manage and serve custom tools within a web
-application. It allows for the dynamic addition, editing, deletion, and execution of tools, which are defined by
-user-provided Kotlin code and associated metadata. This class is part of a larger application framework aimed at
-providing a flexible environment for tool management and execution.
-
-### Key Components
-
-#### Tool Data Class
-
-- **Purpose**: Represents a tool with its path, API description, interpreter string, and servlet code.
-- **Fields**:
-    - `path`: The URL path associated with the tool.
-    - `openApiDescription`: Metadata describing the tool's API in OpenAPI format.
-    - `interpreterString`: A string that specifies how the tool's code should be interpreted.
-    - `servletCode`: The Kotlin code that implements the tool's functionality.
-
-#### Main Methods
-
-##### doGet(HttpServletRequest?, HttpServletResponse?)
-
-- **Description**: Handles GET requests. It serves different pages based on the query parameters, such as the tool index
-  page, tool details page, edit form, and performs actions like tool deletion.
-- **Parameters**:
-    - `req`: The HttpServletRequest object.
-    - `resp`: The HttpServletResponse object.
-
-##### doPost(HttpServletRequest?, HttpServletResponse?)
-
-- **Description**: Handles POST requests for editing existing tools or importing tools from a JSON file.
-- **Parameters**:
-    - `req`: The HttpServletRequest object.
-    - `resp`: The HttpServletResponse object.
-
-##### service(HttpServletRequest?, HttpServletResponse?)
-
-- **Description**: Overrides the default `service` method to provide custom routing logic. It finds the requested tool
-  and, if authorized, uses a dynamically constructed servlet to handle the request.
-- **Parameters**:
-    - `req`: The HttpServletRequest object.
-    - `resp`: The HttpServletResponse object.
-
-#### Utility Methods
-
-##### indexPage(): String
-
-- **Description**: Generates the HTML content for the index page listing all available tools.
-
-##### toolDetailsPage(tool: Tool): String
-
-- **Description**: Generates the HTML content for the details page of a specific tool.
-
-##### serveEditPage(HttpServletRequest, HttpServletResponse, Tool)
-
-- **Description**: Serves the edit page for a specific tool, allowing users to modify its properties.
-
-### Security
-
-The servlet checks for user authentication and authorization before allowing access to tool management functionalities.
-It ensures that only users with admin rights can add, edit, or delete tools.
-
-### Extensibility
-
-Developers can extend `ToolServlet` to implement additional functionalities specific to their application's
-requirements. The dynamic nature of tool management allows for a high degree of customization and extensibility.
-
-### Example Usage
-
-To use `ToolServlet`, one must define a concrete implementation that provides specific functionalities for tool
-management. This involves creating a subclass that might override some methods or add new ones, depending on the
-application's needs.
-
-```kotlin
-class MyToolServlet(app: ApplicationDirectory) : ToolServlet(app) {
-  // Implement additional methods or override existing ones
-}
-```
-
-This class can then be registered with the web application's servlet container to handle requests for the tools' URLs.
-
-### Conclusion
-
-`ToolServlet` provides a robust framework for managing and executing custom tools within a web application. Its flexible
-design and security features make it suitable for applications requiring dynamic tool management and execution
-capabilities.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\UserSettingsServlet.kt
-
-## UserSettingsServlet Documentation
-
-The `UserSettingsServlet` class is a part of the web UI module designed to handle user settings within a web
-application. It extends `HttpServlet` and overrides the `doGet` and `doPost` methods to provide functionality for
-retrieving and updating user settings, respectively. This servlet is designed to work within an application that
-utilizes a cookie-based authentication system and a settings management system.
-
-### Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.servlet`
-- **Dependencies**:
-    - `com.simiacryptus.jopenai.util.JsonUtil` for JSON serialization and deserialization.
-    - `com.simiacryptus.skyenet.core.platform.*` for accessing application services like authentication and user
-      settings management.
-    - `jakarta.servlet.http.*` for servlet functionalities.
-
-### Functionality
-
-#### doGet(HttpServletRequest req, HttpServletResponse resp)
-
-This method is invoked when an HTTP GET request is made to the servlet's URL. It is responsible for displaying the
-current user settings in a web form.
-
-- **Parameters**:
-    - `HttpServletRequest req`: The request object containing the request data.
-    - `HttpServletResponse resp`: The response object used to send data back to the client.
-
-- **Process**:
-    1. Sets the response content type to `text/html`.
-    2. Retrieves the user information based on the cookie provided in the request.
-    3. If the user is not authenticated, it sets the response status to `SC_BAD_REQUEST`.
-    4. For authenticated users, it retrieves the user settings, masks sensitive information, and displays them in a form
-       within an HTML page.
-
-- **Sensitive Information Handling**:
-    - API keys are masked with a predefined mask value for security.
-    - API base URLs are defaulted to `https://api.openai.com/v1` if not specified.
-
-#### doPost(HttpServletRequest req, HttpServletResponse resp)
-
-This method is invoked when an HTTP POST request is made to the servlet's URL, typically when the user submits the
-settings form.
-
-- **Parameters**:
-    - `HttpServletRequest req`: The request object containing the form data.
-    - `HttpServletResponse resp`: The response object used to send data back to the client.
-
-- **Process**:
-    1. Retrieves the user information based on the cookie provided in the request.
-    2. If the user is not authenticated, it sets the response status to `SC_BAD_REQUEST`.
-    3. For authenticated users, it parses the updated settings from the request, reconciles them with the previous
-       settings (especially for masked fields like API keys), and updates the user settings in the system.
-    4. Redirects the user to the root URL upon successful update.
-
-- **Sensitive Information Handling**:
-    - Handles the masking and unmasking of API keys to ensure that actual values are preserved unless explicitly changed
-      by the user.
-
-### Security Considerations
-
-- The servlet relies on cookie-based authentication. Ensure that cookies are secured and handled properly to prevent
-  unauthorized access.
-- Sensitive information like API keys is masked in the user interface to prevent exposure. Care should be taken to
-  ensure that this masking is securely implemented.
-
-### Usage
-
-This servlet is intended to be mapped to a specific URL pattern within a web application. Users can navigate to this URL
-to view and update their settings. The actual URL mapping and integration depend on the web application's configuration.
-
-### Conclusion
-
-The `UserSettingsServlet` provides a secure and user-friendly interface for managing user settings within a web
-application. It ensures that sensitive information is handled carefully while offering users the ability to update their
-settings as needed.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\WelcomeServlet.kt
-
-## WelcomeServlet Class Documentation
-
-The `WelcomeServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends the `HttpServlet`
-class to serve as the entry point for handling HTTP GET and POST requests for the SkyeNet web application. This servlet
-is responsible for rendering the homepage, user information, user settings, and serving static resources based on the
-request URI.
-
-### Constructor
-
-- `WelcomeServlet(ApplicationDirectory parent)`: Initializes a new instance of the `WelcomeServlet` class with a
-  reference to the `ApplicationDirectory` that contains it. This reference is used to access other servlets and
-  resources within the application.
-
-### Methods
-
-#### doGet(HttpServletRequest req, HttpServletResponse resp)
-
-Handles HTTP GET requests by determining the request URI and responding with the appropriate content. The method
-supports rendering the homepage, redirecting to the user information page, and serving static resources.
-
-- **Parameters:**
-    - `HttpServletRequest req`: The request object containing the client's request information.
-    - `HttpServletResponse resp`: The response object used to send data back to the client.
-
-#### doPost(HttpServletRequest req, HttpServletResponse resp)
-
-Handles HTTP POST requests, specifically for updating user settings. If the request URI starts with `/userSettings`, it
-forwards the request to the `userSettingsServlet`. Otherwise, it responds with a 404 error.
-
-- **Parameters:**
-    - `HttpServletRequest req`: The request object containing the client's request information.
-    - `HttpServletResponse resp`: The response object used to send data back to the client.
-
-#### homepage(User user)
-
-Generates the HTML content for the homepage, including the navigation bar, application list, and footer. It dynamically
-renders markdown content for the welcome message and post-application list message.
-
-- **Parameters:**
-    - `User user`: The current user object, which may be `null` if the user is not authenticated.
-
-- **Returns:** A `String` containing the HTML content for the homepage.
-
-#### appRow(ApplicationDirectory.ChildWebApp app, User user)
-
-Generates the HTML content for a row in the application list table. It checks if the user is authorized to access the
-application and displays links for listing sessions and creating new sessions.
-
-- **Parameters:**
-    - `ApplicationDirectory.ChildWebApp app`: The application to generate a row for.
-    - `User user`: The current user object, which may be `null` if the user is not authenticated.
-
-- **Returns:** A `String` containing the HTML content for an application row in the list.
-
-### Fields
-
-- `protected open val welcomeMarkdown`: A markdown string that is rendered on the homepage above the application list.
-- `protected open val postAppMarkdown`: A markdown string that is rendered on the homepage below the application list.
-
-### Usage
-
-To use the `WelcomeServlet`, it must be registered with a servlet container (e.g., Tomcat, Jetty) in the web
-application's deployment descriptor (`web.xml`) or programmatically through a `ServletContext`. Once registered, it will
-respond to HTTP GET and POST requests at its designated URL pattern.
-
-### Security
-
-The servlet uses the `ApplicationServices.authorizationManager` to check if the current user is authorized to perform
-specific operations, such as reading or writing to an application. It ensures that sensitive actions and information are
-protected and only accessible to authorized users.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\UserInfoServlet.kt
-
-## UserInfoServlet Documentation
-
-The `UserInfoServlet` class is a part of the web UI module designed to handle HTTP GET requests to retrieve user
-information in JSON format. This servlet extends `HttpServlet` from the Jakarta Servlet API, enabling it to respond to
-HTTP requests within a web application.
-
-### Package
-
-```plaintext
-com.simiacryptus.skyenet.webui.servlet
-```
-
-### Dependencies
-
-- `com.simiacryptus.jopenai.util.JsonUtil`: Utilized for converting user objects to JSON strings.
-- `com.simiacryptus.skyenet.core.platform.ApplicationServices`: Provides access to application-wide services, including
-  authentication management.
-- `com.simiacryptus.skyenet.core.platform.User`: Represents the user entity whose information is to be retrieved.
-- `com.simiacryptus.skyenet.webui.application.ApplicationServer.Companion.getCookie`: A helper method to extract cookies
-  from the request, used for authentication purposes.
-- `jakarta.servlet.http.HttpServlet`: The base class for HTTP servlets.
-- `jakarta.servlet.http.HttpServletRequest`: Represents the client's request.
-- `jakarta.servlet.http.HttpServletResponse`: Represents the response that the servlet sends to the client.
-
-### Class Overview
-
-#### UserInfoServlet
-
-A servlet class designed to handle HTTP GET requests by providing user information in JSON format. It checks the user's
-authentication status using a cookie and returns the user's details if authenticated.
-
-##### Methods
-
-- `doGet(HttpServletRequest req, HttpServletResponse resp)`: Handles the GET request by retrieving the user's
-  information based on the authentication cookie and responding with the user details in JSON format.
-
-### Usage
-
-1. **Deployment**: This servlet needs to be deployed as part of a Java web application running in a servlet container (
-   e.g., Tomcat, Jetty).
-2. **Configuration**: Ensure that the servlet is mapped to a URL pattern in the web application's deployment
-   descriptor (`web.xml`) or through annotations.
-3. **Request Handling**: When a GET request is made to the servlet's mapped URL, the servlet attempts to authenticate
-   the user based on a cookie.
-4. **Response**:
-    - If the user is not authenticated or the cookie is invalid, the servlet responds with an empty JSON object (`{}`).
-    - If the user is authenticated, the servlet responds with the user's information in JSON format.
-
-### Example Response
-
-For an authenticated user, the response might look like:
-
-```json
-{
-  "id": "12345",
-  "username": "johndoe",
-  "email": "johndoe@example.com"
-}
-```
-
-For an unauthenticated request, the response will be:
-
-```json
-{}
-```
-
-### Security Considerations
-
-- Ensure that sensitive user information is protected and not exposed unnecessarily.
-- Validate and sanitize the cookie to prevent security vulnerabilities such as injection attacks.
-- Implement proper error handling to avoid leaking information through error messages.
-
-### Conclusion
-
-The `UserInfoServlet` provides a straightforward way to retrieve authenticated user information in a web application. By
-leveraging cookies for authentication, it ensures that user details are only provided to authenticated sessions,
-enhancing the application's security.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\UsageServlet.kt
-
-## UsageServlet Documentation
-
-The `UsageServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends the `HttpServlet`
-class to provide a web interface for displaying usage statistics related to OpenAI models. This servlet is designed to
-present users with a summary of their usage, including the number of prompt and completion tokens used, as well as the
-associated costs.
-
-### Overview
-
-When a GET request is made to this servlet, it determines whether the request includes a `sessionId` parameter. If
-present, it fetches the session-specific usage summary; otherwise, it attempts to retrieve the user-specific usage
-summary based on the user's cookie. The usage information is then displayed in an HTML table format.
-
-### Key Methods
-
-#### doGet(HttpServletRequest req, HttpServletResponse resp)
-
-This method is overridden from the `HttpServlet` class and is called by the server to allow the servlet to handle a GET
-request.
-
-- **Parameters:**
-    - `HttpServletRequest req`: The request object containing client request data.
-    - `HttpServletResponse resp`: The response object used for sending data back to the client.
-
-- **Functionality:**
-    - Sets the content type of the response to "text/html".
-    - Checks if the request contains a `sessionId` parameter:
-        - If yes, retrieves the session usage summary and calls the `serve` method.
-        - If no, retrieves the user information from the cookie and fetches the user usage summary, then calls
-          the `serve` method.
-    - If the user information is not found, sets the response status to `SC_BAD_REQUEST`.
-
-#### serve(HttpServletResponse resp, Map usage)
-
-This private method generates and sends an HTML response containing a summary of usage statistics.
-
-- **Parameters:**
-    - `HttpServletResponse resp`: The response object used for sending data back to the client.
-    - `Map usage`: A map containing usage statistics for each OpenAI model.
-
-- **Functionality:**
-    - Calculates total prompt tokens, completion tokens, and cost from the usage map.
-    - Constructs an HTML page displaying the usage statistics in a table format.
-    - Writes the constructed HTML to the response writer.
-
-### Usage Example
-
-To use `UsageServlet`, it must be mapped to a URL pattern in the web application's deployment descriptor (web.xml) or
-through annotations. Once mapped, it can be accessed by sending a GET request to the configured URL, optionally
-including a `sessionId` parameter to retrieve session-specific usage statistics.
-
-### Security Considerations
-
-- Ensure that access to usage statistics is properly secured to prevent unauthorized access.
-- Validate and sanitize the `sessionId` parameter to prevent injection attacks.
-
-### Dependencies
-
-- `com.simiacryptus.jopenai.models.OpenAIModel`
-- `com.simiacryptus.skyenet.core.platform.ApplicationServices`
-- `com.simiacryptus.skyenet.core.platform.Session`
-- `com.simiacryptus.skyenet.webui.application.ApplicationServer`
-- `jakarta.servlet.http.HttpServlet`
-- `jakarta.servlet.http.HttpServletRequest`
-- `jakarta.servlet.http.HttpServletResponse`
-
-This documentation provides an overview of the `UsageServlet` class functionality and usage within a web application
-context.
-
-# kotlin\com\simiacryptus\skyenet\webui\servlet\ZipServlet.kt
-
-## ZipServlet Class Documentation
-
-The `ZipServlet` class is part of the `com.simiacryptus.skyenet.webui.servlet` package and extends the `HttpServlet`
-class to provide functionality for dynamically creating and serving ZIP files based on files stored within a
-session-specific directory. This servlet interacts with a storage interface to access session data and generates a ZIP
-file containing the requested files, which is then sent back to the client.
-
-### Dependencies
-
-- `com.simiacryptus.skyenet.core.platform.*`: Utilizes classes for session management, authentication, and storage
-  interaction.
-- `jakarta.servlet.http.*`: For handling HTTP servlet requests and responses.
-- `java.io.File`: For file manipulation.
-- `java.util.zip.*`: For creating ZIP files.
-
-### Constructor
-
-```kotlin
-ZipServlet(val dataStorage : StorageInterface)
-```
-
-- **Parameters:**
-    - `dataStorage`: An implementation of `StorageInterface` to interact with the application's data storage.
-
-### Public Methods
-
-#### `doGet(HttpServletRequest req, HttpServletResponse resp)`
-
-Handles the GET request by creating a ZIP file containing the requested files from the session directory and sending it
-to the client.
-
-- **Parameters:**
-    - `req`: The `HttpServletRequest` object that contains the request the client made to the servlet.
-    - `resp`: The `HttpServletResponse` object that contains the response the servlet returns to the client.
-
-- **Process Flow:**
-    1. Extracts the session ID and path from the request parameters.
-    2. Validates the requested path.
-    3. Determines the session directory based on the session ID and authenticated user.
-    4. Creates a temporary ZIP file and writes the requested files into it.
-    5. Sets the response content type to `application/zip` and sends the ZIP file content back to the client.
-
-### Private Methods
-
-#### `write(basePath: File, file: File, zip: ZipOutputStream)`
-
-Recursively writes files and directories from the specified path into the ZIP output stream.
-
-- **Parameters:**
-    - `basePath`: The base directory from which the relative paths of files in the ZIP are calculated.
-    - `file`: The current file or directory to add to the ZIP file.
-    - `zip`: The `ZipOutputStream` to which the file data is written.
-
-- **Functionality:**
-    - If the `file` is a regular file, it is added to the ZIP file with its relative path from `basePath`.
-    - If the `file` is a directory, the method is called recursively for each file within the directory, excluding
-      hidden files (those starting with a dot).
-
-### Usage Example
-
-To use `ZipServlet`, it must be mapped to a URL pattern in your web application's deployment descriptor or through
-annotations. Once mapped, it can be accessed by sending a GET request to the mapped URL with the required
-parameters (`session` and optionally `path`).
-
-```kotlin
-val storageInterface: StorageInterface = // Obtain an implementation of StorageInterface
-val zipServlet = ZipServlet(storageInterface)
-```
-
-Ensure that the servlet is properly initialized and configured within your web application to handle requests.
-
-### Security Considerations
-
-- Ensure that proper authentication and authorization checks are performed to prevent unauthorized access to sensitive
-  files.
-- Validate and sanitize input parameters (`session` and `path`) to avoid path traversal vulnerabilities.
-
-This documentation provides an overview of the `ZipServlet` class's functionality and usage within a web application for
-dynamically serving ZIP files based on session-specific data.
-
-# kotlin\com\simiacryptus\skyenet\webui\session\SocketManager.kt
-
-## SocketManager Interface Documentation
-
-The `SocketManager` interface is a crucial component within the `com.simiacryptus.skyenet.webui.session` package,
-designed to manage WebSocket connections for chat functionalities. It provides a structured way to handle chat sockets,
-including adding and removing sockets, handling incoming text messages, and retrieving chat history.
-
-### Interface Overview
-
-```kotlin
-package com.simiacryptus.skyenet.webui.session
-
-import com.simiacryptus.skyenet.webui.chat.ChatSocket
-import org.eclipse.jetty.websocket.api.Session
-
-interface SocketManager {
-  fun removeSocket(socket: ChatSocket)
-  fun addSocket(socket: ChatSocket, session: Session)
-  fun getReplay(): List
-  fun onWebSocketText(socket: ChatSocket, message: String)
-}
-```
-
-### Methods
-
-#### `removeSocket(socket: ChatSocket)`
-
-Removes a specified `ChatSocket` from the manager. This method is typically called when a WebSocket connection is closed
-or needs to be terminated for any reason.
-
-- **Parameters:**
-    - `socket: ChatSocket` - The chat socket instance to be removed.
-
-#### `addSocket(socket: ChatSocket, session: Session)`
-
-Adds a new `ChatSocket` to the manager, associating it with a specific `Session`. This method is usually called when a
-new WebSocket connection is established.
-
-- **Parameters:**
-    - `socket: ChatSocket` - The chat socket instance to be added.
-    - `session: Session` - The session associated with the WebSocket connection.
-
-#### `getReplay(): List`
-
-Retrieves the chat history. This method returns a list of strings, each representing a message in the chat history. This
-can be used to replay chat messages to a newly connected client.
-
-- **Returns:** A `List` containing the chat history.
-
-#### `onWebSocketText(socket: ChatSocket, message: String)`
-
-Handles incoming text messages from a WebSocket connection. This method is called whenever a text message is received
-from a client.
-
-- **Parameters:**
-    - `socket: ChatSocket` - The chat socket through which the message was received.
-    - `message: String` - The text message received from the client.
-
-### Usage Example
-
-Below is a hypothetical example of how the `SocketManager` interface might be implemented and used within a chat
-application:
-
-```kotlin
-class ChatSocketManager : SocketManager {
-  private val sockets = mutableListOf()
-  private val chatHistory = mutableListOf()
-
-  override fun addSocket(socket: ChatSocket, session: Session) {
-    sockets.add(socket)
-    // Optionally, send chat history to the newly connected client
-    socket.sendMessages(getReplay())
-  }
-
-  override fun removeSocket(socket: ChatSocket) {
-    sockets.remove(socket)
-  }
-
-  override fun getReplay(): List = chatHistory
-
-  override fun onWebSocketText(socket: ChatSocket, message: String) {
-    chatHistory.add(message)
-    // Broadcast the message to all connected clients
-    sockets.forEach { it.sendMessage(message) }
-  }
-}
-```
-
-This example demonstrates a basic implementation of the `SocketManager` interface, managing chat sockets, maintaining a
-chat history, and broadcasting messages to all connected clients.
-
-### Conclusion
-
-The `SocketManager` interface plays a fundamental role in managing WebSocket connections for chat functionalities within
-the application. By defining a clear contract for adding, removing, and handling chat sockets, it facilitates the
-development of robust and scalable chat features.
-
-# kotlin\com\simiacryptus\skyenet\webui\session\SocketManagerBase.kt
-
-## SocketManagerBase Class Documentation
-
-The `SocketManagerBase` class is an abstract class designed to manage WebSocket connections for a chat application,
-handling message sending, receiving, and authorization. It serves as a foundation for building WebSocket managers that
-require session management, user authentication, and message handling capabilities.
-
-### Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.session`
-- **Imports**: Various, including platform services, chat utilities, and standard Java utilities.
-- **Inheritance**: Implements the `SocketManager` interface.
-
-### Key Components
-
-#### Fields
-
-- `session`: Represents the current user session.
-- `dataStorage`: Interface for data storage operations, nullable.
-- `owner`: The user who owns this session, nullable.
-- `messageStates`: A map storing the state of messages.
-- `applicationClass`: The class of the application for authorization purposes.
-- `sockets`: A map linking `ChatSocket` instances to Jetty WebSocket sessions.
-- `sendQueues`: A map managing message queues for each `ChatSocket`.
-- `messageVersions`: Tracks version numbers for messages to manage updates.
-
-#### Constructor
-
-The constructor initializes the class with a session, optional data storage, an owner, a map of message states, and the
-application class. It retrieves initial message states from the data storage if available.
-
-#### Methods
-
-##### Public
-
-- `removeSocket(socket: ChatSocket)`: Removes a WebSocket from the manager.
-- `addSocket(socket: ChatSocket, session: org.eclipse.jetty.websocket.api.Session)`: Adds a WebSocket to the manager
-  after authorization checks.
-- `send(out: String)`: Sends a message to all connected WebSockets.
-- `getReplay()`: Retrieves a list of all messages with their current state and version.
-- `newTask(cancelable: Boolean = false)`: Initializes a new task, optionally cancelable, and returns a `SessionTask`
-  instance.
-- `hrefLink(linkText: String, classname: String, handler: Consumer)`: Generates HTML for a hyperlink that triggers
-  a specified action.
-- `textInput(handler: Consumer)`: Generates HTML for a text input form that triggers a specified action.
-
-##### Protected
-
-- `onRun(userMessage: String, socket: ChatSocket)`: Abstract method to be implemented by subclasses, defining behavior
-  for incoming messages.
-- `canWrite(user: User?)`: Checks if the given user has write access.
-
-##### Private
-
-- `publish(out: String)`: Internal method to queue messages for sending.
-- `setMessage(key: String, value: String)`: Updates or sets the value of a message, returning its new version.
-- `onCmd(id: String, code: String)`: Handles special command messages.
-
-#### Inner Classes
-
-- `SessionTaskImpl`: Implementation of `SessionTask`, handling task-specific operations like sending messages and saving
-  files.
-
-### Utility Functions
-
-- `randomID()`: Generates a random ID string.
-- `divInitializer(operationID: String, cancelable: Boolean)`: Generates initial HTML for a task, including a cancel
-  button if requested.
-- `getUser(session: org.eclipse.jetty.websocket.api.Session)`: Retrieves the `User` associated with a WebSocket session.
-
-### Usage
-
-This class is designed to be extended by specific WebSocket manager implementations that require session and user
-management, message handling, and authorization. Implementors will need to provide functionality for the
-abstract `onRun` method to define how incoming messages are processed.
-
-### Example
-
-```kotlin
-class MySocketManager(session: Session, dataStorage: StorageInterface?, owner: User?) :
-  SocketManagerBase(session, dataStorage, owner) {
-  override fun onRun(userMessage: String, socket: ChatSocket) {
-    // Implementation for handling user messages
-  }
-}
-```
-
-This example demonstrates how to extend `SocketManagerBase` to create a custom WebSocket manager that processes user
-messages according to application-specific logic.
-
-# kotlin\com\simiacryptus\skyenet\webui\test\ImageActorTestApp.kt
-
-## ImageActorTestApp Developer Documentation
-
-The `ImageActorTestApp` class is an extension of the `ApplicationServer` designed to facilitate testing and interaction
-with `ImageActor` instances within a web application context. This class allows users to send messages to
-an `ImageActor` and receive responses, including text and images, through a web interface.
-
-### Overview
-
-- **Package**: `com.simiacryptus.skyenet.webui.test`
-- **Imports**: Utilizes a variety of imports from `com.simiacryptus.jopenai`, `com.simiacryptus.skyenet.core`,
-  and `com.simiacryptus.skyenet.webui` packages, among others.
-- **Dependencies**: Requires an instance of `ImageActor`, `ApplicationInterface`, and `API` for its operations.
-
-### Key Components
-
-#### Constructor
-
-The constructor initializes the `ImageActorTestApp` with a specific `ImageActor` instance and optional parameters for
-application name and temperature. The application name defaults to "ImageActorTest_" followed by the simple name of
-the `ImageActor` class. The temperature parameter influences the behavior of the `ImageActor` but is set to 0.3 by
-default.
-
-```kotlin
-ImageActorTestApp(
-  private val actor : ImageActor,
-applicationName: String = "ImageActorTest_" + actor.javaClass.simpleName,
-temperature: Double = 0.3,
-)
-```
-
-#### Settings Data Class
-
-Defines a data class `Settings` with a nullable `ImageActor` property. This class is used to manage application-specific
-settings.
-
-```kotlin
-data class Settings(
-  val actor: ImageActor? = null,
-)
-```
-
-#### Overridden Methods
-
-##### initSettings
-
-Initializes settings for a session by returning an instance of the `Settings` data class with the `ImageActor` specified
-at construction.
-
-```kotlin
-override fun  initSettings(session: Session): T = Settings(actor = actor) as T
-```
-
-##### userMessage
-
-Handles messages from users. It sets a budget for the API client, processes the user message, sends it to
-the `ImageActor`, and then displays the response through the UI. It also handles any errors that occur during this
-process.
-
-```kotlin
-override fun userMessage(
-  session: Session,
-  user: User?,
-  userMessage: String,
-  ui: ApplicationInterface,
-  api: API
-)
-```
-
-#### Companion Object
-
-Contains a logger for the class, used to log warnings and errors.
-
-```kotlin
-companion object {
-  private val log = LoggerFactory.getLogger(ImageActorTestApp::class.java)
-}
-```
-
-### Usage
-
-To use `ImageActorTestApp`, an instance of `ImageActor` must be provided. This instance is then used to interact with
-users through a web interface, where users can send messages and receive responses that include both text and images.
-
-### Error Handling
-
-Errors during message processing are caught and logged using the class's logger. Additionally, error messages are
-displayed to the user through the UI.
-
-### Conclusion
-
-The `ImageActorTestApp` class provides a structured way to test and interact with `ImageActor` instances within a web
-application, facilitating the development and debugging of image-based conversational agents.
-
-# kotlin\com\simiacryptus\skyenet\webui\test\CodingActorTestApp.kt
-
-## CodingActorTestApp Class Documentation
-
-The `CodingActorTestApp` class extends the `ApplicationServer` to create a specialized server application designed to
-test `CodingActor` instances. This application allows users to send messages, which are then processed by
-the `CodingActor` to generate code responses. These responses can be executed if the user has the necessary permissions.
-
-### Constructor
-
-```kotlin
-CodingActorTestApp(
-  private val actor : CodingActor,
-applicationName: String = "CodingActorTest_" + actor.name,
-temperature: Double = 0.3,
-)
-```
-
-#### Parameters:
-
-- `actor`: The `CodingActor` instance that will be used to generate code responses.
-- `applicationName`: The name of the application. Defaults to "CodingActorTest_" followed by the name of the actor.
-- `temperature`: A parameter influencing the randomness of the generated code. Lower values make the code more
-  deterministic.
-
-### Methods
-
-#### userMessage
-
-```kotlin
-override fun userMessage(
-  session: Session,
-  user: User?,
-  userMessage: String,
-  ui: ApplicationInterface,
-  api: API
-)
-```
-
-Handles messages sent by users. It processes the user message through the `CodingActor`, checks if the user has
-execution permissions, and displays the generated code along with an optional execution link.
-
-##### Parameters:
-
-- `session`: The current user session.
-- `user`: The user sending the message. Can be `null`.
-- `userMessage`: The message sent by the user.
-- `ui`: The application interface used to interact with the user interface.
-- `api`: The API instance used for making external API calls.
-
-### Usage
-
-1. **Initialization**: Create an instance of `CodingActorTestApp` by providing a `CodingActor` instance and optionally
-   specifying the application name and temperature.
-
-2. **User Interaction**: The application listens for messages from users. When a message is received, it is processed as
-   follows:
-    - The message is echoed back to the user.
-    - The `CodingActor` generates a code response based on the user message.
-    - The application checks if the user has permission to execute the generated code.
-    - If execution is permitted, a link is provided to execute the code, and the results are displayed.
-
-3. **Error Handling**: If an error occurs during message processing, it is logged, and an error message is displayed to
-   the user.
-
-### Companion Object
-
-- `log`: A logger instance for logging error messages.
-
-### Example
-
-```kotlin
-val codingActor = CodingActor(/* initialization parameters */)
-val testApp = CodingActorTestApp(codingActor)
-testApp.start()
-```
-
-This example creates a new instance of `CodingActorTestApp` with a specified `CodingActor` and starts the application
-server to listen for user messages.
-
-### Note
-
-This class requires external dependencies such as `ApplicationServer`, `CodingActor`, and `API` to be properly set up in
-your project environment. Ensure that all necessary permissions and configurations are in place for the application to
-function correctly.
-
-# kotlin\com\simiacryptus\skyenet\webui\test\ParsedActorTestApp.kt
-
-## ParsedActorTestApp Class Documentation
-
-The `ParsedActorTestApp` class is a specialized application server designed for testing `ParsedActor` instances within
-the Skyenet framework. It extends the `ApplicationServer` class, providing a web interface to interact with and test the
-responses of a given `ParsedActor`.
-
-### Overview
-
-This class is designed to facilitate the testing of `ParsedActor` instances by allowing users to send messages through a
-web interface and view the responses generated by the `ParsedActor`. It integrates with the Skyenet core platform and
-utilizes the JOpenAI library for processing natural language inputs.
-
-### Constructor
-
-```kotlin
-ParsedActorTestApp < T : Any > (
-    private
-val actor: ParsedActor,
-applicationName: String = "ParsedActorTest_" + actor.resultClass.simpleName,
-temperature: Double = 0.3,
-)
-```
-
-#### Parameters
-
-- `actor`: The `ParsedActor` instance to be tested.
-- `applicationName`: (Optional) The name of the application, defaulting to "ParsedActorTest_" followed by the simple
-  name of the result class of the actor.
-- `temperature`: (Optional) A parameter influencing the randomness of the response generation, defaulting to 0.3.
-
-### Methods
-
-#### userMessage
-
-```kotlin
-override fun userMessage(
-  session: Session,
-  user: User?,
-  userMessage: String,
-  ui: ApplicationInterface,
-  api: API
-)
-```
-
-Handles messages sent by users through the web interface. It processes the user's message using the `ParsedActor`,
-generates a response, and displays it back to the user.
-
-##### Parameters
-
-- `session`: The current session object.
-- `user`: The user object, which can be `null`.
-- `userMessage`: The message string sent by the user.
-- `ui`: The application interface for interacting with the UI.
-- `api`: The API client used for processing the message.
-
-### Companion Object
-
-#### Properties
-
-- `log`: A logger instance for logging warnings and errors.
-
-### Usage Example
-
-To use `ParsedActorTestApp`, you first need to instantiate a `ParsedActor` with the desired configuration. Then, create
-an instance of `ParsedActorTestApp` with the `ParsedActor` as a parameter. Finally, start the application server to
-begin testing.
-
-```kotlin
-val myActor = ParsedActor(...)
-val testApp = ParsedActorTestApp(myActor)
-testApp.start()
-```
-
-This will start a web server where users can send messages to be processed by the `ParsedActor`, and view the generated
-responses.
-
-### Conclusion
-
-The `ParsedActorTestApp` class provides a convenient way to test and interact with `ParsedActor` instances, making it
-easier for developers to debug and refine their natural language processing applications within the Skyenet framework.
-
-# kotlin\com\simiacryptus\skyenet\webui\test\SimpleActorTestApp.kt
-
-## SimpleActorTestApp Documentation
-
-The `SimpleActorTestApp` class is a part of the `com.simiacryptus.skyenet.webui.test` package, designed to integrate
-a `SimpleActor` into a web-based application server environment. This class extends the `ApplicationServer` to provide a
-specialized application that interacts with users through a web interface, leveraging the capabilities of
-a `SimpleActor` for processing user messages.
-
-### Overview
-
-The `SimpleActorTestApp` class is designed to facilitate testing and interaction with instances of `SimpleActor`. It
-provides a web UI for users to send messages to the `SimpleActor`, and receive responses. This setup is useful for
-debugging, testing, or demonstrating the capabilities of `SimpleActor` implementations.
-
-### Key Components
-
-#### Constructor
-
-The constructor initializes the application with a specific `SimpleActor` instance, an optional application name, and a
-temperature parameter for the actor's response generation process.
-
-```kotlin
-SimpleActorTestApp(
-  private val actor : SimpleActor,
-applicationName: String = "SimpleActorTest_" + actor.javaClass.simpleName,
-temperature: Double = 0.3
-)
-```
-
-- `actor`: The `SimpleActor` instance to be used for answering user messages.
-- `applicationName`: An optional name for the application, defaulting to "SimpleActorTest_" followed by the simple name
-  of the actor's class.
-- `temperature`: A parameter influencing the randomness of the actor's responses, with a default value of 0.3.
-
-#### Settings Data Class
-
-The `Settings` data class encapsulates the configuration settings for the application, currently only including
-the `SimpleActor` instance.
-
-```kotlin
-data class Settings(
-  val actor: SimpleActor? = null,
-)
-```
-
-#### User Message Handling
-
-The `userMessage` method is overridden to process messages from users. It sets a budget for the API client, sends the
-user's message to the `SimpleActor`, and returns the actor's response.
-
-```kotlin
-override fun userMessage(
-  session: Session,
-  user: User?,
-  userMessage: String,
-  ui: ApplicationInterface,
-  api: API
-)
-```
-
-- `session`: The current user session.
-- `user`: The user sending the message, which can be `null`.
-- `userMessage`: The message from the user.
-- `ui`: The application interface for interacting with the UI.
-- `api`: The API client for external communications.
-
-#### Logging
-
-A companion object provides a logger for the class, facilitating logging throughout the application lifecycle.
-
-```kotlin
-companion object {
-  private val log = LoggerFactory.getLogger(SimpleActorTestApp::class.java)
-}
-```
-
-### Usage
-
-To use `SimpleActorTestApp`, instantiate it with a `SimpleActor` and optionally specify an application name and
-temperature. Deploy the application on a server capable of running Kotlin applications, and navigate to the specified
-path (`/simpleActorTest` by default) to interact with the `SimpleActor` through the web UI.
-
-### Conclusion
-
-The `SimpleActorTestApp` class provides a convenient way to test and demonstrate the capabilities of `SimpleActor`
-instances within a web application context. By extending the `ApplicationServer`, it integrates seamlessly with web UI
-components, allowing for interactive user engagement.
-
-# kotlin\com\simiacryptus\skyenet\webui\util\MarkdownUtil.kt
-
-## MarkdownUtil Documentation
-
-The `MarkdownUtil` object in the `com.simiacryptus.skyenet.webui.util` package provides utility functions for rendering
-Markdown content into HTML. It leverages the Flexmark Java library to parse and render Markdown. This utility is
-designed to enhance the display of Markdown content by optionally incorporating interactive tabs for different content
-views, such as separating visual diagrams from their source code or providing a side-by-side view of Markdown and its
-HTML rendering.
-
-### Functions
-
-#### renderMarkdown
-
-```kotlin
-fun renderMarkdown(markdown: String, options: MutableDataSet = defaultOptions(), tabs: Boolean = true): String
-```
-
-Renders the given Markdown string into HTML. It allows customization of the parsing and rendering process
-through `options` and can optionally wrap the output in interactive tabs for enhanced viewing.
-
-##### Parameters:
-
-- `markdown`: The Markdown content to be rendered as a `String`. If this parameter is blank, the function returns an
-  empty string.
-- `options`: A `MutableDataSet` specifying options for the Markdown parser and renderer. Defaults to `defaultOptions()`
-  if not provided.
-- `tabs`: A `Boolean` indicating whether to wrap the rendered HTML and the original Markdown in interactive tabs.
-  Defaults to `true`.
-
-##### Returns:
-
-A `String` containing the rendered HTML. If `tabs` is `true`, the HTML includes additional markup for interactive tabs.
-
-##### Usage Example:
-
-```kotlin
-val markdownContent = "# Hello World\nThis is a sample Markdown."
-val htmlContent = MarkdownUtil.renderMarkdown(markdownContent)
-println(htmlContent)
-```
-
-#### defaultOptions
-
-```kotlin
-private fun defaultOptions(): MutableDataSet
-```
-
-Generates a default set of options for the Markdown parser and renderer. This includes enabling table support via
-the `TablesExtension`.
-
-##### Returns:
-
-A `MutableDataSet` with the default configuration for parsing and rendering Markdown.
-
-### Implementation Details
-
-- The `renderMarkdown` function first checks if the provided Markdown string is blank, returning an empty string if
-  true.
-- It then creates a parser and renderer using the provided or default options.
-- The Markdown content is parsed into a document, which is then rendered into HTML.
-- If the `tabs` parameter is true, the function replaces specific HTML patterns (e.g., code blocks for mermaid diagrams)
-  with tabbed interfaces. This allows users to switch between different views (e.g., diagram and source code).
-- Additionally, if `tabs` is true, the entire content is wrapped in another layer of tabs, separating the rendered HTML,
-  the original Markdown, and an option to hide the content.
-- The function uses regular expressions to safely insert the Markdown content into the HTML template, escaping `<`
-  and `>` characters to prevent HTML injection.
-
-### Notes
-
-- The `@Language("HTML")` annotation is used to inform the IDE about the language of the string literals for better
-  syntax highlighting and error checking.
-- The `RegexOption.DOT_MATCHES_ALL` option allows the dot (`.`) in regular expressions to match newline characters,
-  enabling the pattern to match multi-line code blocks.
-
-# kotlin\com\simiacryptus\skyenet\webui\session\SessionTask.kt
-
-## SessionTask Class Documentation
-
-The `SessionTask` class is an abstract class designed to manage and display task outputs in a web UI environment. It
-provides a structured way to append messages, errors, images, and other HTML elements to the task output, with support
-for real-time updates through a spinner animation. This class is part of the `com.simiacryptus.skyenet.webui.session`
-package.
-
-### Overview
-
-`SessionTask` serves as a base class for tasks that require output to be displayed to the user in a web interface. It
-manages a buffer of messages and provides methods to append different types of content to this buffer. The class also
-handles the display of a loading spinner to indicate that a task is in progress.
-
-### Properties
-
-- `buffer`: A mutable list of `StringBuilder` objects that holds the HTML content to be displayed.
-- `spinner`: A string representing the HTML for a loading spinner animation.
-
-### Methods
-
-#### Abstract Methods
-
-- `send(html: String)`: Sends the current HTML content to be displayed. Implementations should define how this content
-  is displayed in the UI.
-- `saveFile(relativePath: String, data: ByteArray)`: Saves the given data to a file and returns the URL of the file.
-  Implementations should handle the file saving process and URL generation.
-
-#### Public Methods
-
-- `add(message: String, showSpinner: Boolean, tag: String, className: String)`: Adds a message to the task output with
-  customizable HTML tag and CSS class.
-- `hideable(ui: ApplicationInterface, message: String, showSpinner: Boolean, tag: String, className: String)`: Adds a
-  hideable message to the task output, which can be dismissed by the user.
-- `echo(message: String, showSpinner: Boolean, tag: String)`: Echos a user message to the task output, using a specific
-  HTML tag.
-- `header(message: String, showSpinner: Boolean, tag: String, classname: String)`: Adds a header to the task output.
-- `verbose(message: String, showSpinner: Boolean, tag: String)`: Adds a verbose message to the task output, intended for
-  detailed or debug information.
-- `error(ui: ApplicationInterface, e: Throwable, showSpinner: Boolean, tag: String)`: Displays an error message in the
-  task output, with support for rendering exceptions in a user-friendly format.
-- `complete(message: String, tag: String, className: String)`: Displays a final message in the task output and hides the
-  spinner.
-- `image(image: BufferedImage)`: Displays an image in the task output by saving the image to a file and embedding it
-  using an `` tag.
-
-#### Companion Object
-
-The companion object contains a logger instance for the class and a constant for the spinner HTML. It also includes an
-extension function `toPng()` for `BufferedImage` objects to convert them to PNG format as a byte array.
-
-### Usage
-
-To use the `SessionTask` class, one must extend it and implement the abstract methods `send(html: String)`
-and `saveFile(relativePath: String, data: ByteArray)`. Once implemented, the
-various `add`, `echo`, `header`, `verbose`, `error`, `complete`, and `image` methods can be used to append content to
-the task output.
-
-#### Example
-
-```kotlin
-class MySessionTask : SessionTask() {
-  override fun send(html: String) {
-    // Implementation to send HTML to the UI
-  }
-
-  override fun saveFile(relativePath: String, data: ByteArray): String {
-    // Implementation to save a file and return its URL
-  }
-}
-```
-
-In this example, `MySessionTask` extends `SessionTask` and provides implementations for the abstract methods. This setup
-allows for the appending of messages, errors, images, and other content to the task output in a structured and
-interactive manner.
-
-# kotlin\com\simiacryptus\skyenet\webui\util\EncryptFiles.kt
-
-## EncryptFiles Utility Documentation
-
-The `EncryptFiles` utility is a part of the `com.simiacryptus.skyenet.webui.util` package, designed to encrypt files
-using AWS Key Management Service (KMS) and write the encrypted data to a specified location. This utility leverages
-the `ApplicationServices` for encryption, making it a convenient tool for securing sensitive information.
-
-### Overview
-
-The utility consists of a Kotlin object named `EncryptFiles` that contains a `main` function. This function demonstrates
-how to encrypt a string (in this case, an empty string for demonstration purposes) using a specified AWS KMS key and
-then write the encrypted data to a file. Additionally, two extension functions, `write` and `encrypt`, are provided to
-facilitate writing byte data to a file and encrypting strings, respectively.
-
-### Usage
-
-#### Prerequisites
-
-- AWS Account: Ensure you have an AWS account and have access to the AWS Key Management Service (KMS).
-- AWS KMS Key: You need the ARN (Amazon Resource Name) of the KMS key that will be used for encryption.
-- Kotlin Environment: The utility is written in Kotlin, so ensure your development environment supports Kotlin projects.
-
-#### Encrypting and Writing Data
-
-1. **Main Function**: The `main` function in the `EncryptFiles` object is the entry point. It demonstrates encrypting a
-   string and writing the encrypted data to a file. Replace the empty string `""` with the content you wish to encrypt
-   and specify the correct KMS key ARN in place
-   of `"arn:aws:kms:us-east-1:470240306861:key/a1340b89-64e6-480c-a44c-e7bc0c70dcb1"`.
-
-2. **Writing to a File**: The `write` extension function on `String` takes a file path as its argument and writes the
-   calling string to the specified path. Ensure the path is correctly set to where you want the encrypted file to be
-   saved.
-
-3. **Encrypting Data**: The `encrypt` extension function on `String` takes a KMS key ARN as its argument and returns the
-   encrypted version of the calling string. It utilizes the `ApplicationServices.cloud!!.encrypt` method for encryption,
-   which must be properly configured to interact with AWS KMS.
-
-#### Extension Functions
-
-- **write(outpath: String)**: Writes the calling string to the specified output path. It converts the string to a byte
-  array before writing.
-
-- **encrypt(keyId: String)**: Encrypts the calling string using the specified AWS KMS key. The string is first encoded
-  to a byte array, which is then encrypted. If encryption fails, a `RuntimeException` is thrown.
-
-### Example
-
-```kotlin
-fun main() {
-  // Example content to encrypt
-  val content = "Hello, World!"
-
-  // Encrypt the content using a specified KMS key
-  val encryptedContent = content.encrypt("arn:aws:kms:your-region:your-account-id:key/your-key-id")
-
-  // Write the encrypted content to a file
-  encryptedContent.write("/path/to/your/encrypted_file.kms")
-}
-```
-
-Replace `"Hello, World!"` with the actual content you wish to encrypt, and adjust the KMS key ARN and file path
-accordingly.
-
-### Conclusion
-
-The `EncryptFiles` utility provides a straightforward way to encrypt data using AWS KMS and write the encrypted data to
-a file. It is essential to handle encryption keys and sensitive data securely and ensure that AWS KMS permissions are
-correctly configured for your application.
-
-# kotlin\com\simiacryptus\skyenet\webui\util\OpenAPI.kt
-
-## Skyenet WebUI Util - OpenAPI Data Classes Documentation
-
-This documentation provides an overview of the data classes used in the Skyenet WebUI Util package for representing
-OpenAPI specifications. These classes are designed to model the structure of OpenAPI documents, allowing for easy
-serialization and deserialization of API specifications.
-
-### Overview
-
-The OpenAPI specification is a standard format for describing RESTful APIs. The classes documented here represent the
-various components of an OpenAPI document, including information about the API itself, its paths, operations, and
-reusable components.
-
-#### OpenAPI
-
-The root document of an OpenAPI specification.
-
-- **openapi**: The OpenAPI specification version.
-- **info**: Metadata about the API.
-- **paths**: The available paths and operations for the API.
-- **components**: Reusable components for the API specification.
-
-#### Info
-
-Metadata about the API.
-
-- **title**: The title of the application.
-- **version**: The version of the OpenAPI document.
-- **description**: A short description of the application.
-- **termsOfService**: A URL to the Terms of Service for the API.
-- **contact**: Contact information for the API.
-- **license**: License information for the API.
-
-#### Contact
-
-Contact information.
-
-- **name**: The identifying name of the contact person/organization.
-- **url**: The URL pointing to the contact information.
-- **email**: The email address of the contact person/organization.
-
-#### License
-
-License information for the API.
-
-- **name**: The name of the license.
-- **url**: The URL to the license text.
-
-#### PathItem
-
-Defines an API endpoint and its operations.
-
-- **get, put, post, delete, options, head, patch**: The operations available on this path.
-
-#### Operation
-
-An API operation (endpoint).
-
-- **summary**: A short summary of what the operation does.
-- **description**: A verbose explanation of the operation behavior.
-- **responses**: The list of possible responses from the operation.
-- **parameters**: A list of parameters that are applicable for this operation.
-- **operationId**: Unique string used to identify the operation.
-- **requestBody**: The request body applicable for the operation.
-- **security**: A declaration of which security mechanisms can be used for this operation.
-- **tags**: A list of tags for API documentation control.
-- **callbacks**: Possible callbacks for the operation.
-- **deprecated**: Declares this operation to be deprecated.
-
-#### Response
-
-Describes a single response from an API Operation.
-
-- **description**: A short description of the response.
-- **content**: A map containing descriptions of potential response payloads.
-
-#### Components
-
-Holds a set of reusable objects for different aspects of the OpenAPI spec.
-
-- **schemas, responses, parameters, examples, requestBodies, headers, securitySchemes, links, callbacks**: Maps of their
-  respective types, allowing for reuse across the API.
-
-#### Schema
-
-Defines a schema for a parameter or response body.
-
-- **type**: The type of the schema.
-- **properties**: A map containing the properties of the schema.
-- **items**: A schema or reference to a schema for items in an array.
-- **`$ref`**: A reference to a defined schema.
-- **format**: The extending format for the previously mentioned type.
-- **description**: A short description of the schema.
-
-#### Parameter
-
-Describes a single operation parameter.
-
-- **name**: The name of the parameter.
-- **in**: The location of the parameter.
-- **description**: A brief description of the parameter.
-- **required**: Determines whether this parameter is mandatory.
-- **schema**: The schema defining the type used for the parameter.
-- **content**: A map containing the representations for the parameter.
-- **example**: An example of the parameter's potential value.
-
-#### Example, RequestBody, Header, SecurityScheme, Link, Callback, MediaType
-
-These classes represent various components that can be used within the OpenAPI specification, such as examples of
-request bodies, headers, security schemes, links between operations, callbacks for webhooks, and media types for request
-and response content.
-
-Each of these classes has fields appropriate to their use within the API specification, providing a comprehensive
-toolkit for defining and documenting APIs in a structured and reusable manner.
-
-# kotlin\com\simiacryptus\skyenet\webui\util\TensorflowProjector.kt
-
-## TensorflowProjector Class Documentation
-
-The `TensorflowProjector` class is designed to facilitate the generation of embeddings using OpenAI's API and to
-visualize these embeddings using TensorFlow's Embedding Projector. This class provides a seamless integration between
-the storage of embedding vectors, metadata, and the generation of a web-based visualization.
-
-### Constructor
-
-The constructor of the `TensorflowProjector` class requires several parameters for initialization:
-
-- `api`: An instance of `API` from the `com.simiacryptus.jopenai` package, used for generating embeddings.
-- `dataStorage`: An implementation of `StorageInterface` for storing the generated files.
-- `sessionID`: A `Session` object representing the current session.
-- `host`: A `String` representing the host address.
-- `session`: An instance of `ApplicationInterface` representing the current application session.
-- `userId`: An optional `User` object representing the current user.
-
-### Methods
-
-#### `toVectorMap(vararg words: String): Map`
-
-This private method generates a map of word embeddings. Each word in the input is mapped to its corresponding embedding
-vector.
-
-- **Parameters**: `words` - A variable number of `String` objects representing the words to be embedded.
-- **Returns**: A `Map` where each key is a word and its value is the embedding vector.
-
-#### `writeTensorflowEmbeddingProjectorHtml(vararg words: String): String`
-
-Generates the necessary files for visualizing embeddings in TensorFlow's Embedding Projector and returns HTML code to
-embed the projector in a web page.
-
-- **Parameters**: `words` - A variable number of `String` objects representing the words to be visualized.
-- **Returns**: A `String` containing HTML code. This code includes links to the generated files (vectors, metadata, and
-  projector configuration) and an iframe embedding the TensorFlow Projector.
-
-### Usage Example
-
-```kotlin
-val apiClient = OpenAIClient("YourAPIKey")
-val storage = YourStorageImplementation()
-val sessionID = Session("YourSessionID")
-val host = "http://yourhost.com"
-val session = YourApplicationInterface()
-val userId = User("YourUserID")
-
-val projector = TensorflowProjector(apiClient, storage, sessionID, host, session, userId)
-val words = arrayOf("word1", "word2", "word3")
-val html = projector.writeTensorflowEmbeddingProjectorHtml(*words)
-
-// Output the HTML to your web page
-println(html)
-```
-
-### Notes
-
-- Ensure that the `API` instance (`apiClient` in the example) is properly authenticated with your OpenAI API key.
-- The `StorageInterface` (`storage` in the example) should be implemented to handle file storage according to your
-  application's requirements.
-- The generated HTML code can be embedded in any web page to display the TensorFlow Embedding Projector with your
-  embeddings.
-
-This class simplifies the process of generating and visualizing embeddings, making it accessible to integrate advanced
-natural language processing features into your applications.
-
-# kotlin\com\simiacryptus\skyenet\webui\util\Selenium2S3.kt
-
-## Developer Documentation for Selenium2S3 Class
-
-The `Selenium2S3` class is a comprehensive utility designed for web scraping, content processing, and saving the fetched
-data to an S3-compatible storage system. It leverages Selenium for web interaction, Apache HttpClient for asynchronous
-HTTP requests, and Jsoup for HTML parsing.
-
-### Overview
-
-This class is part of the `com.simiacryptus.skyenet.webui.util` package and extends the `Selenium` interface. It is
-designed to navigate web pages, process content, and save the processed data to cloud storage. The class supports
-handling HTML, JSON, and media content types, and it provides mechanisms for link replacement and content editing.
-
-### Key Features
-
-- **Web Navigation and Content Fetching**: Uses Selenium WebDriver for navigating web pages and fetching content.
-- **Asynchronous HTTP Requests**: Utilizes Apache HttpClient for asynchronous fetching of web resources.
-- **Content Processing**: Offers methods for processing HTML and JSON content, including link replacement and content
-  editing.
-- **Cloud Storage Integration**: Supports saving processed content to an S3-compatible cloud storage system.
-- **Customizable WebDriver**: Allows customization of the Selenium WebDriver, including headless operation and image
-  loading preferences.
-
-### Initialization
-
-The class constructor accepts a `ThreadPoolExecutor` for managing asynchronous tasks and an optional array of cookies
-for session management.
-
-```kotlin
-val selenium2S3 = Selenium2S3(Executors.newCachedThreadPool() as ThreadPoolExecutor, cookies)
-```
-
-### Core Methods
-
-#### save
-
-The `save` method is the primary entry point for processing and saving web content. It navigates to a specified URL,
-processes the page content, and saves the processed content to cloud storage.
-
-```kotlin
-selenium2S3.save(URL("http://example.com"), "filename.html", "saveRoot")
-```
-
-#### process
-
-The `process` method handles the processing of individual links found on the web page. It determines the content type
-and delegates to the appropriate method for further processing.
-
-#### getHtml, getJson, getMedia
-
-These methods are responsible for fetching and processing HTML, JSON, and media content, respectively. They perform
-asynchronous HTTP requests and process the responses.
-
-#### saveJS, saveHTML
-
-These methods handle the final processing and saving of JavaScript and HTML content to cloud storage.
-
-### Utility Methods
-
-- **get**: Prepares an HTTP GET request with cookies.
-- **currentPageLinks**: Extracts links from the current page using Selenium WebDriver or Jsoup.
-- **toAbsolute**, **toRelative**: Methods for converting links to absolute or relative forms.
-- **toArchivePath**: Transforms links to a format suitable for archiving.
-- **validate**: Validates the content type of fetched resources.
-- **mimeType**: Determines the MIME type based on file extension.
-- **editPage**: Performs custom edits on an HTML document.
-
-### Closing Resources
-
-The `close` method ensures proper shutdown of the WebDriver and HttpClient, releasing all associated resources.
-
-### Companion Object
-
-The companion object includes utility methods for initializing the Chrome WebDriver and managing cookies.
-
-#### chromeDriver
-
-A static method for configuring and initializing a Chrome WebDriver instance.
-
-#### setCookies
-
-A static method for setting cookies in the WebDriver.
-
-### Usage Example
-
-```kotlin
-val cookies = arrayOf(/* Your cookies here */)
-val executor = Executors.newCachedThreadPool() as ThreadPoolExecutor
-val selenium2S3 = Selenium2S3(executor, cookies)
-try {
-  selenium2S3.save(URL("http://example.com"), null, "mySaveRoot")
-} finally {
-  selenium2S3.close()
-}
-```
-
-This documentation provides an overview of the `Selenium2S3` class and its capabilities. For detailed usage and
-customization, refer to the method descriptions and parameters within the class source code.
-
-# resources\application\chat.js
-
-## WebSocket Communication Module
-
-This module provides a simple interface for establishing WebSocket connections, sending messages, and managing session
-IDs for real-time web applications. It includes functions for connecting to a WebSocket server, sending messages through
-the WebSocket, and handling session IDs for maintaining the state of the connection.
-
-### Functions
-
-#### getSessionId()
-
-Retrieves or generates a new session ID for the WebSocket connection.
-
-- If the current URL contains a hash (representing a session ID), it extracts and returns this ID.
-- If no session ID is present in the URL, it requests a new session ID from the server at the endpoint `'newSession'`.
-  Upon receiving a new session ID, it updates the URL hash with this ID and initiates a WebSocket connection using this
-  ID.
-
-**Returns:** The session ID as a string if it exists in the URL hash; otherwise, it initiates the process to fetch and
-set a new session ID but does not return a value.
-
-#### send(message)
-
-Sends a message through the established WebSocket connection.
-
-- **Parameters:**
-    - `message`: The message to be sent through the WebSocket. This should be a string.
-
-- **Throws:** An error if the WebSocket is not in the `OPEN` state (readyState !== 1), indicating that the connection is
-  not ready to send messages.
-
-#### connect(sessionId, customReceiveFunction)
-
-Establishes a WebSocket connection using the provided session ID and sets up event listeners for the connection.
-
-- **Parameters:**
-    - `sessionId`: The session ID to be used for the WebSocket connection. This should be a string obtained
-      from `getSessionId()`.
-    - `customReceiveFunction` (optional): A custom function to handle incoming WebSocket messages. If not provided, the
-      default handler `onWebSocketText` will be used.
-
-- **Behavior:** This function constructs the WebSocket URL based on the current window location and the provided session
-  ID, then initiates the WebSocket connection. It also sets up event listeners for `open`, `message`, `close`,
-  and `error` events on the WebSocket.
-
-#### showDisconnectedOverlay(show)
-
-Controls the visibility of a hypothetical "disconnected" overlay based on the WebSocket connection state.
-
-- **Parameters:**
-    - `show`: A boolean indicating whether to show (true) or hide (false) the disconnected overlay.
-
-- **Behavior:** This function enables or disables elements with the class `ws-control` based on the `show` parameter.
-  This can be used to disable user interface controls when the WebSocket connection is lost.
-
-### Usage Example
-
-```javascript
-// Attempt to connect using an existing session ID or fetch a new one
-const sessionId = getSessionId();
-if (sessionId) {
-    connect(sessionId);
-}
-
-// Custom function to handle incoming messages
-function onCustomMessage(event) {
-    console.log('Received message:', event.data);
-}
-
-// Send a message through the WebSocket
-function sendMessage() {
-    try {
-        send('Hello, world!');
-    } catch (error) {
-        console.error(error.message);
-    }
-}
-
-// Reconnect with a custom message handler
-function reconnectWithCustomHandler() {
-    const sessionId = getSessionId();
-    if (sessionId) {
-        connect(sessionId, onCustomMessage);
-    }
-}
-
-// Example of using the showDisconnectedOverlay function
-function handleConnectionChange(isConnected) {
-    showDisconnectedOverlay(!isConnected);
-}
-```
-
-This module simplifies the process of managing WebSocket connections and session IDs, making it easier to build
-real-time interactive web applications.
-
-# resources\application\index.html
-
-## WebSocket Client Web Application Documentation
-
-This document provides an overview and detailed explanation of the WebSocket Client Web Application's HTML structure,
-including its dependencies, layout, and functionality. This application is designed to facilitate real-time
-communication between clients and a server using WebSockets.
-
-### Overview
-
-The application's front-end is structured in HTML and makes extensive use of external libraries and stylesheets for
-enhanced functionality and aesthetics. It includes a dynamic user interface for sending and receiving messages in
-real-time, managing sessions, and customizing themes.
-
-### Dependencies
-
-The application relies on several external resources and libraries:
-
-- **Prism.js**: Used for syntax highlighting in the application. It enhances the readability of code snippets shared
-  within the chat.
-- **Mermaid.js**: A JavaScript-based diagramming and charting tool that renders markdown-inspired text definitions to
-  create and modify diagrams dynamically.
-- **Main CSS**: The primary stylesheet for the application, defining the visual aspects of the UI.
-- **Main JS & Chat JS**: JavaScript files containing the logic for WebSocket communication and UI interactions.
-
-#### External Libraries and Stylesheets
-
-- Prism.js related stylesheets and scripts for syntax highlighting and additional features like line numbers, match
-  braces, and toolbar.
-- Mermaid.js for diagramming, loaded as an ES module.
-- Application-specific styles (`main.css`) and scripts (`main.js`, `chat.js`).
-
-### HTML Structure
-
-#### Head Section
-
-Contains metadata, links to stylesheets for Prism.js and the application, and script tags for the application logic.
-
-#### Body Section
-
-##### Toolbar
-
-A navigation bar with dropdown menus for navigating through the application, managing sessions, changing themes, and
-accessing about sections like privacy policy and terms of service.
-
-##### Namebar
-
-Displays login information and user-specific options like settings, usage, and logout, which become visible upon login.
-
-##### Session
-
-The main interactive area where users can input and send messages. It includes a form with a textarea for message input
-and a submit button.
-
-##### Modal
-
-A generic modal component for displaying additional information or forms in an overlay.
-
-##### Footer
-
-Contains attribution and links to external resources or the project's repository.
-
-### Functionality
-
-- **Real-Time Communication**: Utilizes WebSockets for real-time messaging between clients and the server.
-- **Theme Customization**: Users can select different themes for the UI from the "Themes" dropdown.
-- **Session Management**: Options for managing and sharing sessions, including settings, files, usage, and deletion.
-- **User Interaction**: Login/logout functionality and user-specific settings accessible through the namebar.
-
-### Scripts
-
-#### Prism.js
-
-Integrated for syntax highlighting and additional features to enhance the display of code snippets within messages.
-
-#### Mermaid.js
-
-Initialized in a script tag as an ES module, it allows for the rendering of diagrams based on textual descriptions.
-
-#### Application Scripts
-
-- **main.js**: Contains the core functionality for WebSocket communication and handling UI interactions.
-- **chat.js**: Dedicated to handling chat-specific functionalities, including sending, receiving, and displaying
-  messages.
-
-### Conclusion
-
-This WebSocket Client Web Application provides a robust platform for real-time communication, enhanced with features
-like syntax highlighting and diagram rendering. Its modular structure and use of external libraries make it a flexible
-and visually appealing solution for online chat applications.
-
-# resources\application\main.js
-
-## Developer Documentation
-
-This documentation provides an overview and detailed explanation of the JavaScript functions and event listeners used in
-a web application for managing modals, fetching data, handling WebSocket messages, and dynamically updating the user
-interface.
-
-### Overview
-
-The application includes functionality for:
-
-- Displaying and hiding modals
-- Fetching data from an endpoint and displaying it within a modal
-- Handling WebSocket messages and updating the UI accordingly
-- Dynamically updating tabs, verbose states, and reply forms
-- Managing user settings and themes
-- Handling user input and form submissions
-
-### Functions
-
-#### showModal(endpoint, useSession = true)
-
-Displays a modal by setting its display style to 'block'. It fetches data from the specified endpoint and displays it
-within the modal. The `useSession` parameter indicates whether to append a session ID to the endpoint.
-
-**Parameters:**
-
-- `endpoint`: The URL endpoint from which to fetch data.
-- `useSession`: A boolean indicating whether to use the session ID. Defaults to `true`.
-
-#### closeModal()
-
-Hides the modal by setting its display style to 'none'.
-
-#### async fetchData(endpoint, useSession = true)
-
-Fetches data from the specified endpoint asynchronously. If `useSession` is true, it appends the session ID to the
-endpoint. The function updates the modal content with the fetched data or an error message if the fetch fails.
-
-**Parameters:**
-
-- `endpoint`: The URL endpoint from which to fetch data.
-- `useSession`: A boolean indicating whether to use the session ID. Defaults to `true`.
-
-#### onWebSocketText(event)
-
-Handles WebSocket messages by parsing the message data and updating the UI accordingly. It supports message versioning,
-input visibility, and sticky input behavior.
-
-**Parameters:**
-
-- `event`: The WebSocket message event.
-
-#### updateTabs()
-
-Attaches click event listeners to tab buttons to switch between tab contents within a tab container.
-
-#### toggleVerbose()
-
-Toggles the visibility of elements with the 'verbose' class based on the text of the 'verbose' button.
-
-#### refreshReplyForms()
-
-Attaches keydown event listeners to reply input fields to submit the form when the Enter key is pressed without the
-Shift key.
-
-#### refreshVerbose()
-
-Refreshes the visibility of verbose elements based on the current state of the 'verbose' button.
-
-### Event Listeners
-
-The application attaches several event listeners on the DOMContentLoaded event to handle user interactions such as
-clicking on UI elements, submitting forms, and changing themes.
-
-#### Theme Change
-
-Event listeners are attached to theme selection elements to change the application's theme by updating the stylesheet
-link and saving the selected theme to localStorage.
-
-#### Modal Triggers
-
-Click event listeners are attached to various elements to display modals related to settings, usage, deletion,
-cancellation, threads, and sharing.
-
-#### Form Submission
-
-A submit event listener is attached to the main input form to prevent the default form submission behavior and handle
-the submission manually.
-
-#### Input Field Auto-Resize
-
-An input event listener is attached to a text input field to automatically adjust its height based on the content.
-
-#### Fetch User Information
-
-Fetches user information on page load and updates the UI based on whether the user is logged in.
-
-#### Privacy and Terms Links
-
-Click event listeners are attached to the privacy policy and terms of service links to display the respective modals.
-
-### Conclusion
-
-This documentation covers the core functionality of the web application related to modals, data fetching, WebSocket
-communication, and dynamic UI updates. Developers can use this guide to understand and extend the application's
-features.
-
-# resources\shared\schemes\_alien_spaceship.scss
-
-## Alien Spaceship Theme - Developer Documentation
-
-Welcome to the developer documentation for the Alien Spaceship theme. This theme is designed to bring a cosmic and
-adventurous feel to your application with its carefully chosen color palette inspired by the vastness of space and the
-mystery of alien worlds. Below, you will find detailed information on the theme's color scheme, including base colors,
-derived colors, and specific use cases such as buttons, messages, and modals.
-
-### Base Colors
-
-The Alien Spaceship theme is built around a set of base colors that evoke the depth and mystery of space:
-
-- **Deep Space Black** (`#0b0c10`): A deep black that captures the void of space.
-- **Galactic Blue** (`#1f2833`): A dark blue reminiscent of the galaxy's vast expanse.
-- **Nebula Pink** (`#c5c6c7`): A soft pink inspired by distant nebulae.
-- **Asteroid Gray** (`#4e4e50`): A muted gray that mimics the rocky surface of asteroids.
-- **Alien Green** (`#66fcf1`): A bright green for highlights, inspired by classic alien aesthetics.
-- **Laser Red** (`#fc4445`): A striking red for important elements, inspired by laser beams.
-- **Space Dust** (`#ccc`): A light grey used for background contrasts, reminiscent of cosmic dust.
-- **Comet Tail** (`#45a29e`): A medium teal for borders and lines, inspired by comet tails.
-
-### Derived Colors
-
-Derived colors are variations and combinations of the base colors, tailored for specific UI elements:
-
-- **Backgrounds, Text, and Borders**:
-    - Secondary Background Color: Deep Space Black
-    - Secondary Text Color: Space Dust
-    - Border Color: Comet Tail
-
-- **Application List**:
-    - Header Background: Galactic Blue
-    - Header Text: Space Dust
-    - Link Color: Laser Red
-    - Row Even Background: Nebula Pink
-    - Row Hover Background: Asteroid Gray
-
-- **Buttons**:
-    - Background Color: Laser Red
-    - Text Color: Deep Space Black
-    - Hover Background Color: Darkened Laser Red
-
-- **Messages and Modals**:
-    - Info Background Color: Lightened Nebula Pink
-    - Success Background Color: Lightened Alien Green
-    - Error Background Color: Lightened Laser Red
-    - Modal Content Background Color: Deep Space Black
-
-### Usage
-
-#### Buttons
-
-To style buttons within the Alien Spaceship theme, use the following Sass variables:
-
-```scss
-.button {
-  background-color: $button-bg-color;
-  color: $button-text-color;
-  border-radius: $button-border-radius;
-  box-shadow: $button-box-shadow;
-
-  &:hover {
-    background-color: $button-hover-bg-color;
-    color: $button-hover-text-color;
-  }
-}
-```
-
-#### Messages
-
-For informational, success, and error messages, apply the corresponding background and text color variables:
-
-```scss
-.message-info {
-  background-color: $message-info-bg-color;
-  color: $message-info-text-color;
-}
-
-.message-success {
-  background-color: $message-success-bg-color;
-  color: $message-success-text-color;
-}
-
-.message-error {
-  background-color: $message-error-bg-color;
-  color: $message-error-text-color;
-}
-```
-
-#### Modals
-
-Modals should use the modal content background and text color variables for consistency:
-
-```scss
-.modal-content {
-  background-color: $modal-content-bg-color;
-  color: $modal-content-text-color;
-  box-shadow: $modal-content-shadow;
-}
-```
-
-### Conclusion
-
-The Alien Spaceship theme offers a unique and engaging aesthetic for applications, drawing inspiration from the
-mysteries of outer space and alien worlds. By adhering to the color scheme and usage guidelines provided in this
-documentation, developers can ensure a cohesive and visually appealing user interface.
-
-# resources\shared\schemes\_forest.scss
-
-## Forest Canopy Theme - Developer Documentation
-
-The Forest Canopy theme is a comprehensive styling framework designed to provide a rich, earthy aesthetic to web
-applications. This document outlines the variables and their intended uses within the theme, facilitating easy
-customization and application.
-
-### Importing the Theme
-
-Ensure you import the theme at the beginning of your stylesheet to access its variables and mixins:
-
-```scss
-@import 'night';
-```
-
-### Typography Variables
-
-Typography variables define the font styles used throughout the theme.
-
-- `$font-family-primary`: Primary font family, set to `fantasy`.
-- `$font-family-secondary`: Secondary font family, set to `sans-serif`.
-- `$font-size-base`: Base font size, set to `1em`.
-- `$font-size-large`: Large font size, set to `1.5px`.
-- `$font-size-small`: Small font size, set to `0.8px`.
-- `$font-weight-normal`: Normal font weight, set to `100`.
-- `$font-weight-bold`: Bold font weight, set to `500`.
-
-### Base Colors
-
-Base colors define the primary palette for the Forest Canopy theme.
-
-- `$base-forest-floor`: Rich dark earth, used for main backgrounds.
-- `$base-tree-bark`: Warm brown, used for headers and footers.
-- `$base-canopy-leaf`: Deep green, used for accents and active elements.
-- `$base-sunbeam`: Light khaki, used for highlights and special features.
-- `$base-dew`: Pale green, used for subtle background contrasts.
-- `$base-underbrush`: Dark olive green, used for secondary backgrounds and buttons.
-- `$base-fern`: Muted medium green, used for borders and separators.
-- `$base-acorn`: Nutty brown, used for text and icons.
-
-### Derived Colors
-
-Derived colors are calculated based on the base colors to maintain a cohesive theme.
-
-- `$secondary-bg-color`: Matches `$base-forest-floor`.
-- `$secondary-text-color`: Matches `$base-acorn`.
-- `$applist-header-bg`: Matches `$base-tree-bark`.
-- ... (and so on for each derived color variable, explaining its intended use and base color reference).
-
-### Buttons
-
-Button styling variables provide a consistent look and feel for interactive elements.
-
-- `$button-bg-color`: Background color for buttons, set to `$base-canopy-leaf`.
-- `$button-text-color`: Text color for buttons, set to `$base-forest-floor`.
-- `$button-hover-bg-color`: Background color for buttons on hover, darkened by 10%.
-- `$button-hover-text-color`: Text color for buttons on hover, lightened by 10%.
-- `$button-border-radius`: Border radius for buttons, matches `$border-radius`.
-- `$button-box-shadow`: Box shadow for buttons, matches `$box-shadow`.
-
-### Forms
-
-Form styling variables ensure inputs and form elements align with the theme.
-
-- `$input-border-color`: Border color for inputs, set to `$base-fern`.
-- `$input-border-focus-color`: Border color for inputs on focus, set to `$base-dew`.
-- `$input-bg-color`: Background color for inputs, set to `$base-forest-floor`.
-- `$input-text-color`: Text color for inputs, set to `$base-acorn`.
-- `$input-padding`: Padding for inputs, set to `10px`.
-
-### Messages and Modals
-
-Variables for messages and modals provide feedback and interaction cues to users.
-
-- `$message-info-bg-color`: Background color for info messages, lightened by 40% from `$base-canopy-leaf`.
-- `$message-info-text-color`: Text color for info messages, set to `$base-tree-bark`.
-- ... (and so on for each message and modal variable, explaining its intended use and base color reference).
-
-### Customization
-
-To customize the theme, you can override the default values of the variables before importing the theme or in your
-custom stylesheets. This allows for a personalized look while maintaining the integrity of the theme's design.
-
-### Conclusion
-
-The Forest Canopy theme offers a rich set of variables for comprehensive styling of web applications. By adhering to
-this documentation, developers can ensure consistent application of the theme and ease of customization.
-
-# resources\shared\schemes\_night.scss
-
-## Nighttime Theme Color Scheme Documentation
-
-This documentation provides an overview of the color variables defined for the Nighttime theme. The theme is designed
-with a dark aesthetic, suitable for applications that require a dark mode or wish to offer a less bright, eye-friendly
-interface for nighttime usage. The color scheme is composed of base colors, derived colors, and specific component
-styling for buttons, messages, and modals.
-
-### Base Colors
-
-The foundation of the Nighttime theme is built upon these base colors:
-
-- `$base-night`: #141414 - A very dark gray, almost black, used primarily for backgrounds.
-- `$base-midnight-blue`: #2c3e50 - A dark blue, suitable for headers and footers.
-- `$base-twilight-blue`: #34495e - A slightly lighter shade of dark blue for accents.
-- `$base-dark-grey`: #7f8c8d - A muted grey for readable text that isn't too harsh on the eyes.
-- `$base-moonlit-aqua`: #1abc9c - A muted aqua for subtle highlights and accents.
-- `$base-starlight-yellow`: #f1c40f - A soft yellow for important buttons or icons.
-- `$base-cloudy-grey`: #bdc3c7 - A lighter grey for background contrasts.
-- `$base-meteorite-border`: #95a5a6 - A medium grey for borders and lines.
-
-### Derived Colors
-
-Derived colors are calculated based on the base colors to maintain a cohesive theme across various UI components:
-
-- Backgrounds, text, headers, links, and more have specific derived colors for consistency.
-- Special UI elements like buttons, dropdowns, modals, and messages utilize these derived colors for background, text,
-  and border styling.
-- Interaction states such as hover and active are also defined with slight modifications to the base or derived colors
-  to provide visual feedback.
-
-### Component Styling
-
-#### Buttons
-
-- Background, text, and hover states for buttons are defined to ensure they stand out and provide necessary user
-  feedback.
-
-#### Forms
-
-- Form elements retain the base theme colors for consistency but can be customized as needed.
-
-#### Messages and Modals
-
-- Information, success, and error messages have specific background and text color styling for clear communication.
-- Modals utilize the dark theme colors for background and text, with added shadows for depth.
-
-### Utility Variables
-
-- `$border-radius`, `$box-shadow`, and `$transition-speed` are utility variables provided for consistent styling and
-  animations across the theme.
-
-### Usage
-
-To use these variables in your project, ensure you import this stylesheet at the beginning of your Sass or SCSS files.
-You can then reference these variables throughout your styles to maintain a consistent theme.
-
-```scss
-@import 'path/to/nighttime-theme.scss';
-
-body {
-  background-color: $primary-bg-color;
-  color: $primary-text-color;
-}
-```
-
-This documentation should serve as a reference for developers looking to implement the Nighttime theme in their
-projects. The defined colors and variables offer a comprehensive palette for creating dark-themed UIs with a focus on
-readability and user experience during nighttime usage.
-
-# resources\shared\schemes\_normal.scss
-
-## Developer Documentation: UI Theme Variables
-
-This documentation outlines the variables defined for styling a web application's user interface (UI). These variables
-are categorized into typography, base colors, derived colors, buttons, forms, and messages & modals. Utilizing these
-variables ensures a consistent and easily maintainable theme across the application.
-
-### Typography Variables
-
-Typography variables define the fonts, sizes, and weights used throughout the application.
-
-- `$font-family-primary`: Primary font stack. Default: `'Helvetica Neue', Helvetica, Arial, sans-serif`.
-- `$font-family-secondary`: Secondary font stack. Default: `'Segoe UI', Tahoma, Geneva, Verdana, sans-serif`.
-- `$font-size-base`: Base font size. Default: `1em`.
-- `$font-size-large`: Large font size. Default: `1.5px`.
-- `$font-size-small`: Small font size. Default: `0.8px`.
-- `$font-weight-normal`: Normal font weight. Default: `400`.
-- `$font-weight-bold`: Bold font weight. Default: `700`.
-
-### Base Colors
-
-Base colors define the core palette from which the application's color scheme is derived.
-
-- `$base-white`: Pure white. Default: `#ffffff`.
-- `$base-dark`: Dark color, typically used for text. Default: `#333333`.
-- `$base-light-blue`: Light blue, used for subtle backgrounds. Default: `#dae8fc`.
-- `$base-grey`: Grey, adjusted for readability. Default: `#7f8c8d`.
-- `$base-dark-blue`: Dark blue, suitable for headers or footers. Default: `#2c3e50`.
-- `$base-light-grey`: Light grey, good for backgrounds. Default: `#ecf0f1`.
-- `$base-blue`: Vibrant blue, used for links and buttons. Default: `#3498db`.
-- `$base-green`: Success color. Default: `#2ecc71`.
-- `$base-red`: Error color. Default: `#e74c3c`.
-- `$base-orange`: Warm, friendly color for buttons. Default: `#e67e22`.
-- `$base-dark-grey`: For subtle text. Default: `#95a5a6`.
-
-### Derived Colors
-
-Derived colors are based on the base colors and are used for specific UI elements like backgrounds, text, borders, and
-more.
-
-- `$secondary-bg-color`: Secondary background color. Inherits from `$base-white`.
-- `$secondary-text-color`: Secondary text color. Inherits from `$base-dark`.
-- (Further derived colors follow a similar pattern, adjusting for specific UI elements like headers, links, buttons,
-  etc.)
-
-### Buttons
-
-Button variables define the styling for buttons within the application.
-
-- `$button-bg-color`: Background color for buttons. Inherits from `$base-blue`.
-- `$button-text-color`: Text color for buttons. Inherits from `$base-white`.
-- `$button-hover-bg-color`: Background color for buttons on hover. Darkened by 10% from `$button-bg-color`.
-- `$button-hover-text-color`: Text color for buttons on hover. Inherits from `$button-text-color`.
-- `$button-border-radius`: Border radius for buttons. Inherits from `$border-radius`.
-- `$button-box-shadow`: Box shadow for buttons. Inherits from `$box-shadow`.
-
-### Forms
-
-Form variables define the styling for form elements like inputs.
-
-- `$input-border-color`: Border color for inputs. Inherits from `$base-grey`.
-- `$input-border-focus-color`: Border color for inputs on focus. Inherits from `$base-blue`.
-- `$input-bg-color`: Background color for inputs. Inherits from `$base-white`.
-- `$input-text-color`: Text color for inputs. Inherits from `$secondary-text-color`.
-- `$input-padding`: Padding for inputs. Default: `10px`.
-
-### Messages and Modals
-
-Variables for messages and modals define the styling for informational, success, and error messages, as well as modal
-dialogs.
-
-- `$message-info-bg-color`: Background color for informational messages. Lightened by 50% from `$base-blue`.
-- `$message-info-text-color`: Text color for informational messages. Inherits from `$base-dark-blue`.
-- (Further message and modal variables adjust similarly for success, error messages, and modal content.)
-
-This documentation provides a comprehensive overview of the UI theme variables used in the application. Developers can
-refer to this guide to understand and utilize the predefined variables for consistent styling across the application.
-
-# resources\shared\schemes\_pony.scss
-
-## Developer Documentation: Theme Customization
-
-This documentation provides an overview of the theme customization options available through the use of SCSS variables.
-These variables allow for easy adjustments to typography, colors, buttons, forms, messages, and modals to maintain
-consistency and facilitate rapid development.
-
-### Typography Variables
-
-Customize the typography across your application by adjusting these variables.
-
-- `$font-family-primary`: Primary font family. Default is `cursive`.
-- `$font-family-secondary`: Secondary font family. Default is `cursive`.
-- `$font-size-base`: Base font size for standard text. Default is `1.2em`.
-- `$font-size-large`: Font size for larger text elements. Default is `1.6px`.
-- `$font-size-small`: Font size for smaller text elements. Default is `0.9px`.
-- `$font-weight-normal`: Weight for normal text. Default is `400`.
-- `$font-weight-bold`: Weight for bold text. Default is `700`.
-
-### Base Colors
-
-Define the core palette for your application with these base color variables.
-
-- `$base-bubblegum`: Playful pink, ideal for primary elements. Default is `#ff77a9`.
-- `$base-light-blue`: Vibrant blue, used for links and buttons. Default is `#3471FF`.
-- `$base-candy-red`: Soft red, suitable for calls to action and highlights. Default is `#ff6b6b`.
-- `$base-vanilla`: Soft off-white, for backgrounds to ensure legibility. Default is `#f3f3f3`.
-- `$base-charcoal`: Dark grey, softer than black for better readability. Default is `#454545`.
-- Additional pastel colors for backgrounds and highlights include pastel pink (`$base-pastel-pink`), pastel
-  yellow (`$base-pastel-yellow`), and pastel red (`$base-pastel-red`).
-- `$base-soft-grey`: Soft grey for less important elements. Default is `#ced4da`.
-
-### Derived Colors
-
-These variables utilize the base colors to maintain a cohesive theme across various UI components.
-
-- `$secondary-bg-color`, `$secondary-text-color`: Colors for secondary backgrounds and text.
-- `$applist-header-bg`, `$applist-header-text`: Background and text colors for app list headers.
-- `$border-color`, `$border-radius`, `$box-shadow`: Defaults for borders and shadows.
-- `$error-color`, `$success-color`: Colors for error messages and success indicators.
-- `$link-color`, `$link-hover-color`: Default and hover colors for links.
-- `$modal-overlay-color`: Color for modal overlays.
-- More variables are available for specific UI elements like buttons, dropdowns, messages, and forms.
-
-### Buttons
-
-Customize button appearance with these variables.
-
-- `$button-bg-color`: Background color for buttons.
-- `$button-text-color`: Text color for buttons.
-- `$button-hover-bg-color`, `$button-hover-text-color`: Background and text colors for buttons on hover.
-- `$button-border-radius`, `$button-box-shadow`: Border radius and box shadow for buttons.
-
-### Forms
-
-Adjust the appearance of form elements.
-
-- `$input-border-color`, `$input-border-focus-color`: Border colors for input fields, including focus state.
-- `$input-bg-color`, `$input-text-color`, `$input-padding`: Background color, text color, and padding for input fields.
-
-### Messages and Modals
-
-Define the styling for informational messages and modal dialogs.
-
-- `$message-info-bg-color`, `$message-info-text-color`: Background and text colors for informational messages.
-- `$message-success-bg-color`, `$message-success-text-color`: Background and text colors for success messages.
-- `$message-error-bg-color`, `$message-error-text-color`: Background and text colors for error messages.
-- `$modal-content-bg-color`, `$modal-content-text-color`, `$modal-content-shadow`: Styling for modal content.
-
-### Usage
-
-To use these variables, simply assign them to the desired CSS properties within your SCSS files. For example:
-
-```scss
-body {
-  font-family: $font-family-primary;
-  background-color: $primary-bg-color;
-  color: $primary-text-color;
-}
-```
-
-This approach ensures that your application maintains a consistent look and feel, while also providing the flexibility
-to adapt the theme as needed.
-
-# resources\application\favicon.svg
-
-#### Developer Documentation: SVG Graphic Implementation
-
-##### Overview
-
-This document provides an overview and implementation details for integrating the provided SVG graphic into a web
-application or website. The SVG graphic is a complex illustration with multiple layers, colors, and styles, designed to
-be scalable and adaptable across different platforms and devices.
-
-##### SVG Graphic Description
-
-The SVG graphic is a detailed illustration composed of various elements, including paths, shapes, and styled components.
-It features a rich color palette and intricate design elements, making it suitable for high-resolution displays. The
-graphic is designed with versatility in mind, allowing for easy customization and scaling.
-
-##### File Information
-
-- **File Name:** `custom_graphic.svg`
-- **Dimensions:** 700x700 pixels
-- **File Size:** Varies depending on the optimization level
-
-##### Implementation Guide
-
-1. **Adding the SVG to HTML:**
-   Embed the SVG code directly into your HTML file to ensure the best performance and flexibility. This method allows
-   for easy manipulation of the SVG properties using CSS and JavaScript.
-
-   ```html
-   
-   

-       
-   

-   ```
-
-2. **Styling the SVG:**
-   The SVG graphic comes with predefined classes for styling various elements within the graphic. You can override these
-   styles or add new ones by targeting the specific classes in your CSS file.
-
-   ```css
-   /* Example of overriding the fill color of elements with class 'st0' */
-   .st0 {
-       fill: #newColor;
-   }
-   ```
-
-3. **Interactivity and Animation:**
-   Enhance the SVG graphic with interactivity and animation using JavaScript or CSS. For example, you can add hover
-   effects, click events, or animate certain elements within the SVG.
-
-   ```javascript
-   // Example of adding a hover effect using JavaScript
-   document.querySelector('.st0').addEventListener('mouseover', function() {
-       this.style.fill = '#hoverColor';
-   });
-   ```
-
-   ```css
-   /* Example of adding a simple animation using CSS */
-   .st0 {
-       transition: fill 0.3s ease;
-   }
-   .st0:hover {
-       fill: #hoverColor;
-   }
-   ```
-
-4. **Optimization:**
-   To ensure optimal loading times, especially for complex SVG graphics, consider using SVG optimization tools like
-   SVGO. This can significantly reduce the file size without compromising the quality of the graphic.
-
-5. **Accessibility:**
-   Enhance the accessibility of the SVG graphic by adding descriptive titles and descriptions using the ``
-   and `<desc>` tags within the SVG code. This provides context for screen readers and helps meet accessibility
-   standards.
-
-   ```html
-   <svg>
-       <title>Descriptive Title of the SVG Graphic
-       Description of the SVG Graphic, highlighting key elements.
-       
-   
-   ```
-
-##### Conclusion
-
-Integrating the provided SVG graphic into your project enhances the visual appeal of your application or website. By
-following the implementation guide, you can ensure that the graphic is displayed correctly, optimized for performance,
-and accessible to all users.
-
-# resources\shared\_main.scss
-
-## Developer Documentation: SCSS Mixins and Styles
-
-This documentation provides an overview of the SCSS mixins and styles used in the project. It covers the purpose and
-usage of each mixin, as well as the key styling sections that apply these mixins for UI components.
-
-### Mixins
-
-#### 1. `typography`
-
-**Purpose:** Sets the font properties for elements.
-
-**Parameters:**
-
-- `$font-family`: Font family (default: `$font-family-primary`)
-- `$font-size`: Font size (default: `$font-size-base`)
-- `$font-weight`: Font weight (default: `$font-weight-normal`)
-
-**Usage:**
-
-```scss
-@include typography($font-family-secondary, 16px, bold);
-```
-
-#### 2. `flex-container`
-
-**Purpose:** Creates a flex container with an optional direction.
-
-**Parameters:**
-
-- `$direction`: Flex direction (default: `column`)
-
-**Usage:**
-
-```scss
-@include flex-container(row);
-```
-
-#### 3. `fixed-full`
-
-**Purpose:** Applies a fixed position to an element, covering the entire viewport.
-
-**Parameters:** None
-
-**Usage:**
-
-```scss
-@include fixed-full;
-```
-
-#### 4. `link-hover-transition`
-
-**Purpose:** Adds a color transition effect to links on hover.
-
-**Parameters:** None
-
-**Usage:**
-
-```scss
-@include link-hover-transition;
-```
-
-#### 5. `message-style`
-
-**Purpose:** Applies common styling to message elements.
-
-**Parameters:** None
-
-**Usage:**
-
-```scss
-@include message-style;
-```
-
-### Key Styling Sections
-
-#### Body
-
-Applies typography, color, and spacing to the body element.
-
-```scss
-body {
-  @include typography($font-family-secondary);
-  color: $primary-text-color;
-  background-color: $primary-bg-color;
-  margin: 0;
-  padding: 30px 0 50px;
-}
-```
-
-#### Messages Container
-
-Styles the container for messages, including padding, background color, and shadow.
-
-```scss
-
-#
-#messages {
-  @include flex-container;
-  padding: 10px;
-  background-color: $secondary-bg-color;
-  box-shadow: $box-shadow;
-}
-```
-
-#### Input Fields
-
-Styles for chat and reply input fields, including background color, text color, and border properties.
-
-```scss
-.chat-input,
-.reply-input {
-  background-color: $secondary-bg-color;
-  color: $primary-text-color;
-  border-radius: $border-radius;
-  padding: 10px;
-  margin-bottom: 10px;
-  overflow: auto;
-  resize: vertical;
-  flex: 1;
-  border: 1px solid $border-color;
-  box-shadow: $box-shadow;
-}
-```
-
-#### Disconnected Overlay
-
-Styles for a modal overlay displayed when disconnected, including positioning, background color, and typography.
-
-```scss
-
-#
-#disconnected-overlay {
-  @include fixed-full;
-  display: none;
-  background-color: $modal-overlay-color;
-  z-index: 50;
-  @include flex-container;
-  color: white;
-  font-size: $font-size-large;
-
-  p {
-    @include typography($font-size: $font-size-large, $font-weight: $font-weight-bold);
-    line-height: 1.5;
-    margin-bottom: 20px;
-    animation: bounce $transition-speed infinite alternate;
-    position: relative;
-    color: firebrick;
-  }
-}
-```
-
-#### Buttons
-
-Styles for various buttons, including play, regenerate, cancel, and close buttons. Applies typography, border,
-background, and interaction effects.
-
-```scss
-.play-button,
-.regen-button,
-.cancel-button,
-.close-button {
-  @include typography($font-size: 1.5rem, $font-weight: $font-weight-bold);
-  border: 2px solid transparent;
-  background: $primary-bg-color;
-  cursor: pointer;
-  transition: all $transition-speed;
-  padding: 5px 10px;
-  border-radius: $border-radius;
-  text-decoration: unset;
-
-  &:focus,
-  &:hover {
-    outline: none;
-    background-color: darken($primary-bg-color, 5%);
-    border-color: $link-color;
-  }
-
-  &:active {
-    transform: scale(0.95);
-  }
-}
-```
-
-### Keyframes
-
-Animations for elements, such as `bounce` for bouncing effects and `spin` for continuous rotation.
-
-```scss
-@keyframes bounce {
-  0% {
-    transform: translateY(0);
-  }
-  100% {
-    transform: translateY(-10px);
-  }
-}
-
-@keyframes spin {
-  0% {
-    transform: rotate(0deg);
-  }
-  100% {
-    transform: rotate(360deg);
-  }
-}
-```
-
-This documentation provides a comprehensive overview of the SCSS mixins and styles used throughout the project,
-facilitating easier understanding and modification by developers.
-
-# resources\welcome\index.html
-
-## Developer Documentation: HTML Page Auto-Redirect
-
-This document provides an overview and implementation details for creating an HTML page that automatically redirects
-users to a new page. This technique is useful for redirecting traffic from old URLs to new ones, maintaining user
-experience during site maintenance, or simply guiding users through a multi-step process on a website.
-
-### Overview
-
-The provided HTML template employs a meta refresh tag for redirection and offers a fallback link if the automatic
-redirection does not occur. This ensures compatibility across various browsers and situations where JavaScript is
-disabled.
-
-### Implementation
-
-#### HTML Structure
-
-The HTML document is structured with the basic required elements: ``, ``, ``, and ``
-tags. The core functionality of redirection is implemented within the `` section of the document.
-
-##### Meta Refresh Tag
-
-```html
-
-
-```
-
-- `http-equiv="refresh"`: This attribute specifies that the HTTP-equiv meta tag is being used for automatic page
-  refresh.
-- `content="0;url=/index.html"`: The `content` attribute defines the time delay and target URL for the redirection. `0`
-  indicates that the redirection should occur immediately. Replace `/index.html` with the desired target URL.
-
-#### Fallback Link
-
-```html
-
If you are not redirected, click here.

-```
-
-In the body of the document, a paragraph (`
`) provides a textual message explaining the action to users. It includes
-an anchor (``) tag that serves as a manual fallback method for redirection. Users can click this link if the
-automatic redirection fails for any reason.
-
-### Usage
-
-1. **Customize the URL**: Change the `url` parameter in the meta tag and the `href` attribute in the anchor tag to the
-   desired destination URL.
-2. **Adjust the Delay**: If a delay before redirection is desired, modify the number in the `content` attribute of the
-   meta tag. For example, `content="5;url=/index.html"` will delay the redirection by 5 seconds.
-3. **Deployment**: Save the HTML code in a file and upload it to your web server. Ensure that the target URL is
-   accessible and correct.
-
-### Considerations
-
-- **SEO Implications**: Frequent use of meta refresh redirection can have negative implications for search engine
-  optimization (SEO), as search engines may view it as an attempt to manipulate page rankings.
-- **Accessibility**: Ensure that the fallback link is clearly visible and accessible for users who might have automatic
-  redirection disabled or are using screen readers.
-- **Browser Compatibility**: While the meta refresh tag is widely supported, testing across different browsers and
-  devices is recommended to ensure consistent behavior.
-
-### Conclusion
-
-The provided HTML template offers a simple and effective method for redirecting users to a new page. By following the
-implementation details and considering the usage notes, developers can seamlessly integrate this functionality into
-their web projects.
-
-# resources\welcome\main.js
-
-## Developer Documentation
-
-This documentation provides an overview and detailed explanation of the JavaScript functions used to enhance the
-interactivity of a web application. The application includes features such as modal pop-ups, tab navigation, dynamic
-content fetching, and theme switching.
-
-### Overview
-
-The script is structured around several key functionalities:
-
-- **Modal Operations**: Displaying and hiding modal pop-ups.
-- **Data Fetching**: Asynchronously fetching and displaying data within modals.
-- **Tab Navigation**: Dynamically updating tabbed content based on user interaction.
-- **User Information Handling**: Fetching and displaying user information upon page load.
-- **Theme Switching**: Allowing users to switch between different theme styles.
-
-### Functions
-
-#### showModal(endpoint)
-
-Displays a modal pop-up and fetches content to be displayed within the modal.
-
-- **Parameters**:
-    - `endpoint`: The API endpoint or URL from which to fetch the content to be displayed in the modal.
-
-- **Behavior**: This function first calls `fetchData(endpoint)` to asynchronously fetch content from the specified
-  endpoint. It then makes the modal visible by setting its display style to `block`.
-
-#### closeModal()
-
-Hides the modal pop-up.
-
-- **Behavior**: Sets the display style of the modal to `none`, effectively hiding it from view.
-
-#### async fetchData(endpoint)
-
-Asynchronously fetches data from a specified endpoint and displays it within the modal content area.
-
-- **Parameters**:
-    - `endpoint`: The API endpoint or URL from which to fetch content.
-
-- **Behavior**: This function attempts to fetch data from the specified endpoint. Upon success, it updates the modal's
-  content area with the fetched data. In case of an error, it logs the error to the console.
-
-#### updateTabs()
-
-Sets up event listeners for tab navigation, allowing users to switch between tabs.
-
-- **Behavior**: This function adds click event listeners to all elements with the class `.tab-button`. When a tab button
-  is clicked, the function updates the active state of both the tabs and the corresponding tab content, making the
-  clicked tab and its content visible while hiding others.
-
-#### Event Listeners Setup
-
-Upon the DOM content being fully loaded, the script sets up various event listeners for handling modal operations, user
-information fetching, and theme switching.
-
-- **Modal Operations**: Event listeners are added to close the modal either by clicking the close button or by clicking
-  outside the modal area.
-- **User Information Handling**: Fetches user information and updates the UI accordingly. It also sets up modals for
-  user settings, usage, privacy, and terms of service.
-- **Theme Switching**: Allows users to switch between predefined themes by clicking on theme buttons. The selected theme
-  is saved to `localStorage` for persistence across sessions.
-
-### Usage
-
-To utilize this script, ensure it is included in the HTML file where these functionalities are required. The HTML
-structure must include elements with the IDs and classes referenced by the script (
-e.g., `#modal`, `.tab-button`, `#theme_style`).
-
-### Conclusion
-
-This script provides a comprehensive set of functionalities for enhancing user interaction within a web application
-through modals, tab navigation, dynamic content fetching, and theme customization. By following the outlined
-documentation, developers can effectively implement and customize these features according to their application's needs.
-
-# resources\welcome\main.scss
-
-### Developer Documentation: App Type Styles
-
-This section of the stylesheet is dedicated to defining the appearance of elements classified under the `.app-type`
-class. This class is designed to stylize text elements in a specific way, making them stand out within the application's
-UI. Below is a detailed breakdown of the `.app-type` class and its properties.
-
-#### Import Statements
-
-Before defining the `.app-type` styles, the stylesheet imports two other files:
-
-1. `@import '../shared/schemes/normal';` - This line imports a stylesheet from a relative path that likely contains
-   color schemes or other styling constants used throughout the application. The 'normal' scheme could imply default or
-   primary styling options.
-
-2. `@import '../shared/main';` - This import statement brings in the main stylesheet, which could include global styles,
-   variables, mixins, or other foundational styles that are necessary for the application's consistent look and feel.
-
-#### `.app-type` Class Definition
-
-The `.app-type` class is applied to elements requiring a specific typographic treatment. The properties defined within
-this class are as follows:
-
-- `text-transform: uppercase;` - This property transforms the text of the element to uppercase, making it stand out and
-  providing a visual cue of its importance or category.
-
-- `font-size: 0.6em;` - Sets the font size to 0.6 times the size of the font of its parent element. This relative sizing
-  ensures that the `.app-type` elements maintain proportionality within different contexts.
-
-- `font-weight: 600;` - Applies a font weight of 600, making the text semi-bold. This weight is heavier than normal but
-  not as heavy as bold, striking a balance that enhances readability while drawing attention.
-
-- `background-color: rgba(0, 200, 255, 0.25);` - The background color is set to a semi-transparent light blue, using the
-  RGBA color model. The alpha value of 0.25 provides a subtle hint of color, ensuring the text remains the focal point.
-
-- `border-radius: 5px;` - Applies rounded corners to the element with a radius of 5 pixels, softening its appearance and
-  making it more visually appealing.
-
-- `padding: 2px 5px;` - Adds padding inside the element, with 2 pixels on the top and bottom and 5 pixels on the left
-  and right. This padding creates space around the text, improving legibility and aesthetic appeal.
-
-- `color: white;` - Sets the text color to white, ensuring high contrast against the semi-transparent light blue
-  background, which aids in readability and draws attention to the text.
-
-#### Usage
-
-To apply these styles, add the `class="app-type"` attribute to the desired HTML element. This class is particularly
-suited for labels, tags, or any small pieces of text that need to be highlighted within the application's interface.
-
-#### Example
-
-```html
-New
-```
-
-This example demonstrates how to apply the `.app-type` class to a `` element, which would render the text "NEW" in
-uppercase, with the specified styling applied, making it stand out as a distinct element within the UI.
-
-# resources\welcome\favicon.svg
-
-## Developer Documentation for SVG Illustration
-
-This SVG illustration is a complex graphic designed using XML-based markup. It features a variety of shapes, colors, and
-styles to create a detailed image. The SVG is structured with various elements such as ``, `
-

-
-
-
-```
-
-#### Manipulation and Styling
-
-SVGs can be styled and manipulated using CSS and JavaScript, respectively. For instance, to change the fill color of a
-path with a class `st0`, you can use:
-
-```css
-.st0 {
-    fill: #FF0000; /* Changes the fill color to red */
-}
-```
-
-Using JavaScript, you can add interactivity or dynamic changes to the SVG:
-
-```javascript
-document.querySelector('.st0').addEventListener('click', function () {
-    this.style.fill = '#00FF00'; // Changes the fill color to green on click
-});
-```
-
-#### Performance Considerations
-
-While SVGs are excellent for scalability and resolution independence, it's essential to keep the file size in check for
-web use, especially for complex graphics. Optimize SVGs using tools like SVGO to remove unnecessary code and reduce file
-size without affecting the visual quality.
-
-#### Conclusion
-
-This SVG graphic provides a scalable and customizable way to incorporate high-quality illustrations into your digital
-projects. By understanding the structure and integration methods, developers and designers can effectively leverage SVGs
-to enhance their applications' visual appeal and user experience.
-
-# main\resources\shared\_main.scss
-
-## Developer Documentation: SCSS Mixins and Styles
-
-This documentation outlines the SCSS mixins and styles used to create a cohesive and maintainable stylesheet for a web
-application. The SCSS code provided defines a set of mixins for common CSS patterns, applies these mixins throughout the
-stylesheet, and establishes a consistent theming approach using variables for colors, fonts, and other properties.
-
-### SCSS Mixins
-
-#### Typography Mixin
-
-```scss
-@mixin typography($font-family: $font-family-primary, $font-size: $font-size-base, $font-weight: $font-weight-normal) {
-  font-family: $font-family;
-  font-size: $font-size;
-  font-weight: $font-weight;
-}
-```
-
-This mixin allows for easy application of font styles, including family, size, and weight. Default values are set but
-can be overridden when the mixin is included.
-
-#### Flex Container Mixin
-
-```scss
-@mixin flex-container($direction: column) {
-  display: flex;
-  flex-direction: $direction;
-}
-```
-
-Simplifies the creation of flex containers, with an optional direction parameter that defaults to `column`.
-
-#### Fixed Full Mixin
-
-```scss
-@mixin fixed-full {
-  position: fixed;
-  top: 0;
-  left: 0;
-  width: 100vw;
-  height: 100%;
-}
-```
-
-Applies a fixed, full-screen layout to an element, useful for overlays and modals.
-
-#### Link Hover Transition Mixin
-
-```scss
-@mixin link-hover-transition {
-  transition: color $transition-speed;
-  &:hover {
-    color: $link-hover-color;
-  }
-}
-```
-
-Facilitates a smooth color transition on hover for link elements, leveraging a variable for transition speed.
-
-#### Message Style Mixin
-
-```scss
-@mixin message-style {
-  padding: 10px;
-  margin-bottom: 10px;
-  border-radius: $border-radius;
-}
-```
-
-Provides a consistent styling base for message elements, including padding, margin, and border-radius.
-
-### Usage Examples
-
-#### Applying a Mixin to an Element
-
-```scss
-body {
-  @include typography($font-family-secondary);
-  color: $primary-text-color;
-  background-color: $primary-bg-color;
-  margin: 0;
-  padding: 30px 0 50px;
-}
-```
-
-This example shows how to include the `typography` mixin with a custom font family for the `` element, along with
-additional styles.
-
-#### Flex Container with Row Direction
-
-```scss
-
-#
-#main-input,
-.reply-form,
-.code-execution {
-  @include flex-container(row);
-  padding: 5px;
-  width: -webkit-fill-available;
-  background-color: $primary-bg-color;
-}
-```
-
-Demonstrates the use of the `flex-container` mixin with a `row` direction for layout alignment.
-
-### Keyframes and Animations
-
-Animations are defined using `@keyframes` for specific UI effects:
-
-#### Bounce Animation
-
-```scss
-@keyframes bounce {
-  0% {
-    transform: translateY(0);
-  }
-  100% {
-    transform: translateY(-10px);
-  }
-}
-```
-
-#### Spin Animation
-
-```scss
-@keyframes spin {
-  0% {
-    transform: rotate(0deg);
-  }
-  100% {
-    transform: rotate(360deg);
-  }
-}
-```
-
-These keyframes are used to create dynamic visual effects, such as a bouncing or spinning animation, and can be applied
-to any element via the `animation` property.
-
-### Conclusion
-
-This documentation provides an overview of the SCSS mixins and styles used within the web application. By utilizing
-mixins, variables, and consistent styling practices, the stylesheet remains maintainable and scalable. Remember to
-replace the variable values with your project's specific design tokens and adjust the mixins as needed to fit your
-application's requirements.
-
-# main\resources\welcome\main.js
-
-## Developer Documentation
-
-### Overview
-
-This script is designed to enhance user interaction within a web application by managing modal windows, fetching and
-displaying user information, and handling theme changes dynamically. It leverages asynchronous JavaScript to fetch data
-and modify the DOM in response to user actions.
-
-#### Functions
-
-##### `showModal(endpoint)`
-
-Opens a modal window and fetches content from a specified endpoint to display within the modal.
-
-- **Parameters**
-    - `endpoint`: A string representing the URL from which to fetch content to be displayed in the modal.
-
-##### `closeModal()`
-
-Closes the modal window by setting its display style to 'none'.
-
-##### `fetchData(endpoint)`
-
-Fetches data asynchronously from a specified endpoint and displays it within the modal content area. It also handles
-loading states and errors.
-
-- **Parameters**
-    - `endpoint`: A string representing the URL from which to fetch content.
-
-##### Event Listeners
-
-Sets up various event listeners once the DOM content is fully loaded, handling:
-
-- Closing the modal when the close button or area outside the modal is clicked.
-- Fetching and displaying user information.
-- Handling theme changes based on user selection.
-
-#### Usage
-
-##### Modal Functionality
-
-To open a modal with content fetched from a specific endpoint, call `showModal('/your-endpoint')`. To close the modal,
-either call `closeModal()` directly, click the close button, or click outside the modal area.
-
-##### Fetching User Information
-
-Upon page load, the script automatically fetches user information from the 'userInfo' endpoint. If the user is logged
-in (indicated by the presence of a name in the response data), it updates various elements to reflect the user's state (
-e.g., displaying the username, showing logout link, and hiding the login link).
-
-##### Theme Management
-
-Users can switch themes by clicking on theme options. The script supports 'main', 'night', 'forest', and 'pony' themes.
-The chosen theme is saved to `localStorage` and persisted across sessions.
-
-#### Error Handling
-
-Errors during data fetch operations are logged to the console to aid in debugging.
-
-#### DOM Elements
-
-The script manipulates various DOM elements identified by specific IDs (`modal`, `modal-content`, `theme_style`, etc.)
-and classes (`.close`). Ensure these IDs and classes are present in your HTML to avoid script errors.
-
-#### Best Practices
-
-- Ensure all endpoints used with `showModal` are secure and authenticated as needed.
-- For accessibility, consider adding ARIA attributes and roles to the modal and ensuring keyboard navigability.
-
-#### Dependencies
-
-This script requires a modern browser with support for ES6 features (e.g., `async/await`, `fetch` API, template
-literals) and the DOM `ContentLoaded` event.
-
----
-This documentation provides an overview of the script's functionality and usage. For more detailed integration and
-customization, refer to the specific sections above.
-
-# main\resources\welcome\index.html
-
-## Developer Documentation: HTML Auto-Redirect Page
-
-This document provides an overview and code example for setting up an HTML page that automatically redirects users to a
-new page. This technique is useful for redirecting traffic from old URLs to new ones, maintaining user experience during
-site restructuring, or simply guiding users directly to a main or updated page.
-
-### Overview
-
-The HTML meta refresh tag is utilized to implement an automatic redirect. This tag instructs the browser to refresh the
-page and load a new URL after a specified number of seconds. In this example, the redirection is immediate (0 seconds).
-
-### Code Example
-
-```html
-
-
-
-    
-    
-
-
-
-
If you are not redirected, click here.

-
-
-```
-
-#### Components
-
-- **DOCTYPE Declaration**: `` specifies the document type and HTML version. This ensures the browser
-  renders the page in standards mode.
-
-- **Head Section**: Contains the meta refresh tag used for redirection.
-
-    - **Meta Refresh Tag**: `` is placed within the ``
-      section. It has two main components:
-        - `content="0"`: The number (in seconds) before the page is redirected. `0` makes the redirect instant.
-        - `url=/index.html`: Specifies the destination URL. Replace `/index.html` with your target URL.
-
-- **Body Section**: Provides a fallback for users whose browsers do not support meta refresh or have it disabled.
-
-    - **Fallback Link**: A simple message with a hyperlink (`click here`) guiding users to
-      manually click if the redirect does not work automatically.
-
-### Usage
-
-1. **Customization**: Replace `/index.html` in both the meta tag and the fallback hyperlink with the path to your new
-   page.
-2. **Deployment**: Save this code in an HTML file and upload it to your server or hosting environment in place of the
-   old page you wish to redirect from.
-3. **Testing**: Access the URL of the page where this redirect is set up to ensure it redirects immediately to the new
-   page. Verify the fallback link works by disabling meta refresh in your browser settings and accessing the page again.
-
-### Best Practices
-
-- **SEO Considerations**: Frequent use of meta refresh redirects can affect your site's SEO negatively. For permanent
-  redirects, a server-side 301 redirect is recommended.
-- **Accessibility**: Ensure the fallback link is clearly visible and accessible for users who might need it.
-- **Testing**: Test the redirect across different browsers and devices to ensure compatibility and user experience.
-
-### Conclusion
-
-The HTML meta refresh tag offers a simple method for page redirection. While useful for quick fixes or temporary
-redirects, consider more SEO-friendly methods for permanent solutions. Always provide a clear fallback option for the
-best user experience.
-
-# main\resources\welcome\favicon.png
-
-## Developer Documentation
-
-### Overview
-
-This document provides essential information for developers working on a project that involves handling PNG image files,
-extracting XMP metadata embedded within them, and potentially modifying or utilizing this metadata for further
-processing or analysis.
-
-### PNG File Format
-
-Portable Network Graphics (PNG) is a raster graphics file format that supports lossless data compression. PNG was
-designed for transferring images on the Internet, not for professional-quality print graphics, and therefore does not
-support non-RGB color spaces such as CMYK.
-
-#### Structure
-
-A PNG file consists of a PNG signature followed by a series of chunks. A chunk consists of four parts:
-
-1. **Length**: A 4-byte field indicating the number of bytes in the chunk's data field.
-2. **Chunk Type**: A 4-byte field defining the type of the chunk.
-3. **Chunk Data**: The actual data of the chunk, of length specified by the length field.
-4. **CRC**: A 4-byte field used to check for errors in the chunk.
-
-The first chunk after the PNG signature must be the IHDR chunk, which contains basic information about the PNG image
-such as width, height, bit depth, and color type.
-
-### XMP Metadata in PNG
-
-Extensible Metadata Platform (XMP) is a standard created by Adobe Systems for embedding metadata into digital media. In
-PNG files, XMP metadata is stored in an iTXt, tEXt, or zTXt chunk with the keyword 'XML:com.adobe.xmp'.
-
-#### Extracting XMP Metadata
-
-To extract XMP metadata from a PNG file, follow these steps:
-
-1. Read the PNG file and identify iTXt, tEXt, or zTXt chunks.
-2. Look for chunks with the keyword 'XML:com.adobe.xmp'.
-3. Extract the chunk data, which contains the XMP metadata in XML format.
-
-#### Parsing XMP Metadata
-
-Once you have extracted the XMP metadata, you can parse the XML to read the metadata fields. The metadata can include
-information such as the creator, creation date, modification date, and more.
-
-### Modifying XMP Metadata
-
-To modify XMP metadata:
-
-1. Parse the existing XMP metadata from the PNG file.
-2. Modify the XML document according to your needs.
-3. Encode the modified XML document back into the PNG file, replacing or adding to the existing metadata.
-
-### Example Code
-
-Below is a pseudocode example demonstrating how to extract XMP metadata from a PNG file:
-
-```python
-def extract_xmp_metadata(png_file_path):
-    with open(png_file_path, 'rb') as file:
-        content = file.read()
-    
-    # Look for the XMP metadata chunk
-    start_keyword = b'XML:com.adobe.xmp'
-    start_index = content.find(start_keyword)
-    
-    if start_index == -1:
-        return None
-    
-    # Extract and return the XMP metadata
-    metadata_start = start_index + len(start_keyword)
-    metadata_end = content.find(b'\x00', metadata_start)
-    xmp_metadata = content[metadata_start:metadata_end]
-    
-    return xmp_metadata
-
-
-## Example usage
-xmp_metadata = extract_xmp_metadata('example.png')
-if xmp_metadata:
-    print("XMP Metadata found:", xmp_metadata)
-else:
-    print("No XMP Metadata found.")
-```
-
-**Note:** This pseudocode is simplified for clarity and does not handle all aspects of PNG file parsing or XMP
-extraction, such as dealing with compressed iTXt chunks or validating chunk CRCs.
-
-### Conclusion
-
-This document outlines the basic approach for handling PNG files and extracting, parsing, and modifying XMP metadata
-within them. Developers can use this guide as a starting point for implementing more comprehensive solutions tailored to
-their specific project requirements.
-
-# main\resources\welcome\pony.scss
-
-## Developer Documentation: Styling App-Type Elements
-
-This guide provides an overview of the CSS styles applied to elements with the `.app-type` class. These styles are part
-of a larger styling framework that may include imports from shared schemes and main stylesheets. Below you will find a
-detailed description of each style property used and its impact on the appearance of `.app-type` elements.
-
-### Style Overview
-
-The `.app-type` class is designed to stylize specific text elements within an application, giving them a distinct
-appearance that includes uppercase lettering, a specific font size and weight, a background color with transparency,
-rounded corners, padding, and a text color set to white.
-
-#### Code Snippet
-
-```css
-.app-type {
-    text-transform: uppercase;
-    font-size: 0.6em;
-    font-weight: 600;
-    background-color: rgba(0, 200, 255, 0.25);
-    border-radius: 5px;
-    padding: 2px 5px;
-    color: white;
-}
-```
-
-### Style Properties
-
-- `text-transform: uppercase;` - This property transforms all the text of the element to uppercase, making it stand out
-  and indicating a specific category or type within the application.
-
-- `font-size: 0.6em;` - Sets the font size to 0.6 times the size of the font applicable to the element's parent, making
-  the text smaller and ensuring it doesn't overpower other text or elements nearby.
-
-- `font-weight: 600;` - Applies a font weight of 600, making the text semi-bold. This increases the text's prominence
-  without making it as heavy as bold text.
-
-- `background-color: rgba(0, 200, 255, 0.25);` - Specifies a background color using the RGBA color model. This color is
-  a light shade of cyan with a 25% opacity, allowing for the background behind the text to be partially visible, adding
-  depth to the design.
-
-- `border-radius: 5px;` - Rounds the corners of the element's background with a radius of 5px, softening the overall
-  look and making it more visually appealing.
-
-- `padding: 2px 5px;` - Adds padding inside the element, with 2px on the top and bottom, and 5px on the left and right.
-  This creates space around the text, making it more readable and aesthetically pleasing.
-
-- `color: white;` - Sets the text color to white, ensuring high contrast against the semi-transparent cyan background
-  for better readability.
-
-### Usage
-
-To apply these styles, add the `class="app-type"` attribute to any HTML element that you want to stylize accordingly.
-Typically, this class is used for small, categorizing text elements within larger applications to denote different
-sections or functionalities.
-
-#### Example
-
-```html
-Free Version
-```
-
-This will render the text "Free Version" with all the styles specified under the `.app-type` class, distinguishing it
-from other text elements on the page.
-
-### Conclusion
-
-The `.app-type` class provides a set of predefined styles aimed at enhancing the visual hierarchy and readability of
-specific types of text within an application. By following the guidelines outlined in this document, developers can
-ensure consistent and effective application of these styles across different parts of their projects.
-
-# main\resources\welcome\night.scss
-
-### Documentation for App Type Styles
-
-This section of the stylesheet is dedicated to defining the appearance of elements assigned with the `.app-type` class.
-This class is tailored to modify textual elements, giving them a distinctive look that aids in their identification or
-emphasis within the application. Below is a detailed breakdown of the `.app-type` class and its properties:
-
-#### Import Statements
-
-```css
-@import '../shared/schemes/night';
-@import '../shared/main';
-```
-
-Before defining the `.app-type` styles, the stylesheet imports two external files. These imports likely contain
-variables, mixins, or foundational styles that are necessary for the `.app-type` class to align with the overall design
-system of the application.
-
-- **`../shared/schemes/night`**: This import suggests that the `.app-type` class is part of a theme or scheme, possibly
-  tailored for a night or dark mode appearance.
-- **`../shared/main`**: This import likely contains global styles, variables, and mixins used throughout the
-  application, ensuring consistency in design.
-
-#### `.app-type` Class Properties
-
-```css
-.app-type {
-    text-transform: uppercase;
-    font-size: 0.6em;
-    font-weight: 600;
-    background-color: rgba(0, 200, 255, 0.25);
-    border-radius: 5px;
-    padding: 2px 5px;
-    color: white;
-}
-```
-
-- **`text-transform: uppercase;`**: Converts the text of the element to uppercase, making it stand out and improving
-  readability or emphasis.
-
-- **`font-size: 0.6em;`**: Sets the font size to 0.6 times the size of the font of its parent element. This relative
-  size ensures that the `.app-type` elements scale appropriately with their context.
-
-- **`font-weight: 600;`**: Applies a font weight of 600, making the text semi-bold. This increases the text's prominence
-  without making it overly bold.
-
-- **`background-color: rgba(0, 200, 255, 0.25);`**: Assigns a semi-transparent light blue background color to the
-  element. The use of RGBA color allows for transparency, letting the element's background blend with its surroundings
-  while still maintaining a distinct appearance.
-
-- **`border-radius: 5px;`**: Rounds the corners of the element with a radius of 5 pixels, softening its appearance and
-  making it more visually appealing.
-
-- **`padding: 2px 5px;`**: Adds padding inside the element, with 2 pixels on the top and bottom and 5 pixels on the
-  sides. This creates space around the content, making it more legible and aesthetically pleasing.
-
-- **`color: white;`**: Sets the text color to white, ensuring high contrast against the semi-transparent light blue
-  background, which aids in readability.
-
-#### Usage
-
-To apply these styles, add the `.app-type` class to the desired HTML elements. This class is primarily intended for
-textual elements that require emphasis or a distinctive appearance to stand out from other content. For example:
-
-```html
-Sample Text
-```
-
-This will transform the text within the `` tag according to the defined properties, making it uppercase,
-semi-bold, and applying the specified background color, padding, and border radius.
-
-#### Conclusion
-
-The `.app-type` class is designed to create a specific visual identity for text elements within the application. By
-defining a consistent style for these elements, the application maintains a cohesive look and feel, enhancing the user
-experience.
-
-# test\kotlin\com\simiacryptus\skyenet\webui\ActorTestAppServer.kt
-
-## ActorTestAppServer Documentation
-
-The `ActorTestAppServer` is a component of the SimiaCryptus Skyenet web application framework, designed to facilitate
-the testing of various actor implementations within a web server context. This document provides an overview of its
-functionality, setup, and usage.
-
-### Overview
-
-`ActorTestAppServer` extends the `ApplicationDirectory` class, setting up a web server environment on port `8082` by
-default. It is primarily used for testing different types of actors,
-including `SimpleActor`, `ParsedActor`, `ImageActor`, and various coding actors (`CodingActor`) with support for Scala,
-Kotlin, and Groovy interpreters.
-
-#### Key Components
-
-- **TestJokeDataStructure**: A data class for storing joke structure including setup, punchline, and type.
-- **JokeParser**: An interface extending `Function`, designed for parsing jokes.
-- **childWebApps**: A lazy-initialized list of `ChildWebApp` instances, each corresponding to a specific test
-  application for different actor types.
-- **toolServlet**: A property returning `null`, indicating no specific tool servlet is used in this application.
-
-#### Actors and Test Applications
-
-- **SimpleActorTestApp**: Tests `SimpleActor` capabilities, such as translating user requests into Pig Latin.
-- **ParsedActorTestApp**: Utilizes `ParsedActor` with a `JokeParser` for telling jokes.
-- **ImageActorTestApp**: For testing `ImageActor` functionalities.
-- **CodingActorTestApp**: Tests `CodingActor` with support for Scala, Kotlin, and Groovy interpreters.
-
-### Setup and Running
-
-To set up and run the `ActorTestAppServer`, follow these steps:
-
-1. **Mock User Configuration**: A mock user is configured for authentication and authorization testing purposes.
-2. **Authentication and Authorization Services**: `ApplicationServices` are set up with mock implementations
-   for `authenticationManager` and `authorizationManager` to bypass actual authentication and authorization mechanisms
-   for testing.
-3. **Starting the Server**: The server is started by invoking the `main` method with necessary arguments, if any.
-
-### Usage
-
-Once the server is running, you can access the different test applications through their respective endpoints:
-
-- `/test_simple`: Access the SimpleActor test application.
-- `/test_parsed_joke`: Interact with the ParsedActor for jokes.
-- `/images`: Test image processing capabilities with the ImageActor.
-- `/test_coding_scala`, `/test_coding_kotlin`, `/test_coding_groovy`: Test coding actors with Scala, Kotlin, and Groovy
-  support, respectively.
-
-### Customization
-
-To customize the `ActorTestAppServer`, consider modifying the `childWebApps` list to include additional test
-applications or actors as needed. You can also implement custom authentication and authorization mechanisms by modifying
-the `ApplicationServices.authenticationManager` and `ApplicationServices.authorizationManager` assignments in the `main`
-method.
-
-### Conclusion
-
-The `ActorTestAppServer` provides a flexible and straightforward way to test various actor implementations within a web
-server context. By following the setup and usage guidelines provided in this document, developers can effectively
-leverage this tool to enhance the development and testing of actor-based applications within the SimiaCryptus Skyenet
-framework.
-
-# main\resources\welcome\main.scss
-
-### Developer Documentation: App Type Styles
-
-This section of the stylesheet is dedicated to defining the appearance of elements classified under the `.app-type`
-class. The style rules specified here aim to create a distinctive look for these elements, making them stand out within
-the application's interface. Below is a detailed breakdown of the `.app-type` class and its properties:
-
-#### Import Statements
-
-Before defining the `.app-type` styles, the stylesheet imports two other files:
-
-1. **Normal Schemes**: `@import '../shared/schemes/normal';`
-    - This import statement includes the normal color schemes and potentially other shared style properties used across
-      the application. It ensures consistency in the visual design.
-
-2. **Main Shared Styles**: `@import '../shared/main';`
-    - This import includes the main shared styles that could consist of common typography, layout, or utility classes
-      that are reused throughout the application.
-
-#### `.app-type` Class
-
-The `.app-type` class is designed to be applied to textual elements within the application that require emphasis or
-differentiation from other text types. The styling attributes are as follows:
-
-- **Text Transformation**: `text-transform: uppercase;`
-    - Converts the text to uppercase for a more standout and uniform appearance.
-
-- **Font Size**: `font-size: 0.6em;`
-    - Sets the font size to 0.6 times the size of the font of the element's parent, making the text relatively smaller
-      and suitable for labels or tags.
-
-- **Font Weight**: `font-weight: 600;`
-    - Applies a font weight of 600, making the text semi-bold. This enhances readability and draws attention.
-
-- **Background Color**: `background-color: rgba(0, 200, 255, 0.25);`
-    - Sets a semi-transparent background color using RGBA (Red, Green, Blue, Alpha) values. The color is a light shade
-      of blue with 25% opacity, providing a subtle highlight without overwhelming the text color.
-
-- **Border Radius**: `border-radius: 5px;`
-    - Applies rounded corners with a radius of 5 pixels to the element, softening its appearance and making it more
-      visually appealing.
-
-- **Padding**: `padding: 2px 5px;`
-    - Adds padding inside the element, with 2 pixels on the top and bottom and 5 pixels on the left and right. This
-      creates space around the text, improving legibility and aesthetics.
-
-- **Color**: `color: white;`
-    - Sets the text color to white, ensuring high contrast against the semi-transparent blue background for optimal
-      readability.
-
-#### Usage
-
-To apply these styles, add the `class="app-type"` attribute to the desired HTML element. Example:
-
-```html
-Free
-```
-
-This will render the text "FREE" in uppercase, with a semi-bold font weight, smaller size, white color, and a light blue
-background, enclosed within a box with rounded corners and padding for spacing.
-
-#### Conclusion
-
-The `.app-type` class provides a standardized way to highlight specific types of information within the application,
-making them easily recognizable to users. By combining text transformations, color, and spacing, it creates a visually
-appealing element that can be used across various parts of the interface.
-
-# test\resources\logback.xml
-
-## Logback Configuration Documentation
-
-This document explains the structure and settings of a Logback XML configuration file used for logging in Java
-applications. The provided XML configuration snippet specifies how log messages are formatted and where they are
-outputted.
-
-### Overview
-
-Logback is a widely used logging framework in Java, offering a powerful mechanism for capturing and managing application
-logs. The configuration file (`logback.xml` or `logback-test.xml`) is where you define the behavior of the logging
-system, such as log levels, output formats, and destinations (console, files, etc.).
-
-### Configuration Elements
-
-#### ``
-
-The root element of the Logback configuration file. All configuration properties and appender definitions are nested
-within this element.
-
-#### Appender Configuration
-
-Appenders are responsible for delivering log events to their destination. This configuration defines a console appender
-named `STDOUT`.
-
-##### ``
-
-- `name`: A unique name for the appender, `STDOUT` in this case.
-- `class`: The fully qualified name of the appender class. `ch.qos.logback.core.ConsoleAppender` is used here, meaning
-  logs will be written to the console.
-
-###### ``
-
-Encoders format log events before they are written to their destination.
-
-- ``: Defines the format of the log message. The pattern provided here includes:
-    - `%d{HH:mm:ss.SSS}`: Timestamp in hours, minutes, seconds, and milliseconds.
-    - `[%thread]`: Name of the thread that generated the log event.
-    - `%-5level`: Log level (e.g., DEBUG, INFO) padded to 5 characters.
-    - `%logger{36}`: Name of the logger, truncated or padded to 36 characters.
-    - `- %msg%n`: The log message followed by a new line.
-
-#### Root Logger Configuration
-
-The root logger is the parent of all other loggers in the application, and it captures log messages from all levels
-unless configured otherwise.
-
-##### ``
-
-- `level`: The minimum level of messages that will be logged. Setting this to `debug` means that DEBUG, INFO, WARN,
-  ERROR, and TRACE level messages will be logged.
-- ``: This element links the root logger to the `STDOUT` appender, indicating that log
-  messages accepted by the root logger should be outputted to the console as defined by the `STDOUT` appender.
-
-### Conclusion
-
-This Logback configuration provides a basic setup for console logging in a Java application. By adjusting the appender
-configurations, log patterns, and log levels, developers can customize the logging behavior to suit their needs. This
-setup is particularly useful for development environments where immediate feedback in the console is valuable for
-troubleshooting and debugging.
-
-# test\resources\client_secret_google_oauth.json.kms
-
-#### Developer Documentation: Decoding and Understanding Encrypted Data
-
-This documentation provides an overview and guide on how to decode and understand an encrypted data string. The provided
-encrypted data is a complex string that requires a specific methodology to decode and interpret correctly. This document
-is intended for developers who need to work with and understand encrypted data within their applications.
-
-##### Overview
-
-Encrypted data is used to secure information so that it is not accessible or readable by unauthorized users. The example
-provided is an encrypted string that likely contains sensitive information. Decoding this data requires knowledge of the
-encryption method used and access to the necessary decryption keys or tokens.
-
-##### Prerequisites
-
-- Familiarity with encryption and decryption concepts.
-- Access to the decryption keys or tokens.
-- Knowledge of the encryption algorithm used (e.g., AES, RSA).
-- Required libraries or tools installed for decryption.
-
-##### Decryption Process
-
-The decryption process involves several steps, which are outlined below. Please note that the specific steps may vary
-depending on the encryption algorithm and tools used.
-
-1. **Identify the Encryption Algorithm**: Determine the encryption algorithm used for encrypting the data. This
-   information is crucial for selecting the correct decryption tool or library.
-
-2. **Access Decryption Keys/Tokens**: Ensure you have access to the necessary decryption keys or tokens. These keys are
-   essential for decrypting the data successfully.
-
-3. **Prepare the Environment**: Set up your development environment with the necessary decryption libraries or tools.
-   For example, if the encryption algorithm is AES, ensure that you have an AES decryption library available.
-
-4. **Decode the Encrypted String**: Use the decryption tool or library to decode the encrypted string. This step
-   typically involves passing the encrypted data and the decryption key to the decryption function provided by the
-   library.
-
-   ```python
-   from cryptography.fernet import Fernet
-
-   # Example decryption code
-   def decrypt_message(encrypted_message, key):
-       f = Fernet(key)
-       decrypted_message = f.decrypt(encrypted_message)
-       return decrypted_message.decode()
-
-   # Replace 'encrypted_data' and 'decryption_key' with the actual data and key
-   decrypted_data = decrypt_message(encrypted_data, decryption_key)
-   print("Decrypted message:", decrypted_data)
-   ```
-
-5. **Interpret the Decoded Data**: Once the data is decrypted, you will need to interpret or parse it based on its
-   format (e.g., JSON, XML) and the structure of the information it contains.
-
-6. **Implement Security Measures**: Ensure that the decrypted information is handled securely. Avoid logging sensitive
-   information or exposing it to unauthorized users.
-
-##### Best Practices
-
-- **Key Management**: Securely manage decryption keys. Limit access to these keys to only those who absolutely need it.
-- **Use Secure Channels**: When transmitting sensitive information, even in its encrypted form, always use secure
-  channels (e.g., HTTPS).
-- **Stay Updated**: Encryption algorithms and best practices evolve. Stay updated on the latest security advisories and
-  update your encryption methods accordingly.
-
-##### Conclusion
-
-Working with encrypted data requires a thorough understanding of encryption algorithms, access to decryption keys, and
-the use of appropriate tools or libraries. By following the steps outlined in this documentation, developers can decode
-and understand encrypted data securely and effectively.
-
-# test\kotlin\com\simiacryptus\skyenet\webui\TestOpenAPITool.kt
-
-### Developer Documentation: OpenAPI Tool Testing with JUnit
-
-This documentation provides a comprehensive guide on how to test OpenAPI Tool generation within a JUnit test
-environment. The example code demonstrates how to generate OpenAPI client code in Java by using the OpenAPI Generator
-tool programmatically within a test case.
-
-#### Overview
-
-The test case `TestOpenAPITool` is part of a larger project aimed at testing the functionality of the OpenAPI Generator
-tool. The test involves creating a temporary OpenAPI specification file and using the OpenAPI Generator tool to generate
-client code based on this specification.
-
-#### Pre-requisites
-
-- Java Development Kit (JDK) installed.
-- OpenAPI Generator CLI installed or accessible via project dependencies.
-- JUnit 5 test framework set up in the project.
-
-#### Test Case: `TestOpenAPITool`
-
-##### Step 1: Creating a Temporary OpenAPI Specification File
-
-The test begins by creating a temporary file to store an OpenAPI specification. This specification defines a simple
-API, "Gmail Labels API", which includes an endpoint for fetching Gmail labels.
-
-```java
-val tempFile = File.createTempFile("openapi", ".json").apply {
-
-writeText(
-    """
-    {
-      "openapi" : "3.0.0",
-      ...
-    }
-    """.trimIndent()
-  )
-
-deleteOnExit()
-}
-```
-
-This temporary file is set to be deleted when the JVM exits to ensure no residual files are left.
-
-##### Step 2: Generating Client Code
-
-The main part of the test involves invoking the OpenAPI Generator tool programmatically to generate client code based on
-the temporary OpenAPI specification file created in Step 1.
-
-```java
-try{
-val generator = "java"
-
-File("C:/Users/andre/Downloads/openapi/build/openapi-$generator").
-
-mkdirs()
-  org.openapitools.codegen.OpenAPIGenerator.
-
-main(
-        arrayOf(
-        "generate",
-      "--skip-validate-spec",
-                "-i","C:/Users/andre/Downloads/openapi.yaml",
-                "-g",generator,
-      "-o","C:/Users/andre/Downloads/openapi/build/openapi-$generator",
-)
-  )
-          }catch(e:SpecValidationException){
-        e.
-
-printStackTrace()
-}
-```
-
-In this snippet:
-
-- A target directory for the generated code is created.
-- The `OpenAPIGenerator.main` method is called with arguments to generate the client code. The arguments specify the
-  input file (`-i`), the generator type (`-g`, in this case, `java`), and the output directory (`-o`).
-- The `--skip-validate-spec` option is used to skip OpenAPI specification validation. This can be useful for testing or
-  development purposes but should be used with caution.
-
-##### Step 3: Handling Exceptions
-
-The code generation process is wrapped in a try-catch block to handle `SpecValidationException`, which is thrown if the
-OpenAPI specification is invalid. In a real-world scenario, additional exception handling might be necessary depending
-on the requirements.
-
-```java
-}catch(e:SpecValidationException){
-        e.
-
-printStackTrace()
-}
-```
-
-#### Conclusion
-
-This documentation outlines the steps to create a JUnit test case for generating client code using the OpenAPI Generator
-tool. It demonstrates creating a temporary OpenAPI specification file, invoking the OpenAPI Generator programmatically,
-and handling exceptions.
-
-This approach can be adapted and extended to test various OpenAPI specifications and generator configurations within a
-continuous integration pipeline or a development workflow.
-
-# test\resources\permissions\read.txt
-
-To create effective developer documentation for a given piece of code, it's essential to include several key components.
-These include a clear title or heading, a brief description of the functionality, any prerequisites or dependencies, a
-detailed explanation of the code, usage examples, parameters or inputs descriptions, outputs or return values
-explanations, and any additional notes or warnings. Without a specific code example from you, I'll outline a general
-structure that you can follow to document any piece of code.
-
-#### Developer Documentation Template
-
----
-
-##### Title or Heading
-
-*Provide a concise and descriptive title for the functionality or the piece of code.*
-
-##### Brief Description
-
-*Offer a short summary of what the code does and its purpose.*
-
-##### Prerequisites or Dependencies
-
-*List any required libraries, frameworks, or any other prerequisites needed to run the code.*
-
-##### Detailed Explanation
-
-*Explain how the code works in detail. This section can include the logic behind the code, the flow of execution, and
-any important algorithms or techniques used.*
-
-##### Code Example
-
-```language
-// Insert your code snippet here
-```
-
-##### Usage Examples
-
-*Provide one or more examples of how to use the code in real-world scenarios. Include any variations in use cases if
-applicable.*
-
-##### Parameters or Inputs
-
-*Detail each parameter or input the code accepts. Include data types, default values, and whether a parameter is
-optional or required.*
-
-- **param1** (`type`): Description. Default: `defaultValue`.
-- **param2** (`type`, Optional): Description.
-
-##### Outputs or Return Values
-
-*Describe what the code returns after execution. Include data types and what each type represents.*
-
-- **Return 1** (`type`): Description.
-- **Return 2** (`type`): Description of when this value is returned.
-
-##### Additional Notes or Warnings
-
-*Include any additional information that might be helpful for the user, such as edge cases, limitations,
-version-specific features, or upcoming deprecations.*
-
----
-
-This template is a starting point, and depending on the complexity and nature of the code, some sections may need to be
-expanded, or additional sections might be necessary. For more specific documentation, please provide the code snippet or
-more details about the functionality you wish to document.
-
-# test\resources\permissions\execute.txt
-
-To create comprehensive developer documentation for a given piece of code, it's essential to break down the
-documentation process into several key components. These components typically include an overview of the functionality,
-prerequisites for using the code, a detailed explanation of the code's workings, examples of use cases, and any
-additional notes or warnings that might be relevant. Below is a structured approach to documenting a hypothetical piece
-of code:
-
-#### Overview
-
-The `calculateInterest` function is designed to calculate the interest accrued on a principal amount over a specified
-period at a given interest rate. This function is useful for financial applications, banking software, and personal
-finance tools where interest calculation is required.
-
-#### Prerequisites
-
-Before using the `calculateInterest` function, ensure that the following conditions are met:
-
-- The programming environment supports JavaScript ES6 or later, as the function uses ES6 syntax.
-- Basic understanding of interest calculation principles.
-
-#### Function Signature
-
-```javascript
-function calculateInterest(principal, annualRate, timeInYears, compoundFrequency = 1) {
-    // Function body
-}
-```
-
-#### Parameters
-
-- `principal` (Number): The initial amount of money on which the interest is calculated.
-- `annualRate` (Number): The annual interest rate, expressed as a decimal (e.g., 0.05 for 5%).
-- `timeInYears` (Number): The time period in years over which the interest is calculated.
-- `compoundFrequency` (Number, optional): The number of times the interest is compounded per year. Defaults to 1 (simple
-  interest).
-
-#### Returns
-
-- (Number): The amount of interest accrued over the specified period.
-
-#### Example Usage
-
-```javascript
-// Calculating simple interest
-const simpleInterest = calculateInterest(1000, 0.05, 1);
-console.log(simpleInterest); // Output: 50
-
-// Calculating compound interest
-const compoundInterest = calculateInterest(1000, 0.05, 1, 12);
-console.log(compoundInterest); // Output will vary based on the compound frequency
-```
-
-#### Detailed Explanation
-
-The `calculateInterest` function calculates the interest by first determining if the interest is simple or compounded.
-For simple interest (when `compoundFrequency` is 1), the formula used is:
-
-\[ \text{Interest} = \text{Principal} \times \text{Rate} \times \text{Time} \]
-
-For compound interest, the formula applied is:
-
-\[ \text{Interest} = \text{Principal} \times \left(1 + \frac{\text{Rate}}{\text{Compound Frequency}}\right)
-^{\text{Compound Frequency} \times \text{Time}} - \text{Principal} \]
-
-The function then returns the calculated interest, excluding the principal amount.
-
-#### Notes and Warnings
-
-- Ensure that all numerical inputs are positive values to avoid unexpected results.
-- The function does not validate input types; it assumes all inputs are of the correct type and format.
-- The compound interest calculation assumes that compounding occurs uniformly over the specified time period.
-
-#### Conclusion
-
-The `calculateInterest` function is a versatile tool for calculating both simple and compound interest. By adjusting the
-parameters, users can tailor the function to suit various financial scenarios. This documentation should provide all the
-necessary information to understand and utilize the function effectively in your projects.
-
-# test\resources\permissions\globalkey.txt
-
-To create developer documentation for a piece of code, it's important to provide comprehensive, clear, and useful
-information that can help other developers understand and use the code effectively. The documentation should include an
-overview of what the code does, how it works, its dependencies, how to set it up, and examples of how to use it. Below
-is a template for documenting a generic piece of code, followed by a specific example based on the assumption that
-you're documenting a simple API.
-
-#### Template for Developer Documentation
-
----
-
-##### Name/Title
-
-**[Name or title of the code/library/API]**
-
-##### Overview
-
-Provide a brief description of what the code does and its purpose. Explain in simple terms but be precise.
-
-##### Features
-
-List the key features and functionalities of the code.
-
-##### Requirements
-
-Mention any dependencies, environment requirements, or prerequisites needed to run the code.
-
-##### Installation
-
-Step-by-step guide on how to install or set up the environment and the code.
-
-##### Configuration
-
-Details on how to configure the code or system if necessary.
-
-##### Usage
-
-Provide examples of how to use the code. Include simple code snippets and explain the parameters and return values.
-
-##### API Reference (if applicable)
-
-Document the API endpoints, methods, request parameters, and response objects if you're documenting an API.
-
-##### Examples
-
-Provide more comprehensive examples that show how to use the code in real-world scenarios.
-
-##### Troubleshooting
-
-Common issues and their solutions or workarounds.
-
-##### Contributing
-
-Guidelines for contributing to the codebase, including code style, tests, and review process.
-
-##### License
-
-Specify the license under which the code is released.
-
----
-
-#### Example: Simple API Documentation
-
----
-
-##### Name/Title
-
-**Simple Weather API**
-
-##### Overview
-
-The Simple Weather API allows developers to retrieve current weather information and forecasts. It supports multiple
-cities worldwide.
-
-##### Features
-
-- Current weather data retrieval
-- 5-day weather forecast
-- Support for multiple units (metric, imperial)
-
-##### Requirements
-
-- Internet connection
-- API key (obtainable after free registration)
-
-##### Installation
-
-This API is accessible over HTTP; no installation is required. Register at [website] to obtain your API key.
-
-##### Configuration
-
-Include your API key in the header of each request:
-
-```
-x-api-key: YOUR_API_KEY
-```
-
-##### Usage
-
-**Get Current Weather:**
-
-```
-GET /weather/current?city={cityName}&units={units}
-```
-
-Parameters:
-
-- `cityName` - Name of the city (e.g., "London")
-- `units` - Units of measurement ("metric" or "imperial")
-
-**Get 5-Day Forecast:**
-
-```
-GET /weather/forecast?city={cityName}&units={units}
-```
-
-##### Examples
-
-**Request Current Weather in London (Metric):**
-
-```
-GET /weather/current?city=London&units=metric
-```
-
-**Response:**
-
-```json
-{
-  "temperature": 15,
-  "description": "Partly cloudy",
-  "humidity": 73
-}
-```
-
-##### Troubleshooting
-
-- **Issue:** API key not recognized.  
-  **Solution:** Ensure your API key is correct and included in the request header.
-
-##### Contributing
-
-Contributions are welcome! Please fork the repository and submit a pull request with your changes. Ensure your code
-adheres to the project's code style and includes tests.
-
-##### License
-
-This API is released under the MIT License.
-
----
-
-This template and example should give you a good starting point for documenting your own code. Remember, the goal of
-documentation is to make it easier for others (and your future self) to understand and use your code, so clarity and
-completeness are key.
-
-# test\resources\permissions\write.txt
-
-To create comprehensive developer documentation for a piece of code, it's essential to include several key components
-that will make it easier for other developers to understand, use, and possibly contribute to the code. These components
-typically include an introduction or overview, installation instructions, usage examples, API documentation,
-contribution guidelines, license information, and contact information for the maintainers. Below is a structured
-approach to writing developer documentation, using a hypothetical piece of code as an example.
-
-#### Example Code
-
-```python
-def add_numbers(a, b):
-    """
-    Adds two numbers together.
-
-    Parameters:
-    a (int): The first number.
-    b (int): The second number.
-
-    Returns:
-    int: The sum of a and b.
-    """
-    return a + b
-```
-
-#### Developer Documentation Structure
-
-##### Introduction
-
-Start with an introduction to your code. Explain what it does, why it's useful, and any background information that
-might be relevant.
-
-**Example Introduction:**
-> Welcome to the `NumberOperations` library, a simple Python module designed to perform basic arithmetic operations.
-> This library is perfect for educational purposes or for small-scale projects that require basic mathematical
-> operations
-> without the overhead of more complex libraries.
-
-##### Installation Instructions
-
-Provide clear, step-by-step instructions on how to install your code. Include any prerequisites or dependencies if
-applicable.
-
-**Example Installation Instructions:**
-> To install `NumberOperations`, you'll need Python installed on your system. This library has been tested with Python
-> 3.6 and above. You can install it directly from the source by cloning this repository and running:
-> ```
-> python setup.py install
-> ```
-
-##### Usage Examples
-
-Showcase how to use your code with simple examples. This helps users understand how to implement your code in their
-projects.
-
-**Example Usage:**
-> Here's a quick example of how to use the `add_numbers` function from the `NumberOperations` library:
-> ```python
-> from number_operations import add_numbers
->
-> result = add_numbers(5, 7)
-> print(f"The sum is {result}")
-> ```
-> This will output: `The sum is 12`
-
-##### API Documentation
-
-Detail each function, class, or module provided by your code. Include parameters, return types, and a short description
-of what each does.
-
-**Example API Documentation:**
-> **`add_numbers(a, b)`**
-> - **Parameters:** `a` (int) - The first number, `b` (int) - The second number.
-> - **Returns:** (int) - The sum of `a` and `b`.
-
-##### Contribution Guidelines
-
-Encourage contributions by providing guidelines on how others can contribute to your project. Include instructions for
-submitting pull requests, coding standards, and how to report bugs.
-
-**Example Contribution Guidelines:**
-> Contributions to `NumberOperations` are welcome! Please submit pull requests on GitHub and ensure that your code
-> follows the PEP 8 coding standards. For bug reports, please use the GitHub issues tab to report them.
-
-##### License Information
-
-Include the license under which your code is released. This informs users how they can legally use your code.
-
-**Example License Information:**
-> `NumberOperations` is released under the MIT License. See the LICENSE file for more details.
-
-##### Contact Information
-
-Provide a way for users to contact you or the project maintainers for further assistance or inquiries.
-
-**Example Contact Information:**
-> For any questions or concerns regarding `NumberOperations`, please contact us at example@email.com.
-
-By following this structured approach and adapting each section to fit your specific project, you can create effective
-and useful developer documentation that will help your project gain users and contributors.
-
-# test\resources\permissions\admin.txt
-
-Certainly! To create effective developer documentation for a given code example, it's essential to cover several key
-aspects, including an overview of the functionality, prerequisites, how to set up the environment, a step-by-step guide
-on how to use the code, and any additional notes that might help the developer understand the nuances of the code. Let's
-proceed with an example code snippet you might want to document.
-
-For demonstration purposes, let's assume the code snippet is a simple Python function that calculates and returns the
-factorial of a number using recursion:
-
-```python
-def factorial(n):
-    """Calculate the factorial of a number using recursion.
-
-    Args:
-        n (int): The number to calculate the factorial of.
-
-    Returns:
-        int: The factorial of the number.
-    """
-    if n == 1:
-        return 1
-    else:
-        return n * factorial(n-1)
-```
-
-#### Developer Documentation for Factorial Function
-
-##### Overview
-
-This document provides details on the `factorial` function, a Python implementation that calculates the factorial of a
-given number using recursion. The factorial of a number is the product of all positive integers less than or equal to
-the number. It is denoted by n! and is defined for all positive integers and zero.
-
-##### Prerequisites
-
-- Python 3.x installed on the system.
-
-##### Environment Setup
-
-No additional environment setup is required for this function, as it uses Python's standard library.
-
-##### Using the Code
-
-1. **Import the Function**: First, ensure that the `factorial` function is accessible by your script. If it's defined in
-   a separate module, you'll need to import it.
-
-2. **Call the Function**: Use the function by passing an integer value to it. For example, `factorial(5)` will calculate
-   the factorial of 5.
-
-```python
-result = factorial(5)
-print(f"The factorial of 5 is {result}")
-```
-
-This will output:
-
-```
-The factorial of 5 is 120
-```
-
-##### Notes
-
-- The function uses recursion to calculate the factorial. This means it calls itself with a decremented value of `n`
-  until `n` equals 1, at which point it returns 1.
-- Ensure that the input is a positive integer, as the factorial is not defined for negative numbers, and the function
-  does not handle non-integer inputs or zero.
-- The maximum value of `n` that can be handled is limited by Python's recursion depth limit, which can be checked or set
-  using `sys.getrecursionlimit()` and `sys.setrecursionlimit(limit)` respectively.
-
-##### Conclusion
-
-The `factorial` function provides a straightforward and efficient method to calculate the factorial of a number using
-recursion. It is a fundamental example of both recursion and the calculation of a mathematical series in computer
-science.
-
----
-
-This template can be adapted for documenting other code snippets by adjusting the sections to fit the context and
-requirements of the new code.
-
diff --git a/webui/src/main/kotlin/com/simiacryptus/diff/FileValidationUtils.kt b/webui/src/main/kotlin/com/simiacryptus/diff/FileValidationUtils.kt
index f5460535..580c6d99 100644
--- a/webui/src/main/kotlin/com/simiacryptus/diff/FileValidationUtils.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/diff/FileValidationUtils.kt
@@ -92,6 +92,7 @@ class FileValidationUtils {
                 !file.exists() -> false
                 file.isDirectory -> false
                 file.name.startsWith(".") -> false
+                file.name.endsWith(".data") -> true
                 file.length() > (256 * 1024) -> false
                 isGitignore(file.toPath()) -> false
                 file.extension.lowercase(Locale.getDefault()) in setOf(
@@ -114,6 +115,7 @@ class FileValidationUtils {
             return data.flatMap {
                 (when {
                     it.name.startsWith(".") -> arrayOf()
+                    it.name.endsWith(".data") -> arrayOf(it)
                     isGitignore(it.toPath()) -> arrayOf()
                     it.length() > 1e6 -> arrayOf()
                     it.extension.lowercase(Locale.getDefault()) in
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/AutoPlanChatApp.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/AutoPlanChatApp.kt
index 7d2af73f..0dd2b725 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/AutoPlanChatApp.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/AutoPlanChatApp.kt
@@ -2,11 +2,13 @@ package com.simiacryptus.skyenet.apps.general
 
 import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.models.ChatModel
 import com.simiacryptus.skyenet.TabbedDisplay
 import com.simiacryptus.skyenet.apps.plan.PlanCoordinator
 import com.simiacryptus.skyenet.apps.plan.PlanSettings
 import com.simiacryptus.skyenet.apps.plan.PlanTaskBase
+import com.simiacryptus.skyenet.apps.plan.TaskSettings
 import com.simiacryptus.skyenet.apps.plan.TaskType
 import com.simiacryptus.skyenet.apps.plan.file.FileModificationTask.FileModificationTaskData
 import com.simiacryptus.skyenet.core.actors.ParsedActor
@@ -17,7 +19,6 @@ import com.simiacryptus.skyenet.util.MarkdownUtil.renderMarkdown
 import com.simiacryptus.skyenet.webui.application.ApplicationInterface
 import com.simiacryptus.skyenet.webui.session.SessionTask
 import com.simiacryptus.util.JsonUtil
-import org.slf4j.LoggerFactory
 import java.io.File
 import java.util.Date
 import java.util.UUID
@@ -33,6 +34,7 @@ open class AutoPlanChatApp(
     domainName: String = "localhost",
     showMenubar: Boolean = true,
     api: API? = null,
+    api2: OpenAIClient,
     val maxTaskHistoryChars: Int = 20000,
     val maxTasksPerIteration: Int = 3,
     val maxIterations: Int = 100
@@ -45,6 +47,7 @@ open class AutoPlanChatApp(
     domainName = domainName,
     showMenubar = showMenubar,
     api = api,
+    api2 = api2
 ) {
     override val stickyInput = true
     override val singleInput = false
@@ -220,7 +223,7 @@ open class AutoPlanChatApp(
                         tabbedDisplay["Task Execution $currentTaskId"] = taskExecutionTask.placeholder
                         val future = executor.submit {
                             try {
-                                runTask(api, task, coordinator, currentTask, currentTaskId, userMessage, taskExecutionTask, thinkingStatus.get())
+                                runTask(api, api2, task, coordinator, currentTask, currentTaskId, userMessage, taskExecutionTask, thinkingStatus.get())
                             } catch (e: Exception) {
                                 taskExecutionTask.error(ui, e)
                                 log.error("Error executing task", e)
@@ -265,6 +268,7 @@ open class AutoPlanChatApp(
 
     protected open fun runTask(
         api: ChatClient,
+        api2: OpenAIClient,
         task: SessionTask,
         coordinator: PlanCoordinator,
         currentTask: PlanTaskBase,
@@ -283,14 +287,22 @@ open class AutoPlanChatApp(
         val taskImpl = TaskType.Companion.getImpl(coordinator.planSettings, currentTask)
         val result = StringBuilder()
         taskImpl.run(
-            agent = coordinator,
+            agent = coordinator.copy(
+                planSettings = coordinator.planSettings.copy(
+                    taskSettings = coordinator.planSettings.taskSettings.toList().toTypedArray().toMap().toMutableMap().apply {
+                        this["TaskPlanning"] = TaskSettings(enabled = false, model = null)
+                    }
+                )
+            ),
             messages = listOf(
                 userMessage,
                 "Current thinking status:\n${formatThinkingStatus(thinkingStatus!!)}"
             ) + formatEvalRecords(),
             task = taskExecutionTask,
             api = api,
-            resultFn = { result.append(it) }
+            resultFn ={ result.append(it) },
+            api2 = api2,
+            planSettings = planSettings,
         )
         return result.toString()
     }
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanAheadApp.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanAheadApp.kt
index 58b1f092..ab725922 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanAheadApp.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanAheadApp.kt
@@ -2,6 +2,7 @@ package com.simiacryptus.skyenet.apps.general
 
 import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.models.ChatModel
 import com.simiacryptus.jopenai.models.TextModel
 import com.simiacryptus.skyenet.apps.plan.PlanCoordinator
@@ -29,6 +30,7 @@ open class PlanAheadApp(
     showMenubar: Boolean = true,
     val initialPlan: TaskBreakdownWithPrompt? = null,
     val api: API? = null,
+    val api2: OpenAIClient,
 ) : ApplicationServer(
     applicationName = applicationName,
     path = path,
@@ -59,7 +61,7 @@ open class PlanAheadApp(
                         root = planSettings?.workingDir?.let { File(it).toPath() } ?: dataStorage.getDataDir(user, session).toPath(),
                         planSettings = planSettings!!
                     )
-                    coordinator.executeTaskBreakdownWithPrompt(JsonUtil.toJson(initialPlan), api!!, ui.newTask())
+                    coordinator.executeTaskBreakdownWithPrompt(JsonUtil.toJson(initialPlan), api!!, api2, ui.newTask())
                 } catch (e: Throwable) {
                     ui.newTask().error(ui, e)
                     log.warn("Error", e)
@@ -98,7 +100,7 @@ open class PlanAheadApp(
                 planSettings = coordinator.planSettings,
                 api = api
             )
-            coordinator.executePlan(plan.plan, task, userMessage = userMessage, api = api)
+            coordinator.executePlan(plan.plan, task, userMessage = userMessage, api = api, api2 = api2)
         } catch (e: Throwable) {
             ui.newTask().error(ui, e)
             log.warn("Error", e)
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanChatApp.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanChatApp.kt
index 086826c8..a6649382 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanChatApp.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/general/PlanChatApp.kt
@@ -2,6 +2,7 @@ package com.simiacryptus.skyenet.apps.general
 
 import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.models.ChatModel
 import com.simiacryptus.jopenai.models.TextModel
 import com.simiacryptus.skyenet.apps.plan.*
@@ -24,6 +25,7 @@ open class PlanChatApp(
     showMenubar: Boolean = true,
     initialPlan: TaskBreakdownWithPrompt? = null,
     api: API? = null,
+    api2: OpenAIClient,
 ) : PlanAheadApp(
     applicationName = applicationName,
     path = path,
@@ -34,6 +36,7 @@ open class PlanChatApp(
     showMenubar = showMenubar,
     initialPlan = initialPlan,
     api = api,
+    api2 = api2,
 ) {
     override val stickyInput = true
     override val singleInput = false
@@ -52,7 +55,8 @@ open class PlanChatApp(
                     ui = ui,
                     session = session,
                     user = user,
-                    api = api
+                    api = api,
+                    api2 = api2,
                 )
             }
             sessionHandler.handleUserMessage(
@@ -65,7 +69,8 @@ open class PlanChatApp(
         val ui: ApplicationInterface,
         val session: Session,
         val user: User?,
-        val api: API
+        val api: API,
+        val api2: OpenAIClient,
     ) {
         val messageHistory: MutableList = mutableListOf()
 
@@ -118,7 +123,8 @@ open class PlanChatApp(
                     plan = modifiedPlan,
                     task = sessionTask,
                     userMessage = userMessage,
-                    api = api
+                    api = api,
+                    api2 = api2,
                 )
                 val response = planProcessingState.taskResult["respond_to_chat"] as? String
                 if (response != null) {
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/CodeParsingModel.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/CodeParsingModel.kt
index c4d02e3c..34f59e1a 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/CodeParsingModel.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/CodeParsingModel.kt
@@ -18,17 +18,7 @@ open class CodeParsingModel(
         val newData = newData as CodeData
         return CodeData(
             id = newData.id ?: runningDocument.id,
-            content = mergeContent(runningDocument.content, newData.content).takeIf { it.isNotEmpty() },
-            entities = mergeEntities(runningDocument.entities, newData.entities).takeIf { it.isNotEmpty() },
-            metadata = mergeMetadata(runningDocument.metadata, newData.metadata)
-        )
-    }
-
-    protected open fun mergeMetadata(existing: CodeMetadata?, new: CodeMetadata?): CodeMetadata {
-        return CodeMetadata(
-            language = new?.language ?: existing?.language,
-            libraries = ((existing?.libraries ?: emptyList()) + (new?.libraries ?: emptyList())).distinct(),
-            properties = ((existing?.properties ?: emptyMap()) + (new?.properties ?: emptyMap())).takeIf { it.isNotEmpty() }
+            content_list = mergeContent(runningDocument.content_list, newData.content_list).takeIf { it.isNotEmpty() },
         )
     }
 
@@ -49,41 +39,16 @@ open class CodeParsingModel(
     }
 
     protected open fun mergeContentData(existing: CodeContent, new: CodeContent) = existing.copy(
-        content = mergeContent(existing.content, new.content).takeIf { it.isNotEmpty() },
-        entities = ((existing.entities ?: emptyList()) + (new.entities ?: emptyList())).distinct()
-            .takeIf { it.isNotEmpty() },
+        content_list = mergeContent(existing.content_list, new.content_list).takeIf { it.isNotEmpty() },
         tags = ((existing.tags ?: emptyList()) + (new.tags ?: emptyList())).distinct().takeIf { it.isNotEmpty() }
     )
 
-    protected open fun mergeEntities(
-        existingEntities: Map?,
-        newEntities: Map?
-    ) = ((existingEntities?.keys ?: emptySet()) + (newEntities?.keys ?: emptySet())).associateWith { key ->
-        val existing = existingEntities?.get(key)
-        val new = newEntities?.get(key)
-        when {
-            existing == null -> new!!
-            new == null -> existing
-            else -> mergeEntityData(existing, new)
-        }
-    }
-
-    protected open fun mergeEntityData(existing: CodeEntity, new: CodeEntity) = existing.copy(
-        aliases = ((existing.aliases ?: emptyList()) + (new.aliases ?: emptyList())).distinct()
-            .takeIf { it.isNotEmpty() },
-        properties = ((existing.properties ?: emptyMap()) + (new.properties ?: emptyMap())).takeIf { it.isNotEmpty() },
-        relations = ((existing.relations ?: emptyMap()) + (new.relations ?: emptyMap())).takeIf { it.isNotEmpty() },
-        type = new.type ?: existing.type
-    )
-
     open val promptSuffix = """
 Parse the code into a structured format that describes its components:
-1. Identify functions, classes, and other code structures.
-2. Extract comments and document them with their associated code.
-3. Capture any dependencies or libraries used in the code.
-4. Extract metadata such as programming language and version if available.
-5. Assign relevant tags to each code section to improve searchability and categorization.
-6. Do not copy data from the accumulated code JSON to your response; it is provided for context only.
+1. Separate the content into sections, paragraphs, statements, etc.
+2. All source content should be included in the output, with paraphrasing, corrections, and context as needed
+3. Each content leaf node text should be simple and self-contained
+4. Assign relevant tags to each node to improve searchability and categorization.
         """.trimMargin()
 
     open val exampleInstance = CodeData()
@@ -105,32 +70,16 @@ Parse the code into a structured format that describes its components:
 
     data class CodeData(
         @Description("Code identifier") override val id: String? = null,
-        @Description("Entities extracted") val entities: Map? = null,
-        @Description("Hierarchical structure and data") override val content: List? = null,
-        @Description("Code metadata") override val metadata: CodeMetadata? = null
+        @Description("Hierarchical structure and data") override val content_list: List? = null,
     ) : ParsingModel.DocumentData
 
-    data class CodeEntity(
-        @Description("Aliases for the entity") val aliases: List? = null,
-        @Description("Entity attributes extracted from the code") val properties: Map? = null,
-        @Description("Entity relationships extracted from the code") val relations: Map? = null,
-        @Description("Entity type (e.g., function, class, variable)") val type: String? = null
-    )
-
     data class CodeContent(
         @Description("Content type, e.g. function, class, comment") override val type: String = "",
         @Description("Brief, self-contained text either copied, paraphrased, or summarized") override val text: String? = null,
-        @Description("Sub-elements") override val content: List? = null,
-        @Description("Related entities by ID") val entities: List? = null,
+        @Description("Sub-elements") override val content_list: List? = null,
         @Description("Tags - related topics and non-entity indexing") override val tags: List? = null
     ) : ParsingModel.ContentData
 
-    data class CodeMetadata(
-        @Description("Programming language") val language: String? = null,
-        @Description("Libraries or dependencies associated with the code") val libraries: List? = null,
-        @Description("Other metadata") val properties: Map? = null,
-    ) : ParsingModel.DocumentMetadata
-
     companion object {
         val log = org.slf4j.LoggerFactory.getLogger(CodeParsingModel::class.java)
     }
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParserApp.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParserApp.kt
index 4953fdc1..60b6c51f 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParserApp.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParserApp.kt
@@ -3,6 +3,8 @@ package com.simiacryptus.skyenet.apps.parse
 import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
 import com.simiacryptus.skyenet.TabbedDisplay
+import com.simiacryptus.skyenet.apps.parse.ParsingModel.DocumentData
+import com.simiacryptus.skyenet.apps.parse.ProgressState.Companion.progressBar
 import com.simiacryptus.skyenet.core.platform.Session
 import com.simiacryptus.skyenet.core.platform.model.User
 import com.simiacryptus.skyenet.util.MarkdownUtil
@@ -12,13 +14,11 @@ import com.simiacryptus.skyenet.webui.application.ApplicationSocketManager
 import com.simiacryptus.skyenet.webui.session.SessionTask
 import com.simiacryptus.skyenet.webui.session.SocketManager
 import com.simiacryptus.util.JsonUtil
-import org.intellij.lang.annotations.Language
 import java.awt.image.BufferedImage
 import java.io.File
 import java.nio.file.Path
 import java.util.*
 import javax.imageio.ImageIO
-import kotlin.io.path.name
 import kotlin.math.min
 
 open class DocumentParserApp(
@@ -32,7 +32,7 @@ open class DocumentParserApp(
             else -> TextReader(it)
         }
     },
-    val fileInput: Path? = null,
+    val fileInputs: List? = null,
 ) : ApplicationServer(
     applicationName = applicationName,
     path = path,
@@ -45,35 +45,33 @@ open class DocumentParserApp(
         val socketManager = super.newSession(user, session)
         val ui = (socketManager as ApplicationSocketManager).applicationInterface
         val settings = getSettings(session, user, Settings::class.java) ?: Settings()
-        if (null == (fileInput ?: settings.fileInput)) {
+        val app = this
+        if (null == (fileInputs ?: settings.fileInputs)) {
             log.info("No file input provided")
-        } else socketManager.pool.submit {
-            run(
-                task = ui.newTask(),
-                ui = ui,
-                fileInput = (this.fileInput ?: settings.fileInput?.let { File(it).toPath() }
-                ?: error("File input not provided")).apply {
-                    if (!toFile().exists()) error("File not found: $this")
-                },
-                maxPages = settings.maxPages.coerceAtMost(Int.MAX_VALUE),
-                settings = settings,
-                pagesPerBatch = settings.pagesPerBatch,
-            )
+        } else (fileInputs ?: settings.fileInputs).apply {
+            val progressBar = progressBar(ui.newTask())
+            socketManager.pool.submit {
+                run(
+                    mainTask = ui.newTask(),
+                    ui = ui,
+                    fileInputs = (app.fileInputs ?: settings.fileInputs?.map { File(it).toPath() } ?: error("File input not provided")),
+                    maxPages = settings.maxPages.coerceAtMost(Int.MAX_VALUE),
+                    settings = settings,
+                    pagesPerBatch = settings.pagesPerBatch,
+                    progressBar = progressBar,
+                )
+            }
         }
         return socketManager
     }
 
     override fun userMessage(session: Session, user: User?, userMessage: String, ui: ApplicationInterface, api: API) {
         val settings = getSettings(session, user, Settings::class.java) ?: Settings()
-        val fileInput =
-            (fileInput ?: settings.fileInput?.let { File(it).toPath() } ?: error("File input not provided")).apply {
-                if (!toFile().exists()) error("File not found: $this")
-            }
         ui.socketManager!!.pool.submit {
             run(
-                task = ui.newTask(),
+                mainTask = ui.newTask(),
                 ui = ui,
-                fileInput = fileInput,
+                fileInputs = (this.fileInputs ?: settings.fileInputs?.map { File(it).toPath() } ?: error("File input not provided")),
                 maxPages = settings.maxPages.coerceAtMost(Int.MAX_VALUE),
                 settings = settings,
                 pagesPerBatch = settings.pagesPerBatch,
@@ -82,172 +80,174 @@ open class DocumentParserApp(
     }
 
     private fun run(
-        task: SessionTask,
+        mainTask: SessionTask,
         ui: ApplicationInterface,
-        fileInput: Path,
+        fileInputs: List,
         maxPages: Int,
         settings: Settings,
         pagesPerBatch: Int,
+        progressBar: ProgressState? = null
     ) {
         try {
-            val pdfFile = fileInput.toFile()
-            if (!pdfFile.exists() || !pdfFile.isFile || !pdfFile.name.endsWith(".pdf", ignoreCase = true)) {
-                throw IllegalArgumentException("Invalid PDF file: $pdfFile")
+            mainTask.header("PDF Extractor")
+            val api = (api as ChatClient).getChildClient().apply {
+                val createFile = mainTask.createFile(".logs/api-${UUID.randomUUID()}.log")
+                createFile.second?.apply {
+                    logStreams += this.outputStream().buffered()
+                    mainTask.verbose("API log: $this")
+                }
             }
-            task.add(MarkdownUtil.renderMarkdown("# PDF Extractor", ui = ui))
-            val outputDir = root.resolve("output").apply { mkdirs() }
-            lateinit var runningDocument: ParsingModel.DocumentData
-            reader(pdfFile).use { reader ->
-                runningDocument = parsingModel.newDocument()
-                var previousPageText = "" // Keep this for context
-                task.add(
-                    MarkdownUtil.renderMarkdown(
-                        """
-                        ## Processing PDF: ${pdfFile.name}
-                        Total pages: ${reader.getPageCount()}
-                        """.trimIndent(), ui = ui
-                    )
-                )
-                val pageCount = minOf(reader.getPageCount(), maxPages)
-                val tabs = TabbedDisplay(task)
-                for (batchStart in 0 until pageCount step pagesPerBatch) {
-                    val batchEnd = min(batchStart + pagesPerBatch, pageCount)
-                    val pageTask = ui.newTask(false)
-                    val pageTabs = TabbedDisplay(pageTask.apply {
-                        val label =
-                            if ((batchStart + 1) != batchEnd) "Pages ${batchStart + 1}-${batchEnd}" else "Page ${batchStart + 1}"
-                        tabs[label] = this.placeholder
-                    })
-                    try {
-                        val text = reader.getText(batchStart + 1, batchEnd)
-                        if (settings.saveTextFiles) {
-                            outputDir.resolve("pages_${batchStart + 1}_to_${batchEnd}_text.txt").writeText(text)
-                        }
-                        val promptList = mutableListOf()
-                        promptList.add(
-                            """
-                            |# Accumulated Prior JSON:
-                            |
-                            |FOR INFORMATIVE CONTEXT ONLY. DO NOT COPY TO OUTPUT.
-                            |```json
-                            |${JsonUtil.toJson(runningDocument)}
-                            |```
-                            """.trimMargin()
-                        )
-                        promptList.add(
-                            """
-                            |# Prior Text
-                            |
-                            |FOR INFORMATIVE CONTEXT ONLY. DO NOT COPY TO OUTPUT.
-                            |```text
-                            |$previousPageText
-                            |```
-                            |""".trimMargin()
-                        )
-                        promptList.add(
-                            """
-                            |# Current Page
-                            |
-                            |```text
-                            |$text
-                            |```
-                            """.trimMargin()
-                        )
-                        @Language("Markdown") val jsonResult = parsingModel.getParser(api).let {
-                            it(promptList.toList().joinToString("\n\n"))
-                        }
-                        if (settings.saveTextFiles) {
-                            val jsonFile = outputDir.resolve("pages_${batchStart + 1}_to_${batchEnd}_content.json")
-                            jsonFile.writeText(JsonUtil.toJson(jsonResult))
-                        }
-                        ui.newTask(false).apply {
-                            pageTabs["Text"] = this.placeholder
-                            add(
-                                MarkdownUtil.renderMarkdown(
-                                    "\n```text\n${
-                                        text
-                                    }\n```\n", ui = ui
-                                )
-                            )
-                        }
-                        ui.newTask(false).apply {
-                            pageTabs["JSON"] = this.placeholder
-                            add(
-                                MarkdownUtil.renderMarkdown(
-                                    "\n```json\n${
-                                        JsonUtil.toJson(jsonResult)
-                                    }\n```\n", ui = ui
-                                )
-                            )
-                        }
-                        for (pageIndex in batchStart until batchEnd) {
-                            val image = reader.renderImage(pageIndex, settings.dpi)
-                            if (settings.showImages) {
-                                ui.newTask(false).apply {
-                                    pageTabs["Image ${pageIndex + 1}"] = this.placeholder
-                                    image(image)
+            val docTabs = TabbedDisplay(mainTask)
+            fileInputs.map { it.toFile() }.forEach { file ->
+                if (!file.exists()) {
+                    mainTask.error(ui, IllegalArgumentException("File not found: $file"))
+                    return
+                }
+                ui.socketManager?.pool?.submit {
+                    val docTask = ui.newTask(false).apply { docTabs[file.toString()] = this.placeholder }
+                    val pageTabs = TabbedDisplay(docTask)
+                    val outputDir = root.resolve("output").apply { mkdirs() }
+                    reader(file).use { reader ->
+                        var previousPageText = "" // Keep this for context
+                        val pageCount = minOf(reader.getPageCount(), maxPages)
+                        val pageSets = 0 until pageCount step pagesPerBatch
+                        progressBar?.add(0.0, pageCount.toDouble())
+                        val futures = pageSets.toList().mapNotNull { batchStart ->
+                            val pageTask = ui.newTask(false)
+                            val api = api.getChildClient().apply {
+                                val createFile = pageTask.createFile(".logs/api-${UUID.randomUUID()}.log")
+                                createFile.second?.apply {
+                                    logStreams += this.outputStream().buffered()
+                                    pageTask.verbose("API log: $this")
                                 }
                             }
-                            if (settings.saveImageFiles) {
-                                val imageFile =
-                                    outputDir.resolve("page_${pageIndex + 1}.${settings.outputFormat.lowercase(Locale.getDefault())}")
-                                when (settings.outputFormat.uppercase(Locale.getDefault())) {
-                                    "PNG" -> ImageIO.write(image, "PNG", imageFile)
-                                    "JPEG", "JPG" -> ImageIO.write(image, "JPEG", imageFile)
-                                    "GIF" -> ImageIO.write(image, "GIF", imageFile)
-                                    "BMP" -> ImageIO.write(image, "BMP", imageFile)
-                                    else -> throw IllegalArgumentException("Unsupported output format: ${settings.outputFormat}")
+                            try {
+                                val batchEnd = min(batchStart + pagesPerBatch, pageCount)
+                                val text = reader.getText(batchStart, batchEnd)
+                                val label = if ((batchStart + 1) != batchEnd) "Pages ${batchStart}-${batchEnd}" else "Page ${batchStart}"
+                                val pageTabs = TabbedDisplay(pageTask.apply { pageTabs[label] = placeholder })
+                                if (settings.showImages) {
+                                    for (pageIndex in batchStart until batchEnd) {
+                                        val image = reader.renderImage(pageIndex, settings.dpi)
+                                        ui.newTask(false).apply {
+                                            pageTabs["Image ${1 + (pageIndex - batchStart)}"] = placeholder
+                                            image(image)
+                                        }
+                                        if (settings.saveImageFiles) {
+                                            val imageFile =
+                                                outputDir.resolve("page_${pageIndex}.${settings.outputFormat.lowercase(Locale.getDefault())}")
+                                            when (settings.outputFormat.uppercase(Locale.getDefault())) {
+                                                "PNG" -> ImageIO.write(image, "PNG", imageFile)
+                                                "JPEG", "JPG" -> ImageIO.write(image, "JPEG", imageFile)
+                                                "GIF" -> ImageIO.write(image, "GIF", imageFile)
+                                                "BMP" -> ImageIO.write(image, "BMP", imageFile)
+                                                else -> throw IllegalArgumentException("Unsupported output format: ${settings.outputFormat}")
+                                            }
+                                        }
+                                    }
                                 }
-                            }
-                        }
-                        runningDocument = parsingModel.merge(runningDocument, jsonResult)
-                        ui.newTask(false).apply {
-                            pageTabs["Accumulator"] = this.placeholder
-                            add(
-                                MarkdownUtil.renderMarkdown(
+                                if (text.isBlank()) {
+                                    pageTask.error(ui, IllegalArgumentException("No text extracted from pages $batchStart to $batchEnd"))
+                                    return@mapNotNull null
+                                }
+                                if (settings.saveTextFiles) {
+                                    outputDir.resolve("pages_${batchStart}_to_${batchEnd}_text.txt").writeText(text)
+                                }
+                                val promptList = mutableListOf()
+                                promptList.add(
+                                    """
+                                    |# Prior Text
+                                    |
+                                    |FOR INFORMATIVE CONTEXT ONLY. DO NOT COPY TO OUTPUT.
+                                    |```text
+                                    |$previousPageText
+                                    |```
+                                    |""".trimMargin()
+                                )
+                                promptList.add(
                                     """
-                                    |## Accumulated Document JSON
+                                    |# Current Page
                                     |
-                                    |```json
-                                    |${JsonUtil.toJson(runningDocument)}
+                                    |```text
+                                    |$text
                                     |```
-                                    """.trimMargin(), ui = ui
+                                    """.trimMargin()
+                                )
+                                previousPageText = text
+                                ui.socketManager.pool.submit {
+                                    try {
+                                        val jsonResult = parsingModel.getParser(api)(promptList.toList().joinToString("\n\n"))
+                                        if (settings.saveTextFiles) {
+                                            val jsonFile = outputDir.resolve("pages_${batchStart}_to_${batchEnd}_content.json")
+                                            jsonFile.writeText(JsonUtil.toJson(jsonResult))
+                                        }
+                                        ui.newTask(false).apply {
+                                            pageTabs["Text"] = placeholder
+                                            add(
+                                                MarkdownUtil.renderMarkdown(
+                                                    "\n```text\n${
+                                                        text
+                                                    }\n```\n", ui = ui
+                                                )
+                                            )
+                                        }
+                                        ui.newTask(false).apply {
+                                            pageTabs["JSON"] = placeholder
+                                            add(
+                                                MarkdownUtil.renderMarkdown(
+                                                    "\n```json\n${
+                                                        JsonUtil.toJson(jsonResult)
+                                                    }\n```\n", ui = ui
+                                                )
+                                            )
+                                        }
+                                        jsonResult
+                                    } catch (e: Throwable) {
+                                        pageTask.error(ui, e)
+                                        null
+                                    } finally {
+                                        progressBar?.add(1.0, 0.0)
+                                        pageTask.complete()
+                                    }
+                                }
+                            } catch (e: Throwable) {
+                                pageTask.error(ui, e)
+                                null
+                            }
+                        }.toTypedArray()
+                        val finalDocument = futures.mapNotNull { try { it.get() } catch (e : Throwable) {
+                            mainTask.error(ui, e)
+                            null
+                        } }.fold(parsingModel.newDocument())
+                        { runningDocument, it -> parsingModel.merge(runningDocument, it) }
+                        docTask.add(
+                            MarkdownUtil.renderMarkdown(
+                                """
+                                |## Document JSON
+                                |
+                                |```json
+                                |${JsonUtil.toJson(finalDocument)}
+                                |```
+                                |
+                                |Extracted files are saved in: ${outputDir.absolutePath}
+                                """.trimMargin(), ui = ui
+                            )
+                        )
+                        if (settings.saveFinalJson) {
+                            val finalJsonFile = root.resolve(file.name.reversed().split(delimiters = arrayOf("."), false, 2)[1].reversed() + ".parsed.json")
+                            finalJsonFile.writeText(JsonUtil.toJson(finalDocument))
+                            docTask.add(
+                                MarkdownUtil.renderMarkdown(
+                                    "Final JSON saved to: ${finalJsonFile.absolutePath}",
+                                    ui = ui
                                 )
                             )
                         }
-                        previousPageText = text.takeLast(1000)
-                    } catch (e: Throwable) {
-                        pageTask.error(ui, e)
-                        continue
                     }
                 }
-                task.add(
-                    MarkdownUtil.renderMarkdown(
-                        """
-                        |## Document JSON
-                        |
-                        |```json
-                        |${JsonUtil.toJson(runningDocument)}
-                        |```
-                        |
-                        |Extracted files are saved in: ${outputDir.absolutePath}
-                        """.trimMargin(), ui = ui
-                    )
-                )
-                // Save final JSON if enabled in settings
-                if (settings.saveFinalJson) {
-                    val finalJsonFile = root.resolve(fileInput.name.reversed().split(delimiters = arrayOf("."), false, 2)[1].reversed() + ".parsed.json")
-                    finalJsonFile.writeText(JsonUtil.toJson(runningDocument))
-                    task.add(
-                        MarkdownUtil.renderMarkdown(
-                            "Final JSON saved to: ${finalJsonFile.absolutePath}",
-                            ui = ui
-                        )
-                    )
-                }
             }
         } catch (e: Throwable) {
-            task.error(ui, e)
+            mainTask.error(ui, e)
         }
     }
 
@@ -255,7 +255,7 @@ open class DocumentParserApp(
         val dpi: Float = 120f,
         val maxPages: Int = Int.MAX_VALUE,
         val outputFormat: String = "PNG",
-        val fileInput: String? = "",
+        val fileInputs: List? = null,
         val showImages: Boolean = true,
         val pagesPerBatch: Int = 1,
         val saveImageFiles: Boolean = false,
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParsingModel.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParsingModel.kt
index 717c05aa..a91c64fd 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParsingModel.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentParsingModel.kt
@@ -1,9 +1,15 @@
 package com.simiacryptus.skyenet.apps.parse
 
 import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
+import com.simiacryptus.jopenai.models.ApiModel
 import com.simiacryptus.jopenai.models.ChatModel
+import com.simiacryptus.jopenai.models.EmbeddingModels
 import com.simiacryptus.skyenet.core.actors.ParsedActor
+import com.simiacryptus.util.JsonUtil
+import java.util.concurrent.ExecutorService
+import java.util.concurrent.Future
 
 
 open class DocumentParsingModel(
@@ -19,17 +25,7 @@ open class DocumentParsingModel(
         val newData = newData as DocumentData
         return DocumentData(
             id = newData.id ?: runningDocument.id,
-            content = mergeContent(runningDocument.content, newData.content).takeIf { it.isNotEmpty() },
-            entities = mergeEntities(runningDocument.entities, newData.entities).takeIf { it.isNotEmpty() },
-            metadata = mergeMetadata(runningDocument.metadata, newData.metadata)
-        )
-    }
-
-    protected open fun mergeMetadata(existing: DocumentMetadata?, new: DocumentMetadata?): DocumentMetadata {
-        return DocumentMetadata(
-            title = new?.title ?: existing?.title,
-            keywords = ((existing?.keywords ?: emptyList()) + (new?.keywords ?: emptyList())).distinct(),
-            properties = ((existing?.properties ?: emptyMap()) + (new?.properties ?: emptyMap())).takeIf { it.isNotEmpty() }
+            content_list = mergeContent(runningDocument.content_list, newData.content_list).takeIf { it.isNotEmpty() },
         )
     }
 
@@ -50,42 +46,16 @@ open class DocumentParsingModel(
     }
 
     protected open fun mergeContentData(existing: ContentData, new: ContentData) = existing.copy(
-        content = mergeContent(existing.content, new.content).takeIf { it.isNotEmpty() },
-        entities = ((existing.entities ?: emptyList()) + (new.entities ?: emptyList())).distinct()
-            .takeIf { it.isNotEmpty() },
+        content_list = mergeContent(existing.content_list, new.content_list).takeIf { it.isNotEmpty() },
         tags = ((existing.tags ?: emptyList()) + (new.tags ?: emptyList())).distinct().takeIf { it.isNotEmpty() }
     )
 
-    protected open fun mergeEntities(
-        existingEntities: Map?,
-        newEntities: Map?
-    ) = ((existingEntities?.keys ?: emptySet()) + (newEntities?.keys ?: emptySet())).associateWith { key ->
-        val existing = existingEntities?.get(key)
-        val new = newEntities?.get(key)
-        when {
-            existing == null -> new!!
-            new == null -> existing
-            else -> mergeEntityData(existing, new)
-        }
-    }
-
-    protected open fun mergeEntityData(existing: EntityData, new: EntityData) = existing.copy(
-        aliases = ((existing.aliases ?: emptyList()) + (new.aliases ?: emptyList())).distinct()
-            .takeIf { it.isNotEmpty() },
-        properties = ((existing.properties ?: emptyMap()) + (new.properties ?: emptyMap())).takeIf { it.isNotEmpty() },
-        relations = ((existing.relations ?: emptyMap()) + (new.relations ?: emptyMap())).takeIf { it.isNotEmpty() },
-        type = new.type ?: existing.type
-    )
-
     open val promptSuffix = """
-        |Parse the text into a hierarchical structure that describes the content of the page:
+        |Parse the text into a hierarchical structure:
         |1. Separate the content into sections, paragraphs, statements, etc.
-        |2. The final level of the hierarchy should contain singular, short, standalone sentences.
-        |3. Capture any entities, relationships, and properties that can be extracted from the text of the current page(s).
-        |4. For each entity, include mentions with their exact text and location (start and end indices) in the document.
-        |5. Extract document metadata such as title, author, creation date, and keywords if available.
-        |6. Assign relevant tags to each content section to improve searchability and categorization.
-        |7. Do not copy data from the accumulated document JSON to your response; it is provided for context only.
+        |2. All source content should be included in the output, with paraphrasing, corrections, and context as needed
+        |3. Each content leaf node text should be simple and self-contained
+        |4. Assign relevant tags to each node to improve searchability and categorization.
         """.trimMargin()
 
     open val exampleInstance = DocumentData()
@@ -107,35 +77,60 @@ open class DocumentParsingModel(
 
     data class DocumentData(
         @Description("Document/Page identifier") override val id: String? = null,
-        @Description("Entities extracted") val entities: Map? = null,
-        @Description("Hierarchical structure and data") override val content: List? = null,
-        @Description("Document metadata") override val metadata: DocumentMetadata? = null
+        @Description("Hierarchical structure and data") override val content_list: List? = null,
     ) : ParsingModel.DocumentData
 
-    data class EntityData(
-        @Description("Aliases for the entity") val aliases: List? = null,
-        @Description("Entity attributes extracted from the page") val properties: Map? = null,
-        @Description("Entity relationships extracted from the page") val relations: Map? = null,
-        @Description("Entity type (e.g., person, organization, location)") val type: String? = null
-    )
-
     data class ContentData(
         @Description("Content type, e.g. heading, paragraph, statement, list") override val type: String = "",
         @Description("Brief, self-contained text either copied, paraphrased, or summarized") override val text: String? = null,
-        @Description("Sub-elements") override val content: List? = null,
-        @Description("Related entities by ID") val entities: List? = null,
+        @Description("Sub-elements") override val content_list: List? = null,
         @Description("Tags - related topics and non-entity indexing") override val tags: List? = null
     ) : ParsingModel.ContentData
 
-    data class DocumentMetadata(
-        @Description("Document title") val title: String? = null,
-        @Description("Keywords or tags associated with the document") val keywords: List? = null,
-        @Description("Other metadata") val properties: Map? = null,
-    ) : ParsingModel.DocumentMetadata
-
     companion object {
         val log = org.slf4j.LoggerFactory.getLogger(DocumentParsingModel::class.java)
 
+        fun getRows(
+            inputPath: String,
+            progressState: ProgressState?,
+            futureList: MutableList>,
+            pool: ExecutorService,
+            openAIClient: OpenAIClient,
+            fileData: Map?
+        ): MutableList {
+            val records: MutableList = mutableListOf()
+            fun processContent(content: Map, path: String = "") {
+                val record = DocumentRecord(
+                    text = content["text"] as? String,
+                    metadata = JsonUtil.toJson(content.filter { it.key != "text" && it.key != "content" && it.key != "type" }),
+                    sourcePath = inputPath,
+                    jsonPath = path,
+                    vector = null
+                )
+                records.add(record)
+                if (record.text != null) {
+                    progressState?.add(0.0, 1.0)
+                    futureList.add(pool.submit {
+                        record.vector = openAIClient.createEmbedding(
+                            ApiModel.EmbeddingRequest(
+                                EmbeddingModels.Large.modelName, record.text
+                            )
+                        ).data[0].embedding ?: DoubleArray(0)
+                        progressState?.add(1.0, 0.0)
+                    })
+                }
+                (content["content_list"] as? List>)?.forEachIndexed> { index, childContent ->
+                    processContent(childContent, "$path.content_list[$index]")
+                }
+            }
+            fileData?.get("content_list")?.let { contentList ->
+                (contentList as? List>)?.forEachIndexed> { index, content ->
+                    processContent(content, "content_list[$index]")
+                }
+            }
+            return records
+        }
+
     }
 
 }
\ No newline at end of file
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentRecord.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentRecord.kt
index f098ad48..3e5c6bf8 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentRecord.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/DocumentRecord.kt
@@ -1,8 +1,6 @@
 package com.simiacryptus.skyenet.apps.parse
 
 import com.simiacryptus.jopenai.OpenAIClient
-import com.simiacryptus.jopenai.models.ApiModel
-import com.simiacryptus.jopenai.models.EmbeddingModels
 import com.simiacryptus.util.JsonUtil
 import java.io.File
 import java.io.FileInputStream
@@ -12,6 +10,7 @@ import java.io.ObjectInputStream
 import java.io.ObjectOutputStream
 import java.io.Serializable
 import java.util.concurrent.ExecutorService
+import java.util.concurrent.Future
 import java.util.concurrent.TimeUnit
 
 data class DocumentRecord(
@@ -36,7 +35,7 @@ data class DocumentRecord(
         val metadata = input.readUTF().let { if (it.isEmpty()) null else it }
         val sourcePath = input.readUTF()
         val jsonPath = input.readUTF()
-        val vector = input.readObject() as DoubleArray
+        val vector = input.readObject() as DoubleArray?
         return DocumentRecord(
             text,
             metadata,
@@ -51,65 +50,22 @@ data class DocumentRecord(
 
         fun  saveAsBinary(
             openAIClient: OpenAIClient,
-            outputPath: String,
             pool: ExecutorService,
+            progressState: ProgressState? = null,
             vararg inputPaths: String,
         ) {
-            val records = mutableListOf()
             inputPaths.forEach { inputPath ->
-                processDocument(
-                    inputPath,
-                    JsonUtil.fromJson(File(inputPath).readText(), Map::class.java) as T,
-                    records,
-                    openAIClient,
-                    pool
-                )
-            }
-            writeBinary(outputPath, records)
-        }
-
-        private fun  processDocument(
-            inputPath: String,
-            document: T,
-            records: MutableList,
-            openAIClient: OpenAIClient,
-            pool: ExecutorService
-        ) {
-            fun processContent(content: Map, path: String = "") {
-                val record = DocumentRecord(
-                    text = content["text"] as? String,
-                    metadata = JsonUtil.toJson(content.filter { it.key != "text" && it.key != "content" }),
-                    sourcePath = inputPath,
-                    jsonPath = path,
-                    vector = null
-                )
-                records.add(record)
-                (content["content"] as? List>)?.forEachIndexed { index, childContent ->
-                    processContent(childContent, "$path.content[$index]")
-                }
+                val futureList = mutableListOf>()
+                val infile = File(inputPath)
+                val fileData = JsonUtil.fromJson(infile.readText(), Map::class.java) as T as? Map
+                val records = DocumentParsingModel.getRows(inputPath, progressState, futureList, pool, openAIClient, fileData)
+                val outputPath = infile.parentFile.resolve(infile.name.split("\\.".toRegex(), 2).first() + ".index.data").absolutePath
+                awaitAll(futureList.toTypedArray())
+                writeBinary(outputPath, records)
             }
-            (document as? Map)?.get("content")?.let { contentList ->
-                (contentList as? List>)?.forEachIndexed { index, content ->
-                    processContent(content, "content[$index]")
-                }
-            }
-            addEmbeddings(records, pool, openAIClient)
         }
 
-        private fun addEmbeddings(
-            records: MutableList,
-            pool: ExecutorService,
-            openAIClient: OpenAIClient
-        ) {
-            val futureList = records.map {
-                pool.submit {
-                    it.vector = openAIClient.createEmbedding(
-                        ApiModel.EmbeddingRequest(
-                            EmbeddingModels.Large.modelName, it.text
-                        )
-                    ).data.get(0).embedding ?: DoubleArray(0)
-                }
-            }.toTypedArray()
+        fun awaitAll(futureList: Array>) {
             val start = System.currentTimeMillis()
             for (future in futureList) {
                 try {
@@ -123,6 +79,7 @@ data class DocumentRecord(
             }
         }
 
+
         private fun writeBinary(outputPath: String, records: List) {
             log.info("Writing ${records.size} records to $outputPath")
             ObjectOutputStream(FileOutputStream(outputPath)).use { out ->
@@ -150,4 +107,5 @@ data class DocumentRecord(
             return records
         }
     }
-}
\ No newline at end of file
+}
+
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/PDFReader.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/PDFReader.kt
index 246ec587..6efd64dc 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/PDFReader.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/PDFReader.kt
@@ -9,13 +9,13 @@ import java.io.File
 class PDFReader(pdfFile: File) : DocumentParserApp.DocumentReader {
     private val document: PDDocument = PDDocument.load(pdfFile)
     private val renderer: PDFRenderer = PDFRenderer(document)
-    private val stripper: PDFTextStripper = PDFTextStripper().apply { sortByPosition = true }
 
     override fun getPageCount(): Int = document.numberOfPages
 
     override fun getText(startPage: Int, endPage: Int): String {
-        stripper.startPage = startPage
-        stripper.endPage = endPage
+        val stripper = PDFTextStripper().apply { sortByPosition = true } // Not to be confused with the stripper from last night
+        stripper.startPage = startPage+1
+        stripper.endPage = endPage+1
         return stripper.getText(document)
     }
 
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ParsingModel.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ParsingModel.kt
index a317dde5..f2c6c4e0 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ParsingModel.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ParsingModel.kt
@@ -11,14 +11,14 @@ interface ParsingModel {
     interface ContentData {
         val type: String
         val text: String?
-        val content: List?
+        val content_list: List?
         val tags: List?
     }
 
     interface DocumentData {
         val id: String?
-        val content: List?
-        val metadata: DocumentMetadata?
+        val content_list: List?
+//        val metadata: DocumentMetadata?
     }
 
     companion object {
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ProgressState.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ProgressState.kt
new file mode 100644
index 00000000..34253ce3
--- /dev/null
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/ProgressState.kt
@@ -0,0 +1,71 @@
+package com.simiacryptus.skyenet.apps.parse
+
+import com.simiacryptus.skyenet.set
+import com.simiacryptus.skyenet.webui.session.SessionTask
+
+data class ProgressState(
+  var progress: Double = 0.0,
+  var max: Double = 0.0,
+  val onUpdate: MutableList<(ProgressState) -> Unit> = mutableListOf(),
+) {
+  fun add(progress: Double, max: Double) {
+    this.progress += progress
+    this.max += max
+    onUpdate.forEach { it(this) }
+  }
+
+  companion object {
+    fun progressBar(
+      task: SessionTask,
+    ): ProgressState {
+      val stringBuilder = task.add(
+        """
+                    
+                    

+                      

+                    

+                """
+      )!!
+      return ProgressState(0.0, 0.0).apply {
+        onUpdate += {
+          val progress = it.progress / it.max
+          stringBuilder.set(
+            """
+                    
+                    

+                      

+                    

+                """.trimIndent()
+          )
+          task.update()
+        }
+      }
+    }
+  }
+}
\ No newline at end of file
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/TextReader.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/TextReader.kt
index 9d8c4c1d..5d4d75d2 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/TextReader.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/parse/TextReader.kt
@@ -2,7 +2,6 @@ package com.simiacryptus.skyenet.apps.parse
 
 import java.awt.image.BufferedImage
 import java.io.File
-import kotlin.math.abs
 
 class TextReader(private val textFile: File) : DocumentParserApp.DocumentReader {
     private val pages: List = splitIntoPages(textFile.readLines().joinToString("\n"))
@@ -24,17 +23,17 @@ class TextReader(private val textFile: File) : DocumentParserApp.DocumentReader
     private fun splitIntoPages(text: String, maxChars: Int = 16000): List {
         if (text.length <= maxChars) return listOf(text)
         val lines = text.split("\n")
-        var bestSplitIndex = lines.size / 2
-        var bestFitness = Double.MAX_VALUE
-        for (i in lines.indices) {
+        if (lines.size <= 1) return listOf(text)
+        val splitFitnesses = lines.indices.map { i ->
             val leftSize = lines.subList(0, i).joinToString("\n").length
             val rightSize = lines.subList(i, lines.size).joinToString("\n").length
-            val fitness = abs(leftSize - rightSize) + (if (lines[i].isEmpty()) 0 else 1000)
-            if (fitness < bestFitness) {
-                bestFitness = fitness.toDouble()
-                bestSplitIndex = i
-            }
-        }
+            if (leftSize <= 0) return@map i to Double.MAX_VALUE
+            if (rightSize <= 0) return@map i to Double.MAX_VALUE
+            val fitness = -((leftSize.toDouble() / text.length) * Math.log1p(rightSize.toDouble() / text.length) +
+                (rightSize.toDouble() / text.length) * Math.log1p(leftSize.toDouble() / text.length))
+            i to fitness.toDouble()
+        }.toTypedArray()
+        var bestSplitIndex = splitFitnesses.minByOrNull { it.second }?.first ?: lines.size / 2
         val leftText = lines.subList(0, bestSplitIndex).joinToString("\n")
         val rightText = lines.subList(bestSplitIndex, lines.size).joinToString("\n")
         return splitIntoPages(leftText, maxChars) + splitIntoPages(rightText, maxChars)
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/AbstractTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/AbstractTask.kt
index da8ca8ac..00f86f98 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/AbstractTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/AbstractTask.kt
@@ -1,6 +1,7 @@
 package com.simiacryptus.skyenet.apps.plan
 
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.skyenet.set
 import com.simiacryptus.skyenet.webui.application.ApplicationInterface
 import com.simiacryptus.skyenet.webui.session.SessionTask
@@ -56,8 +57,10 @@ abstract class AbstractTask(
         agent: PlanCoordinator,
         messages: List = listOf(),
         task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+        api: ChatClient,
+        resultFn: (String) -> Unit,
+        api2: OpenAIClient,
+        planSettings: PlanSettings,
     )
 
     companion object {
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/CommandAutoFixTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/CommandAutoFixTask.kt
index 0ec6a4fd..a1b821d9 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/CommandAutoFixTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/CommandAutoFixTask.kt
@@ -1,7 +1,7 @@
 package com.simiacryptus.skyenet.apps.plan
 
-import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.skyenet.Retryable
 import com.simiacryptus.skyenet.apps.general.CmdPatchApp
@@ -52,11 +52,13 @@ ${planSettings.commandAutoFixCommands?.joinToString("\n") { "    * ${File(it).na
     }
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         val semaphore = Semaphore(0)
         val hasError = AtomicBoolean(false)
@@ -70,7 +72,8 @@ ${planSettings.commandAutoFixCommands?.joinToString("\n") { "    * ${File(it).na
                     ?.map { File(it) }?.associateBy { it.name }
                     ?.filterKeys { it.startsWith(alias ?: "") }
                     ?: emptyMap()
-                val executable = cmds.entries.firstOrNull()?.value
+                var executable = cmds.entries.firstOrNull()?.value
+                executable = executable ?: alias?.let { root.resolve(it).toFile() }
                 if (executable == null) {
                     throw IllegalArgumentException("Command not found: $alias")
                 }
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/ForeachTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/ForeachTask.kt
index 71a539a0..2e31ef83 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/ForeachTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/ForeachTask.kt
@@ -1,6 +1,7 @@
 package com.simiacryptus.skyenet.apps.plan
 
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.skyenet.apps.plan.ForeachTask.ForeachTaskData
 import com.simiacryptus.skyenet.apps.plan.PlanUtil.diagram
@@ -37,11 +38,13 @@ ForeachTask - Execute a task for each item in a list
     }
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         val userMessage = messages.joinToString("\n")
         val items =
@@ -66,7 +69,8 @@ ForeachTask - Execute a task for each item in a list
                 pool = agent.pool,
                 userMessage = "$userMessage\nProcessing item $index: $item",
                 plan = itemSubTasks,
-                api = api
+                api = api,
+                api2 = api2,
             )
         }
         subPlanTask.complete("Completed ForeachTask for ${items.size} items")
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/GitHubSearchTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/GitHubSearchTask.kt
similarity index 94%
rename from webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/GitHubSearchTask.kt
rename to webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/GitHubSearchTask.kt
index cdc700e9..3a071ca8 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/GitHubSearchTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/GitHubSearchTask.kt
@@ -1,10 +1,10 @@
-package com.simiacryptus.skyenet.apps.plan.file
+package com.simiacryptus.skyenet.apps.plan
 
 import com.fasterxml.jackson.databind.ObjectMapper
 import com.fasterxml.jackson.module.kotlin.readValue
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
-import com.simiacryptus.skyenet.apps.plan.*
 import com.simiacryptus.skyenet.util.MarkdownUtil
 import com.simiacryptus.skyenet.webui.session.SessionTask
 import org.slf4j.LoggerFactory
@@ -51,16 +51,18 @@ GitHubSearch - Search GitHub for code, commits, issues, repositories, topics, or
     agent: PlanCoordinator,
     messages: List,
     task: SessionTask,
-    api: API,
-    resultFn: (String) -> Unit
+    api: ChatClient,
+    resultFn: (String) -> Unit,
+    api2: OpenAIClient,
+    planSettings: PlanSettings
   ) {
-    val searchResults = performGitHubSearch()
+    val searchResults = performGitHubSearch(planSettings)
     val formattedResults = formatSearchResults(searchResults)
     task.add(MarkdownUtil.renderMarkdown(formattedResults, ui = agent.ui))
     resultFn(formattedResults)
   }
 
-  private fun performGitHubSearch(): String {
+  private fun performGitHubSearch(planSettings: PlanSettings): String {
     val client = HttpClient.newBuilder().build()
     val uriBuilder = StringBuilder("https://api.github.com/search/${planTask?.search_type}?q=${planTask?.search_query}&per_page=${planTask?.per_page}")
     planTask?.sort?.let { uriBuilder.append("&sort=$it") }
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/GoogleSearchTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/GoogleSearchTask.kt
similarity index 86%
rename from webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/GoogleSearchTask.kt
rename to webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/GoogleSearchTask.kt
index 9c7cb29c..31237901 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/GoogleSearchTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/GoogleSearchTask.kt
@@ -1,8 +1,6 @@
-package com.simiacryptus.skyenet.apps.plan.file
+package com.simiacryptus.skyenet.apps.plan
 
-import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.describe.Description
-import com.simiacryptus.skyenet.apps.plan.*
 import com.simiacryptus.skyenet.util.MarkdownUtil
 import com.simiacryptus.skyenet.webui.session.SessionTask
 import org.slf4j.LoggerFactory
@@ -13,6 +11,8 @@ import java.net.http.HttpRequest
 import java.net.http.HttpResponse
 import com.fasterxml.jackson.databind.ObjectMapper
 import com.fasterxml.jackson.module.kotlin.readValue
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.util.JsonUtil
 
 class GoogleSearchTask(
@@ -44,25 +44,22 @@ GoogleSearch - Search Google for web results
     agent: PlanCoordinator,
     messages: List,
     task: SessionTask,
-    api: API,
-    resultFn: (String) -> Unit
+    api: ChatClient,
+    resultFn: (String) -> Unit,
+    api2: OpenAIClient,
+    planSettings: PlanSettings
   ) {
-    val searchResults = performGoogleSearch()
+    val searchResults = performGoogleSearch(planSettings)
     val formattedResults = formatSearchResults(searchResults)
     task.add(MarkdownUtil.renderMarkdown(formattedResults, ui = agent.ui))
     resultFn(formattedResults)
   }
 
-  private fun performGoogleSearch(): String {
+  private fun performGoogleSearch(planSettings: PlanSettings): String {
     val client = HttpClient.newBuilder().build()
     val encodedQuery = URLEncoder.encode(planTask?.search_query, "UTF-8")
     val uriBuilder = "https://www.googleapis.com/customsearch/v1?key=${planSettings.googleApiKey}&cx=${planSettings.googleSearchEngineId}&q=$encodedQuery&num=${planTask?.num_results}"
-    
-    val request = HttpRequest.newBuilder()
-      .uri(URI.create(uriBuilder))
-      .GET()
-      .build()
-
+    val request = HttpRequest.newBuilder().uri(URI.create(uriBuilder)).GET().build()
     val response = client.send(request, HttpResponse.BodyHandlers.ofString())
     if (response.statusCode() != 200) {
       throw RuntimeException("Google API request failed with status ${response.statusCode()}: ${response.body()}")
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanCoordinator.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanCoordinator.kt
index 8e74c914..4e432ca4 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanCoordinator.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanCoordinator.kt
@@ -4,6 +4,7 @@ package com.simiacryptus.skyenet.apps.plan
 import com.simiacryptus.diff.FileValidationUtils
 import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.skyenet.TabbedDisplay
 import com.simiacryptus.skyenet.apps.plan.PlanUtil.buildMermaidGraph
 import com.simiacryptus.skyenet.apps.plan.PlanUtil.filterPlan
@@ -54,7 +55,12 @@ class PlanCoordinator(
                 }
             }
 
-    fun executeTaskBreakdownWithPrompt(jsonInput: String, api: API, task: SessionTask) {
+    fun executeTaskBreakdownWithPrompt(
+        jsonInput: String,
+        api: API,
+        api2: OpenAIClient,
+        task: SessionTask
+    ) {
         try {
             lateinit var taskBreakdownWithPrompt: TaskBreakdownWithPrompt
             val plan = filterPlan {
@@ -73,7 +79,7 @@ class PlanCoordinator(
                 """.trimMargin(), ui = ui
                 )
             )
-            executePlan(plan ?: emptyMap(), task, taskBreakdownWithPrompt.prompt, api)
+            executePlan(plan ?: emptyMap(), task, taskBreakdownWithPrompt.prompt, api, api2)
         } catch (e: Exception) {
             task.error(ui, e)
         }
@@ -83,7 +89,8 @@ class PlanCoordinator(
         plan: Map,
         task: SessionTask,
         userMessage: String,
-        api: API
+        api: API,
+        api2: OpenAIClient,
     ): PlanProcessingState {
         val api = (api as ChatClient).getChildClient().apply {
             val createFile = task.createFile(".logs/api-${UUID.randomUUID()}.log")
@@ -110,7 +117,8 @@ class PlanCoordinator(
                 pool = pool,
                 userMessage = userMessage,
                 plan = plan,
-                api = api
+                api = api,
+                api2 = api2,
             )
         } catch (e: Throwable) {
             log.warn("Error during incremental code generation process", e)
@@ -135,7 +143,8 @@ class PlanCoordinator(
         pool: ThreadPoolExecutor,
         userMessage: String,
         plan: Map,
-        api: API
+        api: API,
+        api2: OpenAIClient,
     ) {
         val sessionTask = ui.newTask(false).apply { task.add(placeholder) }
         val api = (api as ChatClient).getChildClient().apply {
@@ -251,7 +260,9 @@ class PlanCoordinator(
                         messages = messages,
                         task = task1,
                         api = api,
-                        resultFn = { planProcessingState.taskResult[taskId] = it }
+                        api2 = api2,
+                        resultFn = { planProcessingState.taskResult[taskId] = it },
+                        planSettings = planSettings
                     )
                 } catch (e: Throwable) {
                     log.warn("Error during task execution", e)
@@ -274,6 +285,22 @@ class PlanCoordinator(
         }
     }
 
+    fun copy(
+        user: User? = this.user,
+        session: Session = this.session,
+        dataStorage: StorageInterface = this.dataStorage,
+        ui: ApplicationInterface = this.ui,
+        planSettings: PlanSettings = this.planSettings,
+        root: Path = this.root
+    ) = PlanCoordinator(
+        user = user,
+        session = session,
+        dataStorage = dataStorage,
+        ui = ui,
+        planSettings = planSettings,
+        root = root
+    )
+
     companion object : Planner() {
         private val log = LoggerFactory.getLogger(PlanCoordinator::class.java)
     }
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanningTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanningTask.kt
index 70c7cc50..efc25936 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanningTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/PlanningTask.kt
@@ -1,6 +1,8 @@
 package com.simiacryptus.skyenet.apps.plan
 
 import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.jopenai.models.ApiModel
 import com.simiacryptus.jopenai.models.ApiModel.Role
@@ -56,15 +58,17 @@ class PlanningTask(
         agent: PlanCoordinator,
         messages: List,
         task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+        api: ChatClient,
+        resultFn: (String) -> Unit,
+        api2: OpenAIClient,
+        planSettings: PlanSettings
     ) {
         val userMessage = messages.joinToString("\n")
         val newTask = agent.ui.newTask(false).apply { add(placeholder) }
         fun toInput(s: String) = (messages + listOf(s)).filter { it.isNotBlank() }
 
         val subPlan = if (planSettings.allowBlocking && !planSettings.autoFix) {
-            createSubPlanDiscussable(newTask, userMessage, ::toInput, api, agent.ui).call().obj
+            createSubPlanDiscussable(newTask, userMessage, ::toInput, api, agent.ui, planSettings).call().obj
         } else {
             val design = planSettings.planningActor().answer(
                 toInput("Expand ${planTask?.task_description ?: ""}"),
@@ -80,7 +84,7 @@ class PlanningTask(
             )
             design.obj
         }
-        executeSubTasks(agent, userMessage, filterPlan { subPlan.tasksByID } ?: emptyMap(), task, api)
+        executeSubTasks(agent, userMessage, filterPlan { subPlan.tasksByID } ?: emptyMap(), task, api, api2)
     }
 
     private fun createSubPlanDiscussable(
@@ -88,7 +92,8 @@ class PlanningTask(
         userMessage: String,
         toInput: (String) -> List,
         api: API,
-        ui: ApplicationInterface
+        ui: ApplicationInterface,
+        planSettings: PlanSettings
     ) = Discussable(
         task = task,
         userMessage = { "Expand ${planTask?.task_description ?: ""}" },
@@ -116,27 +121,34 @@ class PlanningTask(
     )
 
     private fun executeSubTasks(
-        agent: PlanCoordinator,
+        coordinator: PlanCoordinator,
         userMessage: String,
         subPlan: Map,
         parentTask: SessionTask,
-        api: API
+        api: API,
+        api2: OpenAIClient,
     ) {
-        val subPlanTask = agent.ui.newTask(false)
+        val subPlanTask = coordinator.ui.newTask(false)
         parentTask.add(subPlanTask.placeholder)
-        val subTasks = subPlan ?: emptyMap()
-        val planProcessingState = PlanProcessingState(subTasks.toMutableMap())
-        agent.executePlan(
+        val planProcessingState = PlanProcessingState(subPlan.toMutableMap())
+        coordinator.copy(
+            planSettings = coordinator.planSettings.copy(
+                taskSettings = coordinator.planSettings.taskSettings.toList().toTypedArray().toMap().toMutableMap().apply {
+                    this["TaskPlanning"] = TaskSettings(enabled = false, model = null)
+                }
+            )
+        ).executePlan(
             task = subPlanTask,
-            diagramBuffer = subPlanTask.add(diagram(agent.ui, planProcessingState.subTasks)),
-            subTasks = subTasks,
+            diagramBuffer = subPlanTask.add(diagram(coordinator.ui, planProcessingState.subTasks)),
+            subTasks = subPlan,
             diagramTask = subPlanTask,
             planProcessingState = planProcessingState,
-            taskIdProcessingQueue = executionOrder(subTasks).toMutableList(),
-            pool = agent.pool,
+            taskIdProcessingQueue = executionOrder(subPlan).toMutableList(),
+            pool = coordinator.pool,
             userMessage = userMessage,
             plan = subPlan,
             api = api,
+            api2 = api2,
         )
         subPlanTask.complete()
     }
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/RunShellCommandTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/RunShellCommandTask.kt
index cc2561d4..60d014f0 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/RunShellCommandTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/RunShellCommandTask.kt
@@ -1,6 +1,7 @@
 package com.simiacryptus.skyenet.apps.plan
 
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.jopenai.models.ApiModel
 import com.simiacryptus.skyenet.apps.code.CodingAgent
@@ -65,11 +66,13 @@ Note: This task is for running simple and safe commands. Avoid executing command
     }
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         val semaphore = Semaphore(0)
         object : CodingAgent(
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/TaskType.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/TaskType.kt
index ea13a3e4..d5bfe8ec 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/TaskType.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/TaskType.kt
@@ -17,12 +17,13 @@ import com.simiacryptus.skyenet.apps.plan.file.CodeOptimizationTask.CodeOptimiza
 import com.simiacryptus.skyenet.apps.plan.file.CodeReviewTask.CodeReviewTaskData
 import com.simiacryptus.skyenet.apps.plan.file.DocumentationTask.DocumentationTaskData
 import com.simiacryptus.skyenet.apps.plan.file.FileModificationTask.FileModificationTaskData
-import com.simiacryptus.skyenet.apps.plan.file.GoogleSearchTask.GoogleSearchTaskData
+import com.simiacryptus.skyenet.apps.plan.GoogleSearchTask.GoogleSearchTaskData
 import com.simiacryptus.skyenet.apps.plan.file.InquiryTask.InquiryTaskData
 import com.simiacryptus.skyenet.apps.plan.file.PerformanceAnalysisTask.PerformanceAnalysisTaskData
 import com.simiacryptus.skyenet.apps.plan.file.RefactorTask.RefactorTaskData
 import com.simiacryptus.skyenet.apps.plan.file.SecurityAuditTask.SecurityAuditTaskData
 import com.simiacryptus.skyenet.apps.plan.file.TestGenerationTask.TestGenerationTaskData
+import com.simiacryptus.skyenet.apps.plan.knowledge.EmbeddingSearchTask
 import com.simiacryptus.util.DynamicEnum
 import com.simiacryptus.util.DynamicEnumDeserializer
 import com.simiacryptus.util.DynamicEnumSerializer
@@ -92,11 +93,7 @@ class TaskType(
         fun getImpl(
             planSettings: PlanSettings,
             planTask: PlanTaskBase?
-        ): AbstractTask {
-            val taskType = planTask?.task_type?.let { valueOf(it) }
-                ?: throw RuntimeException("Task type not specified")
-            return getImpl(planSettings, taskType, planTask)
-        }
+        ) = getImpl(planSettings, planTask?.task_type?.let { valueOf(it) } ?: throw RuntimeException("Task type not specified"), planTask)
 
         fun getImpl(
             planSettings: PlanSettings,
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/WebFetchAndTransformTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/WebFetchAndTransformTask.kt
similarity index 93%
rename from webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/WebFetchAndTransformTask.kt
rename to webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/WebFetchAndTransformTask.kt
index cac9445b..48ba78a1 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/WebFetchAndTransformTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/WebFetchAndTransformTask.kt
@@ -1,8 +1,9 @@
-package com.simiacryptus.skyenet.apps.plan.file
+package com.simiacryptus.skyenet.apps.plan
 
 import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
-import com.simiacryptus.skyenet.apps.plan.*
 import com.simiacryptus.skyenet.core.actors.SimpleActor
 import com.simiacryptus.skyenet.util.MarkdownUtil
 import com.simiacryptus.skyenet.webui.session.SessionTask
@@ -42,11 +43,13 @@ open class WebFetchAndTransformTask(
     agent: PlanCoordinator,
     messages: List,
     task: SessionTask,
-    api: API,
-    resultFn: (String) -> Unit
+    api: ChatClient,
+    resultFn: (String) -> Unit,
+    api2: OpenAIClient,
+    planSettings: PlanSettings
   ) {
     val fetchedContent = fetchAndStripHtml(planTask?.url ?: "")
-    val transformedContent = transformContent(fetchedContent, planTask?.transformationGoal ?: "", api)
+    val transformedContent = transformContent(fetchedContent, planTask?.transformationGoal ?: "", api, planSettings)
     task.add(MarkdownUtil.renderMarkdown(transformedContent, ui = agent.ui))
     resultFn(transformedContent)
   }
@@ -112,7 +115,7 @@ open class WebFetchAndTransformTask(
     }
   }
 
-  private fun transformContent(content: String, transformationGoal: String, api: API): String {
+  private fun transformContent(content: String, transformationGoal: String, api: API, planSettings: PlanSettings): String {
     val prompt = """
             Transform the following web content according to this goal: $transformationGoal
             
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/AbstractAnalysisTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/AbstractAnalysisTask.kt
index 98f50877..1469304c 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/AbstractAnalysisTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/AbstractAnalysisTask.kt
@@ -2,6 +2,7 @@ package com.simiacryptus.skyenet.apps.plan.file
 
 import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.skyenet.apps.general.CommandPatchApp
 import com.simiacryptus.skyenet.apps.general.PatchApp
 import com.simiacryptus.skyenet.apps.plan.*
@@ -29,11 +30,13 @@ abstract class AbstractAnalysisTask(
     }
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         val analysisResult = analysisActor.answer(
             messages + listOf(
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/DocumentationTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/DocumentationTask.kt
index ae3740df..70af1f4a 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/DocumentationTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/DocumentationTask.kt
@@ -1,9 +1,9 @@
 package com.simiacryptus.skyenet.apps.plan.file
 
 import com.simiacryptus.diff.addApplyFileDiffLinks
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
-import com.simiacryptus.jopenai.models.chatModel
 import com.simiacryptus.skyenet.Retryable
 import com.simiacryptus.skyenet.apps.plan.*
 import com.simiacryptus.skyenet.apps.plan.file.DocumentationTask.DocumentationTaskData
@@ -66,11 +66,13 @@ class DocumentationTask(
     }
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         if (((planTask?.input_files ?: listOf()) + (planTask?.output_files ?: listOf())).isEmpty()) {
             task.complete("No input or output files specified")
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/EmbeddingSearchTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/EmbeddingSearchTask.kt
deleted file mode 100644
index b00cf31d..00000000
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/EmbeddingSearchTask.kt
+++ /dev/null
@@ -1,122 +0,0 @@
-package com.simiacryptus.skyenet.apps.plan.file
-
-import com.simiacryptus.jopenai.API
-import com.simiacryptus.jopenai.OpenAIClient
-import com.simiacryptus.jopenai.describe.Description
-import com.simiacryptus.jopenai.models.ApiModel
-import com.simiacryptus.jopenai.opt.DistanceType
-import com.simiacryptus.skyenet.apps.parse.DocumentRecord
-import com.simiacryptus.skyenet.apps.plan.*
-import com.simiacryptus.skyenet.util.MarkdownUtil
-import com.simiacryptus.skyenet.webui.session.SessionTask
-import org.slf4j.LoggerFactory
-import java.nio.file.FileSystems
-import java.nio.file.Files
-import kotlin.streams.asSequence
-
-class EmbeddingSearchTask(
-    planSettings: PlanSettings,
-    planTask: EmbeddingSearchTaskData?
-) : AbstractTask(planSettings, planTask) {
-    class EmbeddingSearchTaskData(
-        @Description("The search query to look for in the embeddings")
-        val search_query: String,
-        @Description("The distance type to use for comparing embeddings (Euclidean, Manhattan, or Cosine)")
-        val distance_type: DistanceType = DistanceType.Cosine,
-        @Description("The number of top results to return")
-        val top_k: Int = 5,
-        @Description("The specific index files (or file patterns) to be searched")
-        val input_files: List? = null,
-        task_description: String? = null,
-        task_dependencies: List? = null,
-        state: TaskState? = null,
-    ) : PlanTaskBase(
-        task_type = "EmbeddingSearch",
-        task_description = task_description,
-        task_dependencies = task_dependencies,
-        state = state
-    )
-
-    override fun promptSegment() = """
-EmbeddingSearch - Search for similar embeddings in index files and provide top results
-    ** Specify the search query
-    ** Specify the distance type (Euclidean, Manhattan, or Cosine)
-    ** Specify the number of top results to return
-    ** List input index files or file patterns to be searched
-    """.trimMargin()
-
-    override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
-    ) {
-        val searchResults = performEmbeddingSearch(api as OpenAIClient)
-        val formattedResults = formatSearchResults(searchResults)
-        task.add(MarkdownUtil.renderMarkdown(formattedResults, ui = agent.ui))
-        resultFn(formattedResults)
-    }
-
-    private fun performEmbeddingSearch(api: OpenAIClient): List {
-        val queryEmbedding = api.createEmbedding(
-            ApiModel.EmbeddingRequest(
-                input = planTask?.search_query ?: "",
-                model = (planSettings.getTaskSettings(TaskType.EmbeddingSearch).model
-                    ?: planSettings.defaultModel).modelName
-
-            )
-        ).data[0].embedding
-        val distanceType = planTask?.distance_type ?: DistanceType.Cosine
-
-        return (planTask?.input_files ?: listOf())
-            .flatMap { filePattern ->
-                val matcher = FileSystems.getDefault().getPathMatcher("glob:$filePattern")
-                Files.walk(root).asSequence()
-                    .filter { path ->
-                        matcher.matches(root.relativize(path)) && path.toString().endsWith(".index.data")
-                    }
-                    .flatMap { path ->
-                        val records = DocumentRecord.readBinary(path.toString())
-                        records.mapNotNull { record ->
-                            record.vector?.let { vector ->
-                                EmbeddingSearchResult(
-                                    file = root.relativize(path).toString(),
-                                    record = record,
-                                    distance = distanceType.distance(vector, queryEmbedding ?: DoubleArray(0))
-                                )
-                            }
-                        }
-                    }
-                    .toList()
-            }
-            .sortedBy { it.distance }
-            .take(planTask?.top_k ?: 5)
-    }
-
-    private fun formatSearchResults(results: List): String {
-        return buildString {
-            appendLine("# Embedding Search Results")
-            appendLine()
-            results.forEachIndexed { index, result ->
-                appendLine("## Result ${index + 1}")
-                appendLine("- File: ${result.file}")
-                appendLine("- Distance: ${result.distance}")
-                appendLine("- Text: ${result.record.text}")
-                appendLine("- Source Path: ${result.record.sourcePath}")
-                appendLine("- JSON Path: ${result.record.jsonPath}")
-                appendLine()
-            }
-        }
-    }
-
-    data class EmbeddingSearchResult(
-        val file: String,
-        val record: DocumentRecord,
-        val distance: Double
-    )
-
-    companion object {
-        private val log = LoggerFactory.getLogger(EmbeddingSearchTask::class.java)
-    }
-}
\ No newline at end of file
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/FileModificationTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/FileModificationTask.kt
index dbf99a58..392be5ab 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/FileModificationTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/FileModificationTask.kt
@@ -1,7 +1,8 @@
 package com.simiacryptus.skyenet.apps.plan.file
 
 import com.simiacryptus.diff.addApplyFileDiffLinks
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.skyenet.Retryable
 import com.simiacryptus.skyenet.apps.plan.*
@@ -94,11 +95,13 @@ class FileModificationTask(
     }
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         if (((planTask?.input_files ?: listOf()) + (planTask?.output_files ?: listOf())).isEmpty()) {
             task.complete("CONFIGURATION ERROR: No input files specified")
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/InquiryTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/InquiryTask.kt
index 8ed4899b..be448660 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/InquiryTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/InquiryTask.kt
@@ -1,7 +1,8 @@
 package com.simiacryptus.skyenet.apps.plan.file
 
 import com.simiacryptus.diff.FileValidationUtils
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.jopenai.models.ApiModel
 import com.simiacryptus.jopenai.models.ApiModel.Role
@@ -75,8 +76,10 @@ class InquiryTask(
         agent: PlanCoordinator,
         messages: List,
         task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+        api: ChatClient,
+        resultFn: (String) -> Unit,
+        api2: OpenAIClient,
+        planSettings: PlanSettings
     ) {
 
         val toInput = { it: String ->
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/SearchTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/SearchTask.kt
index 96200c8f..b9ae95a0 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/SearchTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/file/SearchTask.kt
@@ -1,7 +1,8 @@
 package com.simiacryptus.skyenet.apps.plan.file
 
 import com.simiacryptus.diff.FileValidationUtils
-import com.simiacryptus.jopenai.API
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.describe.Description
 import com.simiacryptus.skyenet.apps.plan.*
 import com.simiacryptus.skyenet.util.MarkdownUtil
@@ -44,11 +45,13 @@ Search - Search for patterns in files and provide results with context
     """.trimMargin()
 
     override fun run(
-        agent: PlanCoordinator,
-        messages: List,
-        task: SessionTask,
-        api: API,
-        resultFn: (String) -> Unit
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
     ) {
         val searchResults = performSearch()
         val formattedResults = formatSearchResults(searchResults)
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/knowledge/EmbeddingSearchTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/knowledge/EmbeddingSearchTask.kt
new file mode 100644
index 00000000..9e841c87
--- /dev/null
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/apps/plan/knowledge/EmbeddingSearchTask.kt
@@ -0,0 +1,229 @@
+package com.simiacryptus.skyenet.apps.plan.knowledge
+
+import com.fasterxml.jackson.databind.JsonNode
+import com.fasterxml.jackson.databind.ObjectMapper
+import com.simiacryptus.jopenai.ChatClient
+import com.simiacryptus.jopenai.OpenAIClient
+import com.simiacryptus.jopenai.describe.Description
+import com.simiacryptus.jopenai.models.ApiModel
+import com.simiacryptus.jopenai.models.EmbeddingModels
+import com.simiacryptus.jopenai.opt.DistanceType
+import com.simiacryptus.skyenet.apps.parse.DocumentRecord
+import com.simiacryptus.skyenet.apps.plan.*
+import com.simiacryptus.skyenet.util.MarkdownUtil
+import com.simiacryptus.skyenet.webui.session.SessionTask
+import com.simiacryptus.util.JsonUtil
+import org.slf4j.LoggerFactory
+import java.io.File
+import java.nio.file.Files
+import java.util.regex.Pattern
+import kotlin.streams.asSequence
+
+class EmbeddingSearchTask(
+    planSettings: PlanSettings,
+    planTask: EmbeddingSearchTaskData?
+) : AbstractTask(planSettings, planTask) {
+    class EmbeddingSearchTaskData(
+        @Description("The positive search queries to look for in the embeddings")
+        val positive_queries: List,
+        @Description("The negative search queries to avoid in the embeddings")
+        val negative_queries: List = emptyList(),
+        @Description("The distance type to use for comparing embeddings (Euclidean, Manhattan, or Cosine)")
+        val distance_type: DistanceType = DistanceType.Cosine,
+        @Description("The number of top results to return")
+        val count: Int = 5,
+        @Description("The minimum length of the content to be considered")
+        val min_length: Int = 0,
+        @Description("List of regex patterns that must be present in the content")
+        val required_regexes: List = emptyList(),
+        task_description: String? = null,
+        task_dependencies: List? = null,
+        state: TaskState? = null,
+    ) : PlanTaskBase(
+        task_type = "EmbeddingSearch",
+        task_description = task_description,
+        task_dependencies = task_dependencies,
+        state = state
+    )
+
+    override fun promptSegment() = """
+EmbeddingSearch - Search for similar embeddings in index files and provide top results
+    ** Specify the positive search queries
+    ** Optionally specify negative search queries
+    ** Specify the distance type (Euclidean, Manhattan, or Cosine)
+    ** Specify the number of top results to return
+    """.trim()
+
+    override fun run(
+      agent: PlanCoordinator,
+      messages: List,
+      task: SessionTask,
+      api: ChatClient,
+      resultFn: (String) -> Unit,
+      api2: OpenAIClient,
+      planSettings: PlanSettings
+    ) {
+        val searchResults = performEmbeddingSearch(api2)
+        val formattedResults = formatSearchResults(searchResults)
+        task.add(MarkdownUtil.renderMarkdown(formattedResults, ui = agent.ui))
+        resultFn(formattedResults)
+    }
+
+    private fun performEmbeddingSearch(api: OpenAIClient): List {
+        val positiveEmbeddings = planTask?.positive_queries?.map { query ->
+            api.createEmbedding(
+                ApiModel.EmbeddingRequest(
+                    input = query,
+                    model = EmbeddingModels.Large.modelName
+                )
+            ).data[0].embedding
+        } ?: emptyList()
+        val negativeEmbeddings = planTask?.negative_queries?.map { query ->
+            api.createEmbedding(
+                ApiModel.EmbeddingRequest(
+                    input = query,
+                    model = EmbeddingModels.Large.modelName
+                )
+            ).data[0].embedding
+        } ?: emptyList()
+        if (positiveEmbeddings.isEmpty()) {
+            throw IllegalArgumentException("At least one positive query is required")
+        }
+        val distanceType = planTask?.distance_type ?: DistanceType.Cosine
+        val filtered = Files.walk(root).asSequence()
+            .filter { path ->
+                path.toString().endsWith(".index.data")
+            }.toList().toTypedArray()
+        val minLength = planTask?.min_length ?: 0
+        val requiredRegexes = planTask?.required_regexes?.map { Pattern.compile(it) } ?: emptyList()
+        fun String.matchesAllRegexes(): Boolean {
+            return requiredRegexes.all { regex -> regex.matcher(this).find() }
+        }
+
+        val searchResults = filtered
+            .flatMap { path ->
+                val records = DocumentRecord.readBinary(path.toString())
+                records.mapNotNull { record ->
+                    record.vector?.let { vector ->
+                        val positiveDistances = positiveEmbeddings.filterNotNull().map { embedding ->
+                            distanceType.distance(vector, embedding)
+                        }
+                        val negativeDistances = negativeEmbeddings.filterNotNull().map { embedding ->
+                            distanceType.distance(vector, embedding)
+                        }
+                        val overallDistance = if (negativeDistances.isEmpty()) {
+                            positiveDistances.minOrNull() ?: Double.MAX_VALUE
+                        } else {
+                            (positiveDistances.minOrNull() ?: Double.MAX_VALUE) / (negativeDistances.minOrNull() ?: Double.MIN_VALUE)
+                        }
+                        val content = record.text ?: ""
+                        if (content.length >= minLength && content.matchesAllRegexes()) {
+                            EmbeddingSearchResult(
+                                file = root.relativize(path).toString(),
+                                record = record,
+                                distance = overallDistance
+                            )
+                        } else null
+                    }
+                }
+            }
+            .toList()
+        return searchResults
+            .sortedBy { it.distance }
+            .take(planTask?.count ?: 5)
+    }
+
+    private fun formatSearchResults(results: List): String {
+        return buildString {
+            appendLine("# Embedding Search Results")
+            appendLine()
+            results.forEachIndexed { index, result ->
+                appendLine("## Result ${index + 1}")
+                appendLine("* Distance: %.3f".format(result.distance))
+                appendLine("* File: ${result.record.sourcePath}")
+                appendLine(getContextSummary(result.record.sourcePath, result.record.jsonPath))
+                appendLine("Metadata:\n```json\n${result.record.metadata}\n```")
+                appendLine()
+            }
+        }
+    }
+
+    private fun getContextSummary(sourcePath: String, jsonPath: String): String {
+        val objectMapper = ObjectMapper()
+        val jsonNode = objectMapper.readTree(File(sourcePath))
+        val contextNode = getNodeAtPath(jsonNode, jsonPath)
+        return buildString {
+            appendLine("```json")
+            appendLine(summarizeContext(contextNode, jsonPath, jsonNode))
+            appendLine("```")
+        }
+    }
+
+    private fun getNodeAtPath(jsonNode: JsonNode, path: String): JsonNode {
+        var currentNode = jsonNode
+        path.split(".").forEach { segment ->
+            currentNode = when {
+                segment.contains("[") -> {
+                    val (arrayName, indexPart) = segment.split("[", limit = 2)
+                    val index = indexPart.substringBefore("]").toInt()
+                    val field = currentNode.get(arrayName)
+                    val child = field?.get(index)
+                    if (child == null) {
+                        return currentNode
+                    }
+                    child
+                }
+
+                else -> {
+                    val child = currentNode.get(segment)
+                    if (child == null) {
+                        return currentNode
+                    }
+                    child
+                }
+            }
+        }
+        return currentNode
+    }
+
+    private fun summarizeContext(node: JsonNode, path: String, jsonNode: JsonNode): String {
+        var summary = mutableMapOf()
+        // Add siblings and descendants
+        node.fields().forEach { (key, value) ->
+            if (value.isPrimitive()) {
+                summary[key] = value.asText()
+            }
+        }
+        // Add siblings of parent nodes
+        val pathSegments = path.split(".")
+        for (i in pathSegments.size - 1 downTo 1) {
+            val parentPath = pathSegments.subList(0, i).joinToString(".")
+            val parentNode = getNodeAtPath(jsonNode, parentPath)
+            summary = mutableMapOf(
+                pathSegments[i] to summary
+            )
+            parentNode.fields().forEach { (key, value) ->
+              when {
+                value.isPrimitive() -> summary[key] = value.asText()
+                key == "entities" || key == "tags" || key == "metadata" -> summary[key] = value
+              }
+            }
+        }
+        return JsonUtil.toJson(summary)
+    }
+
+
+    data class EmbeddingSearchResult(
+        val file: String,
+        val record: DocumentRecord,
+        val distance: Double
+    )
+
+    companion object {
+        private val log = LoggerFactory.getLogger(EmbeddingSearchTask::class.java)
+    }
+}
+
+private fun JsonNode.isPrimitive(): Boolean {
+    return this.isNumber || this.isTextual || this.isBoolean
+}
\ No newline at end of file
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/util/TensorflowProjector.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/util/TensorflowProjector.kt
index 03a1f525..aedb1c88 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/util/TensorflowProjector.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/util/TensorflowProjector.kt
@@ -4,12 +4,16 @@ import com.simiacryptus.jopenai.API
 import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.models.ApiModel.EmbeddingRequest
 import com.simiacryptus.jopenai.models.EmbeddingModels
+import com.simiacryptus.skyenet.apps.parse.DocumentRecord
 import com.simiacryptus.skyenet.core.platform.ApplicationServices.cloud
 import com.simiacryptus.skyenet.core.platform.Session
 import com.simiacryptus.skyenet.core.platform.model.StorageInterface
 import com.simiacryptus.skyenet.core.platform.model.User
 import com.simiacryptus.skyenet.webui.application.ApplicationInterface
 import com.simiacryptus.util.JsonUtil
+import java.util.UUID
+import java.io.IOException
+import kotlin.jvm.Throws
 
 class TensorflowProjector(
     val api: API,
@@ -17,22 +21,51 @@ class TensorflowProjector(
     val sessionID: Session,
     val session: ApplicationInterface,
     val userId: User?,
+    private val iframeHeight: Int = 500,
+    private val iframeWidth: String = "100%"
 ) {
+    companion object {
+        private const val VECTOR_FILENAME = "vectors.tsv"
+        private const val METADATA_FILENAME = "metadata.tsv"
+        private const val CONFIG_FILENAME = "projector-config.json"
+        private const val PROJECTOR_URL = "https://projector.tensorflow.org/"
+    }
+    @Throws(IOException::class)
 
     private fun toVectorMap(vararg words: String): Map {
         val vectors = words.map { word ->
             word to (api as OpenAIClient).createEmbedding(
                 EmbeddingRequest(
                     model = EmbeddingModels.AdaEmbedding.modelName,
-                    input = word,
+                    input = word.trim(),
                 )
             ).data.first().embedding!!
         }
         return vectors.toMap()
     }
+    @Throws(IOException::class)
+
+    fun writeTensorflowEmbeddingProjectorHtmlFromRecords(records: List): String {
+        val vectorMap = records
+            .filter { it.text != null && it.vector != null }
+            .associate { record -> 
+                record.text!!.trim() to record.vector!! 
+            }
+        require(vectorMap.isNotEmpty()) { "No valid records found with both text and vector" }
+        return writeTensorflowEmbeddingProjectorHtmlFromVectorMap(vectorMap)
+    }
+    @Throws(IOException::class)
 
     fun writeTensorflowEmbeddingProjectorHtml(vararg words: String): String {
-        val vectorMap = toVectorMap(*words.filter { it.isNotBlank() }.toList().toTypedArray())
+        val filteredWords = words.filter { it.isNotBlank() }.distinct()
+        require(filteredWords.isNotEmpty()) { "No valid words provided" }
+        val vectorMap = toVectorMap(*filteredWords.toTypedArray())
+        return writeTensorflowEmbeddingProjectorHtmlFromVectorMap(vectorMap)
+    }
+
+    private fun writeTensorflowEmbeddingProjectorHtmlFromVectorMap(vectorMap: Map): String {
+        require(vectorMap.isNotEmpty()) { "Vector map cannot be empty" }
+        
         val vectorTsv = vectorMap.map { (_, vector) ->
             vector.joinToString(separator = "\t") {
                 "%.2E".format(it)
@@ -40,41 +73,48 @@ class TensorflowProjector(
         }.joinToString(separator = "\n")
 
         val metadataTsv = vectorMap.keys.joinToString(separator = "\n") {
-            it.replace("\n", " ")
+            it.replace(Regex("\\s+"), " ").trim()
         }
 
-        val vectorFileName = "vectors.tsv"
-        val metadataFileName = "metadata.tsv"
-        val configFileName = "projector-config.json"
+        val uuid = UUID.randomUUID().toString()
         val sessionDir = dataStorage.getSessionDir(userId, sessionID)
-        sessionDir.resolve(vectorFileName).writeText(vectorTsv)
-        sessionDir.resolve(metadataFileName).writeText(metadataTsv)
-        //val vectorURL = """$host/$appPath/fileIndex/$sessionID/$vectorFileName"""
-        val vectorURL = cloud!!.upload("projector/$sessionID/$vectorFileName", "text/plain", vectorTsv)
-        //var metadataURL = """$host/$appPath/fileIndex/$sessionID/$metadataFileName"""
-        val metadataURL = cloud!!.upload("projector/$sessionID/$metadataFileName", "text/plain", metadataTsv)
+        sessionDir.resolve(VECTOR_FILENAME).writeText(vectorTsv)
+        sessionDir.resolve(METADATA_FILENAME).writeText(metadataTsv)
+        val vectorURL = cloud?.upload("projector/$sessionID/$uuid/$VECTOR_FILENAME", "text/plain", vectorTsv)
+            ?: throw IllegalStateException("Cloud storage not initialized")
+        val metadataURL = cloud?.upload("projector/$sessionID/$uuid/$METADATA_FILENAME", "text/plain", metadataTsv)
+
         val projectorConfig = JsonUtil.toJson(
             mapOf(
                 "embeddings" to listOf(
                     mapOf(
                         "tensorName" to "embedding",
-                        "tensorShape" to listOf(vectorMap.values.first().size, 1),
+                        "tensorShape" to listOf(vectorMap.size, vectorMap.values.first().size),
                         "tensorPath" to vectorURL,
                         "metadataPath" to metadataURL,
                     )
                 )
             )
         )
-        sessionDir.resolve(configFileName).writeText(projectorConfig)
-        //val configURL = """$host/$appPath/fileIndex/$sessionID/projector-config.json"""
-        val configURL = cloud!!.upload("projector/$sessionID/$configFileName", "application/json", projectorConfig)
+        sessionDir.resolve(CONFIG_FILENAME).writeText(projectorConfig)
+        val configURL = cloud?.upload("projector/$sessionID/$CONFIG_FILENAME", "application/json", projectorConfig)
+
         return """
-            Projector Config
-            Vectors
-            Metadata
-            Projector
-            
+            

+                

+                    Projector Config |
+                    Vectors |
+                    Metadata |
+                    Open in Projector
+                

+                
+            

             """.trimIndent()
     }
-
 }
\ No newline at end of file
diff --git a/webui/src/main/kotlin/com/simiacryptus/skyenet/webui/session/SessionTask.kt b/webui/src/main/kotlin/com/simiacryptus/skyenet/webui/session/SessionTask.kt
index 3e8bfe17..de957220 100644
--- a/webui/src/main/kotlin/com/simiacryptus/skyenet/webui/session/SessionTask.kt
+++ b/webui/src/main/kotlin/com/simiacryptus/skyenet/webui/session/SessionTask.kt
@@ -205,6 +205,8 @@ abstract class SessionTask(
     }
 
     abstract fun createFile(relativePath: String): Pair
+
+    fun update() = send()
 }
 
 val Throwable.stackTraceTxt: String
diff --git a/webui/src/test/kotlin/com/simiacryptus/skyenet/webui/ActorTestAppServer.kt b/webui/src/test/kotlin/com/simiacryptus/skyenet/webui/ActorTestAppServer.kt
index 4f4004a9..d9bed391 100644
--- a/webui/src/test/kotlin/com/simiacryptus/skyenet/webui/ActorTestAppServer.kt
+++ b/webui/src/test/kotlin/com/simiacryptus/skyenet/webui/ActorTestAppServer.kt
@@ -1,5 +1,6 @@
 package com.simiacryptus.skyenet.webui
 
+import com.simiacryptus.jopenai.OpenAIClient
 import com.simiacryptus.jopenai.models.OpenAIModels
 import com.simiacryptus.jopenai.util.ClientUtil.keyTxt
 import com.simiacryptus.skyenet.apps.parse.DocumentParserApp
@@ -114,6 +115,7 @@ object ActorTestAppServer : com.simiacryptus.skyenet.webui.application.Applicati
                     },
                     model = OpenAIModels.GPT4o,
                     parsingModel = OpenAIModels.GPT4oMini,
+                    api2 = OpenAIClient()
                 )
             ),
             ChildWebApp("/stressTest", StressTestApp()),