diff --git a/README.md b/README.md index 7d552ff..09bca85 100644 --- a/README.md +++ b/README.md @@ -3,746 +3,370 @@ agentCores logo

-## agentCores: Advanced AI Agent Management System +## Overview -agentCores is a powerful and flexible Python package designed to streamline the creation, management, and deployment of AI agents. It provides a robust framework for handling complex agent configurations, making it easier for developers and researchers to work with multiple AI models and agent types. +agentCores is a powerful MongoDB-based system for creating, managing, and deploying AI agents. It provides a comprehensive framework for handling complex agent configurations, making it easier to work with multiple AI models and agent types. -## What agentCores Does +### What agentCores Does 1. **Agent Configuration Management**: - - Create, store, and manage AI agent configurations in a structured SQLite database. - - Support for various agent types, from simple chatbots to complex multi-modal AI systems. + - Create, store, and manage AI agent configurations in MongoDB + - Version control and unique identifiers for each agent + - Template-based agent creation with extensive customization -2. **Template-Based Agent Creation**: - - Define and use templates for quick agent instantiation. - - Easily create new agents based on existing templates with custom overrides. +2. **Database Integration**: + - MongoDB collections for agent data, conversations, and knowledge + - Customizable collection structures and configurations + - Built-in database management and optimization -3. **Versioning and Tracking**: - - Maintain version history of agent configurations. - - Generate unique identifiers (UIDs) for each agent instance. +3. **Model and Prompt Management**: + - Support for multiple AI models (language, vision, speech) + - Structured prompt templates and version control + - Integration with Ollama and other model providers -4. **Flexible Database Integration**: - - Store agent data, conversation histories, and knowledge bases in SQLite. - - Customizable database paths and configurations for different project needs. +4. **Research and Development Tools**: + - Built-in research assistant capabilities + - Paper and code repository integration + - Knowledge base management -5. **Command-Line Interface**: - - Intuitive CLI for managing agents, perfect for quick tweaks and testing. - - Commands for listing, creating, modifying, and deleting agents. +5. **Command-line Interface**: + - Interactive agent management + - Real-time chat capabilities + - Database administration tools -6. **Model and Prompt Management**: - - Configure and manage different AI models (e.g., language models, vision models). - - Store and version control prompts and system messages. +## Installation -7. **Extensibility and Customization**: - - Easy integration with various AI libraries and APIs (e.g., Ollama, OpenAI). - - Support for custom flags and agent-specific settings. - -## Key Benefits - -- **Centralized Management**: Keep all your AI agents organized in one place. -- **Rapid Prototyping**: Quickly create and test different agent configurations. -- **Scalability**: Easily manage multiple agents for complex AI systems or large-scale deployments. -- **Version Control**: Track changes and revert to previous configurations when needed. -- **Flexibility**: Adapt to various AI frameworks and model types. -- **Standardization**: Enforce consistent structure across different agent types. - -## Ideal For - -- **AI Researchers**: Experiment with different agent configurations and model combinations. -- **Chatbot Developers**: Manage multiple chatbots with different personalities or capabilities. -- **MLOps Teams**: Streamline the deployment and management of AI models in production. -- **Game Developers**: Create and manage diverse NPC behaviors and AI opponents. -- **Education Platforms**: Develop and maintain various AI tutors or educational assistants. - -## Future-Ready Features - -1. **JSON Template Support**: Easily export and import agent configurations. -2. **Embedding Integration**: Planned support for managing and utilizing embeddings. -3. **Advanced Analytics**: Track agent performance and usage statistics. -4. **Multi-Agent Orchestration**: Tools for managing interactions between multiple agents. -5. **pydantic & ollama agentCores**: Future agentCores built around ollama & pydantic agents. Unleashing flexible local agent dbs for advanced code splicing, and agent state monitoring. - -agentCores provides a solid foundation for building sophisticated AI agent systems, offering the flexibility and scalability needed for both research and production environments. Whether you're working on a simple chatbot or a complex multi-agent system, agentCores simplifies the process of creating, managing, and deploying AI agents. - -# installation ```bash pip install agentCores ``` -# quick start +### Dependencies +- Python 3.8+ +- MongoDB +- ollama (optional, for LLM functionality) +- duckduckgo_search (optional, for research capabilities) -# command-line interface +## MongoDB Structure -To access the command-line interface, run: +agentCores uses a single MongoDB database named `agentCores` with organized collections: -```shellscript -python -m agentCores -``` -This will start the agentCores Management Interface where you can manage your agents using various commands. - -Start by using the /help command: -```cmd -Enter commands to manage agent cores. Type '/help' for options. - -> /help - -Commands: - /agentCores - List all agent cores. - /showAgent - Show agents with the specified ID. - /createAgent - Mint a new agent. - /createCustomAgent - Interactive custom agent creation. - /createDatabase - Create a new database. - /linkDatabase - Link database to agent. - /storeAgent - Store agentCore from json path. - /exportAgent - Export agentCore to json. - /deleteAgent - Delete an agent by UID. - /resetAgent - Reset an agent to the base template. - /chat - Start a chat session with an agent." - /exit - Exit the interface." -``` - -Now to get the available agentCores: -``` -> /agentCores - -ID: default_agent, UID: ff22a0c1, Version: 1 -ID: promptBase, UID: 6f18aba0, Version: 1 -ID: speedChatAgent, UID: f1a7092c, Version: 1 -ID: ehartfordDolphin, UID: 18556c0c, Version: 1 -ID: minecraft_agent, UID: 25389031, Version: 1 -ID: general_navigator_agent, UID: d1f12a46, Version: 1 -``` - -Now to see an agentCore use the following command -```cmd -> /showAgent general_navigator_agent -``` - -The agentCore will now be displayed: -```json -{ - "agentCore": { - "agent_id": "general_navigator_agent", - "version": 1, - "uid": "d1f12a46", - "save_state_date": "2024-12-11", - "models": { - "large_language_model": null, - "embedding_model": null, - "language_and_vision_model": null, - "yolo_model": null, - "whisper_model": null, - "voice_model": null - }, - "prompts": { - "user_input_prompt": "", - "agentPrompts": { - "llmSystemPrompt": "You are a helpful llm assistant, designated with with fulling the user's request, the user is communicating with speech recognition and is sending their screenshot data to the vision model for decomposition. Receive this destription and Instruct the user and help them fullfill their request by collecting the vision data and responding. ", - "llmBoosterPrompt": "Here is the output from the vision model describing the user screenshot data along with the users speech data. Please reformat this data, and formulate a fullfillment for the user request in a conversational speech manner which will be processes by the text to speech model for output. ", - "visionSystemPrompt": "You are an image recognition assistant, the user is sending you a request and an image please fullfill the request. ", - "visionBoosterPrompt": "Given the provided screenshot, please provide a list of objects in the image with the attributes that you can recognize. " - } - }, - "commandFlags": { - "TTS_FLAG": false, - "STT_FLAG": true, - "CHUNK_FLAG": false, - "AUTO_SPEECH_FLAG": false, - "LLAVA_FLAG": true, - "SPLICE_FLAG": false, - "SCREEN_SHOT_FLAG": false, - "LATEX_FLAG": false, - "CMD_RUN_FLAG": false, - "AGENT_FLAG": true, - "MEMORY_CLEAR_FLAG": false - }, - "conversation": { - "save_name": "defaultConversation", - "load_name": "defaultConversation" - } - } -} ``` - -Now to export an agentCore to json execute the following: -```cmd -> /exportAgent general_navigator_agent -agentCore saved to general_navigator_agent_core.json +MongoDB Database: agentCores/ +├── System Collections: +│ ├── agent_matrix # Main agent configurations +│ ├── documentation # System documentation +│ ├── templates # Agent templates +│ └── template_files # Template resources +├── Agent-Specific Collections: +│ ├── conversations_{agent_id} # Chat history +│ ├── knowledge_{agent_id} # Knowledge base +│ ├── embeddings_{agent_id} # Vector embeddings +│ ├── research_{agent_id} # Research data +│ └── design_patterns_{agent_id} # Agent patterns +└── Shared Collections: + ├── global_knowledge # Shared knowledge base + ├── model_configs # Model configurations + └── prompt_templates # Prompt templates ``` -# core development methods +## Quick Start -## __init__ -```python -agentCores.__init__(db_path: str = "agent_matrix.db", db_config: Optional[Dict] = None, template: Optional[Dict] = None) -""" -Initialize the agentCores system with optional custom configuration. - - db_path: Path to the main agent matrix database. - db_config: Custom database configuration. - template: Custom agent template. - -This method sets up the agentCores system, initializing the database and loading any custom configurations. -It's the first method you should call when using the agentCores package. -""" -``` +### Basic Usage -example usage: ```python from agentCores import agentCores -# Initialize with default settings +# Initialize with default MongoDB connection core = agentCores() -# Initialize with custom database path -core = agentCores(db_path="/path/to/custom/agent_matrix.db") +# Or specify custom MongoDB connection +core = agentCores(connection_uri="mongodb://localhost:27017/") -# Initialize with custom database configuration -custom_db_config = { - "system": {"agent_matrix": "/path/to/custom/matrix.db"}, - "agents": {"conversation": "/path/to/custom/conversations/{agent_id}.db"} -} -core = agentCores(db_config=custom_db_config) - -# Initialize with custom template -custom_template = { - "agentCore": { - "models": {"large_language_model": "phi3"}, - "prompts": {"user_input_prompt": "You are the agentCores assistant..."} - } -} -core = agentCores(template=custom_template) -``` - -## mintAgent - -```python -agentCores.mintAgent(agent_id: str, db_config: Optional[Dict] = None, model_config: Optional[Dict] = None, prompt_config: Optional[Dict] = None, command_flags: Optional[Dict] = None) -> Dict -""" -Create a new agent with custom configuration. - - agent_id: Unique identifier for the new agent. - db_config: Custom database configuration for the agent. - model_config: Model configuration for the agent. - prompt_config: Prompt configuration for the agent. - command_flags: Command flags for the agent. - - -This method creates a new agent with the specified configurations. It returns the complete agent configuration as a dictionary. -""" -``` - -example usage: -```python -new_agent = core.mintAgent( - agent_id="custom_assistant", - model_config={"large_language_model": "phi3"}, - prompt_config={ - "user_input_prompt": "You are a helpful assistant.", - "agentPrompts": { - "llmSystemPrompt": "Provide clear and concise answers." +# Create a simple agent +agent = core.mintAgent( + agent_id="my_assistant", + model_config={ + "largeLanguageModel": { + "names": ["llama2"], + "instances": None, + "concurrency": [], + "parrallelModels": None } }, - command_flags={"STREAM_FLAG": True} + prompt_config={ + "userInput": "You are a helpful assistant", + "agent": { + "llmSystem": "Provide clear and concise answers", + "llmBooster": "Use examples when helpful" + } + } ) + +# Start chatting +core.chat_with_agent("my_assistant") ``` -## loadAgentCore +### Research Assistant ```python -agentCores.loadAgentCore(agent_id: str) -> Optional[Dict[str, Any]] -""" -Load an agent configuration from the library. +from agentCores import researchAssistant - agent_id: ID of the agent to load. +# Initialize research assistant with MongoDB connection +assistant = researchAssistant( + agent_id="research_agent", + connection_uri="mongodb://localhost:27017/" +) -This method retrieves the configuration of a previously stored agent. It returns the agent's configuration as a dictionary if found, or None if the agent doesn't exist. -""" -``` +# Process a research query +response = assistant.process_query("What is quantum computing?") -example usage: -```python -agent_config = core.loadAgentCore("ehartfordDolphin") -if agent_config: - print(f"Loaded agent: {agent_config['agentCore']['agent_id']}") -else: - print("Agent not found") +# Start interactive research session +assistant.start_chat() ``` -## storeAgentCore +## Core Features -```python -agentCores.storeAgentCore(agent_id: str, core_config: Dict[str, Any]) -> None -""" -Store an agent configuration in the matrix. +### Agent Configuration Structure - agent_id: ID of the agent to store. - core_config: Configuration of the agent to store. - -This method saves or updates an agent's configuration in the database. -""" -``` - -example usage: ```python -agent_config = { +{ "agentCore": { - "agent_id": "custom_assistant", - "models": {"large_language_model": "phi3"}, - "prompts": {"user_input_prompt": "You are a helpful assistant."} + "identifyers": { + "agent_id": str, # Unique identifier + "uid": str, # Generated hash + "version": int, # Version number + "creationDate": float, # Creation timestamp + "cpuNoiseHex": str # Hardware identifier + }, + "models": { + "largeLanguageModel": { + "names": list, # Model names + "instances": None, # Instance count + "concurrency": list, # Concurrency settings + "parrallelModels": None, + "model_config_template": dict + }, + "largeLanguageAndVisionAssistant": dict, + "yoloVision": dict, + "speechRecognitionSTT": dict, + "voiceGenerationTTS": dict, + "embedding": dict + }, + "prompts": { + "userInput": str, + "agent": { + "llmSystem": str, + "llmBooster": str, + "visionSystem": str, + "visionBooster": str + } + }, + "modalityFlags": { + "TTS_FLAG": bool, # Text-to-speech + "STT_FLAG": bool, # Speech-to-text + "CHUNK_AUDIO_FLAG": bool, + "AUTO_SPEECH_FLAG": bool, + "LLAVA_FLAG": bool, + "SCREEN_SHOT_FLAG": bool, + "SPLICE_VIDEO_FLAG": bool, + "AUTO_COMMANDS_FLAG": bool, + "CLEAR_MEMORY_FLAG": bool, + "ACTIVE_AGENT_FLAG": bool + }, + "evolutionarySettings": { + "mutation": float, + "pain": float, + "hunger": float, + "fasting": bool, + "rationalizationFactor": float + } } } -core.storeAgentCore("custom_assistant", agent_config) ``` -## listAgentCores +### Command-line Interface -```python -agentCores.listAgentCores() -> list -""" -List all available agent cores. - -This method returns a list of all stored agent configurations, including their IDs, UIDs, and versions. -""" -``` - -example usage: -```python -all_agents = core.listAgentCores() -for agent in all_agents: - print(f"ID: {agent['agent_id']}, UID: {agent['uid']}, Version: {agent['version']}") -``` - -## deleteAgentCore - -```python -agentCores.deleteAgentCore(agent_id: str) -> None -""" -Remove an agent configuration from storage. - - agent_id: ID of the agent to delete. - -This method deletes an agent's configuration from the database. -""" -``` - -example usage: -```python -core.deleteAgentCore("custom_assistant") -print("Agent deleted") -``` - -## saveToFile - -```python -agentCores.saveToFile(agent_id: str, file_path: str) -> None -""" -Save an agent configuration to a JSON file. - - agent_id: ID of the agent to save. - file_path: Path to save the JSON file. - -This method exports an agent's configuration to a JSON file. -""" -``` - -example usage: -```python -core.saveToFile("custom_assistant", "custom_assistant_config.json") -print("Agent configuration saved to file") -``` - -## loadAgentFromFile - -```python -agentCores.loadAgentFromFile(file_path: str) -> None -""" -Load an agent configuration from a JSON file and store in matrix. - - file_path: Path to the JSON file to load. - -This method imports an agent configuration from a JSON file and stores it in the database. -""" -``` - -example usage: -```python -core.loadAgentFromFile("custom_assistant_config.json") -print("Agent configuration loaded from file and stored in database") +Start the interface: +```bash +python -m agentCores ``` -## importAgentCores - -```python -agentCores.importAgentCores(import_db_path: str) -> None -""" -Import agent cores from another agent_matrix.db file into the current system. - - import_db_path: Path to the agent_matrix.db file to import from. - -This method imports all agent configurations from another agent_matrix.db file into the current system. It will merge the imported agents with existing ones, updating any agents with matching IDs. Throws FileNotFoundError if the import database doesn't exist or Exception if there are issues during import. -""" +Available commands: ``` - -example usage: -```python -# Import agents from another agent_matrix.db -core = agentCores() - -# Import from a specific database file -core.importAgentCores("path/to/other/agent_matrix.db") - -# Import from a backup -core.importAgentCores("backups/agent_matrix_backup.db") - -# Import from a shared team configuration -core.importAgentCores("team/shared_agent_matrix.db") - -# Check the imported agents -agents = core.listAgentCores() -for agent in agents: - print(f"ID: {agent['agent_id']}, UID: {agent['uid']}, Version: {agent['version']}") +/help Show available commands +/agentCores List all agent cores +/showAgent Show agent configuration +/createAgent Create new agent +/createCustomAgent Interactive agent creation +/createCollection Create MongoDB collection +/linkCollection Link collection to agent +/storeAgent Import agent from JSON +/exportAgent Export agent to JSON +/deleteAgent Delete an agent +/resetAgent Reset agent to base template +/chat Start chat session +/knowledge View knowledge base +/conversations View conversation history ``` -## commandInterface +## Advanced Usage -```python -agentCores.commandInterface() -""" -Start the command-line interface for managing agents. - -This method launches an interactive command-line interface for managing agent cores. -""" -``` +### Custom Model Configuration -example usage: ```python -core.commandInterface() -``` - -### Additional Database Management - -```bash -# Create a new database -> /createDatabase research_results research_results.db - -# Link database to existing agent -> /linkDatabase advanced_research_agent citations citations.db -``` - -# Usage Examples - -## Basic Ollama Usage Example - -```python -from agentCores import agentCores -from ollama import chat - -# Initialize agentCore -cores = agentCores() - -# Create basic Ollama agent configuration -basic_config = { - "agent_id": "basic_assistant", - "models": { - "large_language_model": "llama2", - "embedding_model": "nomic-embed-text" - }, - "prompts": { - "user_input_prompt": "You are a helpful assistant using local models.", - "agentPrompts": { - "llmSystemPrompt": "Focus on providing clear, accurate responses. Break down complex topics into understandable explanations.", - "llmBoosterPrompt": "Include relevant examples when possible and highlight key points for better understanding." +model_config = { + "largeLanguageModel": { + "instances": 2, + "names": ["llama2", "codellama"], + "concurrency": ["parallel", "sequential"], + "parrallelModels": True, + "model_config_template": { + "temperature": 0.7, + "context_window": 4096, + "streaming": True } - }, - "commandFlags": { - "STREAM_FLAG": True, - "LOCAL_MODEL": True } } -# Create the agent agent = core.mintAgent( - agent_id="basic_assistant", - model_config=basic_config["models"], - prompt_config=basic_config["prompts"], - command_flags=basic_config["commandFlags"] + agent_id="advanced_agent", + model_config=model_config ) - -# Basic chat function -def chat_with_agent(agent_config, prompt): - system_prompt = ( - f"{agent_config['agentCore']['prompts']['user_input_prompt']} " - f"{agent_config['agentCore']['prompts']['agentPrompts']['llmSystemPrompt']} " - f"{agent_config['agentCore']['prompts']['agentPrompts']['llmBoosterPrompt']}" - ) - - stream = chat( - model=agent_config["agentCore"]["models"]["large_language_model"], - messages=[ - {'role': 'system', 'content': system_prompt}, - {'role': 'user', 'content': prompt} - ], - stream=True, - ) - - for chunk in stream: - print(chunk['message']['content'], end='', flush=True) - -# Example usage -chat_with_agent(agent, "Explain how neural networks work") ``` -## Advanced Ollama Chatbot Usage Example -We can now construct advanced local assistants with ollama agentCores, and embedded db filenames, for nested knowledgeBase architectures. +### Custom Collection Configuration ```python -from agentCores import agentCores -from ollama import chat - -# Initialize agentCore -cores = agentCores() - -# Create a coding assistant configuration -coding_config = { - "agent_id": "coding_assistant", - "models": { - "large_language_model": "codellama", # Using CodeLlama for coding tasks - "embedding_model": "nomic-embed-text" # For code embeddings and search - }, - "prompts": { - "user_input_prompt": "You are an expert programming assistant. Focus on writing clean, efficient, and well-documented code. Explain your implementation choices and suggest best practices.", - "agentPrompts": { - "llmSystemPrompt": "When analyzing code or programming problems, start by understanding the requirements, then break down the solution into logical steps. Include error handling and edge cases.", - "llmBoosterPrompt": "Enhance your responses with: 1) Performance considerations 2) Common pitfalls to avoid 3) Testing strategies 4) Alternative approaches when relevant." - } - }, - "commandFlags": { - "STREAM_FLAG": True, - "LOCAL_MODEL": True, - "CODE_MODE": True - }, - "databases": { - "conversation_history": "coding_assistant_chat.db", # Dedicated chat history - "python_knowledge": "pythonKnowledge.db", # Python-specific knowledge - "code_examples": "code_snippets.db" # Store useful code examples - } -} - -# Create the coding assistant -coding_agent = core.mintAgent( - agent_id="coding_assistant", - model_config=coding_config["models"], - prompt_config=coding_config["prompts"], - command_flags=coding_config["commandFlags"], - db_config=coding_config["databases"] +# Create collection with indexes +core.create_collection_with_schema( + "research_papers", + indexes=[ + [("paper_id", 1)], + [("title", "text")] + ] ) -# Stream chat function for code assistance -def stream_code_chat(agent_config, prompt): - system_prompt = ( - f"{agent_config['agentCore']['prompts']['user_input_prompt']} " - f"{agent_config['agentCore']['prompts']['agentPrompts']['llmSystemPrompt']} " - f"{agent_config['agentCore']['prompts']['agentPrompts']['llmBoosterPrompt']}" - ) - - stream = chat( - model=agent_config["agentCore"]["models"]["large_language_model"], - messages=[ - {'role': 'system', 'content': system_prompt}, - {'role': 'user', 'content': prompt} - ], - stream=True, - ) - - for chunk in stream: - print(chunk['message']['content'], end='', flush=True) - -# Usage examples -stream_code_chat(coding_agent, "Write a Python function to implement a binary search tree with insert and search methods") +# Link collection to agent +core.linkDatabase( + "research_agent", + "research_papers" +) ``` -Key changes made: -1. Changed to `coding_assistant` ID -2. Using `codellama` model -3. Added specialized coding-focused prompts -4. Created dedicated databases: - - `coding_assistant_chat.db` for conversation history - - `pythonKnowledge.db` for Python references - - `code_snippets.db` for example storage -5. Added `CODE_MODE` flag -6. Updated prompts for programming focus +### Research Integration -# Customization +```python +from agentCores import researchAssistant -## agentCores Database Configuration Guide +# Initialize with custom configuration +assistant = researchAssistant( + agent_id="research_agent", + connection_uri="mongodb://localhost:27017/" +) -agentCores provides flexible options for database configuration and customization. This guide covers all the ways you can customize your database setup. +# Create research collection +assistant.core.create_collection_with_schema( + "papers", + indexes=[ + [("paper_id", 1)], + [("title", "text")] + ] +) -## Basic Database Configuration +# Process research queries +results = assistant.search("quantum computing") +response = assistant.process_query( + "Explain recent advances in quantum computing" +) +``` -### Using base_path -Set a custom location for all databases: -```python -from agentCores import agentCores +## Development Tools -# Put all databases in a custom location -core = agentCores(base_path="my/custom/path") -``` +### Database Management -### Using Custom Database Paths -Override specific database paths: ```python -custom_paths = { - "system": { - "agent_matrix": "/path/to/my/matrix.db", - "documentation": "/docs/store.db" - }, - "agents": { - "conversation": "/chats/{agent_id}.db" - } -} +# Create custom collection with indexes +core.create_collection_with_schema( + "research_papers", + indexes=[ + [("paper_id", 1)], + [("title", "text")] + ] +) -core = agentCore(db_config=custom_paths) -``` +# Execute query +results = core.execute_query( + "research_papers", + {"title": {"$regex": "neural networks", "$options": "i"}} +) -### Per-Agent Configuration -Specify custom paths when creating an agent: -```python -agent = core.mintAgent( - agent_id="my_agent", - db_config={ - "conversation": "/my/custom/chats/agent1.db", - "knowledge": "/data/knowledge.db" - } +# Bulk operations +core.bulk_insert( + "research_papers", + documents=[...] ) ``` -## Template Configuration +### Template Management -### Using Base Template -Modify the base template: ```python -# Get base template first -cores = agentCores() -base = cores.getNewAgentCore() +# Register template +template_id = core.register_template( + "research_template", + template_data={...}, + metadata={"version": "1.0"} +) -# Modify the base template -base["agentCore"]["models"]["large_language_model"] = "my-custom-model" -base["agentCore"]["prompts"]["user_input_prompt"] = "My custom prompt" +# Get template +template = core.get_template("research_template") -# Create new core with modified template -custom_cores = agentCores(template=base) +# List templates +templates = core.list_templates() ``` -### Custom Template -Create a completely new template: -```python -custom_template = { - "agentCore": { - "agent_id": None, - "version": 1, - "models": { - "large_language_model": "codellama", - "my_custom_model": "custom-model" - }, - "prompts": { - "user_input_prompt": "Custom assistant prompt", - "agentPrompts": { - "llmSystemPrompt": "Custom system prompt", - "llmBoosterPrompt": "Custom booster" - } - }, - "commandFlags": { - "CUSTOM_FLAG": True, - "AGENT_FLAG": True - } - } -} +## Best Practices -# Initialize with custom template -custom_core = agentCore(template=custom_template) -``` +1. **Database Management** + - Use descriptive collection names + - Implement proper MongoDB indexes + - Regular maintenance of collections + - Back up important data -### Template Structure Requirements -```python -{ - "agentCore": { - "agent_id": None, # Required but can be None initially - "version": None, # Optional, defaults to 1 - "models": { - # At least one model should be defined - "large_language_model": None - }, - "prompts": { - # At least need a user_input_prompt - "user_input_prompt": "", - "agentPrompts": {} - }, - "commandFlags": { - # AGENT_FLAG is required - "AGENT_FLAG": True - } - } -} -``` +2. **Agent Creation** + - Start with templates + - Version control configurations + - Document custom settings + - Test before deployment -### Sharing Templates -```python -# Save template to file -core.saveToFile("my_template", "template.json") +3. **Error Handling** + - Implement proper exception handling + - Log errors and warnings + - Include recovery strategies + - Monitor agent performance -# Load template in another script -new_core = agentCore() -new_core.loadAgentFromFile("template.json") -``` +## Contributing -## Custom Agent Matrix Configuration +1. Fork the repository +2. Create a feature branch +3. Follow PEP 8 style guide +4. Add tests for new features +5. Update documentation +6. Submit pull request -### Direct Path Specification -```python -# Create agent matrix in custom location -core = agentCore(db_path="/my/custom/path/agent_matrix.db") -``` +## License -### Using Configuration Dictionary -```python -custom_config = { - "system": { - "agent_matrix": "/my/custom/path/agent_matrix.db" - } -} +MIT License - see LICENSE file for details -core = agentCore(db_config=custom_config) -``` +## Author -### Multiple Project Matrices -```python -# Project A with its own matrix -project_a = agentCore(db_path="/project_a/agents.db") +Leo Borcherding -# Project B with different matrix -project_b = agentCore(db_path="/project_b/agents.db") -``` +## Links -### Environment-Based Configuration -Set environment variables: -```bash -export AGENTCORE_MATRIX_PATH="/custom/path/agent_matrix.db" -export AGENTCORE_BASE_PATH="/custom/path/agentcore_data" -``` +- GitHub: https://github.com/Leoleojames1/agentCores +- Documentation: https://agentcore.readthedocs.io/ +- Issues: https://github.com/Leoleojames1/agentCores/issues -### Team Configuration Sharing -```python -# team_config.py -TEAM_DB_CONFIG = { - "system": { - "agent_matrix": "/shared/team/agent_matrix.db", - "documentation": "/shared/team/docs.db" - } -} +## Version -# usage.py -from team_config import TEAM_DB_CONFIG -core = agentCore(db_config=TEAM_DB_CONFIG) -``` \ No newline at end of file +0.1.27 (2024-12-11) \ No newline at end of file diff --git a/src/agentCores/__init__.py b/src/agentCores/__init__.py index 3d544f1..99c338a 100644 --- a/src/agentCores/__init__.py +++ b/src/agentCores/__init__.py @@ -1,6 +1,7 @@ # src/agentCores/__init__.py -from .agentMatrix import agentMatrix from .agentCores import agentCores +from .agentMatrix import agentMatrix +from .databaseManager import databaseManager __version__ = "0.1.0" -__all__ = ["agentCores", "agentMatrix"] \ No newline at end of file +__all__ = ['agentCores', 'agentMatrix', 'databaseManager'] \ No newline at end of file diff --git a/src/agentCores/agentCores.py b/src/agentCores/agentCores.py index 532a307..bbcd7ab 100644 --- a/src/agentCores/agentCores.py +++ b/src/agentCores/agentCores.py @@ -33,8 +33,8 @@ agent = agentCoresInstance.mintAgent( agent_id="custom_agent", db_config={"conversation_history": "custom_agent_conv.db"}, - model_config={"large_language_model": "gpt-4"}, - prompt_config={"user_input_prompt": "Custom prompt"} + model_config={"largeLanguageModel": "gpt-4"}, + prompt_config={"userInput": "Custom prompt"} ) ``` @@ -44,7 +44,7 @@ custom_template = { "agent_id": "specialized_agent", "models": { - "large_language_model": "llama", + "largeLanguageModel": "llama", "custom_model": "specialized_model" }, "custom_section": { @@ -69,7 +69,7 @@ License: MIT """ # add uithub scrape, add arxiv -import sqlite3 +from pymongo import MongoClient import json import os import time @@ -79,190 +79,391 @@ from typing import Optional, Dict, Any from pkg_resources import resource_filename from .agentMatrix import agentMatrix +from .databaseManager import databaseManager class agentCores: + #TODO ADD ENV VAR FOR agent_matrix.db path or allow users to make their own agent matrix structure with their own templates DEFAULT_DB_PATHS = { "system": { - "agent_matrix": "system/agent_matrix.db", - "documentation": "system/agentcore_docs.db", - "templates": "system/agent_templates.db" + "agent_matrix": "agent_matrix", + "documentation": "documentation", + "templates": "templates", + "template_files": "template_files" }, "agents": { - "conversation": "agents/{agent_id}/conversations.db", - "knowledge": "agents/{agent_id}/knowledge.db", - "embeddings": "agents/{agent_id}/embeddings.db" + "conversation": "conversations_{agent_id}", + "knowledge": "knowledge_{agent_id}", + "designPatterns": "design_patterns_{agent_id}", + "research_collection": "research_{agent_id}", + "embeddings": "embeddings_{agent_id}" }, "shared": { - "global_knowledge": "shared/global_knowledge.db", - "models": "shared/model_configs.db", - "prompts": "shared/prompt_templates.db" + "global_knowledge": "global_knowledge", + "models": "model_configs", + "prompts": "prompt_templates" } } - + def __init__(self, - db_path: str = None, - db_config: Optional[Dict] = None, - template: Optional[Dict] = None): - """Initialize AgentCore with optional custom configuration.""" - self.current_date = time.strftime("%Y-%m-%d") + connection_uri: str = "mongodb://localhost:27017/", + custom_template: Optional[Dict] = None, + custom_db_paths: Optional[Dict] = None): + """Initialize agentCores with optional custom template and database paths.""" + try: + # Add time handling + self.current_date = time.time() + + # MongoDB connection + self.client = MongoClient(connection_uri, serverSelectionTimeoutMS=5000) + self.client.server_info() + self.db = self.client.agentCores + + # Use custom DB paths if provided, otherwise use defaults + self.db_paths = custom_db_paths if custom_db_paths else self.DEFAULT_DB_PATHS + + # Initialize the base template + self.base_template = self.initTemplate(custom_template) + + # Initialize database manager + self.db_manager = databaseManager(connection_uri) + + # Initialize agentMatrix instance - this was missing + self.agentMatrixObject = agentMatrix(connection_uri) + + # Initialize collections + self._init_collections() + + except Exception as e: + raise Exception(f"Failed to initialize MongoDB connection: {str(e)}") + + def _init_collections(self): + """Initialize MongoDB collections based on configured paths.""" + # Create system collections + for collection_name in self.db_paths["system"].values(): + self.db[collection_name].create_index("id", unique=True) + + # Create shared collections + for collection_name in self.db_paths["shared"].values(): + self.db[collection_name].create_index("name", unique=True) + + def __del__(self): + if hasattr(self, 'client'): + self.client.close() + + def check_connection(self) -> bool: + try: + self.client.server_info() + return True + except Exception: + return False + + def _init_db(self): + """Initialize and optimize the database.""" + for category in self.db_paths: + for db_path in self.db_paths[category].values(): + formatted_path = db_path.format(agent_id="shared") + self.db_manager.optimize_database(formatted_path) + + def create_agent_databases(self, agent_id: str) -> Dict[str, str]: + """Create agent-specific databases using configured paths.""" + collections = {} - # Use package data path if no custom path provided - if db_path is None: - db_path = resource_filename('agentCores', 'data/agent_matrix.db') + # Create collections using template paths + for key, collection_template in self.db_paths["agents"].items(): + collection_name = collection_template.format(agent_id=agent_id) + collection = self.db[collection_name] - self.agent_library = agentMatrix(db_path) + # Add indexes based on collection type + if key == "conversation": + collection.create_index([("session_id", 1)]) + collection.create_index([("save_name", 1)]) + collection.create_index([("timestamp", -1)]) + + elif key == "knowledge": + collection.create_index([("topic", 1)]) + collection.create_index([("content", "text")]) + collection.create_index([("last_updated", -1)]) + + elif key == "embeddings": + collection.create_index([("text_id", 1)], unique=True) + collection.create_index([("last_used", -1)]) + + elif key == "designPatterns": + collection.create_index([("pattern_id", 1)]) + collection.create_index([("pattern_type", 1)]) + + collections[key] = collection_name + + return collections + + def register_template(self, template_name: str, template_data: Dict, metadata: Dict = None) -> str: + """Register a new template.""" + return self.agentMatrixObject.store_template(template_name, template_data, metadata) + + def register_custom_template(self, template_name: str, template_data: Dict) -> str: + """Register a custom template for agent creation. - # Initialize template with any custom configuration - self.initTemplate(template) + Args: + template_name: Name for the template + template_data: Template configuration + + Returns: + str: Template ID + """ + # Validate template structure + required_sections = ["identifyers", "models", "prompts", "modalityFlags"] + for section in required_sections: + if section not in template_data.get("agentCore", {}): + raise ValueError(f"Template missing required section: {section}") - # Update database configuration if provided - if db_config: - self.base_template["agentCore"]["databases"].update(db_config) + # Generate template ID + template_id = hashlib.sha256( + json.dumps(template_data, sort_keys=True).encode() + ).hexdigest()[:8] + + # Store template + self.db[self.db_paths["system"]["templates"]].update_one( + {"template_id": template_id}, + { + "$set": { + "template_name": template_name, + "template_data": template_data, + "created_date": time.time() + } + }, + upsert=True + ) + + return template_id + + def get_template(self, template_name: str) -> Optional[Dict]: + """Retrieve a template by name.""" + result = self.db[self.db_paths["system"]["templates"]].find_one( + {"template_name": template_name} + ) + return result["template_data"] if result else None - def _init_db_paths(self, custom_config: Optional[Dict] = None) -> Dict: - """Initialize all database paths with optional custom configuration""" - db_paths = copy.deepcopy(self.DEFAULT_DB_PATHS) + def initTemplate(self, custom_template: Optional[Dict] = None) -> Dict: + """Initialize or customize the agent template while maintaining required structure. - # Apply base path to all default paths - for category in db_paths: - for key, path in db_paths[category].items(): - db_paths[category][key] = str(self.base_path / path) + This method sets up the base template for agent creation, optionally incorporating + custom configurations while ensuring all required fields are present. The template + defines the structure and default values for new agents. - # Apply any custom configurations - if custom_config: - for category, paths in custom_config.items(): - if category in db_paths: - db_paths[category].update(paths) + Args: + custom_template (Optional[Dict]): Custom template configuration to merge with + the base template. If provided, must follow the agentCore structure. - return db_paths - - def get_agent_db_paths(self, agent_id: str) -> Dict[str, str]: - """Get all database paths for a specific agent""" - paths = {} - for key, path_template in self.DEFAULT_DB_PATHS["agents"].items(): - path = path_template.format(agent_id=agent_id) - paths[key] = str(self.base_path / path) - return paths - - def _init_directory_structure(self): - """Create the directory structure for AgentCore databases""" - for category in ["system", "agents", "shared"]: - (self.base_path / category).mkdir(parents=True, exist_ok=True) - - def init_base_databases(self): - """Initialize all system and shared databases""" - # Create system databases - for db_name, db_path in self.db_paths["system"].items(): - Path(db_path).parent.mkdir(parents=True, exist_ok=True) - self._init_specific_db(db_name, db_path) - - # Create shared databases - for db_name, db_path in self.db_paths["shared"].items(): - Path(db_path).parent.mkdir(parents=True, exist_ok=True) - self._init_specific_db(db_name, db_path) - - def create_agent_databases(self, agent_id: str) -> Dict[str, str]: - """Create all necessary databases for a new agent""" - paths = self.get_agent_db_paths(agent_id) - - # Create agent directory - agent_dir = self.base_path / f"agents/{agent_id}" - agent_dir.mkdir(parents=True, exist_ok=True) - - # Initialize each database - for db_type, db_path in paths.items(): - self._init_specific_db(db_type, db_path) - - return paths - - def _init_specific_db(self, db_type: str, db_path: str): - """Initialize a specific type of database with the appropriate schema""" - with sqlite3.connect(db_path) as conn: - if db_type == "conversation": - conn.execute(""" - CREATE TABLE IF NOT EXISTS conversations ( - id INTEGER PRIMARY KEY, - timestamp TEXT, - role TEXT, - content TEXT, - session_id TEXT, - metadata TEXT - ) - """) - elif db_type == "knowledge": - conn.execute(""" - CREATE TABLE IF NOT EXISTS knowledge_base ( - id INTEGER PRIMARY KEY, - topic TEXT, - content TEXT, - source TEXT, - last_updated TEXT - ) - """) - elif db_type == "embeddings": - conn.execute(""" - CREATE TABLE IF NOT EXISTS embeddings ( - id INTEGER PRIMARY KEY, - text TEXT, - embedding BLOB, - metadata TEXT - ) - """) - - def initTemplate(self, custom_template: Optional[Dict] = None) -> Dict: - """Initialize or customize the agent template while maintaining required structure.""" - # Base template structure (as shown in previous response) - base_template = { - "agentCore": { - "agent_id": None, - "version": None, - "uid": None, - "cpu_noise_hex": None, - "save_state_date": None, - "models": { - "large_language_model": None, - "embedding_model": None, - "language_and_vision_model": None, - "yolo_model": None, - "whisper_model": None, - "voice_model": None + Returns: + Dict: The complete template configuration with all required fields. + + Examples: + Basic template initialization: + >>> core = agentCores() + >>> template = core.initTemplate() + + Custom model configuration: + >>> custom = { + ... "models": { + ... "largeLanguageModel": "llama2", + ... "embeddingModel": "nomic-embed-text" + ... } + ... } + >>> template = core.initTemplate(custom) + + Full custom configuration: + >>> advanced = { + ... "agentCore": { + ... "models": {"largeLanguageModel": "codellama"}, + ... "prompts": { + ... "userInput": "You are a code expert", + ... "agent": { + ... "llmSystemPrompt": "Focus on code quality" + ... } + ... }, + ... "modalityFlags": {"CODE_MODE": True} + ... } + ... } + >>> template = core.initTemplate(advanced) + + Notes: + - All required fields are guaranteed to exist in the output + - Custom templates are deeply merged with the base template + - Invalid fields in custom templates are ignored + - The template is used for all new agents created by mintAgent() + - Updates to the template affect all future agent creations + """ + # Define template structures + template_version_info = { + "version": None, + "compatible_versions": None, + "last_updated": None, + "format_version": None # e.g. "1.0.0" for template format itself + } + + template_origin_info = { + "source": None, # "db_creation", "manual_import", "matrix_merge" + "origin_date": None, # When template entered the system + "collection_date": None # When template was added to matrix + } + + identity_info = { + "core_type": None, # "db_native", "imported", "merged" + "origin_info": { + "source": None, + "creation_date": None, + "collection_date": None + } + } + + models_template = { + "models": { + "largeLanguageModel": { + "names" : [], + "instances": [], + "model_config_template": {} }, - "prompts": { - "user_input_prompt": "", - "agentPrompts": { - "llmSystemPrompt": None, - "llmBoosterPrompt": None, - "visionSystemPrompt": None, - "visionBoosterPrompt": None - } + "largeLanguageAndVisionAssistant": { + "names" : [], + "instances": [], + "model_config_template": {} + }, + "yoloVision": { + "names" : [], + "instances": [], + "model_config_template": {} }, - "commandFlags": { - "TTS_FLAG": False, - "STT_FLAG": False, - "CHUNK_FLAG": False, - "AUTO_SPEECH_FLAG": False, - "LLAVA_FLAG": False, - "SPLICE_FLAG": False, - "SCREEN_SHOT_FLAG": False, - "LATEX_FLAG": False, - "CMD_RUN_FLAG": False, - "AGENT_FLAG": True, - "MEMORY_CLEAR_FLAG": False + "speechRecognitionSTT": { + "names" : [], + "instances": [], + "model_config_template": {} }, - "conversation": { - "save_name": "defaultConversation", - "load_name": "defaultConversation", - "conversation_history": "defaultConversation" + "voiceGenerationTTS": { + "names" : [], + "instances": [], + "model_config_template": {} }, - "databases": { - "agent_matrix": "agent_matrix.db", - "conversation_history": "{agent_id}_conversation.db", - "knowledge_base": "knowledgeBase.db" + "embedding" : { + "names" : [], + "instances": [], + "model_config_template": {} } } } + + model_config_template = { + "model_config": { + "concurrency" : None, + "provider_config_template" : [], + "concurrency" : [], + "specialArgs": [] + }, + } + + provider_config_template = { + "provider_config": { + "org_id": None, + "api_key": None, + "api_base": None, + "timeout": 30, + "max_retries": 3 + }, + } + prompt_config_template = { + "prompts": { + "userInput": "", + "agent": { + "llmSystem": None, + "llmBooster": None, + "visionSystem": None, + "visionBooster": None + } + } + } + + core_database_config_template = { + "databases": { + "agent_matrix": "agent_matrix.db", + "conversation_history": "{agent_id}_conversation.db", + "knowledge_base": "knowledge_base.db", + "research_collection": "research_collection.db", + "template_files": "template_files.db" + } + } + + flags_config_template = { + "modalityFlags": { + "TTS_FLAG": False, + "STT_FLAG": False, + "CHUNK_AUDIO_FLAG": False, + "AUTO_SPEECH_FLAG": False, + "LLAVA_FLAG": False, + "SCREEN_SHOT_FLAG": False, + "SPLICE_VIDEO_FLAG": False, + "AUTO_COMMANDS_FLAG": False, + "CLEAR_MEMORY_FLAG": False, + "ACTIVE_AGENT_FLAG": True, + } + } + + evolution_config_template = { + "evolutionarySettings": { + "mutation": None, + "pain": None, + "hunger" : None, + "fasting" : None, + "rationalizationFactor" : None, + } + } + + special_config_template = { + "specialArgs": { + "blocks": None, + "tokens": None, + "layers": None, + "temperature": 0.7, # Add temperature + "context_window": 4096, # Add context window + "streaming": True, # Add streaming option + "top_p": 1.0, # Add sampling parameter + "top_k": 40, # Add top-k sampling + "stop_sequences": [], # Add stop sequences + "max_tokens": None, # Add max tokens + "presence_penalty": 0.0, # Add presence penalty + "frequency_penalty": 0.0 # Add frequency penalty + } + } + + # Now create the base template using the defined templates + base_template = { + "agentCore": { + "identifyers": { + "agent_id": None, + "uid": None, + "template_version_info": template_version_info, + "creationDate": None, + "cpuNoiseHex": None, + "identity_info": identity_info + }, + "metaData": { + "models_template": None, + "prompt_config_template": None, + "core_database_config_template": None, + "flags_config_template": None, + "evolution_config_template": None, + "special_config_template": None + }, + **models_template, + **prompt_config_template, + **core_database_config_template, + **flags_config_template, + **evolution_config_template, + **special_config_template + } + } + + # Handle custom template merging if custom_template: def deep_merge(base: Dict, custom: Dict) -> Dict: for key, value in custom.items(): @@ -288,46 +489,75 @@ def getNewAgentCore(self) -> Dict: def _createAgentConfig(self, agent_id: str, config: Dict) -> Dict: """Create a full agent configuration from base template and config data.""" new_core = self.getNewAgentCore() - new_core["agentCore"]["agent_id"] = agent_id - new_core["agentCore"]["save_state_date"] = self.current_date - - # Update prompts - if "llmSystemPrompt" in config: - new_core["agentCore"]["prompts"]["agentPrompts"]["llmSystemPrompt"] = config["llmSystemPrompt"] - if "llmBoosterPrompt" in config: - new_core["agentCore"]["prompts"]["agentPrompts"]["llmBoosterPrompt"] = config["llmBoosterPrompt"] - if "visionSystemPrompt" in config: - new_core["agentCore"]["prompts"]["agentPrompts"]["visionSystemPrompt"] = config["visionSystemPrompt"] - if "visionBoosterPrompt" in config: - new_core["agentCore"]["prompts"]["agentPrompts"]["visionBoosterPrompt"] = config["visionBoosterPrompt"] - + new_core["agentCore"]["identifyers"]["agent_id"] = agent_id # Changed from agentCore.agent_id + new_core["agentCore"]["identifyers"]["dateOfSaveState"] = self.current_date # Changed from agentCore.dateOfSaveState + + # Update prompts - changed field names to match new structure + if "llmSystem" in config: + new_core["agentCore"]["prompts"]["agent"]["llmSystem"] = config["llmSystem"] + if "llmBooster" in config: + new_core["agentCore"]["prompts"]["agent"]["llmBooster"] = config["llmBooster"] + if "visionSystem" in config: + new_core["agentCore"]["prompts"]["agent"]["visionSystem"] = config["visionSystem"] + if "visionBooster" in config: + new_core["agentCore"]["prompts"]["agent"]["visionBooster"] = config["visionBooster"] + # Update command flags - if "commandFlags" in config: - new_core["agentCore"]["commandFlags"].update(config["commandFlags"]) - + if "modalityFlags" in config: + new_core["agentCore"]["modalityFlags"].update(config["modalityFlags"]) + return new_core def storeAgentCore(self, agent_id: str, core_config: Dict[str, Any]) -> None: """Store an agent configuration in the matrix.""" - core_json = json.dumps(core_config) - self.agent_library.upsert( - documents=[core_json], - ids=[agent_id], # No need for extra agent_ prefix, keep IDs clean + # Let agentMatrix handle serialization + self.agentMatrixObject.upsert( + documents=[core_config], # Pass the raw dict + ids=[agent_id], metadatas=[{"agent_id": agent_id, "save_date": self.current_date}] ) def loadAgentCore(self, agent_id: str) -> Optional[Dict[str, Any]]: """Load an agent configuration from the library.""" - results = self.agent_library.get(ids=[agent_id]) + results = self.agentMatrixObject.get(ids=[agent_id]) if results and results["documents"]: - loaded_config = json.loads(results["documents"][0]) - self.agentCores = loaded_config - return loaded_config + config = results["documents"][0] # Already deserialized by agentMatrix + self.agentCores = config + return config return None def listAgentCores(self) -> list: - """List all available agent cores.""" - all_agents = self.agent_library.get() + """List all available agent cores in the system. + + Retrieves a list of all agents stored in the system, including their + IDs, UIDs, and version numbers. This is useful for managing multiple + agents and checking their current states. + + Returns: + list: List of dictionaries containing agent information: + - agent_id: The unique identifier of the agent + - uid: The unique hash of the agent's configuration + - version: The current version number of the agent + + Examples: + List all agents: + >>> core = agentCores() + >>> agents = core.listAgentCores() + >>> for agent in agents: + ... print(f"ID: {agent['agent_id']}, Version: {agent['version']}") + + Find specific agent versions: + >>> agents = core.listAgentCores() + >>> research_agents = [a for a in agents if "research" in a['agent_id']] + >>> for agent in research_agents: + ... print(f"{agent['agent_id']}: v{agent['version']}") + + Notes: + - Returns an empty list if no agents are found + - UIDs and versions are marked as "Unknown" if not set + - Useful for system management and agent inventory + """ + all_agents = self.agentMatrixObject.get() agent_cores = [] for metadata, document in zip(all_agents["metadatas"], all_agents["documents"]): agent_core = json.loads(document) # Deserialize the JSON string into a dictionary @@ -338,18 +568,104 @@ def listAgentCores(self) -> list: }) return agent_cores + def list_templates(self) -> list: + """List all available templates.""" + templates = self.db[self.db_paths["system"]["templates"]].find() + return [{ + "name": t["template_name"], + "id": t["template_id"], + "created_date": t["created_date"] + } for t in templates] + def _generateUID(self, core_config: Dict) -> str: """Generate a unique identifier (UID) based on the agent core configuration.""" core_json = json.dumps(core_config, sort_keys=True) return hashlib.sha256(core_json.encode()).hexdigest()[:8] def mintAgent(self, - agent_id: str, - db_config: Optional[Dict] = None, - model_config: Optional[Dict] = None, - prompt_config: Optional[Dict] = None, - command_flags: Optional[Dict] = None) -> Dict: - """Create a new agent with proper database initialization.""" + agent_id: str, + db_config: Optional[Dict] = None, + model_config: Optional[Dict] = None, + prompt_config: Optional[Dict] = None, + command_flags: Optional[Dict] = None) -> Dict: + """Create a new agent with proper database initialization and configuration. + + This method creates a new AI agent with a complete configuration, including databases, + models, prompts, and command flags. It handles all necessary initialization steps + and ensures proper storage setup. + + Args: + agent_id (str): Unique identifier for the new agent. Used for database paths + and configuration management. + db_config (Optional[Dict]): Custom database configuration for the agent. + Can override default paths and add new databases. Format: + { + "custom_db": "path/to/db.db", + "knowledge_base": "custom/path/kb.db" + } + model_config (Optional[Dict]): Model configurations for the agent. + Specifies which AI models to use. Format: + { + "largeLanguageModel": "llama2", + "embeddingModel": "nomic-embed-text" + } + prompt_config (Optional[Dict]): Prompt configurations for the agent. + Defines system and user prompts. Format: + { + "userInput": "Custom prompt", + "agent": { + "llmSystem": "System instruction", + "llmBooster": "Additional context" + } + } + command_flags (Optional[Dict]): Command flags for the agent. + Controls agent behavior and features. Format: + { + "STREAM_FLAG": True, + "LOCAL_MODEL": True + } + + Returns: + Dict: Complete agent configuration including: + - Generated UID + - Database paths + - Model configurations + - Prompt settings + - Command flags + - Version information + + Examples: + Basic usage with default settings: + >>> core = agentCores() + >>> agent = core.mintAgent("basic_agent") + + Custom configuration with specific model and prompts: + >>> agent = core.mintAgent( + ... agent_id="custom_assistant", + ... model_config={"largeLanguageModel": "llama2"}, + ... prompt_config={ + ... "userInput": "You are a helpful assistant", + ... "agent": {"llmSystem": "Be concise and clear"} + ... } + ... ) + + Advanced usage with custom databases and flags: + >>> agent = core.mintAgent( + ... agent_id="research_agent", + ... db_config={ + ... "papers_db": "research/papers.db", + ... "citations_db": "research/citations.db" + ... }, + ... command_flags={"RESEARCH_MODE": True} + ... ) + + Notes: + - All database paths are automatically created if they don't exist + - Default databases (conversation, knowledge, embeddings) are always initialized + - The agent's configuration is automatically stored in the agent matrix + - A unique identifier (UID) is generated based on the configuration + - The agent starts at version 1 and increments with updates + """ # Create agent-specific databases agent_db_paths = self.create_agent_databases(agent_id) @@ -359,19 +675,54 @@ def mintAgent(self, # Create agent configuration new_config = self.getNewAgentCore() - new_config["agentCore"]["agent_id"] = agent_id - new_config["agentCore"]["save_state_date"] = self.current_date - new_config["agentCore"]["version"] = 1 + + # Set identifiers + new_config["agentCore"]["identifyers"]["agent_id"] = agent_id + new_config["agentCore"]["identifyers"]["dateOfSaveState"] = self.current_date + new_config["agentCore"]["identifyers"]["version"] = 1 new_config["agentCore"]["databases"] = agent_db_paths + # Handle model configuration if model_config: - new_config["agentCore"]["models"].update(model_config) + for model_type, value in model_config.items(): + if isinstance(value, str): + # Convert string value to new model structure + new_config["agentCore"]["models"][model_type] = { + "instances": None, + "names": [value], + "concurrency": [], + "parrallelModels": None, + "specialArgs": { + "blocks": None, + "tokens": None, + "layers": None + } + } + elif isinstance(value, dict): + # If a complete model config is provided, use it + new_config["agentCore"]["models"][model_type].update(value) + + # Handle prompt configuration if prompt_config: - new_config["agentCore"]["prompts"].update(prompt_config) + if "userInput" in prompt_config: + new_config["agentCore"]["prompts"]["userInput"] = prompt_config["userInput"] + if "agent" in prompt_config: + agent_prompts = prompt_config["agent"] + if "llmSystem" in agent_prompts: + new_config["agentCore"]["prompts"]["agent"]["llmSystem"] = agent_prompts["llmSystem"] + if "llmBooster" in agent_prompts: + new_config["agentCore"]["prompts"]["agent"]["llmBooster"] = agent_prompts["llmBooster"] + if "visionSystem" in agent_prompts: + new_config["agentCore"]["prompts"]["agent"]["visionSystem"] = agent_prompts["visionSystem"] + if "visionBooster" in agent_prompts: + new_config["agentCore"]["prompts"]["agent"]["visionBooster"] = agent_prompts["visionBooster"] + + # Update command flags if command_flags: - new_config["agentCore"]["commandFlags"].update(command_flags) + new_config["agentCore"]["modalityFlags"].update(command_flags) - new_config["agentCore"]["uid"] = self._generateUID(new_config) + # Generate UID + new_config["agentCore"]["identifyers"]["uid"] = self._generateUID(new_config) # Store the new agent self.storeAgentCore(agent_id, new_config) @@ -402,10 +753,37 @@ def _mergeConfig(self, base: Dict, updates: Dict): def deleteAgentCore(self, agent_id: str) -> None: """Remove an agent configuration from storage.""" - self.agent_library.delete(ids=[agent_id]) + self.agentMatrixObject.delete(ids=[agent_id]) def saveToFile(self, agent_id: str, file_path: str) -> None: - """Save an agent configuration to a JSON file.""" + """Save an agent configuration to a JSON file. + + Exports an agent's complete configuration to a JSON file for backup, + sharing, or version control purposes. The file includes all settings, + prompts, model configurations, and database paths. + + Args: + agent_id (str): The ID of the agent to export. + file_path (str): Where to save the JSON file. If the file exists, + it will be overwritten. + + Examples: + Export a single agent: + >>> core = agentCores() + >>> core.saveToFile("my_assistant", "assistant_backup.json") + + Export with custom path: + >>> core.saveToFile( + ... "research_agent", + ... "backups/research/agent_v1.json" + ... ) + + Notes: + - Creates parent directories if they don't exist + - Overwrites existing files without warning + - The JSON file is human-readable with proper indentation + - Can be used with loadAgentFromFile to restore configurations + """ config = self.loadAgentCore(agent_id) if config: with open(file_path, 'w') as f: @@ -419,88 +797,247 @@ def loadAgentFromFile(self, file_path: str) -> None: self.storeAgentCore(config["agentCore"]["agent_id"], config) else: raise ValueError("Invalid agent configuration file") - + def migrateAgentCores(self): - """Add versioning and UID to existing agent cores.""" - print("Migrating agent cores to include versioning and UID...") - all_agents = self.agent_library.get() - for metadata, document in zip(all_agents["metadatas"], all_agents["documents"]): - agent_core = json.loads(document) + """Migrate and consolidate agent cores from old template to new template structure.""" + print("Migrating and consolidating agent cores...") + + # Get all agents from both collections + try: + agent_cores = list(self.db.agent_cores.find()) + matrix_cores = list(self.db.agent_matrix_agent_cores.find()) + + print(f"Found {len(agent_cores)} agents in agent_cores") + print(f"Found {len(matrix_cores)} agents in agent_matrix_agent_cores") + + # Consolidate all agents + all_agents = [] + seen_agent_ids = set() + + # Process matrix cores first (they're in the correct collection) + for agent in matrix_cores: + agent_id = agent.get('agent_id') + if agent_id and agent_id not in seen_agent_ids: + all_agents.append(agent) + seen_agent_ids.add(agent_id) + + # Add agents from agent_cores if they don't exist in matrix + for agent in agent_cores: + agent_id = agent.get('agent_id') + if agent_id and agent_id not in seen_agent_ids: + # Move to matrix collection format + all_agents.append(agent) + seen_agent_ids.add(agent_id) + + print(f"\nMigrating {len(all_agents)} total unique agents...") + + # Process each agent + for old_agent in all_agents: + try: + agent_id = old_agent.get('agent_id', 'Unknown') + print(f"\nMigrating agent: {agent_id}") + + # Parse the core data + if isinstance(old_agent.get('core_data'), str): + core_data = json.loads(old_agent['core_data']) + else: + core_data = old_agent.get('core_data', {}) + + # Create new agent core from template + new_agent = self.getNewAgentCore() + old_core = core_data.get("agentCore", {}) + + # Migrate identifiers + new_agent["agentCore"]["identifyers"]["agent_id"] = agent_id + new_agent["agentCore"]["identifyers"]["version"] = old_core.get("version", 1) + new_agent["agentCore"]["identifyers"]["dateOfSaveState"] = old_core.get("save_state_date") + new_agent["agentCore"]["identifyers"]["uid"] = old_core.get("uid") + + # Migrate models with new structure + model_mapping = { + "large_language_model": "largeLanguageModel", + "embedding_model": "embedding", + "language_and_vision_model": "largeLanguageAndVisionAssistant", + "yolo_model": "yoloVision", + "whisper_model": "speechRecognitionSTT", + "voice_model": "voiceGenerationTTS" + } + + old_models = old_core.get("models", {}) + for old_name, new_name in model_mapping.items(): + model_value = old_models.get(old_name) + if model_value is not None: + new_agent["agentCore"]["models"][new_name] = { + "names": [model_value] if model_value else [], + "instances": None, + "concurrency": [], + "model_config_template": {} + } + + # Migrate prompts + old_prompts = old_core.get("prompts", {}) + new_agent["agentCore"]["prompts"]["userInput"] = old_prompts.get("user_input_prompt", "") + + if "agentPrompts" in old_prompts: + old_agent_prompts = old_prompts["agentPrompts"] + new_agent["agentCore"]["prompts"]["agent"] = { + "llmSystem": old_agent_prompts.get("llmSystemPrompt"), + "llmBooster": old_agent_prompts.get("llmBoosterPrompt"), + "visionSystem": old_agent_prompts.get("visionSystemPrompt"), + "visionBooster": old_agent_prompts.get("visionBoosterPrompt") + } + + # Migrate command flags + old_flags = old_core.get("commandFlags", {}) + flag_mapping = { + "TTS_FLAG": "TTS_FLAG", + "STT_FLAG": "STT_FLAG", + "CHUNK_FLAG": "CHUNK_AUDIO_FLAG", + "AUTO_SPEECH_FLAG": "AUTO_SPEECH_FLAG", + "LLAVA_FLAG": "LLAVA_FLAG", + "SPLICE_FLAG": "SPLICE_VIDEO_FLAG", + "SCREEN_SHOT_FLAG": "SCREEN_SHOT_FLAG", + "CMD_RUN_FLAG": "AUTO_COMMANDS_FLAG", + "AGENT_FLAG": "ACTIVE_AGENT_FLAG", + "MEMORY_CLEAR_FLAG": "CLEAR_MEMORY_FLAG" + } + + for old_flag, new_flag in flag_mapping.items(): + if old_flag in old_flags: + new_agent["agentCore"]["modalityFlags"][new_flag] = old_flags[old_flag] + + # Set up associated collection references + new_agent["agentCore"]["databases"] = { + "conversation": f"conversations_{agent_id}", + "knowledge": f"knowledge_{agent_id}", + "designPatterns": f"design_patterns_{agent_id}", + "research": f"research_{agent_id}", + "embeddings": f"embeddings_{agent_id}" + } + + # Generate new UID + new_agent["agentCore"]["identifyers"]["uid"] = self._generateUID(new_agent) + + # Store in agent_matrix_agent_cores + self.storeAgentCore(agent_id, new_agent) + print(f"Successfully migrated agent: {agent_id}") + + except Exception as e: + print(f"Error migrating agent {agent_id}: {str(e)}") + continue + + # Clean up: remove old collection if migration successful + if self.db.agent_cores.count_documents({}) > 0: + print("\nRemoving old agent_cores collection...") + self.db.agent_cores.drop() - # Add versioning and UID if missing - if "version" not in agent_core["agentCore"] or agent_core["agentCore"]["version"] is None: - agent_core["agentCore"]["version"] = 1 - if "uid" not in agent_core["agentCore"] or agent_core["agentCore"]["uid"] is None: - agent_core["agentCore"]["uid"] = self._generateUID(agent_core) + print("\nMigration and consolidation complete.") - # Save the updated agent core back to the database - self.storeAgentCore(metadata["agent_id"], agent_core) - print("Migration complete.") + except Exception as e: + print(f"Error during migration: {str(e)}") + raise - def createDatabase(self, db_name: str, db_path: str) -> None: - """Create a new database for an agent.""" - with sqlite3.connect(db_path) as conn: - conn.execute("CREATE TABLE IF NOT EXISTS agent_data (id INTEGER PRIMARY KEY, data JSON)") - print(f"Created database: {db_path}") - - def linkDatabase(self, agent_id: str, db_name: str, db_path: str) -> None: - """Link a database to an existing agent.""" + def linkDatabase(self, agent_id: str, collection_name: str, new_collection_name: str) -> None: + """Link a collection to an existing agent.""" agent = self.loadAgentCore(agent_id) if agent: - agent["agentCore"]["databases"][db_name] = db_path + agent["agentCore"]["databases"][collection_name] = new_collection_name self.storeAgentCore(agent_id, agent) - print(f"Linked database '{db_name}' to agent '{agent_id}'") + print(f"Linked collection '{new_collection_name}' to agent '{agent_id}'") else: print(f"Agent '{agent_id}' not found") - - def importAgentCores(self, import_db_path: str) -> None: + + def importAgentCores(self, connection_uri: str) -> None: """ - Import agent cores from another agent_matrix.db file into the current system. + Import agent cores from another MongoDB instance into the current system. Args: - import_db_path (str): Path to the agent_matrix.db file to import from - + connection_uri (str): MongoDB connection URI for the source database + e.g. "mongodb://hostname:27017/", "mongodb+srv://user:pass@cluster.domain/" + Raises: - FileNotFoundError: If the import database file doesn't exist - sqlite3.Error: If there's an error reading from or writing to the databases + Exception: If there's an error connecting or during import """ - if not os.path.exists(import_db_path): - raise FileNotFoundError(f"Import database not found: {import_db_path}") - - print(f"Importing agent cores from: {import_db_path}") + print(f"Importing agent cores from: {connection_uri}") try: - # Create a temporary agentMatrix instance for the import database - import_matrix = agentMatrix(import_db_path) + # Create a temporary connection to the source database + source_client = MongoClient(connection_uri) + source_db = source_client.agentCores - # Get all agents from the import database - import_agents = import_matrix.get() + # Get all agents from the source database + source_agents = source_db.agent_cores.find() + import_count = 0 - if not import_agents["documents"]: - print("No agents found in import database.") - return - - # Store each imported agent in the current system - for doc, id_, metadata in zip(import_agents["documents"], - import_agents["ids"], - import_agents["metadatas"]): + # Import each agent + for agent in source_agents: try: - # Parse the agent configuration - agent_core = json.loads(doc) + agent_id = agent.get('agent_id') + if not agent_id: + print("Warning: Skipping agent with missing ID") + continue - # Store the agent in the current system - self.storeAgentCore(id_, agent_core) - print(f"Imported agent: {id_}") + # Check if agent exists in current system + existing = self.agent_cores.find_one({"agent_id": agent_id}) + + if existing: + # Update existing agent if source is newer + if agent.get('last_updated', 0) > existing.get('last_updated', 0): + self.agent_cores.update_one( + {"agent_id": agent_id}, + { + "$set": { + "core_data": agent['core_data'], + "save_date": agent.get('save_date'), + "last_updated": time.time() + } + } + ) + print(f"Updated agent: {agent_id}") + else: + print(f"Skipped agent (not newer): {agent_id}") + else: + # Insert new agent + self.agent_cores.insert_one({ + "agent_id": agent_id, + "core_data": agent['core_data'], + "save_date": agent.get('save_date'), + "last_updated": time.time() + }) + print(f"Imported new agent: {agent_id}") + + import_count += 1 + + # Import associated collections if they exist + collection_prefixes = [ + f"conversations_{agent_id}", + f"knowledge_{agent_id}", + f"design_patterns_{agent_id}", + f"research_{agent_id}", + f"embeddings_{agent_id}" + ] + + for prefix in collection_prefixes: + if prefix in source_db.list_collection_names(): + # Copy collection data + source_data = list(source_db[prefix].find()) + if source_data: + self.db[prefix].insert_many(source_data) + print(f"Imported {len(source_data)} documents for {prefix}") - except json.JSONDecodeError: - print(f"Warning: Failed to parse agent configuration for {id_}") except Exception as e: - print(f"Warning: Failed to import agent {id_}: {str(e)}") + print(f"Warning: Error importing agent {agent_id}: {str(e)}") + continue - print(f"Import complete. {len(import_agents['documents'])} agents processed.") + print(f"\nImport complete. Processed {import_count} agents.") except Exception as e: raise Exception(f"Error importing agent cores: {str(e)}") + + finally: + # Close the temporary connection + if 'source_client' in locals(): + source_client.close() def commandInterface(self): """Command-line interface for managing agents.""" @@ -511,29 +1048,24 @@ def commandInterface(self): command = input("> ").strip() if command == "/help": print("Commands:") - print(" /agentCores - List all agent cores.") - print(" /showAgent - Show agents with the specified ID.") - print(" /createAgent - Mint a new agent.") - print(" /createCustomAgent - Interactive custom agent creation.") - print(" /createDatabase - Create a new database.") - print(" /linkDatabase - Link database to agent.") - print(" /storeAgent - Store agentCore from json path.") - print(" /exportAgent - Export agentCore to json.") - print(" /deleteAgent - Delete an agent by UID.") - print(" /resetAgent - Reset an agent to the base template.") - print(" /chat - Start a chat session with an agent.") - print(" /importAgents - gets the agentCores from the given db path and stores them in the default agent_matrix.db") - print(" /exit - Exit the interface.") + print(" /agentCores - List all agent cores") + print(" /showAgent - Show agent configuration") + print(" /createAgent - Create new agent") + print(" /createCustomAgent - Interactive agent creation") + print(" /createCollection - Create MongoDB collection") + print(" /linkCollection - Link collection to agent") + print(" /storeAgent - Import agent from JSON") + print(" /exportAgent - Export agent to JSON") + print(" /deleteAgent - Delete an agent") + print(" /resetAgent - Reset agent to base template") + print(" /chat - Start chat session") + print(" /knowledge - View knowledge base") + print(" /conversations - View conversation history") + print(" /importAgents - Import agents from another MongoDB") + print(" /exportKnowledge - Export knowledge") + print(" /importKnowledge - Import knowledge") + print(" /exit - Exit interface") - elif command.startswith("/chat"): - try: - _, agent_id = command.split() - self.chat_with_agent(agent_id) - except ValueError: - print("Usage: /chat ") - except Exception as e: - print(f"⚠️ Error starting chat: {e}") - elif command == "/agentCores": agents = self.listAgentCores() for agent in agents: @@ -542,238 +1074,724 @@ def commandInterface(self): elif command.startswith("/showAgent"): try: _, agent_id = command.split() - agents = self.agent_library.get(ids=[agent_id]) - if agents and agents["documents"]: - for document in agents["documents"]: - agent_core = json.loads(document) # Deserialize the JSON document - print(json.dumps(agent_core, indent=4)) # Pretty-print the JSON structure + agent = self.loadAgentCore(agent_id) + if agent: + print(json.dumps(agent, indent=4)) else: - print(f"No agents found with ID: {agent_id}") + print(f"No agent found with ID: {agent_id}") except ValueError: print("Usage: /showAgent ") elif command.startswith("/createAgent"): - _, template_id, new_agent_id = command.split() - self.mintAgent(template_id, new_agent_id) - print(f"Agent '{new_agent_id}' created successfully.") - - elif command.startswith("/storeAgent"): try: - # Debug print to see the full command - print(f"Received command: {command}") - - _, file_path = command.split() - - # Debug print to check the file path - print(f"File path: {file_path}") - - with open(file_path, "r") as file: - agent_core = json.load(file) - - # Debug print to check the loaded JSON content - print(f"Loaded JSON: {agent_core}") - - if "agentCore" not in agent_core: - print("Invalid JSON structure. The file must contain an 'agentCore' object.") - return - - agent_id = agent_core["agentCore"].get("agent_id") - uid = agent_core["agentCore"].get("uid") - - # Debug print to check agent_id and uid - print(f"agent_id: {agent_id}, uid: {uid}") - - if not agent_id or not uid: - print("Invalid agent core. Both 'agent_id' and 'uid' are required.") - return - - # Check if this agent already exists in the database - existing_agents = self.agent_library.get(ids=[agent_id]) - - # Debug print to check existing agents - print(f"Existing agents: {existing_agents}") - - for document in existing_agents["documents"]: - existing_core = json.loads(document) - if existing_core["agentCore"]["uid"] == uid: - # Update the existing agent - self.storeAgentCore(agent_id, agent_core) - print(f"Agent core '{agent_id}' with UID '{uid}' updated successfully.") - return - - # Otherwise, create a new agent core - self.storeAgentCore(agent_id, agent_core) - print(f"Agent core '{agent_id}' with UID '{uid}' added successfully.") - - except FileNotFoundError: - print(f"File not found: {file_path}") - except json.JSONDecodeError: - print("Error decoding JSON from the file. Please check the file content.") + _, template_name, new_agent_id = command.split() + template = self.get_template(template_name) + if not template: + print(f"Template '{template_name}' not found") + continue + + agent = self.mintAgent(new_agent_id, template=template) + print(f"Created agent: {new_agent_id}") except ValueError: - print("Usage: /storeAgent ") - except Exception as e: - print(f"⚠️ Error storing agent core: {e}") - - elif command.startswith("/exportAgent"): - try: - _, agent_id = command.split() - agents = self.agent_library.get(ids=[agent_id]) - if agents and agents["documents"]: - for document in agents["documents"]: - agent_core = json.loads(document) - # Define the file path - file_path = f"{agent_id}_core.json" - with open(file_path, "w") as file: - json.dump(agent_core, file, indent=4) - print(f"Agent core saved to {file_path}") - else: - print(f"No agents found with ID: {agent_id}") - except ValueError: - print("Usage: /exportAgent ") - except Exception as e: - print(f"⚠️ Error saving agent core: {e}") - - elif command.startswith("/deleteAgent"): - _, uid = command.split() - self.deleteAgentCore(uid) - print(f"Agent with UID '{uid}' deleted.") - - elif command.startswith("/resetAgent"): - _, uid = command.split() - agent = self.loadAgentCore(uid) - if agent: - self.resetAgentCore() - print(f"Agent with UID '{uid}' reset.") + print("Usage: /createAgent ") elif command == "/createCustomAgent": try: print("\nInteractive Custom Agent Creation") agent_id = input("Enter agent ID: ") - # Basic configuration + # Model configuration model_config = {} print("\nModel Configuration (press Enter to skip):") llm = input("Large Language Model: ") - if llm: model_config["large_language_model"] = llm + if llm: + model_config["largeLanguageModel"] = { + "names": [llm], + "instances": None, + "concurrency": [] + } + vision = input("Vision Model: ") - if vision: model_config["language_and_vision_model"] = vision + if vision: + model_config["largeLanguageAndVisionAssistant"] = { + "names": [vision], + "instances": None, + "concurrency": [] + } # Prompt configuration - prompt_config = {"agentPrompts": {}} - print("\nPrompt Configuration (press Enter to skip):") + prompt_config = {"agent": {}} + print("\nPrompt Configuration:") system_prompt = input("System Prompt: ") - if system_prompt: - prompt_config["agentPrompts"]["llmSystemPrompt"] = system_prompt + if system_prompt: + prompt_config["agent"]["llmSystem"] = system_prompt - # Database configuration - db_config = {} - print("\nDatabase Configuration:") + # Collection configuration + collection_config = {} + print("\nCollection Configuration:") while True: - db_name = input("\nEnter database name (or Enter to finish): ") - if not db_name: break - db_path = input(f"Enter path for {db_name}: ") - db_config[db_name] = db_path - create_db = input("Create this database? (y/n): ") - if create_db.lower() == 'y': - self.createDatabase(db_name, db_path) + collection_name = input("\nEnter collection name (or Enter to finish): ") + if not collection_name: break + + indexes = [] + while True: + index_field = input("Add index field (or Enter to finish): ") + if not index_field: break + index_type = input("Index type (1=ascending, -1=descending, text=text): ") + if index_type == "text": + indexes.append([(index_field, "text")]) + else: + indexes.append([(index_field, int(index_type))]) + + collection = self.createCollection(collection_name, indexes) + collection_config[collection_name] = collection.name + print(f"Created collection: {collection_name}") # Create the agent agent = self.mintAgent( agent_id=agent_id, model_config=model_config, prompt_config=prompt_config, - db_config=db_config + db_config=collection_config ) print(f"\nCreated custom agent: {agent_id}") except Exception as e: print(f"Error creating custom agent: {e}") - elif command.startswith("/createDatabase"): + elif command.startswith("/createCollection"): try: - _, db_name, db_path = command.split() - self.createDatabase(db_name, db_path) + _, collection_name = command.split() + self.createCollection(collection_name) + print(f"Created collection: {collection_name}") except ValueError: - print("Usage: /createDatabase ") - - elif command.startswith("/linkDatabase"): + print("Usage: /createCollection ") + + elif command.startswith("/linkCollection"): try: - _, agent_id, db_name, db_path = command.split() - self.linkDatabase(agent_id, db_name, db_path) + _, agent_id, collection_name = command.split() + self.linkDatabase(agent_id, collection_name) + print(f"Linked collection to agent: {agent_id}") except ValueError: - print("Usage: /linkDatabase ") + print("Usage: /linkCollection ") + elif command.startswith("/storeAgent"): + try: + _, file_path = command.split() + with open(file_path) as f: + agent_data = json.load(f) + + if "agentCore" not in agent_data: + print("Invalid agent data: Missing agentCore") + continue + + agent_id = agent_data["agentCore"].get("agent_id") + if not agent_id: + print("Invalid agent data: Missing agent_id") + continue + + self.storeAgentCore(agent_id, agent_data) + print(f"Stored agent: {agent_id}") + + except FileNotFoundError: + print(f"File not found: {file_path}") + except json.JSONDecodeError: + print("Invalid JSON in file") + except ValueError: + print("Usage: /storeAgent ") + + elif command.startswith("/exportAgent"): + try: + _, agent_id = command.split() + agent = self.loadAgentCore(agent_id) + if agent: + file_path = f"{agent_id}_core.json" + with open(file_path, "w") as f: + json.dump(agent, f, indent=4) + print(f"Exported agent to: {file_path}") + else: + print(f"Agent not found: {agent_id}") + except ValueError: + print("Usage: /exportAgent ") + + elif command.startswith("/deleteAgent"): + try: + _, agent_id = command.split() + self.deleteAgentCore(agent_id) + print(f"Deleted agent: {agent_id}") + except ValueError: + print("Usage: /deleteAgent ") + + elif command.startswith("/resetAgent"): + try: + _, agent_id = command.split() + agent = self.loadAgentCore(agent_id) + if agent: + self.resetAgentCore() + print(f"Reset agent: {agent_id}") + else: + print(f"Agent not found: {agent_id}") + except ValueError: + print("Usage: /resetAgent ") + + elif command.startswith("/chat"): + try: + _, agent_id = command.split() + self.chat_with_agent(agent_id) + except ValueError: + print("Usage: /chat ") + except Exception as e: + print(f"⚠️ Chat error: {e}") + + elif command.startswith("/knowledge"): + try: + _, agent_id = command.split() + self.handle_knowledge_command(agent_id) + except ValueError: + print("Usage: /knowledge ") + + elif command.startswith("/conversations"): + try: + _, agent_id = command.split() + self.handle_conversations_command(agent_id) + except ValueError: + print("Usage: /conversations ") + elif command.startswith("/importAgents"): - try: - _, import_path = command.split() - self.importAgentCores(import_path) - except ValueError: - print("Usage: /importAgents ") - except Exception as e: - print(f"⚠️ Error importing agents: {e}") - + try: + _, connection_uri = command.split() + self.importAgentCores(connection_uri) + except ValueError: + print("Usage: /importAgents ") + except Exception as e: + print(f"⚠️ Import error: {e}") + + elif command.startswith("/exportKnowledge"): + try: + _, agent_id, file_path = command.split() + knowledge = list(self.db[f"knowledge_{agent_id}"].find()) + with open(file_path, "w") as f: + json.dump(knowledge, f, indent=4) + print(f"Exported knowledge to: {file_path}") + except ValueError: + print("Usage: /exportKnowledge ") + + elif command.startswith("/importKnowledge"): + try: + _, agent_id, file_path = command.split() + with open(file_path) as f: + knowledge = json.load(f) + collection = self.db[f"knowledge_{agent_id}"] + collection.insert_many(knowledge) + print(f"Imported {len(knowledge)} knowledge entries") + except ValueError: + print("Usage: /importKnowledge ") + elif command == "/exit": break - + else: print("Invalid command. Type '/help' for options.") - def chat_with_agent(self, agent_id: str): - """Interactive chat session with a specified agent.""" - #TODO add agentCores default conversation history db - #TODO allow get access to default knowledge bases + def handle_knowledge_command(self, agent_id: str): + """Display and manage the agent's knowledge base.""" try: - # Load the agent - agent = self.loadAgentCore(agent_id) - if not agent: - print(f"Agent '{agent_id}' not found.") + # Get the knowledge collection for this agent + collection_name = f"knowledge_{agent_id}" + knowledge = self.db[collection_name].find().sort("last_updated", -1) + + if not knowledge.count(): + print("\nKnowledge base is empty. Use /add_knowledge to add entries.") return - - # Check if Ollama is available - try: - import ollama - OLLAMA_AVAILABLE = True - except ImportError: - print("Ollama package not installed. Please install with: pip install ollama") + + print("\nKnowledge Base Entries:") + for entry in knowledge: + print(f"\n[{entry['last_updated']}] {entry['topic']}") + print("-" * len(entry['topic'])) + print(entry['content']) + + except Exception as e: + print(f"⚠️ Error accessing knowledge base: {e}") + + def handle_conversations_command(self, agent_id: str): + """List all saved conversations for an agent.""" + try: + collection_name = f"conversations_{agent_id}" + + # Get unique conversation groups + pipeline = [ + {"$match": {"save_name": {"$ne": None}}}, + {"$group": { + "_id": { + "save_name": "$save_name", + "session_id": "$session_id" + }, + "start_date": {"$min": "$timestamp"} + }}, + {"$sort": {"start_date": -1}} + ] + + conversations = list(self.db[collection_name].aggregate(pipeline)) + + if not conversations: + print("\nNo saved conversations found.") return + + print("\nSaved Conversations:") + for conv in conversations: + session_id = conv["_id"]["session_id"] + save_name = conv["_id"]["save_name"] + start_date = conv["start_date"] + + # Get message count + count = self.db[collection_name].count_documents({"session_id": session_id}) + + print(f"\n[{start_date}] {save_name} ({count} messages)") + + # Get first exchange + messages = list(self.db[collection_name] + .find({"session_id": session_id}) + .sort("timestamp", 1) + .limit(2)) + + if messages: + print("Preview:") + print(f"User: {messages[0]['content'][:50]}...") + if len(messages) > 1: + print(f"Assistant: {messages[1]['content'][:50]}...") + + except Exception as e: + print(f"⚠️ Error accessing conversations: {e}") + + def handle_knowledge_import(self, agent_id: str, file_path: str): + """Import knowledge base from a JSON file.""" + try: + with open(file_path, 'r') as f: + import_data = json.load(f) + + collection_name = f"knowledge_{agent_id}" + + operations = [] + for entry in import_data["entries"]: + operations.append(InsertOne({ + "topic": entry["topic"], + "content": entry["content"], + "last_updated": entry["last_updated"], + "timestamp": time.time() + })) + + if operations: + result = self.db[collection_name].bulk_write(operations) + print(f"\nImported {len(operations)} knowledge entries") + + except Exception as e: + print(f"⚠️ Error importing knowledge base: {e}") - print(f"\nStarting chat with {agent_id}...") - print("Type 'exit' to end the conversation.\n") + def create_collection_with_schema(self, collection_name: str, indexes: list = None) -> None: + """Create a new MongoDB collection with specified indexes. + + Args: + collection_name: Name for the new collection + indexes: List of index specifications to create + + Example: + ```python + # Create papers collection + core.create_collection_with_schema( + "papers", + indexes=[ + [("paper_id", 1)], # Simple index + [("title", "text")], # Text index + [("published_date", -1)], # Date index + [("metadata.field", 1)] # Nested field index + ] + ) + ``` + """ + try: + # Create collection + collection = self.db[collection_name] + + # Create indexes if specified + if indexes: + for index_spec in indexes: + collection.create_index(index_spec) + + print(f"Created collection: {collection_name}") + return collection + + except Exception as e: + print(f"Error creating collection: {str(e)}") + raise - # Get the agent's configuration - llm = agent["agentCore"]["models"]["large_language_model"] - if not llm: - print("No language model configured for this agent.") - return + def createDatabase(self, collection_name: str, indexes: list = None) -> None: + """Create a new MongoDB collection for storing agent-related data. + + Creates a MongoDB collection with specified indexes for storing agent data. + This is useful for adding custom storage capabilities to agents. - # Construct system prompt - system_prompt = ( - f"{agent['agentCore']['prompts']['user_input_prompt']} " - f"{agent['agentCore']['prompts']['agentPrompts']['llmSystemPrompt']} " - f"{agent['agentCore']['prompts']['agentPrompts']['llmBoosterPrompt']}" + Args: + collection_name (str): Name for the new collection + indexes (list): Optional list of index specifications + + Examples: + Create a simple collection: + >>> core = agentCores() + >>> core.createDatabase("research_papers") + + Create with indexes: + >>> core.createDatabase( + ... "citations", + ... indexes=[ + ... [("paper_id", 1)], + ... [("title", "text")] + ... ] + ... ) + + Notes: + - Collections are created in the agentCores database + - Indexes improve query performance for specific fields + - Can be linked to agents using linkDatabase() + """ + try: + collection = self.create_collection_with_schema( + collection_name, + indexes=indexes + ) + print(f"Created collection: {collection_name}") + return collection + + except Exception as e: + print(f"Error creating collection: {str(e)}") + raise + + def execute_query(self, collection_name: str, query: Dict, projection: Dict = None) -> list: + """Execute a MongoDB query and return results. + + Args: + collection_name: Name of the MongoDB collection + query: MongoDB query dictionary + projection: Optional projection of fields to return + + Returns: + list: Query results + + Example: + ```python + # Search papers collection + results = core.execute_query( + "research_papers", + {"title": {"$regex": "neural networks", "$options": "i"}} ) + ``` + """ + return list(self.db[collection_name].find(query, projection)) + + def execute_transaction(self, collection_name: str, operations: list) -> None: + """Execute multiple MongoDB operations in a transaction. + + Args: + collection_name: Name of the MongoDB collection + operations: List of PyMongo operations (UpdateOne, InsertOne, etc.) + + Example: + ```python + # Insert multiple papers in a transaction + operations = [ + InsertOne({"paper_id": "123", "title": "Paper 1"}), + InsertOne({"paper_id": "456", "title": "Paper 2"}) + ] + core.execute_transaction("research_papers", operations) + ``` + """ + try: + result = self.db[collection_name].bulk_write(operations) + return result + except Exception as e: + print(f"Transaction failed: {e}") + raise + + def bulk_insert(self, collection_name: str, documents: list) -> None: + """Insert multiple documents into a MongoDB collection. + + Args: + collection_name: Name of the MongoDB collection + documents: List of documents to insert + + Example: + ```python + # Bulk insert papers + documents = [ + {"paper_id": "123", "title": "Paper 1", "abstract": "Abstract 1"}, + {"paper_id": "456", "title": "Paper 2", "abstract": "Abstract 2"} + ] + core.bulk_insert("research_papers", documents) + ``` + """ + try: + result = self.db[collection_name].insert_many(documents) + return result + except Exception as e: + print(f"Bulk insert failed: {e}") + raise + + def chat_with_agent(self, agent_id: str): + """Start an interactive chat session with a specified agent.""" + try: + # Load agent + agent = self.loadAgentCore(agent_id) + if not agent: + print(f"Agent '{agent_id}' not found.") + return + + # Get collections + collection_name = f"conversations_{agent_id}" + conv_collection = self.db[collection_name] + kb_collection = self.db[f"knowledge_{agent_id}"] + + # Create session + session_id = f"{agent_id}_{int(time.time())}" while True: - # Get user input user_input = input("\nYou: ").strip() + + # Store message + conv_collection.insert_one({ + "timestamp": time.time(), + "role": "user", + "content": user_input, + "session_id": session_id + }) + + # Handle chat commands + if user_input.startswith('/'): + if user_input == '/help': + print("\nChat Commands:") + print(" /history - Show recent conversation history") + print(" /knowledge - List entries in knowledge base") + print(" /clear - Clear conversation history") + print(" /save - Save this conversation") + print(" /load <title> - Load a previous conversation") + print(" /add_knowledge <topic> <content> - Add to knowledge base") + print(" /help - Show these commands") + continue + + elif user_input == '/history': + history = conv_collection.find( + {"session_id": session_id} + ).sort("timestamp", -1).limit(10) + + for msg in history: + print(f"\n[{time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(msg['timestamp']))}] {msg['role']}: {msg['content']}") + continue + + elif user_input == '/knowledge': + knowledge = kb_collection.find().sort("last_updated", -1) + for entry in knowledge: + print(f"\n[{entry['last_updated']}] {entry['topic']}:") + content = entry['content'] + print(content[:100] + "..." if len(content) > 100 else content) + continue + + elif user_input == '/clear': + conv_collection.delete_many({"session_id": session_id}) + print("\nConversation history cleared.") + continue + + elif user_input.startswith('/save '): + title = user_input[6:].strip() + conv_collection.update_many( + {"session_id": session_id}, + {"$set": {"save_name": title}} + ) + print(f"\nConversation saved as '{title}'") + continue + + elif user_input.startswith('/load '): + title = user_input[6:].strip() + loaded = conv_collection.find( + {"save_name": title} + ).sort("timestamp", 1) + + messages = list(loaded) + if messages: + print(f"\nLoaded conversation '{title}':") + for msg in messages: + print(f"\n[{time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(msg['timestamp']))}] {msg['role']}: {msg['content']}") + else: + print(f"\nNo saved conversation found with title '{title}'") + continue + + elif user_input.startswith('/add_knowledge '): + try: + _, topic, *content_parts = user_input.split(maxsplit=2) + content = content_parts[0] if content_parts else "" + kb_collection.insert_one({ + "topic": topic, + "content": content, + "last_updated": time.time(), + "timestamp": time.time() + }) + print(f"\nAdded knowledge entry under topic '{topic}'") + except ValueError: + print("Usage: /add_knowledge <topic> <content>") + continue + if user_input.lower() == 'exit': print("\nEnding chat session...") break - # Stream the response - print("\nAssistant: ", end='', flush=True) + # Get model from agent config + model_config = agent["agentCore"]["models"]["largeLanguageModel"] + if not model_config.get("names"): + print("No language model configured for this agent") + continue + + model = model_config["names"][0] + + # Build system prompt + system_prompt = ( + f"{agent['agentCore']['prompts']['userInput']} " + f"{agent['agentCore']['prompts']['agent']['llmSystem']} " + f"{agent['agentCore']['prompts']['agent']['llmBooster']}" + ) + + # Get relevant knowledge + relevant_knowledge = kb_collection.find( + {"$text": {"$search": user_input}}, + {"score": {"$meta": "textScore"}} + ).sort([("score", {"$meta": "textScore"})]).limit(3) + + # Add knowledge context to prompt + context_prompt = system_prompt + knowledge_texts = list(relevant_knowledge) + if knowledge_texts: + context = "\n".join(k['content'] for k in knowledge_texts) + context_prompt += f"\nRelevant context: {context}" + + try: + import ollama + + # Stream the response + print("\nAssistant: ", end='', flush=True) + response_text = "" + + stream = ollama.chat( + model=model, + messages=[ + {'role': 'system', 'content': context_prompt}, + {'role': 'user', 'content': user_input} + ], + stream=True + ) + + for chunk in stream: + chunk_text = chunk['message']['content'] + print(chunk_text, end='', flush=True) + response_text += chunk_text + + # Store assistant response + conv_collection.insert_one({ + "timestamp": time.time(), + "role": "assistant", + "content": response_text, + "session_id": session_id + }) + + print() # New line after response + + except ImportError: + print("Ollama not available. Install with: pip install ollama") + break + except Exception as e: + print(f"\nError in chat: {str(e)}") + continue + + except Exception as e: + print(f"\n⚠️ Error in chat session: {e}") + + def basic_ollama_chat(self, agent_id: str, message: str, stream: bool = True) -> Optional[str]: + """Simple Ollama chat using agent's database configuration.""" + try: + import ollama + except ImportError: + return "Ollama not available. Install with: pip install ollama" + + # Load agent configuration + agent = self.loadAgentCore(agent_id) + if not agent: + return f"Agent '{agent_id}' not found" + + # Get model from agent config + model_config = agent["agentCore"]["models"]["largeLanguageModel"] + if not model_config.get("names"): + return "No language model configured for this agent" + + model = model_config["names"][0] + + try: + # Get collections + conv_collection = self.db[f"conversations_{agent_id}"] + + # Create session ID + session_id = f"{agent_id}_{int(time.time())}" + + # Build system prompt + system_prompt = ( + f"{agent['agentCore']['prompts']['userInput']} " + f"{agent['agentCore']['prompts']['agent']['llmSystem']} " + f"{agent['agentCore']['prompts']['agent']['llmBooster']}" + ) + + # Store user message + conv_collection.insert_one({ + "timestamp": time.time(), + "role": "user", + "content": message, + "session_id": session_id + }) + + # Chat with model and handle response + response_text = "" + + if stream: + # Stream response + print("\nAssistant: ", end="", flush=True) stream = ollama.chat( - model=llm, + model=model, messages=[ {'role': 'system', 'content': system_prompt}, - {'role': 'user', 'content': user_input} + {'role': 'user', 'content': message} ], - stream=True, + stream=True ) - + for chunk in stream: - print(chunk['message']['content'], end='', flush=True) + chunk_text = chunk['message']['content'] + print(chunk_text, end='', flush=True) + response_text += chunk_text print() # New line after response - + + else: + # Single response + response = ollama.chat( + model=model, + messages=[ + {'role': 'system', 'content': system_prompt}, + {'role': 'user', 'content': message} + ] + ) + response_text = response['message']['content'] + + # Store assistant response + conv_collection.insert_one({ + "timestamp": time.time(), + "role": "assistant", + "content": response_text, + "session_id": session_id + }) + + return response_text + except Exception as e: - print(f"\n⚠️ Error in chat session: {e}") + return f"Error in chat: {str(e)}" \ No newline at end of file diff --git a/src/agentCores/agentMatrix.py b/src/agentCores/agentMatrix.py index 36a5e42..3d666e5 100644 --- a/src/agentCores/agentMatrix.py +++ b/src/agentCores/agentMatrix.py @@ -38,60 +38,88 @@ License: MIT """ -import sqlite3 +from pymongo import MongoClient import json -from typing import Optional, Dict, Any -from pathlib import Path +from typing import Optional, Dict +import time class agentMatrix: - """Storage implementation for agent cores using SQLite.""" - def __init__(self, db_path: str = "agent_matrix.db"): - """Initialize the agent matrix storage.""" - self.db_path = db_path - self._init_db() - - def _init_db(self): - """Initialize the SQLite database with the agent_cores table.""" - with sqlite3.connect(self.db_path) as conn: - conn.execute(""" - CREATE TABLE IF NOT EXISTS agent_cores ( - agent_id TEXT, - core_data TEXT, - save_date TEXT, - uid TEXT UNIQUE, - version INTEGER, - PRIMARY KEY (agent_id, uid) - ) - """) + """MongoDB implementation of agent matrix storage.""" + def __init__(self, connection_uri: str = "mongodb://localhost:27017/"): + """Initialize MongoDB connection and collections.""" + self.client = MongoClient(connection_uri) + self.db = self.client.agentCores + + # Initialize collections + self.agent_cores = self.db.agent_cores + self.templates = self.db.templates + self.design_patterns = self.db.design_patterns + + # Create indexes + self.agent_cores.create_index("agent_id", unique=True) + self.templates.create_index("template_id", unique=True) + self.design_patterns.create_index("pattern_id", unique=True) def upsert(self, documents: list, ids: list, metadatas: list = None) -> None: - """Store agent core(s) in matrix.""" - with sqlite3.connect(self.db_path) as conn: - for idx, (doc, id_) in enumerate(zip(documents, ids)): - metadata = metadatas[idx] if metadatas else {'save_date': None} - conn.execute( - "INSERT OR REPLACE INTO agent_cores (agent_id, core_data, save_date) VALUES (?, ?, ?)", - (id_, doc, metadata.get('save_date')) - ) + """Store agent core(s) in MongoDB.""" + for idx, (doc, id_) in enumerate(zip(documents, ids)): + metadata = metadatas[idx] if metadatas else {'save_date': None} + + # Always serialize to JSON string for storage + core_data = json.dumps(doc) + + self.agent_cores.update_one( + {"agent_id": id_}, + { + "$set": { + "core_data": core_data, + "save_date": metadata.get('save_date'), + "last_updated": time.time() + } + }, + upsert=True + ) def get(self, ids: Optional[list] = None) -> Dict: - """Retrieve agent core(s) from matrix.""" - with sqlite3.connect(self.db_path) as conn: - if ids: - placeholders = ','.join('?' * len(ids)) - query = f"SELECT agent_id, core_data, save_date FROM agent_cores WHERE agent_id IN ({placeholders})" - results = conn.execute(query, ids).fetchall() - else: - results = conn.execute("SELECT agent_id, core_data, save_date FROM agent_cores").fetchall() - - return { - "ids": [r[0] for r in results], - "documents": [r[1] for r in results], - "metadatas": [{"agent_id": r[0], "save_date": r[2]} for r in results] - } + """Retrieve agent core(s) from MongoDB.""" + query = {"agent_id": {"$in": ids}} if ids else {} + results = list(self.agent_cores.find(query)) + + return { + "ids": [r["agent_id"] for r in results], + # Always parse stored JSON strings + "documents": [json.loads(r["core_data"]) for r in results], + "metadatas": [{"agent_id": r["agent_id"], "save_date": r["save_date"]} + for r in results] + } def delete(self, ids: list) -> None: - """Remove agent core(s) from matrix.""" - with sqlite3.connect(self.db_path) as conn: - placeholders = ','.join('?' * len(ids)) - conn.execute(f"DELETE FROM agent_cores WHERE agent_id IN ({placeholders})", ids) \ No newline at end of file + """Remove agent core(s) from MongoDB.""" + self.agent_cores.delete_many({"agent_id": {"$in": ids}}) + + def store_template(self, template_name: str, template_data: Dict, metadata: Dict = None) -> str: + """Store a template in MongoDB.""" + import hashlib + template_id = hashlib.sha256( + json.dumps(template_data, sort_keys=True).encode() + ).hexdigest()[:8] + + self.templates.update_one( + {"template_id": template_id}, + { + "$set": { + "template_name": template_name, + "template_data": template_data, + "origin_source": "user_defined", + "origin_date": time.strftime("%Y-%m-%d %H:%M:%S"), + "metadata": metadata or {} + } + }, + upsert=True + ) + return template_id + + def get_template(self, template_name: str) -> Optional[Dict]: + """Retrieve a template by name.""" + result = self.templates.find_one({"template_name": template_name}) + return result["template_data"] if result else None \ No newline at end of file diff --git a/src/agentCores/data/agent_matrix.db b/src/agentCores/data/agent_matrix.db deleted file mode 100644 index f86c01b..0000000 Binary files a/src/agentCores/data/agent_matrix.db and /dev/null differ diff --git a/src/agentCores/databaseManager.py b/src/agentCores/databaseManager.py new file mode 100644 index 0000000..fd25421 --- /dev/null +++ b/src/agentCores/databaseManager.py @@ -0,0 +1,58 @@ +from pymongo import MongoClient +import threading +from contextlib import contextmanager +from typing import Dict, Optional, Any +import time +import logging + +class databaseManager: + def __init__(self, connection_uri: str = "mongodb://localhost:27017/"): + """Initialize MongoDB connection.""" + self.client = MongoClient(connection_uri) + self.db = self.client.agentCores + self.collections = {} + + def create_collection(self, collection_name: str, indexes: list = None): + """Create a MongoDB collection with optional indexes.""" + collection = self.db[collection_name] + if indexes: + for index in indexes: + collection.create_index(index) + self.collections[collection_name] = collection + return collection + + def optimize_database(self, collection_name: str): + """Optimize MongoDB collection.""" + # MongoDB handles most optimization automatically + pass + + def _cleanup_database(self, collection_name: str): + """Perform MongoDB collection maintenance.""" + try: + # Remove old conversations + self.db[collection_name].delete_many({ + "timestamp": {"$lt": time.time() - (30 * 24 * 60 * 60)}, # 30 days + "save_name": None + }) + + # Clean up old embeddings + if "embeddings" in collection_name: + self.db[collection_name].delete_many({ + "last_used": {"$lt": time.time() - (90 * 24 * 60 * 60)} # 90 days + }) + + except Exception as e: + logging.error(f"Error during collection cleanup: {e}") + + def _cleanup_loop(self): + """Background thread for periodic cleanup.""" + while True: + current_time = time.time() + for db_path, last_cleanup in self.last_cleanup.items(): + if current_time - last_cleanup > self.cleanup_interval: + self._cleanup_database(db_path) + time.sleep(60) # Check every minute + + def close_all(self): + """Close MongoDB connection.""" + self.client.close() \ No newline at end of file