This is really cool! Have you looked into integrating with open interpreter?

[grexzen]

Open interpreter allows you to chat in the console and for the llm to run code, so you could give it as system prompt your commands and mini guide, and then it could simply chat w users and run your commands to do stuff.

Pretty awesome set up.

[danenania]

Open interpreter is great! This is a cool idea. One thing I want to do before long is add an 'auto' mode that disables prompts (prompts in the CLI sense, not LLM sense) for easier scripting and interop in these kind of use cases.