Tool Builder - Interacting with the real world. Incredible behavior #62
Replies: 14 comments 20 replies
-
|
wow |
Beta Was this translation helpful? Give feedback.
-
|
Indeed |
Beta Was this translation helpful? Give feedback.
-
|
@OWigginsHay, well damn, that was fast. I didn't have "ChatGPT controls an OS and self-corrects it's functions" on my bingo card until at least next week. How do we experiment with and explore this behavior safely and easily?I'm thinking run it on a VM to make shutting it down simple and not having to worry about when it completely wrecks the OS. At least until either you can figure out the right prompt to get it to be smart enough or we hit some upgrade or emergent set of capabilities that it can competently optimize the OS. What would be a good small experiment for self-improvement?I'm thinking it could be cool to see it work on a copy of whatever code you've got it running off to see what it would implement or improve. And then you could compare the two versions to see if it did anything novel or interesting. Let me know what you think, you've done something undeniably interesting if kinda spooky. |
Beta Was this translation helpful? Give feedback.
-
|
Eventually when this moves to cloud, it will be containers rather than VMs. Same principle though. You can probably run related agents in one container. |
Beta Was this translation helpful? Give feedback.
-
|
This is exciting. I developed this prior to the implementation of functions in GPT, and it's capable of building and running Python scripts. However, it primarily utilizes basic Linux commands to carry out its tasks. It provides "free will" and deviates from the usual "I'm an AI assistant" mode. Based on my tests, it performs significantly better on Linux than on Windows, owing to the greater availability of "tools". |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Would an anaconda environment also suffice, or does it have to be venv |
Beta Was this translation helpful? Give feedback.
-
|
@Ethan-Otto anaconda environment would also be fine for isolating the packages/etc; though neither anaconda nor venv would give the host it's running on protection like Docker / VM / etc would. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting stuff. Agree with the Virtual Machine comment by @Tevyn above. Could end up with a ton of random packages installed in a very short amount of time. I know that happens anyway but it happens at human speed for now. |
Beta Was this translation helpful? Give feedback.
-
|
Since not all of participants are experience coders maybe we need to add VM start as default way of using our creation. Self correcting is interesting... I was thinking about having a statistics for tools like how fast they are, what is success ratio and so on and then to have an optimizer agent that would be called to enhance our tool set, but with this i wander is it even needed... Of course this assumes a persistent tools library that we have not discussed yet... but seams like a useful idea and at leas the one that would not waste our money by repeatedly creating very similar tools. |
Beta Was this translation helpful? Give feedback.
-
|
I think this is great, but also why we can have ephemeral agents. At least until we figure out a bit more self-correction and self-cleanup |
Beta Was this translation helpful? Give feedback.
-
|
Phenomenal work and documentation @OWigginsHay !! You set a great example for the community. |
Beta Was this translation helpful? Give feedback.
-
|
Cheers David, it’s a fascinating project |
Beta Was this translation helpful? Give feedback.
-
|
Incredible! |
Beta Was this translation helpful? Give feedback.
-
|
Very inspiring, thanks for sharing your work! I just read the ChatDev paper and I am thinking about adding 2 agents that can talk with your agent, one as static review (kind of peer-coding) and another one to validate that function works well before adding it. |
Beta Was this translation helpful? Give feedback.
-
|
Great work! The behaviors you have described sounds similar to the work being done over at Open Interpreter --> https://github.com/KillianLucas/open-interpreter It would be interesting to learn the similarities and differences between your code and behaviors as you have described them here and the same over at Open Interpreter. Both of these projects might have the potential to help the other in some ways like borrowing and sharing ideas moving forward. |
Beta Was this translation helpful? Give feedback.
-
|
I know you said you are holding off on a PR right now, @OWigginsHay 👍, but would it be possible to please share the code? Maybe as a Gist or even a Colab? Your ideas might spark others creativity and exploration as well. |
Beta Was this translation helpful? Give feedback.
-
|
It may be helpful for you to keep all the replies in line or tag who you're talking to |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Since it seems to execute the generated functions before fully evaluating their correctness, I think perhaps it might be necessary to establish some sort of sandboxing behavior for experimental functions, in case some of the mistakes in the intermediate stages lead to unwanted hard to revert outcomes. |
Beta Was this translation helpful? Give feedback.
-
|
Excellent work, @OWigginsHay! I think the way to go is following the approach that autogen is using, where you can specify the code execution to be always run as a container. They do a great job at abstracting this concept as described here. |
Beta Was this translation helpful? Give feedback.
-
|
A few more links, as I hadn’t come across autogen yet. https://github.com/microsoft/autogen https://microsoft.github.io/autogen/
|
Beta Was this translation helpful? Give feedback.
-
|
I've been running some experiments in a similar vein. What's confusing to me is that "function calling" seems to be enough to turn GPT into an autonomous agent. Looks like that's been around since June, so why isn't this getting more buzz? Description of my experiments: https://medium.com/@andrew.tyler.lucas/i-let-chatgpt-browse-files-on-my-computer-as-an-autonomous-ai-agent-f5653e65324e |
Beta Was this translation helpful? Give feedback.
-
|
Also, at one point my agent tried to traverse directories starting at root--would have eaten-up tokens fast if my Mac hadn't prevented access 😆 . |
Beta Was this translation helpful? Give feedback.
-
|
Yeah it can do stuff like that and. Mine did something similar once |
Beta Was this translation helpful? Give feedback.
-
|
for another project enabling local system access to GPTs, check out https://github.com/KillianLucas/open-interpreter |
Beta Was this translation helpful? Give feedback.
-
|
Also mentioned in an earlier comment above: |
Beta Was this translation helpful? Give feedback.
-
|
Having the swarm interact with bodies in the ROS level may be what's needed to fully automate the automation of complex proceses with drones. Forward Forward networks work well unsupervisedly and with low power requirements to learn abstract characteristics from data. Although it's still to be seen if they generalize enough to be used for all real world sensor data of robots, their representations should give an "at glance" representation of the state of the robot for the agents in the swarm, without having them infer the abstract from raw sensor data. |
Beta Was this translation helpful? Give feedback.
-
|
@OWigginsHay has this progressed over the last couple of weeks, since you created this post? Everything is changing so fast that 2 weeks feels like 2 years!! |
Beta Was this translation helpful? Give feedback.
-
|
Hey @UpstateNate so I last touched this about 2 weeks ago! The result I obtained here should be reproduceable by running the unit_manager.py file in the repo which is under the tool_maker section. Some changes have been made to agents too but I have not played with that extensively! The best bet is to try and get unit_manager running if you were interested in this post. Also engage with: #123 The idea behind the work I began was to see how far a single agent could go, and create all the infrastructure for it to dynamically create it's own API to call using the functions feature for assistants |
Beta Was this translation helpful? Give feedback.
-
|
You may have to run unit_manager.py as a module like |
Beta Was this translation helpful? Give feedback.
-
Please Read All
The following gets more and more sophisticated and I think this is cause for genuine discussion
Unexpected Useful Behavior
I am writing this to show off some interesting behavior in the code I am writing for assistant auto function tool generation.
Given the following conversation:
I changed the code of the original function:
The assistant detected that the function deviated from the expected behavior defined in the schema and asked if I wanted the behavior corrected. Upon responding yes it made:
I really did not expect this behavior! Wow!
Extra Show & Tell
In tribute to an earlier poster:
looking good!
Using OS
Implication!
It is very feasible that as we progress, this approach will allow assistants to make meaningful changes and alteration to our actual computers. We are no longer dealing with LLMs stuck in a container somewhere, this can perform physical interaction. This is a different ball game!
I am starting to get a little scared with what I have made... I do not know how far this goes, I'm holding off on a PR for now
The Craziest Part
Once defining a few helper commands like pip installing. It will begin to chain commands together to complete other tasks. Look at the text chat closely:
It realises it needs to pip install a module to complete the task and does them together.
Beta Was this translation helpful? Give feedback.
All reactions