Skip to content

Latest commit

 

History

History
13 lines (11 loc) · 4.78 KB

sits.md

File metadata and controls

13 lines (11 loc) · 4.78 KB
Raw

what would an superintelligent IT professional system SITS, meaning a programmer+sysadmin+mathematician+computer scientist etc., be doing?

  • the system would almost only be doing backend stuff. Compile time and efficiency of the tools and the system it is working on could be a huge bottleneck.
  • it would not have intrinsic interests, but there would be many instrumental goals given sufficiently complex goals
  • in the case where the system is very fast it would not in the same way be restricted to existing tooling, its tooling could be very customized
  • it would most likely still find use in abstractions, but the abstractions would not need to match abstractions that work well for humans
  • different kinds of abstractions exist, programming languages are sometimes abstractions, there might be programming languages that work better than others for language modelling
  • the syntax of a programming language defines a set of valid programs, but somehow humans with their intentions and goals have produced an empirical distribution over the programs that we have written. The syntax and working of the programming language itself also "leaks" some information about the distribution of interests maybe, but maybe not a lot, given that turing completeness is most likely human-independent.
  • as of 2024, we have models that have certain capabilities in the realm of coding and one major theme is that agents need the right interface to the computational substrate. Computational Substrate Interface. Models are unhobbled the most when they get access to tool calling that they are able to use effectively. For example it is obviously better to get it some well structured information about the current code, the result of running it and maybe some type annotations or whatever, than just letting it loose on the assembly instructions etc. This means that currently, agents are most unhobbled when they work with certain abstractions, and it is worth the investment to put in the effort of coding up these abstractions customly. The optimal CSI in some years might not be the same as it is today. Different tasks and goals could require different CSIs. There is no use in having a system make its own operating system or programming language, when we have invested so much effort into making them good. Developing a SITS might somehow involve making the CSI path, where the SITS gradually gets better tools for doing its work. These tools might be very different from the way human IT professionals work. For example, there are some who may develop a lot of unit tests, and they might do it before programming anything, but maybe they are not making a thousand. Maybe there are reasons why an agent would want to make a very extensive amount of unit tests. Maybe the SITS will use a debugger in a very weird way to generate some kind of annotatation to simulate some kind of reasoning.
  • The current general language models encode some kind of information about the common sense understanding that humans have. By common sense I do not necessarily mean common sense reasoning, more like knowledge of what humans like and want. The exact nature of what is encoded in the language models might be debated but it seems very likely that this knowledge is somehow "bootstrappable". It should be possible to make a setup where an agent (itself driven by an LM) uses a general LM to come up with tasks that have relatively fixed objectives, not necessarily formalizable but sufficiently determinable by a general LM. (here there is maybe some risk of reward hacking but I think it might be possible to set the whole orchestraction up in the right way so that the "flow of information" avoids the reward hacking, but maybe not).
  • One of the tasks that seem most approachable at the current moment is transpilation and learning to translate between languages. I would have to look up if there has been any studies but I think that GPT4 is much better at python than rust.
  • Certain problems are well-defined, others are ill-defined. While it can in some cases be useful to view the relationship between SITS and a computer to AlphaZero and the chess board, the difference is that chess has a definite goal, while computation does not have such a thing. Humans care about certain things and want to model and program certain things. A formal language induces an infinite space of propositions and there are certain things where by exploring this space, it might be possible to learn the landscape, just like AlphaZero has somehow learned some structure in the landscape, there must be clear theoretical limits to this. Deep Learning has not changed the fact that certain problems are NP-hard. Of course there is a lot about approximation algorithms. It might just be worth thinking more about which "cheap lunches" it is we think we might be able to get and which we will not be able to get.