Replies: 8 comments 21 replies
-
|
Interesting idea. In my experience, chat AIs like ChatGPT aren't very good at Prolog in general. If an AI was trained to do this specifically, then maybe they would actually be good enough, but training an AI like this is really hard. I also think that Scryer's errors would need to improve drastically with more information before this could be remotely viable, but maybe an sufficiently advanced AI could infer most of it. My intuition is that a "dumb" (in the sense of no trained statistical AI) system like GUPU would be more impactful for less work. |
Beta Was this translation helpful? Give feedback.
-
|
There are some recent efforts of using LLMs to generate Prolog code: https://arxiv.org/abs/2405.17893 I didn't read the paper only abstract. |
Beta Was this translation helpful? Give feedback.
-
|
As I see it (and my reading of the article can be seen as quick & superficial, very selective), an interesting article for a person interested in Prolog and LLM AI, especially in the light of the recent "OpenAI o1", an AI showing its "chain of thought" to a user. As I see it, the article in not about helping people to learn Prolog, but If the LL model can generate Prolog code, "can learn to extract predicates in the mathematical questions [...] which more precisely reflects the nature of the Prolog language.", then, maybe, it can have some pedagogical value: to see how a Prolog program can be constructed from the "mathematical questions". On the other hand, as I see it, yes, Prolog can be seen as a simple programming language, likewise, for example, Scheme can be seen as a simple programming language. There is a simple syntax, a simple logical or simple mathematical core to program with. Yet, there is more to it: “Scheme is a fairly simple language to learn, since it is based on a handful of syntactic forms and semantic concepts and since the interactive nature of most implementations encourages experimentation. Scheme is a challenging language to understand fully, however; developing the ability to use its full potential requires careful study and practice.” R. Kent Dybvig: The Scheme Programming Language, 4th ed., 2009 https://www.scheme.com/tspl4/intro.html#./intro:h0 Or, programming is still programming: “However, the "discipline of programming" method of design carries over directly to Prolog. [...] When should you have an 'if' (a case analysis) in your program? When should you have a loop (a recursive predicate) in your program? What should the loop invariant (the meaning of the predicate) be? What controls the loop (what is the recursion based on)? These are the kinds of question which this method tackles, and has answers (well, heuristics) for. I strongly recommend both of these books [Dijkstra: "A Discipline of Programming", Gries: "The Science of Programming"]; if you master this approach you will be a much more effective programmer, and the techniques do apply to Prolog because for the most part they are concerned with logical formulas.” Richard A. O'Keefe: The Craft of Prolog, 1990 So, as a beginner for a beginner (to programming), I would say, better then LLM AI is this web-book: How to Design Programs |
Beta Was this translation helpful? Give feedback.
-
|
Here is a good compilation of recent work on Logic and LLMs: https://arxiv.org/pdf/2410.19738 Sorry, for hijacking this thread, but I didn't want to create yet another topic for a simple link share. |
Beta Was this translation helpful? Give feedback.
-
|
The biggest problem for beginners is #16. Any other effort is just a distraction from it. One needs to master non-termination first. |
Beta Was this translation helpful? Give feedback.
-
|
As I see it at this moment, in order to be, at the end of some day, a friend with Prolog, so to speak, there are (at least, but I can imagine more) three basic "threshold concepts" for a beginner. Non-determinism (is non-termination part of it, or a "threshold concept" itself?), logical variables, recursion (or, in Prolog context, maybe, in a sense, even a better term/perspective: inductive definitions). |
Beta Was this translation helpful? Give feedback.
-
|
Right, but the other issues do not freeze the computer (if naively called without |
Beta Was this translation helpful? Give feedback.
-
|
PS: I have to apologize; this discussion page is about LLM AI "for training beginners on Prolog and through syntax errors". I was writing (as always) about "a beginner in programming in the Prolog world". My apologies. And thank you... |
Beta Was this translation helpful? Give feedback.
-
|
For anyone here who saw my talk (still working on getting the slides together), you'll know that an LLM is only as good as the dataset it is trained on. We could absolutely make one that tutors people on pure Prolog or Scryer Prolog, or answer questions about the common library, if (and only if) we create the right dataset (unclear if that is possible or not!). Based on the presentation he gave, @UWN probably has the best dataset in the world for such a purpose, but that's not what he created his Prolog training tool GUPU for. But as a community effort, making such a dataset would be the only hurdle to making the LLM we want. |
Beta Was this translation helpful? Give feedback.
-
|
#302 would accelerate the localization of a syntax error very cheaply. As for training data, the data is very biased. I myself "harvest" the logfiles for misspellings and unfitting names every semester thus causing such edits from then on. And the syntax is pretty much restricted, roughly: goals and non-terminals must be on a single line which rules out many of the harder-to-spot errors. And then, it's all in German... |
Beta Was this translation helpful? Give feedback.
-
|
Ah, yes, I should clarify -- we can have an LLM that can give general conversation around Prolog, more like an interactive Q&A. It could not be used to help find logical errors without running Prolog on the backend and possibly handing the work off to GUPU or doing some kind of genetic programming to try to find logical errors. We would actually need the architecture discussed in my talk, and the LLM part would be the easiest (and least useful) part of the system, the Prolog to find the logical errors would be the most difficult part. |
Beta Was this translation helpful? Give feedback.
-
|
👋🏼, Is there a chance you'd be able to share the slides of your presentation somewhere? By the way, I've stumbled upon this talk recently: |
Beta Was this translation helpful? Give feedback.
-
|
Hi Thierry! I got a chance to talk about it with Luke at that conference. Luke is a really smart guy. He is one of the few people with a really strong understanding of where the field is going and not afraid to be one of the first people to say it. I, on the other hand, am a coward, and thats why my slides are not done yet, but hopefully they will be in the next few days. |
Beta Was this translation helpful? Give feedback.
-
😮 Amazing! Nice to learn that we're in the same galaxy :)
Sorry, I didn't mean to rush you. |
Beta Was this translation helpful? Give feedback.
-
|
Off topic, again... but LLM and Prolog style together.... An ACM SIGPLAN SPLASH 2024 talk by Erik Meijer: From AI Software Engineers to AI Knowledge Workers https://www.youtube.com/live/_VF3pISRYRc?&t=15429 Speaking about eliminating the programming work, so to speak, in a safe way, with LLM AI combined with a Prolog-like internal language (a.k.a. neuro-symbolic computing?), generated from a natural language input. Some "slogans" from the presentation: "Tools are Relations/Predicates/Facts"; "Primitive tools are facts"; "Chains are conjunctive goals"; "Derived tools are Horn Clauses". |
Beta Was this translation helpful? Give feedback.
-
There is a TON of confusion about it. Large companies have a lot of money invested in keeping people confused. The more mysterious it seems, the easier it is to get investment dollars. If you consider the idea of an ideal LLM which perfectly captures a generalization over a dataset, you stop focusing on the LLM and start focusing on the dataset. Then it gets a lot easier to reason about what is possible.
If you are interested, please go here https://www.lotusmundi.com/ , scroll down, and send an email to that link from your preferred email address and I will give you early access the slides from the presentation. I agree with all your sentiments, but my goal is to empower local LLM development and to democratize the tools, as well as help provide an understanding about how to use them most effectively. Also, based on your interest in Smalltalk, you would probably be really interested in Torbjorn's talk notes. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @haijinSk apparently the form doesn't show your email address, could you send another email and include in the email the email address you want me to give access to? I need to look into why the form did that :( |
Beta Was this translation helpful? Give feedback.
-
|
@jjtolton Again, thank you very much... I will study your presentation in more details, but my first impressions: very interesting, unexpectedly comprehensive, cool and great graphic and information design... |
Beta Was this translation helpful? Give feedback.
-
Much appreciated!! Working hard on getting it cleaned up for general release. It's really hard trying to convey in writing the content of the talk. Hopefully more concise content to follow. |
Beta Was this translation helpful? Give feedback.
-
|
Personally, I like that it is (for me) comprehensive (and I mean it only in a good/concise way). I like that it is explaining things and showing code. I can even imagine something independent of the talk, a (mini) book... |
Beta Was this translation helpful? Give feedback.
-
An idea occurred to me today. Would it be possible or even a good idea to augment the Scryer Prolog playground with a chatbot AI that could coach beginners through their syntax errors? The effort involved to train an AI up to a good level of quality might preclude realistically doing this.
Beta Was this translation helpful? Give feedback.
All reactions