Integration of Sphinx sites with LLMs

[mgaitan]

Hello Sphinx Community,

I'm exploring the possibility of integrating Sphinx-based documentation with large language models (LLMs) such as Claude, LLaMA, or (chat)GPT. This integration aims to allow users to query the documentation using natural language and receive relevant responses (hopefully with links to the doc) generated by the LLM.

Are there any existing extensions or guides that facilitate the integration of Sphinx documentation with LLMs? If no such solution exists, is there interest within the community to develop one? AFAIK this could be based on tools like LlamaIndex to structure the documentation in a way that is easily accessible for LLMs. I'm not an expert, so I'm open to suggestions on the best practices or strategies to achieve this integration.

Any insights, suggestions, or pointers to existing resources would be greatly appreciated.

Thanks!
Martin

[jayaddison]

Hi @mgaitan - I'm not a Sphinx core developer, although I do contribute when I can, and am interested in some of the aspects of the existing search/query functionality in Sphinx HTML projects.

In my developer mindset there are a few things that I like about the current search functionality that make me hesistant/unsure about LLMs. They are:

• I like the knowledge that although search may not be 'optimal' for all users, it is 'fair' for all users -- given the same HTML project and search query, they'll receive the same results.
• A related property is that when we receive a bugreport about search results -- provided also that the HTML + searchindex is provided or the same version is linked to -- then I like that I know with reasonable confidence that I can replicate the problem and diagnose what happened. I might not immediately have suggestions about how to fix it, but can pinpoint the path to the code and document the decisions that allowed that path to be followed.
• Similar to most developers, I do introduce bugs unfortunately. However, those can be identified by others and/or fixed and/or reverted as necessary. It doesn't feel good to know that I'd introduced a problem, but I do like that it's possible to figure out when and where that occurred, and for what versions of the code.
• To some extent I think I attempt to be (perhaps over-)protective of users, and although I realize that many of them may not have the time or inclination to inspect the search code themselves, I think it's good that they can do that if they want to -- and I worry that they might lose that ability with a less-scrutable query engine such as a language model.

These tend to disincline me towards suggestions to integrate Sphinx projects with LLMs -- either by adding LLM functionality into Sphinx itself, or by exposing additional features that would make it easier for LLMs to consume the contents of them, because I don't think that the maintainers of those LLMs could offer the same kind of support or care to their communities.

Thanks for opening the discussion! I think it's a useful and topical one.

[mgaitan]

I don't know if I understand your response correctly. At no point did I propose that we should replace the integrated Sphinx search engine with an LLM (and I don't know if that's possible for a "static" site).

My intention is simply to facilitate from Sphinx the optimal compilation of the contents so they can be consumed by an LLM (an independent application, for example, a chatbot integrated into Slack) which can be queried and respond including links to the documentation.

For example, in my company, internal documentation generated with Sphinx is automatically deployed when the .md sources change. I would like that in addition to updating the .html output, the "knowledge base" of an LLM chatbot is updated so that it is always up to date to respond.

[jayaddison]

My apologies @mgaitan - I did understand your message, and knowingly replied on something of a tangent. I was feeling grumpy about a sense that proprietary LLMs have/are taking the hard work of volunteers in ways that arguably do provide value but do not necessarily return it to those contributors (this is no doubt debatable on a case-by-case basis).

LLMs can be implemented in software, and any software (even where hardware acceleration supports it) can be implemented in the browser, so an LLM within a statically-deployed HTML documentation project is indeed feasible; I would guess that various client-side chatbots likely exist already, at differing technology maturity levels.

The data pipeline for a documentation-trained LLM would be the same as most search engines, I expect. Markdown isn't really Sphinx's preferred input format, but if you were to use reStructuredText instead, then you could either train your model on the .rst file itself (it is plain text), or perhaps render the documentation using the text builder (to gain some of the benefits of the processing that Sphinx performs).

As a software developer I'd recommend adding frequently-asked questions to a continuous integration process that checks whether the trained model responds with useful answers to commonly-asked questions. What I'm less likely to be able to help with is how to fix the LLM if it stops answering those questions reliably; for that you would have to refer to your chosen LLM vendor/community.