Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NL-to-ESQL] add review/correct steps #193262

Closed
pgayvallet opened this issue Sep 18, 2024 · 2 comments
Closed

[NL-to-ESQL] add review/correct steps #193262

pgayvallet opened this issue Sep 18, 2024 · 2 comments
Labels
Team:AI Infra AppEx AI Infrastructure Team

Comments

@pgayvallet
Copy link
Contributor

pgayvallet commented Sep 18, 2024

Part of https://github.com/elastic/kibana-team/issues/1264

At the moment, we're using the correctCommonEsqlMistakes (non-llm) function to fix common syntax mistakes, but it's fairly limited, and does not allow functional validation of the generated query.

The common pattern for LLM-based review is the review/correct chain prompting: after the generation, you ask the model to review the generation and list errors and possible improvements, when you run another prompt asking to correct the first generation based on the review of the second one.

@pgayvallet pgayvallet changed the title [Nl-to-ESQL] add review/correct steps [NL-to-ESQL] add review/correct steps Sep 18, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 18, 2024
@pgayvallet pgayvallet added the Team:AI Infra AppEx AI Infrastructure Team label Sep 18, 2024
@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 18, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/appex-ai-infra (Team:AI Infra)

@pgayvallet
Copy link
Contributor Author

Note that from various papers (e.g. this one), self-correctness doesn't work very well without external input.

The best approach here could be to perform programmatic validation of the query (ast validation), and then feed the result of the validation to the LLM for correction, instead of just asking it to review its own work without additional insight.

@pgayvallet pgayvallet closed this as not planned Won't fix, can't repro, duplicate, stale Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:AI Infra AppEx AI Infrastructure Team
Projects
None yet
Development

No branches or pull requests

2 participants