Add reflection + validation steps in get_feedback

[wongjingping]

Add call to /reflect_on_error if negative feedback received, and then display the suggestions for validation from the user. Here is a test run where we only updated the glossary but not the column descriptions or reference queries after. Requerying in the CLI after that magically works!:

$ defog query "how many users made long requests"
Generating the query for your question: how many users made long requests...
Query generated, now running it on your database...
Defog generated the following query to answer your question:

SELECT COUNT(DISTINCT defog_api_stats.api_key)
  FROM defog_api_stats
 WHERE defog_api_stats.requested_at::TIMESTAMP > CURRENT_TIMESTAMP - INTERVAL '10 minutes';

Results:

count |
------|
2     |

Did Defog answer your question well? Just hit enter to skip (y/n):
n
Could you tell us why this was a bad query? This will help us improve the model for you. Just hit enter if you want to leave this blank.
long requests are requests where the questions are over 1000 character limit
Thank you for the feedback, let us see how can we improve this for you...

Here is our automated assessment:
The query provided does not filter requests based on their length, which is crucial for identifying 'long requests' as defined by the question. Instead, it filters based on a time interval. To improve the query, it's necessary to include a condition that checks the length of the 'question' column in the 'defog_api_stats' table. Additionally, the instruction set and column descriptions could be enhanced to explicitly mention how to identify long requests and the significance of the 'question' column length. Reference queries should also demonstrate how to filter data based on content length or other content-based criteria.
We came up with the following additions for improving your glossary:
1. When identifying long requests, include a condition to check the length of the 'question' column in the 'defog_api_stats' table.
2. Ensure that reference queries demonstrate how to filter data based on content length or other content-based criteria, such as filtering requests longer than a specific character count.
3. Clarify in the instruction set that 'long requests' are defined by the length of the question, specifically requests where the question exceeds a certain character limit.
If you would like to add these suggestions to your glossary, please enter 'y'. If you would like to amend it, just type in your edits and hit enter. Otherwise, enter 'n'.
long requests = length of the 'question' column in the 'defog_api_stats' table > 1000
Glossary updated successfully.

We came up with the following suggestions for improving your column descriptions:
[{'table_name': 'defog_api_stats', 'column_name': 'question', 'description': 'The question asked in the request. Use the length of this column to identify long requests, specifically those exceeding a certain character limit.'}]

Current description for question:
Suggested description for question: The question asked in the request. Use the length of this column to identify long requests, specifically those exceeding a certain character limit.
Would you like to replace this description with our suggestion? Please hit 'y' to add, or anything else to skip to the next suggestion.
n
No metadata changes to update.

We came up with the following suggestions for adding as your reference queries:
Question: How many users made requests with questions longer than a specific character limit?
SQL: SELECT COUNT(DISTINCT api_key) FROM defog_api_stats WHERE LENGTH(question) > CHARACTER_LIMIT;
Would you like to add this as one of your reference queries? Please hit 'y' to add, or anything else to skip to the next suggestion.
n
No reference queries to update.

Please enter another query, or type 'e' to exit: e
Exiting.

# issue the same query
$ defog query "how many users made long requests"
Generating the query for your question: how many users made long requests...
Query generated, now running it on your database...
Defog generated the following query to answer your question:

SELECT COUNT(DISTINCT defog_api_stats.api_key)
  FROM defog_api_stats
 WHERE length(defog_api_stats.question) > 1000;

Results:

count |
------|
14    |

Did Defog answer your question well? Just hit enter to skip (y/n):

[rishsriv]

Amazing! Thank you! This was a much needed feature – really glad we've implemented this :D

Will make this a part of the UI soon, too!