Refactor leader.c to fix stack growth in handle_exec_sql

[cole-miller]

This PR is another attempt to fix the stack growth issue with handle_exec_sql (related to #679), this time in a more principled way that involves a significant refactor of leader.c. Details follow.

Recall that the issue arises with a call stack like this (callees at the top)

handle_exec_sql_next
handle_exec_sql_cb
leaderExecV2
leader__barrier
leader__exec
handle_exec_sql_next
...

that is, indirect recursion that can generate a number of stack frames proportional to the number of ;-separated statements in an EXEC_SQL request. This happens because leader__barrier and leader__exec can invoke their callbacks synchronously when suspending is not required (respectively, because the FSM is up to date with the raft log and and because sqlite3_step generated no changed pages).

This PR rewrites those two functions, establishing a new contract: the callback is only invoked if we yielded to the event loop at least once while processing the request; if the request was processed synchronously, a magic value LEADER_NOT_ASYNC is returned to indicate that the caller should invoke the callback. Existing callsites are updated to reflect this new contract.

With these new implementations available, we can fix the original problem by rewriting handle_exec_sql_next to iteratively process as many statements as possible until one of them suspends or returns an error. The PR includes a regression test to validate this fix.

The new implementations of the leader.c functions are intentionally different in style from the old ones. Effectively, each barrier request and exec request is a coroutine driven by a state machine. With this approach the LEADER_NOT_ASYNC feature falls out rather naturally, and we get the added benefits of more compact code and built-in observability.

[codecov]

Codecov Report

Attention: Patch coverage is 83.26848% with 43 lines in your changes missing coverage. Please review.

Project coverage is 80.95%. Comparing base (85e494d) to head (06992a6).
Report is 25 commits behind head on master.

Files with missing lines	Patch %	Lines
src/leader.c	81.45%	9 Missing and 19 partials ⚠️
src/gateway.c	78.87%	7 Missing and 8 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #697      +/-   ##
==========================================
- Coverage   81.07%   80.95%   -0.12%     
==========================================
  Files         197      196       -1     
  Lines       29164    29186      +22     
  Branches     4066     4089      +23     
==========================================
- Hits        23644    23627      -17     
- Misses       3875     3895      +20     
- Partials     1645     1664      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cole-miller]

Jepsen run at https://github.com/canonical/jepsen.dqlite/actions/runs/10672883562, looks clean.

[cole-miller]

Status: trying to identify and fix the regression seen here canonical/lxd#14034, namely sqlite3_close returning nonzero in leader__close (indicating that a prepared statement was not finalized).

[cole-miller]

I believe the LXD issue has been fixed, the problem was returning 0 from exec_async in a case where we didn't suspend. This caused handle_exec_sql_cb to not be invoked, leaking a prepared statement and tripping assert(sqlite3_close(conn) == 0).

[letFunny]

Looks good to me, thanks Cole. I just have a few questions to improve my understanding and some very small suggestions.