Bug Report: Incorrect errant GTID detection because of how semi-sync replication works

[GuptaManan100]

Overview of the Issue

In the recent PR #16833, we added the capability to vttablets to run errant GTID detection. But we've found a case where it is marking wrong GTIDs as errant.

Specifically, when running PRS, we've seen in tests that the replica its reading its own position as ff8ecd9a-8f92-11ef-b369-733dd679dde6:1-33 and the primary's position (after reading its own position) as ff8ecd9a-8f92-11ef-b369-733dd679dde6:1-29.

This is happening because it is possible in semi-sync replication for a GTID to be visible on a replica while still being blocked on semi-sync ACKs on the primary.

Reproduction Steps

1. setup examples 101. Read @@gtid_executed on the primary and on the replicas.
2. Then, assuming zone1-0000000100 is the primary, run:
   vtctldclient ChangeTabletType zone1-0000000101 DRAINED
vtctldclient ChangeTabletType zone1-0000000102 DRAINED
3. Then, issue a
   mysql -e "drop view if exists v1"
4. This will block on the primary, because it expects semi-sync acks, which it never gets from the replicas.
   HOWEVER, you will find that @@gtid_executed does increase on the replicas, while on the primary, it stays the same.
   You can run multiple such connections, and each will advance the replica's @@gtid_executed while the primary is still fixed in place.

Binary Version

v21

Operating System and Environment details

any

Log Fragments

No response